The first search for nonresonant production of Higgs boson pairs (HH) with one H decaying into four leptons and the other into a pair of b quarks is presented, using proton-proton collisions recorded at a center-of-mass energy of 13 TeV by the CMS experiment. The analyzed data correspond to an integrated luminosity of 138 fb. A 95% confidence level upper limit of 32.4 is set on the signal strength modifier , defined as the ratio of the observed HH production rate in the decay channel to the standard model expectation. Possible modifications of the H trilinear coupling with respect to the standard model (SM) value are investigated. The coupling modifier , defined as divided by its SM prediction, is constrained to be within the observed (expected) range 8.8 (9.8) 13.4 (15.0) at 95% confidence level.