A search for nonresonant production of Higgs boson pairs via gluon-gluon fusion and vector boson fusion in final states with two bottom quarks and two photons is presented. This search uses data from proton-proton collisions at a center-of-mass energy of {\sqrt{s}=13\,\mathrm{TeV}} recorded by the CMS detector at the LHC from 2016 to 2018, corresponding to an integrated luminosity of {137\,\mathrm{fb}^{-1}}. No signal is observed, and a 95 confidence level upper limit is set on the product of the inclusive Higgs boson pair production cross section and branching fraction into . The observed (expected) upper limit is determined to be (), which corresponds to () times the standard model prediction. Assuming all other Higgs boson couplings are equal to their values in the standard model, the coupling modifiers of the trilinear self-coupling and the coupling between a pair of Higgs bosons and a pair of vector bosons are constrained within the ranges and at 95 confidence level. Constraints on are also set by combining this analysis with a search for single Higgs bosons produced in association with top quark-antiquark pairs, and by performing a simultaneous fit of and the top Yukawa coupling modifier .