A search for the exclusive decays of the Higgs and bosons to a or meson and a photon is performed with a collision data sample corresponding to an integrated luminosity of up to 35.6\,\text{fb}^{-1} collected at \sqrt{s}=13\,\text{TeV} with the ATLAS detector at the CERN Large Hadron Collider. These decays have been suggested as a probe of the Higgs boson coupling to light quarks. No significant excess of events is observed above the background, in agreement with the Standard Model expectations. Upper limits at 95\% confidence level on the branching fractions of the Higgs boson decays to \phi\,\gamma and \rho\,\gamma of and , respectively, are obtained. The corresponding 95\% confidence level upper limits for the boson decays are and for \phi\,\gamma and \rho\,\gamma, respectively.