The possibility of searching for dark matter with quantum sensors has recently received a lot of attention. In this short paper, we discuss the possibility of searching for gravitational dark matter with quantum sensors and identify a very narrow window of opportunity for future quantum sensors with improved sensitivity. Gravitational dark matter candidates with masses in the range [10 , 1]^-3 eV could lead to an effective time variation of the proton mass that could be measured with, e.g., future atomic clocks.