For some people, simple sensory stimuli (e.g., noises, patterns) may reliably evoke intense and aversive reactions. This is common in certain clinical groups (e.g., autism) and varies greatly in the neurotypical population. This paper critically evaluates the concept of individual differences in sensory sensitivity, explores its possible underlying neurobiological basis, and presents a roadmap for future research in this area. A distinction is made between subjective sensory sensitivity (self-reported symptoms); neural sensory sensitivity (the degree of neural activity induced by sensory stimuli); and behavioral sensory sensitivity (detection and discrimination of sensory stimuli). Whereas increased subjective and neural sensory sensitivity are assumed to increase together, the status of behavioral sensory sensitivity depends on the extent to which the increased neural activity is linked to signal or noise. A signal detection framework is presented that offers a unifying framework for exploring sensory sensitivity across different conditions. The framework is discussed, in more concrete terms, by linking it to four existing theoretical accounts of atypical sensory sensitivity (not necessarily mutually exclusive): increased excitation-to-inhibition ratio; predictive coding; increased neural noise; and atypical brain connectivity.