- Phaselocking, temporal scheme, envelope, temporal fine structure, pitch shifts
Abstract
Phase-locked neural activity in the brainstem generating the human frequency following response (FFR) has been shown to preserve neural information about certain spectrotemporal attributes of complex sounds that likely contributes to the perception of speech, language, music, and segregation of concurrent sound streams. Here we review some evidence in support of the view that the temporal pattern of neural activity in the FFR does preserve information relevant to the representation of spectra and pitch of complex sounds. Specifically, FFR spectra for both steady state and time variant complex sounds show clear peaks at formant related harmonics that follow the changes in formant frequencies for time-variant sounds. Similarly, neural information relevant to steady- and time-variant pitch, pitch shifts, and changes in pitch salience are well preserved in the FFR and exhibits a strong correlation with behavioral measures. For inharmonic, frequency shifted, and unresolved harmonics stimuli that produce pitch shifts or multiple pitches, neural activity relevant to these perceived changes in pitch is primarily contained in the neural representation of the temporal fine structure (TFS). Both reverberation and noise degrade the neural representation of envelope and TFS with phase locking to ENV showing greater resilience. Overall, these results clearly suggest that the FFR provides a robust physiologic window to evaluate the nature of neural representation of spectra and pitch of complex sounds in normal and impaired ears, age related changes in neural encoding, and to understand the role of experience in shaping subcortical processing and its application to re-training and perceptual learning.
References
- Aiken, S. J., & Picton, T. W. (2008). Envelope and spectral frequency-following responses to vowel sounds. Hearing Research, 245, 35–47.
- Ananthakrishnan, S., & Krishnan, A. (2018). Human frequency following responses to iterated rippled noise with positive and negative gain: Differential sensitivity to waveform envelope and temporal fine structure. Hearing Research, 367, 113-123.
- Ananthakrishnan, S., Krishnan, A., & Bartlett, E. (2016). Human frequency following response: Neural representation of envelope and temporal fine structure in listeners with normal hearing and sensorineural hearing loss. Ear & Hearing, 37, e91-e103.
- Ananthakrishnan, S., Lou, Xin., & Krishnan, A. (2017). Human frequency following responses to vocoded speech. Ear & Hearing, 38, e256-e267.
- Ananthanarayan, A. K., & Durrant, J. J. (1992). The frequency following response and the onset response: Evaluation of frequency specificity using a forward-masking paradigm. Ear & Hearing, 13, 228-232.
- Anderson, S., Parbery-Clark, A., White-Schwoch, T., Drebohl, S., & Kraus, N. (2013). Effects of hearing loss on the subcortical representation of speech cues. The Journal of the Acoustical Society of America, 133, 3030–38.
- Arthur, R. M., Pffeifer, R. R., & Suga, N. (1971). Properties of two-tone inhibition in primary auditory neurons. Journal of Physiology, 212, 593-609.
- Bidelman, G., & Krishnan, A. (2009). Neural correlates of consonance, dissonance, and the hierarchy of musical pitch in the human brainstem. Journal of Neuroscience, 29, 13165–13171.
- Bidelman, G., Gandour, J., & Krishnan, A. (2011). Cross-domain effects of music and language experience on the representation of pitch in the human auditory brainstem. Journal of Cognitive Neuroscience, 23, 425-434.
- Bidelman, G. M., Davis, M. K., & Pridgen, M. H. (2018). Brainstem-cortical functional connectivity for speech is differentially challenged by noise and reverberation. Hearing Research, 367, 149-160.
- Bidelman, G. M. (2017). Communicating in challenging environments: Noise and reverberation. In: N. Kraus, S. Anderson, T. White-Schwoch, R.R. Fay, & A.N. Popper AN (Eds.), Springer Handbook of Auditory Research: The frequency-following response: A window into human communication. New York, N.Y: Springer Nature.
- Bidelman, G. M. (2018). Subcortical sources dominate the neuroelectric auditory frequency-following response to speech. NeuroImage, 175, 56–69.
- Bidelman, G. M., & Krishnan, A. (2011). Effects of reverberation on brainstem representation of speech in musicians and non-musicians. Brain Research, 1355, 112-125.
- Billings, C. J., Bologna, W. J., Muralimanohar, R. K., Madsen, B., & Molis, M. R. (2019). Frequency following responses to tone glides: effects of frequency extent, direction, and electrode montage. Hearing Research, 375, 25-33.
- Bilsen, F. A., & Ritsma, R. J. (1970). Some parameters influencing the perceptibility of pitch. The Journal of the Acoustical Society of America, 47, 469-475.
- Bilsen, F. A., ten Kate, J. H., Buunen, T. J. F., & Raatgever, J. (1975). Responses of single units in the cochlear nucleus of the cat to cosine noise. The Journal of the Acoustical Society of America, 58, 858–866.
- Blackburn, C. C., & Sachs, M. B. (1990). The representations of the steady-state vowel sound /e/ in the discharge patterns of cat anteroventral cochlear nucleus neurons. Journal of Neurophysiology, 63, 1191–1212.
- Brown, R. (1958). Words and Things. Glencoe, IL: Free Press, 36–42.
- Cariani, P. A., & Delgutte, B. (1996a). Neural correlates of the pitch of complex tones. I. Pitch and pitch salience. Journal of Neurophysiology, 76(3), 1698-1716.
- Cariani, P. A., & Delgutte, B. (1996b). Neural correlates of the pitch of complex tones. II. Pitch shift, pitch ambiguity, phase invariance, pitch circularity, rate pitch, and the dominance region for pitch. Journal of Neurophysiology, 76, 1717-1734.
- Carlson, R., Fant, G., & Granstrom, B. (1975). Two-formant models, pitch and vowel perception. In G. Fant, & M.A.A. Tatham (Eds.), Auditory Analysis and Perception of Speech. London: Academic, pp 55–82.
- Carlyon, R. P., & Shackleton, T. M. (1994). Comparing the fundamental frequencies of resolved and unresolved harmonics: Evidence for two pitch mechanisms. The Journal of the Acoustical Society of America, 95, 3541–3554.
- Caspary, D. M., Rupert, A. L., & Moushegian, G. (1977). Neuronal coding of vowel sounds in the cochlear nuclei. Experimental Neurology, 54, 414–431.
- Cedolin, L., & Delgutte, B. (2005). Pitch of complex tones: rate-place and interspike interval representations in the auditory nerve. Journal of Neurophysiology, 94, 347-362.
- Clinard, C. G., & Cotter, C. M. (2015). Neural representation of dynamic frequency is degraded in older adults. Hearing Research, 323, 91–98.
- Cunningham, J., Nicol, T., King, C., Zecker, S. G., & Kraus, N. (2002). Effects of noise and cue enhancement on neural responses to speech in auditory midbrain, thalamus and cortex. Hearing Research, 169, 97–111.
- Dau, T. (2003). The importance of cochlear processing for the formation of auditory brainstem and frequency following responses. The Journal of the Acoustical Society of America, 113, 936-950.
- De Boer, E. (1956). On the Residue in Hearing. Academic Thesis, University of Amsterdam.
- De Boer, E. (1976). On the “Residue” and auditory pitch perception. Auditory System. 479-583.
- Drgas, S., & Blaszak, M. A. (2009). Perceptual consequences of changes in vocoded speech. Ear & Hearing, 41, 300-311.
- Easwar, V., Purcell, D. W., Aiken, S. J., Parsa, V., & Scollie, S. D. (2015b). Evaluation of speech-evoked envelope following responses as an objective aided outcome measure: Effect of stimulus level, bandwidth, and amplification in adults with hearing loss. Ear and Hearing, 36, 635–652.
- Fastl, H., & Stoll, G. (1979). Scaling of pitch strength. Hearing Research, 1, 293–301.
- Fay, R. R., Yost, W. A., & Coombs, S. (1983). Psychophysics and neurophysiology of repetition noise processing in a vertebrate auditory system. Hearing Research, 12, 31–55.
- Galbraith, G., Jhaveri, S. P., & Kuo, J. (1997). Speech-evoked brainstem frequency-following responses during verbal transformations due to word repetition. Electroencephalography and Clinical Neurophysiology, 102, 46–53.
- Gelfand, S. A., & Silman, S. (1979). Effects of small room reverberation upon the recognition of some consonant features. The Journal of the Acoustical Society of America, 66, 22–29.
- Goblick, T. J., & Pffeifer, R. R. (1969). Time-domain measurements of the cochlear nonlinearities using combination click stimuli. The Journal of the Acoustical Society of America, 46, 924-938.
- Gockel, H. E., Carlyon, R. P., Mehta, A., & Plack, C. J. (2011). The frequency following response (FFR) may reflect pitch-bearing information but is not a direct representation of pitch. Journal of the Association for Research in Otolaryngology, 12, 767-782
- Greenberg, S. (1980). Neural temporal coding of pitch and vowel quality. UCLA Working Papers in Phonetics, 52, (doctoral dissertation). Los Angeles.
- Greenberg, S., Marsh, J. T., Brown, W. S., & Smith, J. C. (1987). Neural temporal coding of low pitch. I. Human frequency-following responses to complex tones. Hearing Research, 25, 91-114.
- Hall, J. W. III. (1979) Auditory brainstem frequency following responses to waveform envelope periodicity. Science, 205, 1297-1299.
- Henry, K. R. (1999). Noise improves transfer of near-threshold, phase-locked activity of the cochlear nerve: Evidence for stochastic resonance? Journal of Comparative Physiology A: Neuroethology, Sensory, Neural, and Behavioral Physiology, 184, 577–584.
- Horst, W., Javel, E., & Farley, G.R. (1990). Coding of spectral fine structure in the auditory nerve. II. Level-dependent nonlinear responses. The Journal of the Acoustical Society of America ,88, 2656-2681.
- Houtgast, T. & Steeneken, H. J. M. (1985). A review of the MTF-concept in room acoustics, The Journal of the Acoustical Society of America, 77, 1069–1077.
- Houtgast, T., & Steeneken, H. J. M. (1973). The modulation transfer function in room acoustics as a predictor of speech intelligibility. Acustica, 28, 66–73.
- Houtsma, A., & Smurzynski, J. (1990). Pitch identification n and discrimination for complex tones with many harmonics. The Journal of the Acoustical Society of America, 87, 304–310.
- Jacobson, R., Fant, G., & Halle, M. (1963). Preliminaries to Speech Analysis. Cambridge, MA: MIT.
- Janssen, T., Steinhoff, H.J., & Bohnke, F. (1991). Zum Entstchungs Mechanisms der Frequenfolgepotentiale. Otorhinolaryngology Nova, 1, 16–25.
- Keilson, S. E., Richards, V. M., Wyman. B. T., & Young, E. D. (1997). The representation of concurrent vowels in the cat anesthetized ventral cochlear nucleus: evidence for a periodicity-tagged spectral representation. The Journal of Acoustical Society of America, 102, 1056–1071.
- Krishnan, A., Gandour, J., & Bidelman, G. (2010). Neural representation of pitch salience in the human brainstem revealed by psychophysical and electrophysiological indices. Hearing Research, 268, 60-66.
- Krishnan, A., Gandour, J., & Bidelman, G. (2010b). Brainstem pitch representation in native speakers of Mandarin is less susceptible to degradation of stimulus temporal regularity. Brain Research, 1313, 124-133.
- Krishnan, A. (1999). Human frequency following responses to two-tone approximations of steady-state vowels. Journal of Audiology and Neuro-Otology, 4, 95-103.
- Krishnan, A. (2002). Human frequency-following responses: representation of steady-state synthetic vowels. Hearing Research, 166, 192-201.
- Krishnan, A. (2006). Human frequency following response. In R.F. Burkard, M. Don, & J.J. Eggermont (Eds.), Auditory evoked potentials: Basic principles and clinical application. (pp. 313-335). Baltimore, MD: Lippincott Williams & Wilkins.
- Krishnan, A., & Agrawal, S. (2010). Human frequency-following response to speech-like sounds: correlates of off-frequency masking. Journal of Audiology & Neuro-Otology, 15, 221-228.
- Krishnan, A., & Gandour, J. T. (2017). Brainstem representation of pitch relevant information is shaped by language experience. In N. Kraus, S. Anderson, T. White-Schwoch, A.N. Popper, & R.R.Fay, (Eds.), The Frequency-following Response: A Window into Human Communication. (pp. 45-73). Auditory Research Series, ASA Press, Springer.
- Krishnan, A., & Gandour, J. T. (2014). Language experience shapes processing of pitch relevant information in the human brainstem and auditory cortex: Electrophysiological evidence. Acoustics Australia, 42, 187-199.
- Krishnan, A., & Parkinson, J. (2000). Human frequency following response: Representation of tonal sweeps. Journal of Audiology & Neuro-Otology, 5, 312-321.
- Krishnan, A., & Plack, C. (2011b). Human frequency following responses: Correlates of the pitch of complex stimuli with inharmonic and frequency-shifted components. Data presented at the Mid- winter meeting of the Association for Research in Otolaryngology (ARO), Baltimore.
- Krishnan, A., & Plack, C. (2011). Neural encoding in the human brainstem relevant to the pitch of complex tones. Hearing Research, 275, 110-119.
- Krishnan, A., Bidelman, G. M., & Gandour, J. T (2010). Neural representation of pitch salience in the human brainstem revealed by psychophysical and electrophysiological indices. Hearing Research, 268, 1-2. doi: 10.1016/j.heares.2010.04.016.
- Krishnan, A., Gandour, J. T., & Chandan, S. (2016). Language-experience plasticity in neural representation of changes in pitch salience. Brain Research, 1637: 102-117.
- Krishnan, A., Gandour, J. T., & Chandan, S. (2017b). Differential sensitivity to changes in pitch acceleration in the auditory brainstem and cortex. Brain & Language, 169, 22-27.
- Krishnan, A., Gandour, J.T., Xu, Y., & Chandan, S. (2016). Language-dependent changes in pitch-relevant neural activity in the auditory cortex reflect differential weighting of temporal attributes of pitch contours. Journal of Neurolinguistics, 41, 38-49.
- Krishnan, A., Suresh, C., & Gandour, J. T (2019). Tone language experience-dependent advantage in pitch representation in brainstem and auditory cortex is maintained under reverberation. Hearing Research, 177, 63-71.
- Krishnan, A., Xu, Y., Gandour, J. T., & Cariani, P. (2005). Encoding of pitch in the human brainstem is sensitive to language experience. Cognitive Brain Research, 25, 161-168.
- Krishnan, A., Xu, Y., Gandour, J. T., & Cariani, P. A. (2004). Human frequency-following response: representation of pitch contours in Chinese tones. Hearing Research, 189, 1-12.
- Krumbholz, K., Patterson, R. D., Seither-Preisler, A., Lammertmann, C., & Lütkenhoner, B. (2003). Neuromagnetic evidence for a pitch processing center in Heschl's gyrus. Cerebral Cortex, 13, 765-772.
- Li, X., & Jeng, F. C. (2011). Noise tolerance in human frequency-following responses to voice pitch. The Journal of the Acoustical Society of America, 129, 21–26.
- Madhavi, B., Krishnan, A., & Weber-Fox, C. (2009). Brainstem correlates of temporal auditory processing in children with specific language impairment. Developmental NeuroScience, 13, 77-91.
- Mesgarani, N., David, S. V., Fritz, J. B., & Shamma, S. A. (2014). Mechanisms of noise robust representation of speech in primary auditory cortex. Proceedings of the National Academy of Science, USA 111, 6792–6797.
- Miller, M. I., & Sachs, M. B. (1983). Representation of stop consonants in the discharge patterns of auditory-nerve fibers. The Journal of the Acoustical Society of America, 74, 502–517.
- Miller, M. I., & Sachs, M. B. (1984). Representation of voice pitch in discharge patterns of auditory-nerve fibers. Hearing Research,14, 257–279.
- Nabelek, A. K., & Dagenais, P. A. (1986). Vowel errors in noise and in reverberation in hearing-impaired listeners. The Journal of the Acoustical Society of America, 80, 741–748.
- Nabelek, A. K., & Letowski, T. R. (1988). Similarities of vowels in nonreverberant and reverberant fields. The Journal of the Acoustical Society of America, 83, 1891–1899.
- Nabelek, A. K., Letowski, T. R., & Tucker, F. M. (1989). Reverberant overlap- and self-masking in consonant identification. The Journal of the Acoustical Society of America, 86, 1259–1265.
- Neuert, V., Verhey, J. L., & Winter, I. M. (2005). Temporal representation of the delay of iterated rippled noise in the dorsal cochlear nucleus. Journal of Neurophysiology, 93, 2766-2776.
- Palmer, A. R., & Russell, I. J. (1986). Phase-locking in the cochlear nerve of the guinea-pig and its relation to the receptor potential of inner hair-cells. Hearing Research, 24, 1-15.
- Palmer, A. R. and Winter, I. M. (1993). Coding of the fundamental frequency of voiced speech sounds and harmonic complexes in the cochlear nerve and ventral cochlear nucleus. In M. A. Merchán, J. M. Juiz, D. A. Godfrey, & E. Mugnaini, (Eds.), The mammalian cochlear nuclei: Organization and Function. pp. 373-384, New York, NY: Plenum Press.
- Palmer, A. R., & Winter, I. M. (1992). Cochlear nerve and cochlear nucleus responses to the fundamental frequency of voiced speech sounds and harmonic complex tones. Auditory Physiology Perception, 231-239.
- Palmer, A. R., Winter, I. M., & Darwin, C. J. (1986). The representation of steady-state vowel sounds in the temporal discharge patterns of the guinea pig cochlear nerve and primary like cochlear nucleus neurons. The Journal of the Acoustical Society of America, 79, 100–113.
- Parbery-Clark, A., Skoe, E., & Kraus, N. (2009). Musical experience limits the degradative effects of background noise on the neural processing of sound. The Journal of Neuroscience, 29, 14100–14107.
- Patterson, R. D., Handel, S., Yost, W. A., & Datta, A. J. (1996). The relative strength of the tone and noise components in iterated ripple noise. The Journal of the Acoustical Society of America,100, 3286–3294.
- Peterson, G. E., & Barney, H. L. (1952). Control methods used in a study of vowels. The Journal of the Acoustical Society of America, 24, 175–184.
- Plyler, P., & Krishnan, A. (2001). Human frequency following response: Representation of CV formant transitions in normal-hearing and hearing-impaired listeners. Journal of the American Academy of Audiology, 12, 523-533.
- Prevost, F., Laroche, M., Marcoux, A. M., & Dajani, H. R. (2013). Objective measurement of physiological signal-to-noise gain in the brainstem response to a synthetic vowel. Clinical Neurophysiology, 124, 52–60.
- Recio, A., & Rhode, W.S. (2000). Representation of vowel stimuli in the ventral cochlear nucleus of the chinchilla. Hearing Research, 146, 167–184.
- Rhode, W. (1998). Neural encoding of single-formant stimuli in the ventral cochlear nucleus of the chinchilla. Hearing Research, 117, 39–56.
- Russo, N., Nicol, T., Musacchia, G., & Kraus, N. (2004). Brainstem responses to speech syllables. Clinical Neurophysiology, 115, 2021–2030.
- Sachs, M. B., Voigt, H. F., & Young, E. D. (1983). Auditory nerve representation of vowels in background noise. Journal of Neurophysiology, 50, 27–45.
- Sayles, M., & Winter, I. M. (2007). The temporal representation of the delay of dynamic iterated rippled noise with positive and negative gain by single units in the ventral cochlear nucleus. Brain Research, 1171, 52-66.
- Sayles, M., & Winter, I. M. (2008). Reverberation challenges the temporal representation of the pitch of complex sounds. Neuron, 58, 789–801.
- Sayles, M., Stasiak, A., & Winter, I. M. (2015). Reverberation impairs brainstem temporal representations of voiced vowel sounds: challenging “periodicity-tagged” segregation of competing speech in rooms. Frontiers of Systems Neuroscience, 8, doi: 10.3389/fnsys.2014.00248. 00248.
- Sayles, M., Stasiak, A., & Winter, I. M. (2016). Neural segregation of concurrent Speech: Effects of background noise and reverberation on auditory scene analysis in the ventral cochlear nucleus. In Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing, pp 389-397. Part of the Advances in Experimental Medicine and Biology book series (AEMB, volume 894).
- Schouten, J. F. (1940). The residue, a new component in the subjective sound analysis. Proceedings of the Koninklijke Nederlandse Akademie van Wetenschappen, 43, 356-365.
- Shackleton, T. M., & Carlyon, R. P. (1994a). The role of resolved and unresolved harmonics in pitch perception and frequency modulation discrimination. The Journal of the Acoustical Society of America, 95, 3529–40
- Shackleton, T. M., Liu, L. F., & Palmer, A. R. (2009). Responses to diotic, dichotic, and alternating phase harmonic stimuli in the inferior colliculus of Guinea pigs. Journal of the Association for Research in Otolaryngology, 10, 76-799.
- Shofner, W. P. (1999). Responses of cochlear nucleus units in the chinchilla to iterated rippled noises: analysis of neural autocorrelograms. Journal of Neurophysiology, 81, 2662-2674.
- Shofner, W. P. (1991). Temporal representation of rippled noise in the anteroventral cochlear nucleus of the chinchilla. The Journal of the Acoustical Society of America, 90, 2450–66.
- Shofner, W. P., & Selas, G. (2002). Pitch strength and Stevens’s power law. Perception Psychophysics, 64, 437–50.
- Smalt, C. J., Krishnan, A., Bidelman, G. M., Ananthakrishnan, S., & Gandour, J. T. (2012). Neural correlates of cochlear distortion products and their influence on representation of pitch relevant information in the human brainstem. Hearing Research, 292, 26-34.
- Soeta, Y., Nakagawa ,S., & Tonoike, M. (2005). Auditory evoked magnetic fields in relation to iterated rippled noise. Hearing Research, 205, 256–61.
- Song, J. H., Skoe, E., Banai, K., & Kraus, N. (2011). Perception of speech in noise: Neural correlates. Journal of Cognitive Neuroscience, 23, 2268–2279.
- Suresh, C., Krishnan, A., & Gandour, J. T. (2017). Language experience-dependent advantage in pitch representation in the auditory cortex is limited to favorable signal-to-noise ratios. Hearing Research, 355, 42-53.
- Suresh, C. H., Krishnan, A., & Luo, X. (2020). Human frequency following responses to vocoded speech with amplitude modulation alone, and amplitude modulation plus frequency modulation.
- Swaminathan, J., Krishnan, A., & Gandour, J. T. (2008). Pitch encoding in speech and nonspeech contexts in the human auditory brainstem. Neuroreport, 19,1163-1167.
- ten Kate J. H., van Bekkum M. F. (1988). Synchrony-dependent autocorrelation in eighth-nerve fiber response to rippled noise. Journal of the Acoustical Society of America, 84, 2092–2102.
- Verhey, J. L., & Winter, I. M., (2006). The temporal representation of the delay of iterated rippled noise with positive or negative gain by chopper units in the cochlear nucleus. Hearing Research, 216, 43-51.
- Wang, D., & Brown, G. J. (2006). In D. Wang, & G.J. Brown (Eds.), Computational Auditory Scene Analysis: Principles, Algorithms, and Applications (pp. 395). New York, NY: Wiley/IEEE Press.
- Winter, I. M, Wiegrebe, L., & Patterson, R. D. (2001). The temporal representation of the delay of iterated rippled noise in the ventral cochlear nucleus of the guinea-pig. Journal of Physiology, 537, 553–66.
- Yost, W. A. (1996). Pitch of iterated rippled noise. The Journal of the Acoustical Society of America, 100, 511-518.
- Yost, W. A. (1996b). Pitch of iterated rippled noise. The Journal of the Acoustical Society of America,100, 511–518.
- Yost, W. A., Patterson, R., & Sheft, S. (1996). A time domain description for the pitch strength of iterated rippled noise. The Journal of the Acoustical Society of America, 99, 1066-1078.
- Young, E. D., & Sachs, M. B. (1979). Representation of steady-state vowels in the temporal aspects of the discharge patterns of populations of auditory-nerve fibers. The Journal of the Acoustical Society of America, 66, 1381–1403.