We measured the neural activity associated with the temporal structure of sound in the human auditory pathway from cochlear nucleus to cortex. The temporal structure includes regularities at the millisecond level and pitch sequences at the hundreds-of-milliseconds level. Functional magnetic resonance imaging (fMRI) of the whole brain with cardiac triggering allowed simultaneous observation of activity in the brainstem, thalamus and cerebrum. This work shows that the process of recoding temporal patterns into a more stable form begins as early as the cochlear nucleus and continues up to auditory cortex.