One of the best films I have ever seen is Christopher Nolan’s Dunkirk, which checks off all of a film critic’s boxes. Cinematography, acting, script, settings and overall gravitas. What impressed most is the soundtrack. Nolan used sound as a foundation, never allowing the viewer’s tension to lag.
The industrial sound motifs were effective at telegraphing the feeling of what it must be like to be caught up in mortal combat.
It is not until the very end of the film, when the viewer is finally let off the acoustic hook, as a human voice reads the words of Winston Churchill.
This acoustic thread woven into the piece lends viewers a participatory element which drives the dramatic intensity all the way through.
The Importance of Scientific Hearing
We are accustomed to using visual models to understand science, but we are far less likely to find or use sound models to understand or explain science. This may be related to the importance of visual perception in our evolution, yet hearing is just as critical to our survival.
Yet we put seeing before hearing in almost all cases, while evolution seems to have made hearing as important. Why would nature give so much complexity and power to hearing if it was less important? Thoughts to keep in mind.
The following video highlights the physiological science behind how we hear.
I bring this up as hearing the shapes and functions of scientific processes became as important as visualization in some theories I built recently. Yet science tends to focus heavily on visual data representation.
Sound is not absent from science. Ultrasound, sonar, NMR, the sonification of gravitational-wave and solar data, and phonon spectroscopy all use sound or vibration.
Science tends to treat sound as an instrument rather than as a way of thinking or explaining. Whereas visual representations are taken for granted, while sound representations are unusual.
This essay seeks to explore why this might be the case, not yet to repair it.
I am accustomed to “seeing” sound represented as a wave form, or as various forms of musical notation. Yet when I visualize sound mentally I see either a shape or a process, always associated with an emotion. Skillful film makers never ignore sound quality as being equally important to the visuals.
Sound carries a dimensional sense of space and time that sight alone struggles to hold; it is the two together, in a well-functioning nervous system and body, that give us the fuller picture.
Sound is a pressure wave that unfolds over time, so the ear takes in space and time together. The ear finds where a sound is from tiny differences in when it reaches each ear, and it reads a room’s size from how the echoes fade. A picture freezes one moment, while sound carries the moment and its motion at once. The brain combines sight and sound, and the combination is more accurate than either sense alone.
There is an interesting asymmetry here. Humans can almost see through sound: some people echolocate, and most of us can hear the size of a room from its echo. Yet we can hardly see sound itself. Our ways of visualizing it (Chladni plates, spectrograms, waveforms) flatten what the ear holds simultaneously. A chord is one experience to the ear and a stack of lines to the eye.
For many, if they visualize sound mentally, they see a shape or a process, one which is frequently tied to an emotion. Musicians build whole careers around seeing a shape and an emotional content in sound, and in knowing how to apply that in composition. A waveform or a line of notation records sound; your inner image keeps its motion and feeling. Does sound reach emotional centers more quickly than other perceptions? Doubtful but interesting. Yet skillful film makers, ever intent on the best methods of telling a story, tend also to be very good at making sure the sound track carries its weight. As a recent example, Christopher Nolan’s Dunkirk, with its score by Hans Zimmer, is a good example. In this film sound is made equal to the visuals rather than placed underneath them. The texture is metallic and industrial, with deep bass, and the ticking of Nolan’s own pocket watch runs through it. Zimmer also uses the Shepard tone, an auditory illusion of pitch that seems to rise endlessly without ever arriving. The soundtrack maximizes the held tension, winds up the tension like a watch, and does not let the observer resolve it. A partial resolution comes only at the end, when a human voice reads Churchill’s speech from the newspaper as the score moves into a slowed version of Elgar’s “Nimrod.” Thus a human voice resolves the tension, in contrast to the heavily industrial music which holds the tension until the resolution. The sound supplies an emotional energy the visuals cannot. In a loose sense it is thermodynamics in disguise: energy stored and withheld, then released. If a filmmaker treats sound as equal to image in order to understand an experience, why doesn’t science do the same to understand a process?
Attempting to contrast what is done in science with the concept of what is required in war, met here by invoking Dunkirk, is that both possess a tension that winds before it resolves, a real cost, and, in the Dunkirk case, a human voice that brings the resolution. The struggle to understand better or to survive a war is a thermodynamically expensive path, while drifting into ignorance or into failing to survive is the cheap thermodynamic path.
Why, then, does science put seeing first?
Part of the answer may be the page itself. Science is kept on paper and on screens, and a page holds a picture far more easily than a sound. A graph can be printed, cited and returned to; a sound has to be played, and it is gone again the moment it ends. What is easy to store becomes what is easy to trust.
Part of the answer may be time. A picture can be taken in at a glance and read in any order the eye chooses. Sound will not be hurried. It must be heard in the order it unfolds, at the speed it unfolds, and a minute of sound costs a minute of listening. A science in a hurry chooses the faster sense.
Part of the answer may be the emotion that travels with sound. If the inner image of a sound is a shape or a process tied to a feeling, then hearing brings feeling into the room with it. Science has long trained itself to keep feeling out of the room, and it may distrust the sense that refuses to leave feeling behind. Seeing is believing; hearing is feeling. The first sounds like evidence and the second sounds like art.
And part of the answer may be habit. Once the instruments, the journals and the training are all built around the eye, each new generation learns to see its data before it ever thinks to listen to it. The habit becomes the method, and the method stops being questioned.
None of these reasons says that sound is less able to carry a scientific idea. They say only that our ways of keeping, sharing and trusting knowledge were built for the eye. The filmmaker has no such excuse, and so the filmmaker listens.
What, then, does hearing add?
The first thing it adds is a mathematics performed before anyone decides to do it. The cochlea is a tapering channel whose stiffness changes along its length, so an incoming pressure wave breaks into its frequencies at different places: high tones near the base, low tones near the far end. Békésy measured this travelling wave and described the cochlea as a frequency analyser, work for which he received the 1961 Nobel Prize. The listener does not choose this analysis and cannot switch it off. A chord arrives at the nerve already separated into its parts. The eye does nothing comparable. A spectrogram is the work of a machine; the ear’s spectrogram is the ear.
The second thing it adds is the fact that the ear spends energy in order to hear. In 1948 Maurice Gold proposed that the cochlea must feed energy back into the wave it is analysing rather than merely receive it. That turned out to be right: the outer hair cells act as an active process, amplifying faint input by more than a hundredfold, sharpening the discrimination between neighbouring frequencies, and compressing a range of amplitude of a million to one into what a person can hear. Hearing is therefore not a passive registration but an expenditure. The organ pays for its own selectivity. This is the thermodynamics I have been looking for, found in the sense itself: the ear does not receive the world cheaply. It buys a picture of it.
Third, the ear keeps better time than the eye. A listener with normal hearing can detect a gap in a sound of about three milliseconds, and can tell one click from two when the interval between them is a few tens of microseconds. The eye, by contrast, fuses a flicker into a steady light at roughly thirty-five to forty flashes a second, which is about twenty-five milliseconds. The two ears compare the arrival time of the same sound to within some ten microseconds, a difference in path of only a few centimetres. Where a process is fast and ordered, hearing resolves what seeing simply averages. Science studies processes far more than it studies surfaces, and the sense best fitted to a process is the one it consults least.
Fourth, hearing separates many sources from one mixture. The difficulty was named the cocktail party problem by Colin Cherry in 1953, and Albert Bregman built the general account of it in 1990 under the name auditory scene analysis: the auditory system recovers individual streams from a single pressure wave. A room full of people talking arrives at the eardrum as one wave and is heard as a dozen people, a heater, a door. Nothing about the wave arrives labelled. The eye, shown a superposition of this kind, cannot recover the components unless it is told beforehand what they were. This is the capacity I have called holding several dimensions at once, and it is neither a metaphor nor a gift of long training. It is ordinary hearing doing what it does without being asked.
Fifth, hearing reports the space and not only the surface. Listeners judge a room’s size from its reverberation, and can often say which room a recording was made in. A photograph of a cathedral shows its walls; the sound of a cathedral tells you what the air inside does to a voice. Sight reports where things are. Hearing reports the medium they sit in and what a disturbance does while crossing it. For anyone whose subject is a fluid, a membrane, a gel, or an atmosphere, this is a sense with a direct line to the thing under study.
Sixth, it cannot be closed. There is no eyelid for the ear. Auditory processing continues through sleep, and the primary auditory regions keep answering sound. Hearing is the sense of the unwanted and the persistent, the one that brings the world in whether or not the observer consents. A discipline that has trained itself to keep feeling out of the room, and that values the ability to look away at will, will naturally prefer the sense it can shut.
Hearing matters to survival for a reason that has nothing to do with pitch. It is the sense that works when the eye cannot. It reports what is behind the head, in the dark, in the tall grass, in the sleep a hunter has to have to be dangerous the next day. There is no eyelid for the ear, and there is no direction the ear has to turn to face. An animal lying still with its eyes shut is not lying helpless, and the reason is that the second sense is still open and still answering.
What the ear delivers about distance is not precise, and it is honest to say so. Out in open ground people systematically underestimate how far off a sound is, and near at hand they do the opposite, bringing distant things closer than they are. The reports are usable, and they are biased. But out in the dark, where the eye delivers no place at all, a coarse and biased place is not a small thing. It is the difference between an animal that knows which way to move and one that does not.
What makes the ear’s report usable is that the brain does not take it as it comes. It fills it in, and it fills it in from the other senses. This is not a guess of mine; it has been measured for years under the name crossmodal correspondences, which are the reliable tendencies people show to match a feature in one sense with a feature in another. High pitch goes with small things, with brighter surfaces, with pointed shapes and with positions higher in space. A sound can also change the feel of a surface under the hand, and it changes the taste of what is in the mouth: the same toffee has been rated significantly more bitter when it is eaten to a soundtrack built to sound bitter than to the identical toffee eaten to a soundtrack built to sound sweet, the food held constant and only the sound altered.
The most famous of these matches is the one where most people, shown a rounded blob and a spiky figure, will call the round one bouba and the spiky one kiki. That effect has now been tested across speakers of twenty-five languages and ten writing systems, and it holds in most of them. It does not hold in all of them, which is the part worth keeping: in the same study the agreement ran from a hundred per cent in one language down to thirty-six in another, and three languages showed no effect at all. The correspondence is a tendency the nervous system brings to the world, not a law the world imposes. It is a habit of filling in, and habits can be absent.
So the dimensions hearing reports are not carried whole to the brain by the ear. Position arrives from a comparison between two channels. Distance arrives under- or over-estimated and is corrected against what the eye knows. Pitch arrives already wearing size, brightness and shape, and it goes on to alter the mouth’s report. This is what I mean when I say the brain must fill in the dimensions: the sense delivers part of a world and the rest is supplied from the ledger the senses share.
That shared ledger is the last and the largest of the relational facts in this essay. The correspondence is not in the tone and not in the shape; it is between them, and it is maintained by a body that has to act on one world rather than five separate streams of data. A science built on the eye keeps its senses in separate departments and calls that rigour. A science that heard would have to account for the accounting between them, which is a harder and more interesting problem, and the one my own work keeps arriving at from the direction of geometry instead of the direction of the ear.
There is one further thing the ear adds, and it is the reason this essay comes back to the ground my own work stands on.
Begin with where a sound is. Neither ear’s signal contains the position of the source. What reaches an eardrum is a pressure wave, and a pressure wave taken by itself says nothing at all about where it came from. The position appears only in the comparison: which ear the wave reached first and by how much, and which ear it reached more strongly. Lord Rayleigh set out the account in 1907 that still stands, that the ear uses the time difference at low frequencies and the level difference at high ones. Two channels, neither of which knows anything about place, and a place in the difference between them.
Consider next what happens in a room. A single handclap leaves the walls and comes back as a dozen arrivals, each carrying its own spatial clues, most of them false. The listener does not hear a dozen claps and does not hear the room’s geometry; one event is heard, at the position of the hand. Wallach, Newman and Rosenzweig showed in 1949 that the first arrival governs the heard place while the later copies are suppressed rather than added in, work still studied under the name law of the first wavefront. The position is not read out of the wave. It is a verdict about the order of several waves.
Now the strangest case, and the one I would most want a reader to sit with. Build a sound out of the harmonics of two hundred cycles a…