Audioscapes: Hearing is Believing?

warning: Creating default object from empty value in /home/buckman/public_html/neo/modules/advanced_forum/advanced_forum.module on line 492.
Matt Barton's picture

As I was playing Dungeons of Daggorath again this morning, I was struck by how much clever audio can enhance a game, more than making up for simplistic graphics and the like. I think we have a tendency to strongly underestimate the power of sound for suggesting certain emotions and sensations. When I was a kid, I bought a cassette tape called "scary sounds," the sort that gets sold around Halloween. When I got home, I turned off all the lights and listened to the tape in a dark room. I didn't last long--about five minutes into the tape and that light was back on, buddy. Hearing spooky sounds and not being able to see what they are is very terrifying, perhaps more terrifying than seeing a scary picture or video.

I usually avoid speculating on evolution and instinct's role in this kind of thing, but it does seem pretty obvious why we might have such strong reactions to certain sounds. After all, in the wild we might very likely hear a predator a long time before we saw it. Imagine being deep in the forest and hearing a panther's cry, for instance. Although your vision would be obscured by all the limbs and leaves, the sound would put you on alert. You'd probably end up turning your head frequently to see if something was creeping up behind you, another very scary feeling. My guess is that these are remnants of survival instincts that kept our kind alive in a more dangerous stage of our development.

In recent games, I was struck by Doom 3 and Bioshock's "scary" audio, though for some reason I've yet to be seriously impressed with surround sound. Perhaps I just don't have a good set of speakers, but it just doesn't seem to be sensitive or well-developed enough to genuinely make me feel (aurally, that is) that I'm in standing in the environment depicted on the screen. I'm not quite sure why this has failed (could just be something like speaker placement), but it would *seem* to be an easy enough thing to do right. Of course, it could also just be that we hear much better if something is in front or beside us than behind. I'm not quite sure the "spatial" aspects are important as the way the sound is being generated.

But, back to the topic at hand. DoD had that great heartbeat sound that keeps you so immersed and responsive in the game. What are some other tricks you can do with audio to heighten immersion and ratchet up the tension? Clearly, music is a great resource here, though we'd really need better procedurally generated music to make any real progress in this area. I know of a few games here and there that have tried to do this, but none (to my knowledge; correct me) that have truly been successful. Seems a bit strange; how hard could it be just to make some tension-inducing "strings" music when a monster is drawing near? I know of some games that have done this, but shouldn't it pretty much be a standard feature? And why not go beyond that, to create a "game audio language" with other kinds of music or sounds to signal a whole variety of moods and sensations?

To wit; we know that different kinds of roads sound different to us as we drive over them. I'm not sure how aware we are that our brains file that kind of information away, so that we can "stay on track" or at least feel comfortable when we hear that particular sound because it means we're on the right road. Surely it wouldn't be too hard to make a game so that different parts of the world sounded different; not just "ambiance" and simple reverb effects and such (although they're rare enough), but genuine audio responses to whatever we were doing there.

I realize that one of the biggest obstacles here is that we're essentially using what amounts to a "tile" system for game audio. That is, in the same way that we used pre-made "tiles" and arranged them to create maps and such in older games, we're using a set amount of sound samples to create audio in modern games. Audio software hasn't caught up with graphics software in this regard. For instance, every time you fire your gun in a FPS, it probably sounds exactly the same (or perhaps only varying a bit in volume or such). However, in reality, no two gunshots are exactly identical. Imagine how distinct two or three different explosions sound in real life. Unfortunately, the "tile" problem has also crept into modern films. Go back and watch an old Western shootout (Two Mules for Sister Sara is a good one) where they have no CGE and are actually using real explosives and blanks rather than just piping in pre-recorded sound effects. Although some parts look faker, I'm always impressed by how these sequences seem to "pop," almost as if I'm hearing a stageplay.

My guess is that most games probably have something like a 10:1 ratio to repeated to unique sounds. That is, pre-recorded dialog and cutscenes and such aside, the great bulk of what you hear is repeated throughout the game. This seems to me to represent a great hurdle, but one that could be overcome if we weren't so worried about "realism" and were willing to experiment with different kinds of audio. Instead of just trying to copy films, we need to think about ways to integrate audio in clever, innovative ways, ways that only games can truly capitalize on. Unfortunately, I too am stumped beyond obvious examples like heartbeats and such, but I'm certain there is great room for development here. If nothing else, it shouldn't be hard to procedurally generate bird calls, murmurs, wind noises, rain splattering, etc., so that none of these sounds were exactly identical.

If you really want to read about some fascinating applications of what I'm talking about here, read the Wikipedia entry on Laugh Track. This article really gets at the problem with using sound samples, but also how we can train ourselves to accept it (i.e., we know full well that we're listening to a laugh track, but that doesn't seem to matter at all and still causes us to laugh more often than without it).

Comments

Chris Kennedy
Chris Kennedy's picture
Offline
Joined: 08/31/2008
Synergy

I think it is quite fair to say that the assembly of all things - play control, gameplay, pacing, the visual aspects, the audio aspects, force feedback, etc can alter, enhance, or sometimes diminish the feeling of the game. I actually want to look at a particular movie to explain how you can grip a viewer by stimulating them in multiple ways.

Back to the Future -

Forgive me if you haven't seen the film, but I think it is safe to assume everyone here has seen it. I like a good many films, and several have certain sequences in them that I can watch over and over and over again. Back to the Future has a scene that most people refer to as the "Clock Tower scene."

In this scene, Marty is awaiting the alarm to go off on his new dashboard alarm clock so he can start his drive and go back to the future. Meanwhile, Doc is struggling to fix the wiring problem at the clock tower after a tree limb unplugged the lead from the lightening rod to the lamp posts. The entire sequence from the time the alarm goes off to when Marty goes back to the future is just cinematic bliss for me. Alan Silvestri's score, the building tension, the constant lightening strikes in the background, the acceleration of the Delorean - all of it comes together until that Lightening bolt strikes and sends Marty forward in time. I cannot tell you how many times I have just cranked that scene on the home theater. Yes - the audio really makes that scene. Playback the whole thing without the music. Play it back without the ambient sound effects. Does the emotion drop out of it? Yes. Now take away the video and just listen to the scene.

I would gamble that you feel more emotion with Audio minus Video than you do Video minus Audio! Surround sound or not, it makes a huge difference.

I remember when they first started putting Dolby Pro Logic/Surround into the console games and Creative Labs and Aureal had competing 3D sound formats. I have to admit that I didn't really do the best job setting up a *second* surround system for my computer in order to enjoy these things. Still - One has to properly calibrate a home theater in order to make it enjoyable. I think it is a safe assumption to say you would need to do the same thing for a computer. You need to be enveloped in the sound.

But! Let me also say this - I think the most overlooked gaming accessory when it comes to computer gaming is your chair. Seriously. If there is any bit of discomfort from your chair, the graphics and sound will not suck you in. It's just a fact. If you are distracted or even subconsciously distracted by some minor (or naturally, a major) annoyance, it just won't work.

Focusing more on Matt's topic, I think sounds can play a pivotal role in gaming. I also think that the ambient sounds are sometimes more important than the gameplay-specific sound effects. How many times do they repeat? How do they overlay? Too much? Too little? Too loud? Aside from the repetition, you still have to have a good mix.

I don't have a lot of examples to give, but I will say that the final battle of Persona 3 was something that really struck a chord (no pun intended) with me. The music during that sequence is great, and the sound effects almost seem to mesh with the music. Specifically, the heroes twirl and fire evokers (they look just like guns) at their own heads to awaken a monster to execute an attack (Yes, it is Japanese). The twirl and fire seems to fit quite well with the energy level of the music. The heroes actions and sound effects have been there the entire game, but wow does it ever pair well with the music.

I think this synergy in gaming is something that can add a lot to the emotions of gaming and leave a lasting impression. I mean, heck - I can't tell you of a lot of *specific* gaming memories I have while playing games - I can mostly just talk about the fun I had playing certain games. But that sequence in Persona 3 - when the battle, gameplay, control, music, and sound effects game together - that left a lasting impression on me.

n/a
Rowdy Rob
Rowdy Rob's picture
Offline
Joined: 09/04/2006
Sound, back to the future
Chris Kennedy wrote:

Back to the Future -

Forgive me if you haven't seen the film, but I think it is safe to assume everyone here has seen it. I like a good many films, and several have certain sequences in them that I can watch over and over and over again. Back to the Future has a scene that most people refer to as the "Clock Tower scene."

Me too, I love watching the "Clock Tower" sequence over and over! It's one of the most perfectly executed sequences in cinema history, in my opinion. I don't think you're giving the visual side enough credit, though. The cinematography, editing, acting, and special FX were all top notch, and brought the movie to an extremely exciting conclusion! The only reason the sequence is working for you with just the sound is because you're seeing the movie in your mind as the sound is playing, I suspect. But, back to the main topic at hand.

Chris Kennedy wrote:

Still - One has to properly calibrate a home theater in order to make it enjoyable. I think it is a safe assumption to say you would need to do the same thing for a computer. You need to be enveloped in the sound.

I've been having problems properly calibrating sound on my computer. I recently installed "Mass Effect," and the sound mix is so screwed up that the music overpowers all other sound, particularly voices. I had subtitles turned off, so I was placed in situations where I had to respond to questions I couldn't hear! No matter what I did, the problem remained unless I turned the music off, which I didn't want to do, because, as we are discussing here, the music is part of the sound experience.

Chris Kennedy wrote:

Focusing more on Matt's topic, I think sounds can play a pivotal role in gaming. I also think that the ambient sounds are sometimes more important than the gameplay-specific sound effects. How many times do they repeat? How do they overlay? Too much? Too little? Too loud? Aside from the repetition, you still have to have a good mix.

Max Payne had very atmospheric sound effects, as did the similar "Bioshock." The GTA games always seemed to have great ambient sound effects that really made you feel like you were in a living city.

As for where we go from here, I don't really know. Algorithmic musical soundtracks? It almost seems to me that we've "topped out" what games can do in the audio department, short of synthesized voice acting.

qoj hpmoj o+ 6uo73q 3Jv 3svq jnoh 77V

Cody Reimer
Cody Reimer's picture
Offline
Joined: 10/14/2008
The Sound of a Swift Death

I agree with Matt. The repetition of sound effects can mire a production, be it game or film, in a way that you'd imagine only the regurgitation of visuals could (the same explosion sequence used for two different explosions). There are, however, exceptions. The Wilhelm scream is a great example. http://en.wikipedia.org/wiki/Wilhelm_scream

I always swore I heard that same person yelling, but not until Cracked wrote an article ("5 Things Hollywood Reuses More Than Plots") did I realize it was the exact same sound effect. Now, whenever I hear that yell, I smile quietly to myself. It's a type of aural allusion or reference, more nuanced and obscure than most, and therefore more satisfying to recognize. But most films don't use the Wilhelm scream more than once, and to do so would be like reiterating the same explosion sound effect.

Doom 3 was nearly unplayable in the dark, the suspension of disbelief was so great. The game is a great example of the convergence of numerous senses to create an ambience of horror. Every time the screen jolted or I had to lower my flashlight or I heard a girl giggling or a monster moaning, I shuddered. Games have a tendency to use sounds in dynamic ways; I never understood why my friends could listen to music while playing games that involved hearing. In WoW, there are few better ways to get the drop on a stealthed rogue than hearing that indicative "stealthed" noise move past your character. In Left 4 Dead 2, each of the special infected have key sound identifiers: the Smoker coughs, hacks, and wheezes; the Hunter shrieks; the Jockey giggles uncontrollably; and so on. Hearing and recogninzing which sound belongs to which zombie is absolutely vital when playing at the advanced levels (or against other players), because with few exceptions, the players will always hear the special infected before they see them.

Sound mapping a game, or creating the equivalent of road's shoulder markers, is an interesting concept, but the implementation would be tricky depending on the genre. It'd be a creative alternative to just throwing up walls or unconquerable obstacles (oops, not supposed to go there yet--that monster just killed me in one hit!). My question would be how to let the player know what the sounds mean. If an adventurer is traversing through a forest and the birdcalls and crickets die away to be replaced by the sounds of blinking eyes, does that mean their adventure has truly begun (ho, danger is afoot) or they should turn back (ho, danger is afoot and I'm only level one!).

The musical scores in game seem to matter as much as, if not more, than the identifying sound effects. Chrono Trigger and some other RPGs have such memorable scores that players remake them using beer bottles http://www.youtube.com/watch?v=7AJMujiJ-UQ or Mario Paint http://www.youtube.com/watch?v=8PHmE8MdTuI&feature=related There is just something about the mood of adventure that some of those scores instill. The nostalgia they produce matches any visual.

Cody Reimer
Freshman Composition TA
St. Cloud State University

Bill Loguidice
Bill Loguidice's picture
Offline
Joined: 12/31/1969
Canned sounds
Cody Reimer wrote:

I agree with Matt. The repetition of sound effects can mire a production, be it game or film, in a way that you'd imagine only the regurgitation of visuals could (the same explosion sequence used for two different explosions). There are, however, exceptions. The Wilhelm scream is a great example. http://en.wikipedia.org/wiki/Wilhelm_scream

There is another canned scream that constantly appears, even in videogames. It's used in Eye of the Beholder for the GameBoy Advance, as well as several other games and media. I would love to know the genesis of that particular scream. It's a manly type of scream, almost like an "aargh!".

We more often think of stock footage and stock photos rather than stock sounds, but if you really think about it, stock anything is noticeable after a while.

***************************
Bill Loguidice, Managing Director
Armchair Arcade, Inc.

n/a
Matt Barton
Matt Barton's picture
Offline
Joined: 01/16/2006
Damn Server!!!

Damnit! I had typed out a long response and it got ate up in some kind of server crash. Grrr....

Anyway, I liked what I typed, so I'll try to quickly rehash it.

So, I have in mind two innovations: PGA (procedurally generated audio) and "sound language."

First off, I'd like to forget the idea of procedurally generated music. I don't think this will ever evolve beyond either very mechanical sounding pieces or the type of "ambient" music with no melody or tune. There will never be a substitute for a composer, and even one of basic skill will likely surpass even the best computer-made music. The problem is that music, like good writing and so on, relies too heavily on seemingly arbitrary choices made by a composer, whose grasp of the whole goes far beyond what is possible in an algorithm. At best, a computer could only mimic or imitate, but I doubt anyone but the most tone deaf person could tell at once it was made by a computer, not a composer. I'm not an expert musician by any means, but I know enough to know it's more than a computer can handle. There may be new kinds of music that a computer could do well, and perhaps the "composer" in that sense would be the one shaping the algorithms. Still, it seems like a dead end to me.

Conversely, computers would seem ideal for PGA for sound effects. It seems easy enough to me to record a few thousand different gunshots, have the computer analyze and data mine, and come out with some type of algorithms that could generate a unique gunshot on-the-fly based on the parameters of the virtual environment (compensating for distance, acoustics of the room, etc.). Screams and laughter would seem more daunting challenges, but again I see no reason why it couldn't be done given a large enough pool of data for the computer to mine. I'm particularly interested in bird calls and nature sounds, since there seems to be some real potential to use data mining to arrive at believable "alien" soundscapes just by blending in all the different sounds from many different zones. I guess you'd need to limit yourself to climates and zones that are roughly equivalent to the virtual ones (i.e., a forest rather than a jungle or desert). There seems to be a point where few people outside of experts care, though--I mean, how often have you watched a movie and heard a bird and thought, "That's impossible. That bird doesn't live in that part of the country" or some such. Then again, since we're just dealing with data and algorithms anyway, I don't see why it would be THAT difficult just to get it right. How hard is it just to look at a book and find out what birds and such you're likely to hear? But, I digress.

So, if you wanted to make a PGA of a particular bird call, just get recordings of a few hundred or thousand such calls from different birds, have the computer analyze the data and come back with algorithms with the correct parameters (i.e., what is possible given the range you recorded), and viola! I realize I'm making it sound trivial whereas I'm sure doing the data gathering and analysis is serious stuff...But, you get the idea.

Now, sound languages. What I have in mind here is the little sounds we're used to in games, such as the "jumping sound," plopping on tiles, sounds that play when you get coins and such...Okay, now I'd say that this stuff, when it's shared across enough games to become conventional or standardized, is the rudiments of a very simple language. My idea is simply to take this concept much, much further, in effect reversing what I said about PGA above. Instead of having the computer learn the sounds of our world, we have the computer teach us the sounds of its world. Given enough time and some PGA techniques, I'm sure we could end up with some very sophisticated and elaborate sound languages that could add a whole new dimension to gameplay, particularly since you can do things with games that you can't with other media. I have a rough time even imagining what this could sound like, but perhaps games like Captain Blood offer a glimpse. Indeed, I was thinking about speech synthesis as well. Again, I don't think we will ever have speech synthesis that is indistinguishable from a real person. But why bother? Instead, we could learn to understand the speech the computer is capable of synthesizing. So, just have a game set in a world of robots where we expect the robots to talk like that. Go with what works.

n/a
sascha/hdrs (not verified)
game soundtracks

For me the audio is more important than great graphics in a game and many recent games do it completely wrong. They throw in some generic orchestral noise or some other prefab music where you can hear in the first minute that the developer didn't care much about the audio of the game.

Luckily there are some games that have received an excellent audio job. Check out Vampire: Bloodlines - The Masquerade for example (not the licensed music in it but the soundscapes by Rik W. Schaffer). Can it get any more atmospheric?!

If a game has poor audio it almost destroys the fun for me. Good Example: Mass Effect 2 - awesome visuals cannot make up for the (mostly) bland synth soundtrack. There are a few tracks that are alright but the bigger part of it is just soo boring. When asked about their motives with that kind of music the devs told that the music is supposed to reflect the vast coldness and loneliness of space, or something along these lines. But that's a poor excuse for me! The music is still dull and boring.

@Matt Always copy your text to the clipboard before hitting submit! ;)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.