Is sound synthesis a relatively new area for computer graphics (i.e. is it currently only in research, or is it being already used in industry)? I was also wondering whether we need the whole animation or simulation to be fully run first before we can generate a sound for it, or if we can go frame by frame of the simulation to create the sound as we are rendering the visuals.
keenan
@Eaglee It's a relatively "young" sub-area of graphics, though people have been thinking about it for quite some time. Actually, a lot of work on sound synthesis was done here at CMU by Doug James (now at Stanford).
As for generating the sound, you can essentially generate a short snippet of sound from a short snippet of motion, though this motion has to be sampled at a much higher rate (e.g., 44,000 times per second) in order to get high-fidelity audio, versus about 30-60fps for video. For this reason, people cook up all sorts of techniques to approximate what happens at the high frequencies, or use different models that are better adapted to the task (e.g., working with modal vibrations).
jkalapos
Before this lecture I had not really thought about sound and touch being a part of graphics. I found defining computer graphics broadly as the synthesis and manipulation of sensory information interesting because I know that, focusing on sound specifically, people have tested out many more physical ways to develop this sensory information for their media. This video I found (https://www.youtube.com/watch?v=Sn6MFI7-gjQ) shows how some old Disney movie sound effects were made back in the day. Contrasted with this idea of simulating the sound effect digitally along with the animation itself, it's kind of neat to see how far we've progressed from using coconuts to recreate the sound of horse hooves.
silentQ
You mentioned smell in lecture as something that was sort of weird or "out there," but it's been around for a while (longer, I think, than haptic feedback). First there was "Smell-o-Vision" and "4D" theaters that sprayed certain scents at you. But now I see a lot of artists doing cool things with scent. A recent one I found really interesting is "Meta Cookie" (https://gizmodo.com/5608457/meta-cookie-uses-virtual-reality-to-make-a-cookie-taste-like-anything-you-want) which used smells to successfully fool people into thinking a bland cookie was actually a variety of flavors.
keenan
@jkalapos Fun! Though this also ties into another interesting issue: for someone like Disney, it may be that the coconuts sound “better” than actual horse hooves, from an artistic point of view. So beyond just doing accurate physical simulation, folks in graphics have to think about ways to provide additional creative (and often non-physical) control over simulations.
keenan
@silentQ Gross! But also really fun. Thanks for the links. :-)
Is sound synthesis a relatively new area for computer graphics (i.e. is it currently only in research, or is it being already used in industry)? I was also wondering whether we need the whole animation or simulation to be fully run first before we can generate a sound for it, or if we can go frame by frame of the simulation to create the sound as we are rendering the visuals.
@Eaglee It's a relatively "young" sub-area of graphics, though people have been thinking about it for quite some time. Actually, a lot of work on sound synthesis was done here at CMU by Doug James (now at Stanford).
As for generating the sound, you can essentially generate a short snippet of sound from a short snippet of motion, though this motion has to be sampled at a much higher rate (e.g., 44,000 times per second) in order to get high-fidelity audio, versus about 30-60fps for video. For this reason, people cook up all sorts of techniques to approximate what happens at the high frequencies, or use different models that are better adapted to the task (e.g., working with modal vibrations).
Before this lecture I had not really thought about sound and touch being a part of graphics. I found defining computer graphics broadly as the synthesis and manipulation of sensory information interesting because I know that, focusing on sound specifically, people have tested out many more physical ways to develop this sensory information for their media. This video I found (https://www.youtube.com/watch?v=Sn6MFI7-gjQ) shows how some old Disney movie sound effects were made back in the day. Contrasted with this idea of simulating the sound effect digitally along with the animation itself, it's kind of neat to see how far we've progressed from using coconuts to recreate the sound of horse hooves.
You mentioned smell in lecture as something that was sort of weird or "out there," but it's been around for a while (longer, I think, than haptic feedback). First there was "Smell-o-Vision" and "4D" theaters that sprayed certain scents at you. But now I see a lot of artists doing cool things with scent. A recent one I found really interesting is "Meta Cookie" (https://gizmodo.com/5608457/meta-cookie-uses-virtual-reality-to-make-a-cookie-taste-like-anything-you-want) which used smells to successfully fool people into thinking a bland cookie was actually a variety of flavors.
@jkalapos Fun! Though this also ties into another interesting issue: for someone like Disney, it may be that the coconuts sound “better” than actual horse hooves, from an artistic point of view. So beyond just doing accurate physical simulation, folks in graphics have to think about ways to provide additional creative (and often non-physical) control over simulations.
@silentQ Gross! But also really fun. Thanks for the links. :-)