Adding Audio

Immersive applications often take advantage of sound to enhance suspension of disbelief. VR Juggler application programmers have many options available to them for adding sound to their virtual environments. In this section, we describe the library Sonix that ships with VR Juggler. Sonix provides a layer of abstraction for third-party audio libraries including OpenAL, Audiere, and AudioWorks. In VR Juggler 2.0, there is one Sound Manager implemented using Sonix. As such, Sonix is the easiest tool to use for adding sound to VR Juggler applications, but it is not the only option.

Sonix provides simple audio sound objects on top of several audio APIs. The interface to Sonix is kept very simple in order to get people up and running with sound as fast as possible. Sonix has the following features:

VR Juggler application programmers can use the full Sonix API directly, or they can take advantage of the VR Juggler Sound Manager to reduce the amount of coding required. We will present both approaches. Example applications can be found in the directory $VJ_BASE_DIR/share/vrjuggler/samples/sound/simple.

Using Sonix Directly

We begin our explanation of adding sound to a VR Juggler application by presenting the direct use of the Sonix API. This requires more programming than using the VR Juggler Sound Manager, but it will be useful to understand how Sonix operates in order to take full advantage of the Sound Manager. In particular, reading this section is a necessary part of understanding general Sonix usage, and readers are encouraged to read both this section and the next before deciding on which approach to use in their VR Juggler applications.

Basic Sonix Usage

In Figure 6.1, “Basic Sonix Interface”, we see the API for Sonix (see Figure 6.2, “The Sonix Design” for the complete software architecture). The main parts that a sound programmer will use are the classes snx::SoundHandle and snx::sonix (a singleton). The snx::SoundHandle class is used to manipulate individual sounds. The snx::sonix singleton class is used to start, stop, and reconfigure the sound system. Both classes must be used for any sound to be heard.

Figure 6.1. Basic Sonix Interface

Basic Sonix Interface

Starting the Sonix system is easy. Basically, the only call needed to start the system is snx::sonix::changeAPI(). The backend audio software is loaded into Sonix through a plug-in system. The plug-in to use is identified by a string name, one of “OpenAL”, “AudioWorks”, or “Audiere”, as shown in Example 6.12, “Initializing the Sonix Sound APi”. The choice of which to use depends on availability and compatibility. For example, AudioWorks is available on the IRIX operating system, but it can only be used with a build of VR Juggler that uses SPROC threads. On the other hand, OpenAL and Audiere on IRIX can only be used with a pthreads build of VR Juggler.

Example 6.12. Initializing the Sonix Sound APi

snx::sonix::instance()->changeAPI("OpenAL");

Setting up a sound is designed to be simple, and an example of doing so is shown in Example 6.13, “Setting Up a Sonix Sound Handle”. Here, we use an instance of snx::SoundInfo to configure the sound object, which is in turn accessed by an instance of snx::SoundHandle.

Example 6.13. Setting Up a Sonix Sound Handle

snx::SoundInfo info;
info.filename = "crack.wav";
info.datasource = snx::SoundInfo::FILESYTEM;

snx::SoundHandle crack_sound("crack");
crack_sound.configure(info);

Periodically, an update function must be called to keep Sonix running. The method is snx::sonix::step(), and it takes a single argument of type float representing the time since the last time it was called. In general, this should be invoked in the frame function of an application. In the context of VR Juggler, this would be either preFrame() or postFrame() in the application object interface. An example of using this function is shown in Example 6.14, “Sonix Frame Update”.

Example 6.14. Sonix Frame Update

void MyApp::preFrame()
{
   // Update application state for the current frame
   // ...

   float time_delta = getTimeChangeInSeconds(); // use a system call or other API
                                                // to get the time delta
   snx::sonix::instance()->step(time_delta);
}

Example 6.15. Complete Sonix Program Using OpenAL

  1 #include <iostream>
    #include <string>
    #include <snx/sonix.h>
    
  5 int main(int argc, char* argv[])
    {
       std::string filename(:808kick.wav"), api("OpenAL");
    
       if ( ! snx::FileIO::fileExists(filename.c_str()) )
 10    {
          std::cout << "File not found: " << filename << "\n" << std::flush;
          return 1;
       }
    
 15    // start sonix using OpenAL
       snx::sonix::instance()->changeAPI(api);
    
       // fill out a description for the sound we want to play
       snx::SoundInfo sound_info;
 20    sound_info.filename = filename;
       sound_info.datasource = snx::SoundInfo::FILESYSTEM;
    
       // create the sound object
       snx::SoundHandle sound_handle;
 25    sound_handle.init("my simple sound");
       sound_handle.configure(sound_info);
    
       // trigger the sound
       sound_handle.trigger();
 30    sleep(1);
    
       // trigger the sound from a different position in 3D space
       sound_handle.setPosition(10.0f, 0.0f, 0.0f);
       sound_handle.trigger();
 35    sleep(1);
    
       // this simulates a running application
       while ( 1 )
       {
 40       snx::sonix::instance()->step(time_delta);
       }
    
       return 0;
    }

Reconfiguration

Sonix is reconfigurable, allowing audio APIs to be swapped out at run time safely without the dependent systems noticing. Applications using Sonix expect to be completely portable. Changing sound APIs at run time can be useful so that the user can experiment with quality and latency differences of different hardware and sound APIs. If no audio API is available on a given platform, calls to Sonix simply are ignored. This means that applications do not have to have special code to enable or disable sounds based on the availability of the backend sound software. These details are handled transparently by Sonix.

Everything in Sonix can be reconfigured behind the scenes during application execution. See Example 6.16, “Reconfiguring Sonix at Run Time” for a trivial example of how to reconfigure Sonix in C++. In a more complex context, this feature of Sonix has powerful implications. Sounds can be changed on the fly, as well as the API being used to render the sound. Such details are hidden behind sound handles and the snx::sonix singleton.

Example 6.16. Reconfiguring Sonix at Run Time

  1 // start sonix using OpenAL
    snx::sonix::instance()->changeAPI("OpenAL");
    
    // fill out a description for the sound we want to play
  5 snx::SoundInfo sound_info;
    sound_info.filename = "808kick.wav";
    sound_info.datasource = snx::SoundInfo::FILESYSTEM;
    
    // create the sound object
 10 snx::SoundHandle sound_handle;
    sound_handle.init("my sound for testing");
    sound_handle.configure(sound_info);
    
    // trigger the sound
 15 sound_handle.trigger();
    sleep(1);
    
    // trigger the sound using a different audio system
    snx::sonix::instance()->changeAPI("AudioWorks");
 20 sound_handle.trigger();
    sleep(1);
    
    // trigger our sound object using different source data
    sound_info.filename = "303riff.wav";
 25 sound_handle.configure(sound_info);
    sound_handle.trigger();
    sleep(1);
    

Design and Implementation of Sonix

Before we conclude this section on using Sonix directly in applications, we examine the design details of Sonix. These details are beyond the scope of what needs to be understood in order for Sonix to be utilized by VR Juggler applications, so readers not interested in these details can skip ahead to the section called “Using the VR Juggler Sound Manager”.

Sonix was designed using modern software design principles including design patterns and object oriented design. The complete design of Sonix is depicted in UML in Figure 6.2, “The Sonix Design”. As evidenced by the diagram, the design of Sonix is quite simple, though the classes tend to have relatively large interfaces. Refer to the Sonix Programmer Reference, found on the Sonix documentaiton web page, for complete documentation of these interfaces.

Figure 6.2. The Sonix Design

The Sonix Design

Design Patterns Overview

Design patterns describe simple and elegant solutions to specific problems in object oriented software design. When designing Sonix, we used many design patterns appropriate for a simple audio system.

  • Adapter (snx::SoundImplementation). This adapter provides a common interface to the underlying sound API.

  • Prototype (snx::SoundImplementation). Making snx::SoundImplementation a Prototype allows a new cloned object to be created from it that has duplicate state.

  • Store/plugin-method (snx::SoundFactory). Each sound implementation is registered with a Store called snx::SoundFactory. This Store allows users to select items from its inventory. Another name for Store is “Abstract Factory.

  • Abstract Factory (snx::SoundFactory). The Store can create new instances of the requested sound implementation. The Abstract Factory consults its Store of registered objects, and if found, makes a clone of that object (Prototype pattern). The Abstract Factory is used in Sonix to configure the Bridge.

  • Bridge (snx::sonix interface class and snx::SoundImplementation). The snx::sonix class is the audio system abstraction which is decoupled from its implementation snx::SoundImplementation. This way the two can vary independently. Bridge also facilitates run-time configuration of the sound API.

  • Proxy (std::string and snx::SoundHandle). snx::SoundHandle is how users manipulate their sound object. snx::SoundHandle is actually a proxy to a std::string proxy. The std::string Proxy is what allows Sonix reconfiguration of resources. Rather than using pointers which can easily be left to dangle, the std::string serves as a lookup for a protected sound resource located internally to the Sonix run-time memory space. The snx::SoundHandle wraps this std::string to provide a simple and familiar C++ object to use as the sound handle. The Sonix class acts as Mediator between every Proxy method and the actual audio system Adapter.

Pluggable Audio Subsystems

Sonix supports the selection of several audio subsystems by the application through implementation plug-ins (see Figure 6.3, “Use of Plug-ins in Sonix”). Each plug-in implements an adapter to an underlying audio subsystem. The adapter supports a common interface that Sonix knows how talk to. Each adapter is then registered with a factory object, which may ask that adapter to clone itself for use by whomever called the factory.

Figure 6.3. Use of Plug-ins in Sonix

Use of Plug-ins in Sonix

Using the VR Juggler Sound Manager

Now that we understand how Sonix works and how to use its simple API, we can go one step further and simplify the usage even more. Similar to the graphics API-specific VR Juggler Draw Managers, there exists the concept of a Sound Manager. There is a Sonix-specific Sound Manager implementation that handles many of the details of Sonix usage.

The Sonix Sound Manager is configured using a config element of type sound_manager_sonix. The config element sets the sound API to use, defines the llistener position, and sets up all the sound objects that will be used by the application. All of this would normally have to be done by the application programmer using the snx::sonix singleton, as presented above in the section called “Basic Sonix Usage”. In Example 6.17, “Example Sonix Sound Manager Conifguration”, we show a small example of configuring the Sonix Sound Manager. Note that some parts of the config file are removed for brevity. Aspects of this will be referenced in the code examples shown below.

Example 6.17. Example Sonix Sound Manager Conifguration

<sound_manager_sonix version="1">
   <api>OpenAL</api>
   <listener_position>0.0</listener_position>
   <listener_position>0.0</listener_position>
   <listener_position>0.0</listener_position>
   <sound>
      <sound name="bump" version="1">
         <filename>${VJ_BASE_DIR}/share/vrjuggler/data/sounds/bump.wav</filename>
         <ambient>false</ambient>
         <retriggerable>false</retriggerable>
         <loop>1</loop>
         <pitch_bend>1.0</pitch_bend>
         <cutoff>1.0</cutoff>
         <volume>1.0</volume>
         <position>0.0</position>
         <position>0.0</position>
         <position>0.0</position>
      </sound>
   </sound>
   <sound>
      <sound name="step" version="1">
         <filename>${VJ_BASE_DIR}/share/vrjuggler/data/sounds/footstep.wav</filename>
         <ambient>false</ambient>
         <retriggerable>false</retriggerable>
         <loop>1</loop>
         <pitch_bend>1.0</pitch_bend>
         <cutoff>1.0</cutoff>
         <volume>1.0</volume>
         <position>0.0</position>
         <position>0.0</position>
         <position>0.0</position>
      </sound>
   </sound>
   <!-- Other sound objects ... -->
</sound_manager_sonix>

All sound files named in the config element are automatically loaded by the Sonix Sound Manager. User applications simply declare sound handles that refer to the loaded sound objects. The VR Juggler application class will then include one snx::SoundHandle instance for each sound object declared in the Sonix Sound Manager config element, as shown in Example 6.18, “Declaring Sound Handles in Application Object Class”. This is very similar to the use of Gadgeteer device interfaces that provide access to device input.

Example 6.18. Declaring Sound Handles in Application Object Class

class MySoundApp : public vrj::GlApp
{
public:
   void init();
   void preFrame();
   // Other application object interface methods ...

private:
   // Determines if the "bump" sound should be triggered.
   bool shouldTriggerBump();

   // Determines if the "step" sound should be triggered.
   bool shouldTriggerStep();

   snx::SoundHandle mBumpSound;
   snx::SoundHandle mStepSound;
   // And so on ...
};

To get access to the sounds through the snx::SoundHandle objects, the sound handles must be initialized. This is done in the same way as described earlier in the explanation of direct use of the Sonix API. The sound handle initialization should be performed in the init() method of the VR Juggler application object, as shown in Example 6.19, “Initializing Sound Handles in an Application Object”. The string value passed to snx::SoundHandle::init() is the name given in the Sonix Sound Manager config element. Again, we see similarity with the usage of Gadgeteer device interfaces.

Example 6.19. Initializing Sound Handles in an Application Object

void MySoundApp::init()
{
   mBumpSound.init("bump");
   mStepSound.init("step");
   // And so on ...
}

Finally, the sounds can be triggered in the application object preFrame() method, shown below in Example 6.20, “Triggering Sounds in an Application Object”. They should not be triggered in the draw() method because, as we have seen in earlier sections, draw() can be invoked multiple times per frame. There is no need to call the function snx::sonix::step() in preFrame() or in postFrame(). This is done automatically within the Sonix Sound Manager each time through the kernel control loop.

Example 6.20. Triggering Sounds in an Application Object

void MySoundApp::preFrame()
{
   if ( shouldTriggerBump() )
   {
      mBumpSound.trigger();
   }

   if ( shouldTriggerStep() )
   {
      mStepSound.trigger();
   }
}

Other methods in the snx::SoundHandle class interface can be invoked on the sound handle objects. We have focused on simple rendering of sounds here. Refer to the Sonix Programmer Reference, found on the Sonix documentation web page, for complete documentation on the interface of snx::SoundHandle.