A Proposed Programming Language for Working with Rhythm

Introduction

Here is a wish list to guide the creation of EARPLUG:

1. Programmers shouldn't be forced to make simplifying assumptions about the shape or density of input. Rather, they should be able to change the kind of input they're using without a lot of recoding. For example, they should be able to switch from constant-interval click tracks (as in Povel and Essens) to 44.1kHz digital recordings of cafeteria noise without tearing apart a program.
2. As with input, programmers should be able to switch easily between different kinds of output--everything from MIDI events for a soundcard to voltages for a robotic conductor's baton.
3. The output should be available as input. For example, the voltages which drive a robotic conductor's baton might be relevant for input back into the system. Note that time lag is of concern.
4. The system should gracefully keep up with real time, even when processing loads become overwhelming. This may mean throwing away unprocessed information. The desired behavior can be compared to a pianist accompanying a performer: a good accompanist will keep up with a performer even when a mistake is made. That is, the accompanist won't try to go back and correctly replay erroneous passages.
5. Also, the system should support multiple processing entities and should allow the simulation of parallel processing. Note that this seriously increases the complexity of real time processing.
6. Programmers should not have to code commonly-used tools from scratch. Since there exists no complete theory of music or speech rhythm, we don't have a full list. However, some things are clearly helpful:
   • DSP tools such as resampling algorithms, Fourier transforms, FIR and IIR filters
   • Dynamical systems tools such as oscillators with plug-in coupling functions and driving functions
   • Connectionist network tools such as common learning algorithms
   • Robotics tools such as functions for interacting with a digital-to-analog robotic arm controller.
   As new models and new methods come to light, corresponding toolkits can be developed.

1. An Operating System (OS) that moves streams of information in real time. For the purposes of this discussion, the OS need do nothing more than shuttle information to and from the world. The shape of the information is not of concern to the OS; it need only keep track of one or more streams of bytes. However, the timing of the information is of central concern. The operating system should also support simulated parallel processing via threads. (Threads are processing entities which share common memory.)
2. An Application Programming Interface (API) which allows low-level interaction with the OS. Specifically, the API should grant the programmer full access to the timing mechanisms of the OS. This would allow the programmer to change how the system responds in the event that processing demands outstrip processing power.
3. A Programming Language built to work with the API. As far as I am concerned, this programming language can be Java and the API can be tailored to work naturally with Java.
4. A Programmers Toolkit programmed using the Programming Language which provides the tools mentioned above (e.g. DSP tools). The toolkit, then, is a series of Java classes which exploit the API in useful ways. This toolkit can now be thought of as a set of objects which take as input a stream and produce as output a stream.

Operating System

Real Time Processing in EARPLUG

Consider how existing real-time OSs handle this problem of delay, or latency. Traditionally, most real time OSs (for example, an OS used to monitor engine parameters during lab tests of a diesel engine) solve the problem of latency by either doing everything in hardware and therefore negating the need for an OS at all or by implementing some sort of cascaded interrupt scheme so that the OS can respond to appropriate tasks at appropriate times. To use the engine testing example, if oil pressure rises above a certain threshold, it may be vital to shut the engine down within some number of milliseconds. It is important, therefore, that the OS have the ability to complete a context switch to the appropriate process in a set amount of time. Once the context switch is completed, it is up to the programmer to write code which is fast enough to shut down the engine in time. The important point is that traditional real time systems are concerned with guaranteeing at all costs some well-defined maximum latency.

The goal of EARPLUG is different. Since EARPLUG is meant to run on a wide variety of platforms I would prefer to allow a wide variety of latencies and build into the system a mechanism for working with whatever time quantization the hardware is able to provide. Also I would prefer to forego having fixed-duration quantization. If the system is to react to input without some intermediate buffering and timestamping then it is inevitable for a variety of reasons that the rhythm process will not see the data in perfect fixed-duration slices. For example, the rhythm process may get swapped out so that an operating system function can run. When the rhythm process gets swapped back in, the system shouldn't try to catch up with lost time but rather throw away (or perhaps compress) data and pick up where it left off. This is similar to what a pianist would do if she made a mistake while performing a hard passage. Playing the passage over would interrupt the rhythmic flow of the piece; it's better to simply move on.

Multi-threading in EARPLUG

But threads come at some price, because they introduce their own timing complexities. Specifically, threads introduce another kind of latency which makes it even less plausible to ensure some fixed quantization. Threads are generally scheduled in a pseudo-stochastic fashion with such variables as thread queue priority and thread state (alive, dead, waiting for data) interacting to make synchronous thread servicing difficult to achieve. Even if there were some mechanism to make the servicing of the threads synchronous during the time when the thread process has the chip, there is still no way to guarantee synchronous servicing overall given that eventually some other process will be serviced, taking all threads off line for a moment. Therefore, different threads will have different amounts of time elapsed between servicing. For example, Thread A may be serviced with the following time differences (in microseconds): [12, 4321, 13, 12] and Thread B may have these time differences: [12, 14, 4353, 12]. Note that a large time delay, presumably due to a context switch, affected Thread A on a different servicing than it did Thread B. This would happen because Thread B got serviced once more than Thread A did before the context switch happened. The main observation is that in a multi-threaded system, each thread has a unique view of real time because each is scheduled in a pseudo-stochastic manner and each is affected differently by context switches.

As was already suggested, the solution to this problem is the following: the operating system should be equipped to inform a real time thread exactly how much time has passed since it was last serviced. Once a thread is in a critical stage of processing (i.e. once it has committed to updating shared memory based on this reported time) the thread should be allowed some reasonable fixed amount of time to complete its processing without being swapped out. In this way, computation in the thread can take into account the fact that there may be slippages due to context switches and the like: unlike a traditional dynamical system, there would be no fixed Delta-T between each time step in the system, but rather a variable Delta-T which is supplied by the operating system. This scheme allows the rhythm threads to modify their computation accordingly as servicing times vary due to processing demands.

Furthermore, this scheme helps EARPLUG be easily adapted to different machine configurations. If EARPLUG is run on a fairly slow or fairly busy machine, the threads will not get serviced very often. Therefore, the computation will be coarsely quantized in time. But, vitally, the computation will still keep up with world time. Even as hardware changes, this scheme of allowing a variable time quantization and reporting the local changes in that quantization to each thread will allow the system to meet its primary goal: EARPLUG will be equipped to react to the world as best it can.

An Application Programming Interface (API)

I believe the only relevant addition is a set of classes which exploit the real time, multi-threaded aspect of the system. Specifically I want the programmer to have the following control over threads:

• Programmable granularity of temporal processing
  Programmers should be able to override the default behavior of the operating system to simply process as many threads as possible and instead kick into a more traditional mode of guaranteeing a certain maximum latency. This may mean that a program asks for better granularity than the OS can provide. In this case the OS should kill the process and exit with some metric of required versus available processing power.
• A mode allowing simulated time processing
  It should be easy to take the system out of real time by simply changing the type of underlying processing thread. Clearly, not all cognitive modeling requires on-line real time processing and EARPLUG should support turning this off. This simply means that the operating system should ensure that (a) all processing is completed (all threads are waiting) before advancing an input or output stream and that (b) the computation is known to be off-line so that no attempt is made by the threads to gauge world time.
• Programmable priority of processing
  Like a standard thread package, it should be possible to mark certain threads as being more important than others.
• Thread-level access to hardware
  Java doesn't support access to hardware, at least not very well. I feel that access to hardware should happen on the thread level. The API should support the creation of something like a AcmeRoboticArmThread which has access to an A-to-D card running the ACME robotic arm. This idea of thread-level hardware support has great advantages because it provides direct access to hardware without using inelegant hacks to work around the Java virtual machine.

A Programming Language

One part of Java that deserves explicit mention is its support of streams. Java allows the creation of new kinds of streams of information and for the transformation of one kind of stream into another. In EARPLUG this would allow a programmer to take, for example, RealAudio from the Internet as a stream, filter it appropriately using the DSP toolkit, and then cast it into a stream that can be used by EARPLUG.

A Programmers Toolkit

A Connectionist Network Toolkit for EARPLUG

• Simple Recurrent Networks (Elman Nets)
• Jordan Nets
• Oscillator Nets

• Word prediction tasks
• Temporal XOR
• Beat tracking of Povel and Essens-style patterns

• Class Network extends Object (A network is a top-level object in the system.
• Class Connection extends Stream. (A connection is a stream.)
  • Class OneWayConnection extends Connection (One way connections are one-way streams.)
  • Class SymmetricalConnection extends Connection (Symmetrical connections are two-way streams.)
• Class Node extends Thread implements StreamFilter (Nodes are filters. Instead of thinking of them as containing input functions, activation functions, and output functions they are thought of as entities which change the properties of the stream(s) which flow through them.)
  • Class InputNode extends Node (Input nodes have the ability to transform different kinds of input streams (e.g. RealAudio) into the kind of stream expected by the rest of the nodes.)
  • Class OutputNode extends Node (Output nodes have the ability to transform internal streams into streams for driving such things as robotic arms and MIDI sound cards.)

Even this sparse description contains a lot of information about how a toolkit for EARPLUG would work. The key is that a toolkit do a good job of taking advantage of the real-time streams of data and real-time processing threads acting on that data. In fact, there are plenty of good ways to implement a connectionist toolkit (as well as all the other toolkits) and EARPLUG supports their creation. The default EARPLUG system would contain one implementation and then other programmers could add to that class hierarchy by creating better ones. A cursory visit to Gamelan (www.gamelan.com), the worldwide repository for Java enhancements, will show that allowing other programmers to add functionality to a good base system (EARPLUG API) holds a lot of promise.

Conclusion

In closing I wish to note that EARPLUG doesn't solve these problems nor should it. Monolithic systems are destined to be unworkable, and accordingly I believe that it is enough to wed an operating system to a programming language in order to achieve some good real-time performance. Forcing users into an EARPLUG environment as well would be too constraining. Besides, since EARPLUG uses Java as its underlying language, it should be compatible with current Java development tools. Concerning input and output signal manipulation, I hope that EARPLUG will blend in well with labs that already have good configurations and will lend itself to elegant new configurations. Consider as a contrast the task of building a lab around a Sparc workstation running Sun Solaris with C++ or Java as the programming language. This system offers no default byte-level (MIDI) input devices and no elegant way to get real time signals into or out of the system. Furthermore, since the OS and the programming languages are separate, the programmer is left with the task of learning how to use the Sun-specific native I/O libraries. Though EARPLUG offers no new hardware alternatives, I hope that the system's solution to the problem of real time reaction provides a foundation which helps cognitive scientists bring more complex and interesting ideas to bear on issues involving time.