An extensible operating system lets an application apply certain customization to tailor the operating system's behavior to the needs of the application. Applications running on personal computers have been taking advantage of the non-existing protection mechanisms in MS-DOS and the Apple Macintosh operating system to extend the operating system. For example, intercepting keystrokes and mouse events, as well as writing to screen memory directly is possible. These simple operating systems are not able to protect themselves or other applications from buggy or malicious user code.
Many variants of the Unix operating system allow a trusted user (the system administrator) to modify the running system by adding new device drivers and kernel services such as different file systems. Flexibility is lost, since the applications cannot dynamically adjust the system to their needs anymore. Furthermore, the operating system still has no way of verifying that the extension will not harm the system or other applications.
Current research in extensible operating systems tries to address these issues. The goal is to let applications safely modify system behavior. For example, it should be possible for an application to specify its own memory page manager. The kernel would call this page manager when the system needs to reclaim memory pages currently allocated to this application. The application specific page manager can then decide which pages should be given up. For this to be efficient, the page manager needs to reside inside the kernel, so no expensive cross-domain calls are necessary to evict memory pages. The page manager has to be isolated so it can not disrupt other kernel services or wreck havoc with the page handling of other applications.
Several recent systems use the methods presented in
Section 2.2 to enable such extensions. In this
section we look at SPIN, the MIT Exo kernel, GLUnix, VINO, and
Choices.