Durable State

The MSpider program MUST be capable of storing its WEB DATABASE (including REVERSE INDEX) durably on the drive beyond the execution of the program.⁴ The save files SHOULD use a .moog extension. Furthermore, the MSpider engine MUST be capable of loading a previously existing .moog save file and continuing the crawl from where it left off. When the MSpider is initialized with a saved state, it MUST NOT repeat any of the previous work. Specifically, it MUST NOT request any PAGEs that were downloaded on its previous run.

The designer MAY use the Java serialization mechanism to save and retrieve the WEB DATABASE and REVERSE INDEX and any other necessary program state.