MSpider

The job of MSpider is to crawl the web domain (for this project, just www.cs.unm.edu) and compile the REVERSE INDEX and WEB DATABASE. Regarding the web domain as a graph, where PAGEs are nodes and URLs are arcs, the crawl will be a breadth-first search with cycle detection. Note that the web domain may be an arbitrary, directed graph, so cycle detection will be necessary to prevent the spider from looping over the same set of pages indefinitely.