URL Spider Pro allows you to create your own Altavistastyle search engine or a Yahoo! style directory. It has the ability to follow links it finds and now supports the robots exclusion. Many features allow you to index only the types of web pages you want. Useful for creating your own mini search engine for any subject you're interested in.
Comments