Search Technology: SasaTAC - Crawler Management System

A typical Crawler is a program or automated script, which browses the World Wide Web in a methodical, automated manner. It is designed for the most general (rather than specific) capture of web content. These crawlers are mainly used to create an indexed copy of all the visited pages in order to fulfill searches of unstructured content.