Crawlers (or bots) are used to collect information available on the internet. By using website navigation menus, and reading internal and external hyperlinks, the bots begin to grasp the context of a page. Of course, the words, photographs, and other information on pages also assist search engines like google and yahoo perceive what they’re about