Our spider will find any URL connected to the main body of the Web through even one link. If you don't want your entire site to be indexed, we strongly advise that you take advantage of the Robots Exclusion Standard by setting up a /robots.txt file. It only takes a minute, and gives you complete control over what fraction of your site is indexed. The file looks like:   User-agent: *   # directed to all spiders,  not just Scooter  Disallow: /cgi-bin/sources  Disallow: /access_stats  Disallow: /cafeteria/lunch_menus/ Any URL matching one of these patterns will be ignored by robots visiting your site. This file is read by Scooter every few days, so changes may not take immediate effect.