SoBot Crawler

What is Sobot?

SoBot is Search Oldsmar's web-crawling spider. It collects Oldsmar related documents to create a searchable index for the Search Oldsmar Search Engine.

How often will SoBot crawl my site? We typically crawl on a monthly schedule. You can submit to have your site crawled by submitting your URL. However, Search Oldsmar is designed to specifically index documents related to Oldsmar, Florida. If your content appears to be unrelated it will not be included in the Search Oldsmar index.

SoBot User Agent String

If SoBot crawls your web site you will see our user agent string: SoBot/3.0 http://www.searcholdsmar.com/sobot/ in your web traffic logs.

Robots.txt

How can I prevent SoBot from crawling pages on my web site? SoBot will obey your Robots file, place this code in your robots.txt file:

User-agent: sobot
Disallow: /

Frame Support

SoBot does not support frames. If your website is designed with either the frameset or iframe tags then there is a very good chance that it cannot be entirely crawled.

Link Crawling

SoBot recognizes all hyperlinks that exist on each page that it discovers. However, SoBot will only follow those links that are pointed at your website. This feature helps us support our Oldsmar Only initiative. Many web pages discovered by SoBot contain links to sites that are not relevant to Oldsmar.

Where do I send any questions that I have regarding SoBot? You may send your questions to: or contact us here.