JittyBot
Jitty is a property search engine. JittyBot visits estate agent websites so we can keep listings on Jitty accurate and up to date. It strictly obeys robots.txt rules and is engineered to minimise its impact on the websites it visits as described in our technical properties.
JittyBot does not fill in forms, attempt to log in, create accounts, make purchases, or try to get around paywalls or CAPTCHAs. It does not collect personal data from the pages it visits.
How to identify JittyBot
JittyBot sends a User-Agent header identifying itself on
every request. There are three variants, depending on what it is doing. All of them contain
JittyBot and a link back to this page, so a substring
match on JittyBot will catch all of our traffic.
Non-interactive mode
Plain HTTP requests, used for the majority of our crawling.
Mozilla/5.0 (compatible; JittyBot/1.0; +https://jitty.com/jittybot)
Interactive mode
A real headless browser, used only when a page needs JavaScript to render its content.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 (compatible; JittyBot/1.0; +https://jitty.com/jittybot)
The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 155.0.8059.16. This version number increases over time to match the latest Chromium release version used by JittyBot.
If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.
Media
Used when fetching images, floorplans and other media referenced by a listing. It is separated out so you can give media its own rules and its own rate limit.
Mozilla/5.0 (compatible; JittyBot-Image/1.0; +https://jitty.com/jittybot)
How JittyBot behaves
JittyBot is deliberately conservative. It rate limits per origin rather than per hostname, so sites sharing a backend share a crawl budget, and it backs off when it sees errors that suggest a site is struggling. Our technical properties page covers the detail, including how we use conditional requests and sitemaps to keep the cost of being crawled low.
Controlling JittyBot
Everything JittyBot does automatically is governed by your robots.txt file. Our robots.txt reference sets out exactly how we fetch, parse and apply those rules, which fields we support, and how quickly changes take effect.
Getting in touch
If JittyBot is causing you problems, if you would like it to crawl your site differently, or if you would like your listings removed from Jitty, email [email protected]. Please include your domain name and, if you can, a sample of the requests in question — it makes it much quicker for us to track down.
User-Agents are easily spoofed. If you see any traffic claiming to be JittyBot that ignores robots.txt or behaves aggressively, it is not us. Please report it to the address above with a sample IP address so we can investigate.