JittyBot
Jitty is a property search engine. JittyBot visits estate agent websites so we can keep listings on Jitty accurate and up to date. It strictly obeys robots.txt rules and is engineered to minimise its impact on the websites it visits as described in our technical properties.
JittyBot does not fill in forms, attempt to log in, create accounts, make purchases, or try to get around paywalls or CAPTCHAs. It does not collect personal data from the pages it visits.
How to identify JittyBot#
JittyBot sends a User-Agent header identifying itself on
every request. There are three variants, depending on what it is doing. All of them contain
JittyBot and a link back to this page, so a substring
match on JittyBot will catch all of our traffic.
Non-interactive mode#
Plain HTTP requests, used for the majority of our crawling.
Mozilla/5.0 (compatible; JittyBot/1.0; +https://jitty.com/jittybot)
Interactive mode#
A real headless browser, used only when a page needs JavaScript to render its content.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 (compatible; JittyBot/1.0; +https://jitty.com/jittybot)
The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 155.0.8059.16. This version number increases over time to match the latest Chromium release version used by JittyBot.
If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.
Media#
Used when fetching images, floorplans and other media referenced by a listing. It is separated out so you can give media its own rules and its own rate limit.
Mozilla/5.0 (compatible; JittyBot-Image/1.0; +https://jitty.com/jittybot)
How JittyBot behaves#
JittyBot is deliberately conservative. It rate limits per origin rather than per hostname, so sites sharing a backend share a crawl budget, and it backs off when it sees errors that suggest a site is struggling. Our technical properties page covers the detail, including how we use conditional requests and sitemaps to keep the cost of being crawled low.
Controlling JittyBot#
Everything JittyBot does automatically is governed by your robots.txt file. Our robots.txt reference sets out exactly how we fetch, parse and apply those rules, which fields we support, and how quickly changes take effect.
Helping JittyBot understand your site#
Supported platforms and plugins#
JittyBot supports estate agency software like Expert Agent and Bright Logic, WordPress plugins and themes like PropertyHive and Houzez, and web agencies like Property Jungle and Homeflow. Most work automatically, and for the few where a small setup change helps, we explain what to do. View all supported platforms and plugins →
For bespoke and custom websites#
JittyBot reads market status, price, property type, location, photos, bedrooms, tenure and more directly from your listing pages. Adding JSON-LD structured data and clearly labelling those facts on the page ensures your listings appear accurately. Our structured data guide explains what to add.
Getting in touch#
If JittyBot is causing you problems, if you would like it to crawl your site differently, or if you would like your listings removed from Jitty, email [email protected]. Please include your domain name and, if you can, a sample of the requests in question — it makes it much quicker for us to track down.
User-Agents are easily spoofed. If you see any traffic claiming to be JittyBot that ignores robots.txt or behaves aggressively, it is not us. Please report it to the address above with a sample IP address so we can investigate.