JittyBot

Jitty is a property search engine. JittyBot visits estate agent websites so we can keep listings on Jitty accurate and up to date. It strictly obeys robots.txt rules and is engineered to minimise its impact on the websites it visits as described in our technical properties.

JittyBot does not fill in forms, attempt to log in, create accounts, make purchases, or try to get around paywalls or CAPTCHAs. It does not collect personal data from the pages it visits.

How to identify JittyBot#

JittyBot sends a User-Agent header identifying itself on every request. There are three variants, depending on what it is doing. All of them contain JittyBot and a link back to this page, so a substring match on JittyBot will catch all of our traffic.

Non-interactive mode#

Plain HTTP requests, used for the majority of our crawling.

Mozilla/5.0 (compatible; JittyBot/1.0; +https://jitty.com/jittybot)

Interactive mode#

A real headless browser, used only when a page needs JavaScript to render its content.

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36 (compatible; JittyBot/1.0; +https://jitty.com/jittybot)

The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 155.0.8059.16. This version number increases over time to match the latest Chromium release version used by JittyBot.

If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.

Media#

Used when fetching images, floorplans and other media referenced by a listing. It is separated out so you can give media its own rules and its own rate limit.

Mozilla/5.0 (compatible; JittyBot-Image/1.0; +https://jitty.com/jittybot)

Verifying JittyBot with Web Bot Auth#

JittyBot also signs its requests cryptographically using Web Bot Auth, which is built on RFC 9421 HTTP Message Signatures. Non-interactive and media requests are signed; interactive mode requests are not yet. A signed request carries three extra headers:

Signature-Agent: "https://jitty.com" Signature-Input: sig1=("@authority" "signature-agent");created=…;expires=…;keyid="…";alg="ed25519";nonce="…";tag="web-bot-auth" Signature: sig1=:…:

To check that a request really came from JittyBot:

  1. Check that Signature-Agent is "https://jitty.com" and that Signature-Input has tag="web-bot-auth".
  2. Fetch our public keys from https://jitty.com/.well-known/http-message-signatures-directory. It is a JSON Web Key Set of Ed25519 keys; the keyid in the request is the JWK thumbprint of the key that signed it. The directory response is itself signed with the same key.
  3. Rebuild the signature base from the request's @authority (its host, with the port if it is not the default) and its Signature-Agent header, followed by the @signature-params line (the Signature-Input value without its sig1= label), and verify the Signature against that key.
  4. Check that the current time is between created and expires. Each signature is valid for 60 seconds.

Most Web Bot Auth libraries and bot management services do all of this for you once they are pointed at the key directory.

How JittyBot behaves#

JittyBot is deliberately conservative. It rate limits per origin rather than per hostname, so sites sharing a backend share a crawl budget, and it backs off when it sees errors that suggest a site is struggling. Our technical properties page covers the detail, including how we use conditional requests and sitemaps to keep the cost of being crawled low.

Controlling JittyBot#

Everything JittyBot does automatically is governed by your robots.txt file. Our robots.txt reference sets out exactly how we fetch, parse and apply those rules, which fields we support, and how quickly changes take effect.

Helping JittyBot understand your site#

Supported platforms and plugins#

JittyBot supports estate agency software like Expert Agent and Bright Logic, WordPress plugins and themes like PropertyHive and Houzez, and web agencies like Property Jungle and Homeflow. Most work automatically, and for the few where a small setup change helps, we explain what to do. View all supported platforms and plugins →

For bespoke and custom websites#

JittyBot reads market status, price, property type, location, photos, bedrooms, tenure and more directly from your listing pages. Adding JSON-LD structured data and clearly labelling those facts on the page ensures your listings appear accurately. Our structured data guide explains what to add.

Getting in touch#

If JittyBot is causing you problems, if you would like it to crawl your site differently, or if you would like your listings removed from Jitty, email [email protected]. Please include your domain name and, if you can, a sample of the requests in question — it makes it much quicker for us to track down.

User-Agents are easily spoofed. If you see any traffic claiming to be JittyBot that ignores robots.txt, behaves aggressively, or carries an invalid Web Bot Auth signature, it is not us. Please report it to the address above with a sample IP address so we can investigate.