JittyBot's technical properties

How JittyBot paces itself, how it reacts when a site is struggling, and how it tries to keep the cost of being crawled as low as possible.

Rate limiting

JittyBot is deliberately conservative about how quickly it crawls. Rather than rate limiting per hostname, it rate limits per origin — the infrastructure actually serving the request. We work this out as best we can from DNS resolution and HTTP header probing, so that a group of sites sharing a single backend is treated as one target rather than many.

In practice this means that if you host hundreds of estate agency websites on the same servers, JittyBot will not crawl them all in parallel at full speed. It will spread the load across the origin as a whole.

Origin detection is a best-effort heuristic. If we have grouped your sites wrongly, in either direction, tell us and we will correct it.

If our default pace is still too fast for you, JittyBot honours Crawl-delay in robots.txt.

Backing off when things go wrong

JittyBot watches the responses it gets and slows itself down when a site looks like it is struggling. It backs off in particular on:

• 429 Too Many Requests — we are going too fast
• 500 Internal Server Error — the site is erroring
• 502 Bad Gateway — the upstream backend server is failing or unreachable
• 503 Service Unavailable — the site is overloaded or in maintenance
• 504 Gateway Timeout — the upstream backend server took too long to respond

Back-off is exponential and applies to the whole origin, not just the individual URL, so one struggling site will slow our crawling of everything else on the same backend. If a Retry-After header is present we use that instead. Repeated errors will pause crawling of that origin altogether until it recovers.

Timeouts, connection resets and DNS failures are treated the same way as server errors.

Keeping the load down

• Sends conditional requests using If-Modified-Since and If-None-Match, so unchanged pages cost you almost nothing to serve. Serving Last-Modified and ETag headers is the easiest way to reduce our impact.
• Accepts compressed responses to reduce bandwidth.
• Uses sitemaps where they are published, in preference to crawling link by link. A sitemap with accurate lastmod dates means we only come back for listings that have actually changed.
• Fetches media only when a listing references it and we have not already stored it.

How requests are made

• Crawls over HTTPS wherever it is available.
• Follows redirects, and respects noindex and nofollow directives, in both <meta> tags and X-Robots-Tag HTTP headers.
• Issues GET and HEAD requests only. JittyBot never submits forms or sends anything that would change state on your site.
• Identifies itself on every request. See the user agent strings for the exact values.

If we are causing you problems

The fastest lever is robots.txt, which we re-read regularly and always obey. If you need something to stop sooner than that, or our pacing is wrong for your infrastructure, email [email protected] with your domain and a sample of the requests, and we will adjust things directly.

Back to JittyBot