How JittyBot interprets robots.txt
JittyBot supports the Robots Exclusion Protocol as defined in RFC 9309. This page describes our interpretation of it, so you know exactly what to expect.
Before we crawl anything
Before JittyBot crawls a site it downloads and parses that site's robots.txt file and extracts the rules that apply to it. If the rules disallow a URL, JittyBot will not request it.
This applies to all of JittyBot's automated crawling, in non-interactive mode, interactive mode and when fetching media. It does not apply to one-off fetches triggered directly by a person—for example a member of our team checking a specific page you have asked us about—which are not crawling and are not made by JittyBot.
File location and scope
Your robots.txt file must live in the top-level directory of a site, fetched over HTTP or HTTPS. The rules in it apply only to the exact host, protocol and port it was served from.
https://example.com/robots.txtcovershttps://example.com/and every path beneath it, but nothttp://example.com/,https://www.example.com/or a non-standard port.- Every subdomain needs its own robots.txt file. If you run many agency sites on subdomains, each one is treated separately.
- We do not look for robots.txt files in subdirectories.
- Internationalised domain names are treated as equivalent to their punycode form.
Note that our rate limiting works differently: it is applied per origin rather than per host, so sites that share a backend share a crawl budget even though they each have their own robots.txt file.
File format
robots.txt must be a UTF-8 encoded plain text file with lines separated by
CR,
LF or
CR/LF. A valid line is a field, a colon and a value.
Field names are case-insensitive; path values are case-sensitive. Anything after a
# is a comment.
We ignore invalid lines and a leading byte order mark, and we read at most the first 500 KiB of the file. If your robots.txt is larger than that, consolidate your rules by moving excluded material into a smaller number of directories.
Fields we support
user-agent- Identifies which crawler the following rules apply to. We answer to
JittyBot, and toJittyBot-Imagewhen fetching media. disallow- A URL path that must not be crawled. A rule with no path is ignored.
allow- A URL path that may be crawled. Useful for carving an exception out of a broader
disallow. sitemap- The absolute URL of a sitemap or sitemap index. You can list as many as you like. We prefer crawling from sitemaps over following links.
crawl-delay- The minimum number of seconds between our requests. Unlike some major crawlers, JittyBot does honour this.
Any other field is ignored.
Which group of rules we follow
Only one group is valid for a given crawler. We look for the group with the most specific
user-agent value that matches, and ignore the rest.
Matching is case-insensitive and on the token only, so
jittybot,
JittyBot and
JittyBot/1.0 are all equivalent. The order of groups in
the file does not matter.
We recognise two tokens. JittyBot covers page crawling,
and JittyBot-Image covers images, floorplans and other
media. When fetching media we use a JittyBot-Image group
if you have written one, then fall back to JittyBot, then
to *. So a single
JittyBot group is enough to control both unless you want
to treat media differently.
If there are several groups naming the same token, we merge their rules into one. Named groups and
* groups are never merged with each other — as soon
as you write a group for JittyBot, the * group stops
applying to us. Remember to repeat any global rules you still want us to follow.
Path matching
Paths are relative to the root of the site and must start with
/. We compare rules against URLs in their
percent-encoded form, canonicalising raw UTF-8 rule paths first, so
/hus/å and
/hus/%C3%A5 are treated identically.
Two wildcards are supported:
*matches zero or more of any character.$matches the end of the URL.
The value we match against includes the query string, as shown in Figure 4 of
RFC 9309.
So $ anchors to the end of the path and query together,
not the end of the path alone.
Some examples of how rule paths match:
/- The root and everything below it.
/$- The root only.
/property- Any path starting with
/property, including/propertiesand/property/12. Case-sensitive, so not/Property. /property/- Anything inside the
/property/folder, but not/property.html. /*.pdf$- Any path ending in
.pdf, but not/brochure.pdf?v=2.
Where rules conflict, the most specific one wins, measured by the length of the rule path. If two rules are equally
specific we take the least restrictive, which means an allow
beats a disallow of the same length.
Caching
We cache robots.txt for up to 24 hours, and longer if we cannot refresh it. We will shorten or extend that based on
the Cache-Control: max-age header you send. Changes you
make can therefore take up to a day to take effect. If something needs to stop sooner than that, email us and we will
action it directly.
Errors and status codes
When we request your robots.txt file, the response we get determines what we do next:
- 2xx
- Parse and apply the file as served.
- 3xx
- Follow up to five redirect hops. If a redirect loop is detected or the limit is exceeded, we treat it as a 5xx server error to avoid crawling an improperly protected site. We do not follow JavaScript, frame or meta refresh redirects.
- 4xx (except 429)
- Treat it as though no robots.txt exists, meaning no crawl restrictions. If you want us out of somewhere, serve a real file — a 404 is not a block.
- 429
- Back off and retry later. We do not assume the site is open.
- 5xx
- Stop crawling the site and keep retrying the file. Where we have a previously fetched copy we keep using it rather than assuming the site is open.
- Network errors
- DNS failures, timeouts, resets and malformed responses are treated the same as a server error (5xx).
Examples
Keep us out of specific sections
User-agent: JittyBot Disallow: /search Disallow: /admin
Block a section but allow part of it
User-agent: JittyBot Disallow: /media/ Allow: /media/brochures/
Ask us to slow down
User-agent: JittyBot Crawl-delay: 10
Treat media differently to pages
User-agent: JittyBot Crawl-delay: 5 User-agent: JittyBot-Image Crawl-delay: 30 Disallow: /media/staff-photos/
Point us at your sitemap
This is the single most useful thing you can do. It means we can find new and updated listings without crawling your site looking for them, which is faster for us and much cheaper for you.
Sitemap: https://example.com/sitemap.xml
What robots.txt will not do
robots.txt controls crawling, not what already exists. Disallowing a path stops us fetching it again, but it does not remove listings we have already collected, and it does not remove anything supplied to us through a property feed. If you want content taken down, or you would prefer to send us a feed instead of being crawled, email [email protected].