# GovIntel robots.txt # https://govintel.ai # ── /api/ is closed to crawlers ────────────────────────────────────────────── # Every public company profile links to /api/public/manage/{claim_id} — the # "Is this your business? Correct or remove this listing" form. That link makes the # manage endpoint a crawl surface the same size as the profile corpus itself: roughly # half the request volume in a sampled hour of backend logs was /api/public/manage/*, # for pages that are a form, carry nothing rankable, and appear in no sitemap. # # THIS DOES NOT AFFECT INDEXING. Profiles live at /companies/* and /:country/companies/*, # which are untouched — Google crawls and ranks exactly what it does today. Only bots read # robots.txt, so real users clicking "Is this your business?" are unaffected. # # THE RULE IS REPEATED IN EVERY GROUP ON PURPOSE. A crawler obeys only the most specific # User-agent group matching its name and ignores the rest, so a Disallow in `*` alone would # leave all eleven named answer engines below still crawling /api/. Within a group, the # longest matching rule wins, so `Disallow: /api/` beats `Allow: /` for those URLs. # ── /app signup and login are closed to crawlers ───────────────────────────── # Every public company profile emits THREE links into /app/research/signup, and only one of # them carries rel="nofollow". One of the followed pair carries the claim id # (?claim=1846654&utm_source=public_profile), so each of ~975,000 US profile pages — and 5.5M # international ones — advertises a DISTINCT crawlable URL. All of them resolve to the same # 1,656-byte SPA shell, which has no canonical tag and no robots meta, because /app is client # rendered and there is nothing per-URL to put one in. # # Search Console filed 922 of these as "Duplicate without user-selected canonical", every one # failing validation, and the ceiling is the size of the whole profile corpus. That is crawl # budget being spent on a signup button instead of the pages worth indexing. # # /app/research itself stays ALLOWED — it is the public marketing and paywall page where # profile traffic lands. Only the signup and login paths below it are closed, and they have no # ranking value: they are gated forms. # # Repeated in every group for the reason given at the top of this file — a crawler obeys only # the most specific User-agent group matching its name. User-agent: * Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login # AI / answer-engine crawlers — explicitly welcome. GovIntel's company intelligence is # meant to be a cited source in generative answers (ChatGPT, Perplexity, Google AI # Overviews, Gemini, Copilot). See https://www.govintel.ai/llms.txt User-agent: OAI-SearchBot Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login User-agent: ChatGPT-User Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login User-agent: Perplexity-User Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login # EXPLICIT, THOUGH * WOULD ALSO ALLOW THEM. ClaudeBot is disallowed below and these are # one token away from it; a matcher that is loose about prefixes must not be able to read # the wrong rule. Both are user-triggered — a person asked a question — and both can cite. User-agent: Claude-User Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login User-agent: DuckAssistBot Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login User-agent: Applebot-Extended Allow: / Disallow: /api/ Disallow: /app/signup Disallow: /app/research/signup Disallow: /app/login # ── Bulk harvesters — disallowed ───────────────────────────────────────────── # NONE OF THIS AFFECTS SEARCH. Googlebot and Bingbot are untouched, and Bing is what # backs Copilot and DuckDuckGo. Nothing below is a search engine. # # THE CRAWLER THAT CITES YOU AND THE ONE THAT COPIES YOU ARE DIFFERENT AGENTS, and the # earlier version of this file missed that — it kept "the answer engines" open as a # single thing. They are not one thing: # # ChatGPT-User, OAI-SearchBot, Claude-User, Perplexity-User, DuckAssistBot # a person asked a question. The fetch can produce a citation and a click. # STILL ALLOWED, above. Every referral this site has ever received came from these. # # PerplexityBot, ClaudeBot, GPTBot, anthropic-ai, Claude-Web # nobody asked. Bulk corpus acquisition, and no link ever comes back. # # Measured over 30 days in geo_crawler_hits, that second group took 3,588,816 fetches — # PerplexityBot 2,040,673 and ClaudeBot 1,419,134 alone — against 107,124 for the whole # first group. 97% of AI traffic here has never sent a single visitor. # # For scale: Googlebot took 2.3M requests over NINETY days. Perplexity alone took 2.0M in # thirty, against a corpus assembled from 18 national registers with no purchased data. # # THIS IS A LICENSING POSITION, NOT A GRUDGE. The corpus is available — the API is at # govintel.ai/api and licensing enquiries go to sanders@govintel.ai. What is not available # is taking all of it, continuously, for nothing. Applications through publisher forms went # unanswered while access stayed open, which is the predictable outcome of asking someone # to pay for what they already have. # # CCBot is the one with a real tradeoff: it feeds Common Crawl, which many smaller and # open-source models train on, so disallowing it slightly narrows long-tail AI reach. # Judged worth it — Common Crawl is also how a competitor bulk-acquires this data free. User-agent: PerplexityBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: GPTBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: Amazonbot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://www.govintel.ai/sitemap.xml Sitemap: https://www.govintel.ai/companies-sitemap.xml Sitemap: https://www.govintel.ai/intl-sitemap.xml