robots.txt generator with AI crawler blocking
Crawler rules, with or without the AI crawlers
Starting point
AI crawlers
Checked means blocked. Blocking a training crawler costs no visibility; blocking a search crawler removes you from the answers.
Add your sitemap address. It is the simplest way to point every engine at it in one go.
Your robots.txt
User-agent: * Disallow:
Place it at the root of the domain, nowhere else. A robots.txt in a subdirectory is never read.
Everything is computed in your browser. Nothing is stored.
robots.txt is the first file a search engine asks a site for, and one of the few where a one-line mistake is catastrophic: a misplaced Disallow deindexes an entire site. This tool composes the file from explicit choices, adds the sitemap address, and offers to block the language model training crawlers that have multiplied in recent years.
How to use it
How it works
Start from a preset
Brochure site, site with a customer area, or a blank file. Each preset sets the usual disallows for that case.
Add your paths
Admin directories, internal search pages, tracking parameters: whatever has no business being crawled.
Decide about AI crawlers
A list of the main training crawlers can be blocked in one click, without touching the classic search engines.
Worth knowing
Four things to keep in mind
Disallowing is not deindexing
A page disallowed in robots.txt can stay in the index if other sites link to it, shown without a description. To really remove it you need a noindex tag, which means allowing the crawl.
Never block CSS or JavaScript
Google renders the page like a browser. Without its stylesheets and scripts it sees a broken page and judges it accordingly, which can cost dearly on mobile.
One Sitemap line, always
It is the simplest way to announce your sitemap without going through a console. The address must be absolute, in https, and point at a file that actually responds.
Test before you deploy
Search Console offers a robots.txt tester. A single misplaced slash after Disallow blocks the whole site, and the effect shows in indexing within days.
FAQ
Frequently asked questions
At the root of the domain, and nowhere else. A robots.txt placed in a subdirectory is never read. It must return a 200 at the root address followed by robots.txt.
By the operators who publicly document their crawler, generally yes. robots.txt remains a convention; it has no binding power over a crawler that decides to ignore it.
It is not mandatory, but a minimal file with a general allow and a Sitemap line avoids a 404 in your logs and hands the sitemap address to every engine.
Yes, by targeting its User-agent. Mind the order: the most specific rule applies, and a targeted group completely ignores the rules of the generic group.
Also
Other tools in the same family
Also
What Lokvi does for you
Now imagine all of it running without you
Review collection, replies, posts and loyalty, on your Google Business listing, automatically.
No credit card required: create your account and explore your real data.
14 days free, then 29 €/month, no commitment.