lilRobots

Scan any domain's robots.txt and llms.txt, see exactly which search engines and AI crawlers are allowed or blocked, then generate clean files of your own below.

Who is allowed to crawl your site?

Try:

The report shows up here: what the robots.txt actually says, an allowed-or-blocked grid for the major AI crawlers, and whether llms.txt exists yet.

Generate your own
A clean robots.txt with your AI-crawler policy, and an llms.txt so language models know what your site is about.
Sitemap URL
Paths to keep crawlers out of (one per line)
Block AI crawlers (your call; search engines stay allowed)
Rather not do it by hand?

Busywork like this is what we automate.

The checks you run every month, the reports you rebuild every week. We set up an agent to do the repetitive part, a person on our team supervises it, and everything runs in accounts you own.

GET STARTED

Find out what you could stop doing.

We take the repetitive work that eats your week and automate as much of it as we can, so the only things left on your plate are the ones that actually need you.

Smiling woman holding a smartphone at the center with orbiting logos of programming languages and platforms including Swift, Ok.ru, and others.