lilRobots
robots.txt & llms.txt checker: see which search and AI crawlers any site allows or blocks, then generate clean files of your own
Who is allowed to crawl your site?
Scan any domain's robots.txt and llms.txt, see exactly which search engines and AI crawlers are allowed or blocked, then generate clean files of your own below.
Try:
The report shows up here: what the robots.txt actually says, an allowed-or-blocked grid for the major AI crawlers, and whether llms.txt exists yet.
Generate your own
A clean robots.txt with your AI-crawler policy, and an llms.txt so language models know what your site is about.
Sitemap URL
Paths to keep crawlers out of (one per line)
Block AI crawlers (your call; search engines stay allowed)
Site name
One-line summary
Key pages (Title: URL, one per line)
Got the sitemap robots.txt points to?
Your robots.txt tells crawlers where the sitemap lives, but only a clean, valid sitemap actually guides them through every page. lilSitemap checks the one you have and builds a fresh one when you need it.
Check your sitemap with lilSitemap
