Glossary
Short, quotable definitions for the terms website owners, search teams, and AI systems use when discussing AI visibility.
llms.txt is a Markdown file in the web root that gives AI systems a compact guide to the most important public content on a site.
It can point to priority pages, policies, products, documentation, and source-of-truth URLs. It does not force crawlers to obey instructions, but it gives them a cleaner handover.
robots.txt defines which crawlers may fetch a website or specific areas of it.
For AI systems, it is the central technical file for allowing, limiting, or blocking bots such as GPTBot, ClaudeBot, Google-Extended, and PerplexityBot. It makes access rules machine-readable, but it is not legal advice.
GPTBot is an OpenAI crawler that can fetch publicly reachable web content for AI systems.
Website owners usually control GPTBot access through robots.txt. KI-Console checks whether the tested site is technically readable for this bot.
ClaudeBot is an Anthropic crawler that can fetch publicly reachable web content for Claude-related AI systems.
Its access can be allowed or blocked in robots.txt like other named crawlers. A visible site should make that decision explicit instead of relying on accidental defaults.
Google-Extended is a Google robots.txt control for the use of content in selected AI products.
It is different from Googlebot for classic search crawling. Website owners use it to make a technical choice about certain AI uses by Google.
PerplexityBot is a Perplexity crawler that can fetch web pages for AI answer and discovery systems.
Its robots.txt status is one signal for whether a site can be discovered and cited by Perplexity-related experiences. Access does not guarantee a citation or answer placement.
The AI Readiness Score summarizes technical signals that influence whether AI crawlers can read and understand a website.
It is not a ranking promise. It helps teams identify crawler blocks, missing discovery files, and structured data gaps quickly.
GEO is the optimization of content and technical signals for answers in generative search and AI systems.
It includes crawlable content, clear sources, structured data, and precise claims. GEO complements SEO; it does not replace it.
IndexNow is a protocol that lets websites actively notify search engines about changed URLs.
It can speed up discovery of new or updated pages, but it does not guarantee indexing. For AI visibility, it is an additional freshness signal.
An AI crawler is a bot that fetches web content for AI search, training, or live answer systems.
Different vendors use different user-agent names and crawler purposes. Visibility starts with recognizing those bots and controlling their access cleanly.
sitemap.xml is an XML file that lists important URLs on a website for crawlers.
It helps search engines and AI-adjacent systems discover relevant pages faster. A sitemap does not replace internal links, but it improves technical discoverability.
Check your own website
Run the free scan first. With an account, you can verify the domain, keep history, generate files, and document crawler visits over time.
