What Amazon Kendra does
Amazon Kendra Web Crawler indexes web documents for Amazon Kendra, the AWS intelligent search service. AWS customers point it at the URLs they want indexed and set other operational parameters, and they are required to obtain authorization before indexing any particular website. The crawler respects standard robots.txt Allow and Disallow directives, so site owners can control which pages it crawls. AWS has announced that Amazon Kendra will no longer be open to new customers starting July 30, 2026.
How to control Amazon Kendra
Target Amazon Kendra in your robots.txt with the amazon-kendra token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Amazon Kendra
User-agent: amazon-kendra
Allow: /Block Amazon Kendra
User-agent: amazon-kendra
Disallow: /How to identify Amazon Kendra
Temso identifies Amazon Kendratraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
amazon-kendraAmazon Kendra FAQ
What is Amazon Kendra?
The amazon-kendra user agent is the web crawler for Amazon Kendra, an AWS search service customers use to index and search documents. It crawls URLs that an AWS customer has configured for their own Kendra search index.
How do I block Amazon Kendra?
Add a robots.txt group with "User-agent: amazon-kendra" and "Disallow: /" to stop it entirely, or use Allow and Disallow rules under that group to restrict it to specific paths. The crawler respects standard robots.txt directives.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
