What Amazon Bedrock does
The Amazon Bedrock Web Crawler fetches HTML pages for use as a web data source in Amazon Bedrock knowledge bases. It crawls only URLs an AWS customer configures, starting from seed URLs and following child links within a chosen scope (same path, same host, or subdomains), and runs when the customer syncs the data source, with later syncs updating new, modified, and removed content incrementally. Customers can set rate limits of 1 to 300 URLs per host per minute and a cap of up to 25,000 pages per sync, and can append a custom user agent suffix. The connector respects robots.txt per RFC 9309 and is in preview release.
How to control Amazon Bedrock
Target Amazon Bedrock in your robots.txt with the bedrockbot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Amazon Bedrock
User-agent: bedrockbot
Allow: /Block Amazon Bedrock
User-agent: bedrockbot
Disallow: /The crawler checks robots.txt for bedrockbot-UUID rules before falling back to generic bedrockbot rules, and if a site has no robots.txt file it disallows crawling by default.
How to identify Amazon Bedrock
Temso identifies Amazon Bedrocktraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
bedrockbotAmazon Bedrock FAQ
What is Amazon Bedrock?
In your logs, bedrockbot is the web crawler AWS provides for Amazon Bedrock knowledge bases. It fetches pages from URLs that an AWS customer has added as a web data source, so its content can be retrieved by that customer's knowledge base.
How do I block Amazon Bedrock?
Add a robots.txt group with "User-agent: bedrockbot" and "Disallow: /". The crawler first looks for rules addressed to a customer-specific bedrockbot-UUID token and then falls back to generic bedrockbot rules, and it disallows crawling by default when no robots.txt file is found.
About AI Search Crawlers
Crawlers that build the retrieval indexes behind AI search products like ChatGPT search and Perplexity. They decide which pages can be cited as sources.
Blocking these keeps your content out of the source index: AI search products can't cite pages they can't crawl.
