What Slurp does
Slurp is the Yahoo Search robot for crawling and indexing web page information. According to Yahoo's help page, it indexes pages for Yahoo Mobile Search results, collects content from partner sites for Yahoo News, Finance, and Sports, and accesses pages across the web to confirm accuracy and improve Yahoo's personalized content. It identifies itself with the user-agent string Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) and obeys the Robot Exclusion Standard, including the crawl-delay directive for sites that want to slow it down.
How to control Slurp
Target Slurp in your robots.txt with the Slurp token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Slurp
User-agent: Slurp
Allow: /Block Slurp
User-agent: Slurp
Disallow: /Slurp obeys the first robots.txt entry whose User-agent contains "Slurp"; if no such record exists, it falls back to the first entry with a User-agent of "*".
How to identify Slurp
Temso identifies Slurptraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
Yahoo!\s*SlurpSlurp FAQ
What is Slurp?
Slurp is the Yahoo Search robot that crawls and indexes web pages. Yahoo uses it for Yahoo Mobile Search results, for collecting partner-site content for Yahoo News, Finance, and Sports, and for checking page accuracy for its personalized content.
How do I block Slurp?
Add a robots.txt group with User-agent: Slurp and the Disallow rules you want, for example Disallow: / to block the whole site. Slurp obeys the first entry whose User-agent contains "Slurp" (falling back to the "*" entry), and it also honors the crawl-delay directive if you only want to slow it down.
About Search Engine Crawlers
Classic search indexers that now also feed AI features: Google's AI Overviews and AI Mode build on Googlebot's index, and Microsoft Copilot builds on Bingbot's.
Blocking these removes you from both traditional rankings and the AI answers built on top of those indexes, usually the most costly block of all.
