Most Major Travel Websites Lack Clear AI Crawler Policies
A Skift audit of 50 major travel sites reveals only 10 explicitly address AI crawlers in their robots.txt files, amid a 119% surge in AI-driven traffic.
Stock photo for illustration only, not from the actual event
- Skift reviewed 50 major travel websites and found only 10 explicitly addressed AI crawlers in their robots.txt files.
- Major travel brands like Expedia and Vrbo allow broad AI search access, while Tripadvisor blocks training crawlers.
- Airbnb shifted its stance between August and October, moving to fully disallow GPTBot and ClaudeBot.
- Adobe reported in July that AI visits to travel websites surged 119% year over year.
As artificial intelligence rapidly transforms how consumers plan trips, a new audit reveals that many travel brands are sending mixed signals—or no signals at all—to the automated bots indexing the web.
A review conducted by Skift of the robots.txt files across 50 major travel websites discovered that only 10 sites explicitly addressed AI crawlers. The remaining 40 websites either lumped AI bots in with every other automated program or relied entirely on default catch-all settings.
Stock photo for illustration only, not from the actual event
This policy gap carries significant weight as travelers increasingly turn to AI assistants to shop and compare options. In July, data from Adobe showed that AI-driven visits to travel websites surged 119% compared to the previous year.
Vivek Pandya, an analyst at Adobe, highlighted a major operational bottleneck, noting that significant portions of travel and retail sites remain unreadable to machines, which restricts their visibility across modern AI search results.
"When an LLM cannot easily read brand content (such as amenities, pricing, or availability), potential revenue is being left on the table."
Vivek Pandya, Adobe Analyst
From a technical perspective, a robots.txt file acts as a digital front door that guides how automated software indexes online content. When travel brands rely on broad catch-all directives instead of tailored rules, they lose the nuanced control needed to differentiate between search discovery bots that drive bookings and heavy data scrapers training foundational models. Furthermore, an October 30 proposal backed by Anthropic, Google, Microsoft, and OpenAI introduced a unified opt-out entry to block AI training across all four platforms at once—a standard that requires explicit implementation because catch-all directives will not inherit the rule.
The audit also revealed a divergence between stated policies and actual enforcement. Eight sites, primarily hotels, blocked automated requests with 403 Forbidden errors. Additionally, data from Cloudflare indicated that Meta's crawler accounted for 25.5% of AI-bot requests directed at travel and tourism sites, outpacing the traffic generated by OpenAI or Anthropic bots.
Source: Skift
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment