The Web's New Doormen
The public web's old crawler bargain, rooted in robots.txt and referral traffic, is being renegotiated as AI crawlers make extraction more visible and less obviously reciprocal. Cloudflare's AI Crawl Control, AI Labyrinth, and Pay Per Crawl show how crawler access is becoming a market and governance layer built from headers, bot identity, payment signals, and enforcement tools.

