Hacker News
- With the rise of AI, web crawlers are suddenly controversial https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders 84 comments
- The text file that runs the internet https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders 2 comments
- The rise and fall of robots.txt https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders 13 comments programming
Linking pages
- The AI Search War Has Begun - The Atlantic https://www.theatlantic.com/technology/archive/2024/07/perplexity-ai-search-media-partners/679294/ 16 comments
- Google Is Staring Down Its First Serious Threats in Years https://nymag.com/intelligencer/article/google-is-staring-down-its-first-serious-threats-in-years.html 11 comments
- Amazon Is Investigating Perplexity Over Claims of Scraping Abuse | WIRED https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ 4 comments
- Reddit, Google, and the Real Cost of the AI Data Rush https://nymag.com/intelligencer/article/reddit-google-and-the-real-cost-of-the-ai-data-rush.html 4 comments
Linked pages
- The Verge https://www.theverge.com/ 326 comments
- Robots.txt meant for search engines don’t work well for web archives - Internet Archive Blogs https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ 144 comments
- A principled approach to evolving choice and control for web content https://blog.google/technology/ai/ai-web-publisher-controls-sign-up/ 111 comments
- https://en.wikipedia.org/robots.txt 44 comments
- https://www.facebook.com/robots.txt 22 comments
- Who blocks OpenAI, Google AI and Common Crawl? — News Homepages documentation https://palewi.re/docs/news-homepages/openai-gptbot-robotstxt.html 3 comments
- https://blog.medium.com/default-no-to-ai-training-on-your-stories-abb5b4589c8 3 comments
- Generative AI at the BBC https://www.bbc.co.uk/mediacentre/articles/2023/generative-ai-at-the-bbc/ 0 comments
Related searches:
Search whole site: site:theverge.com
Search title: With the rise of AI, web crawlers are suddenly controversial - The Verge
See how to search.