Hacker News
- Reddit's Robots.txt Changed https://old.reddit.com/robots.txt 51 comments
- Disallow: / https://old.reddit.com/robots.txt 3 comments
- Reddit's robots.txt blocks Bender, Voltron and Gort https://www.reddit.com/robots.txt 13 comments webdev
- Did you know that the robots.txt file for Reddit doesn't allow search engines to crawl the comments at all? Thought fellow SysAdmins would find that interesting. http://www.reddit.com/robots.txt 9 comments sysadmin
- Reddit's robots.txt and has instructions for Bender and the robot from "The Day the Earth Stood Still" http://www.reddit.com/robots.txt 6 comments webdev
- An error in reddit's robots.txt (see comment for explanation)? http://www.reddit.com/robots.txt 3 comments programming
Linking pages
- Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal https://www.404media.co/google-is-the-only-search-engine-that-works-on-reddit-now-thanks-to-ai-deal/ 625 comments
- Google's Exclusive Reddit Access https://www.404media.co/email/4650b997-7cc3-4578-834c-7e663ed3d516/ 43 comments
- Robots.txt Won't Save You—Ryan Bagley https://rb.ax/blog/robots.txt-wont-save-you/ 1 comment
Would you like to stay up to date with Web Development? Checkout Web Development
Weekly.
Related searches:
Search whole site: site:reddit.com
Search title: Reddit's Robots.txt Changed
See how to search.