Page MenuHomePhabricator

AI has ruined the internet
Open, LowPublic

Description

This issue is perhaps somewhat multifaceted, but this server has been completely slammed by crawler traffic that ignores robots.txt since around the time when AI took over all human labor and forced us into a life of luxury.

As a stopgap, I'm going to restrict access to Diffusion and Multimeter to logged-in users.

Event Timeline

epriestley created this task.

Multimeter was already locked down? Let's see if we can survive health checks now...

ubuntu@secure-01a4c8f6:/core$ grep '22/Oct/2026' /var/log/apache2/other_vhosts_access.log | wc -l
1032536
ubuntu@secure-01a4c8f6:/core$ grep '22/Oct/2026' /var/log/apache2/other_vhosts_access.log | grep developers.facebook.com | wc -l
140014
ubuntu@secure-01a4c8f6:/core$ grep '22/Oct/2026' /var/log/apache2/other_vhosts_access.log | grep reflection.ai | wc -l
69485
ubuntu@secure-01a4c8f6:/core$ grep '22/Oct/2026' /var/log/apache2/other_vhosts_access.log | grep go/applebot | wc -l
8978

~2M requests a day for content which hasn't changed in five years.