![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
jducoeurDuring today's massive update of the Period Games Homepage, I'm discovering a new horror. Many of the sites I point to are now dead, which isn't a surprise. Many of them have been taken over by domain thieves, which also isn't a surprise.
What *is* a surprise is that many of those thieves have turned on robots.txt files that wind up blocking the Wayback Machine from producing results: it appears that archive.org respects robots.txt a little *too* much. The result is that a large number of useful pages are just plain inaccessible -- I can't even get at their archived versions. Grr...
(BTW, time for another reminder that archive.org is one of the most important and unsung sites on the Web -- the Wayback Machine is the only really good archive of the Web's history, and is often invaluable. I've given them another donation today...)
What *is* a surprise is that many of those thieves have turned on robots.txt files that wind up blocking the Wayback Machine from producing results: it appears that archive.org respects robots.txt a little *too* much. The result is that a large number of useful pages are just plain inaccessible -- I can't even get at their archived versions. Grr...
(BTW, time for another reminder that archive.org is one of the most important and unsung sites on the Web -- the Wayback Machine is the only really good archive of the Web's history, and is often invaluable. I've given them another donation today...)