jducoeur: (0)
jducoeur ([personal profile] jducoeur) wrote 2006-06-01 04:55 pm (UTC)

Unfortunately, it wasn't a single incident -- it appears that they originally noticed it a while ago, and then gradually vandalized it over some time, interleaves with genuine changes. So there's no single clean snapshot. (The danger of the Rolls being as well-automated as they are is that I don't need to pay a lot of day-to-day attention to them -- this has probably been brewing over the course of several months, although I suspect that the real wipeout was within the past few days.)

Fortunately, I long ago implemented a detailed logging tool, which tracks all changes. So the theory is that I should be able to edit all of the undesired modifications out of that log, apply it to the last known clean backup, and get a clean state. The only snags there are that (a) the last known clean backup was a *long* time ago, so there's a lot of data to hand-screen, and (b) the tool for applying the log entries hasn't been written yet.

So there's a fair pile of scutwork that needs to be done before I have a clean DB again. Worse, I need to add a good deal of infrastructure to make sure this doesn't happen again, which is going to be a pain in the ass. (Sadly, I think I'm going to have to make the Rolls at least lightly moderated...)

Post a comment in response:

(will be screened)
(will be screened if not validated)
If you don't have an account you can create one now.
HTML doesn't work in the subject.
More info about formatting