Unfortunately, it wasn't a single incident -- it appears that they originally noticed it a while ago, and then gradually vandalized it over some time, interleaves with genuine changes. So there's no single clean snapshot. (The danger of the Rolls being as well-automated as they are is that I don't need to pay a lot of day-to-day attention to them -- this has probably been brewing over the course of several months, although I suspect that the real wipeout was within the past few days.)
Fortunately, I long ago implemented a detailed logging tool, which tracks all changes. So the theory is that I should be able to edit all of the undesired modifications out of that log, apply it to the last known clean backup, and get a clean state. The only snags there are that (a) the last known clean backup was a *long* time ago, so there's a lot of data to hand-screen, and (b) the tool for applying the log entries hasn't been written yet.
So there's a fair pile of scutwork that needs to be done before I have a clean DB again. Worse, I need to add a good deal of infrastructure to make sure this doesn't happen again, which is going to be a pain in the ass. (Sadly, I think I'm going to have to make the Rolls at least lightly moderated...)
no subject
Fortunately, I long ago implemented a detailed logging tool, which tracks all changes. So the theory is that I should be able to edit all of the undesired modifications out of that log, apply it to the last known clean backup, and get a clean state. The only snags there are that (a) the last known clean backup was a *long* time ago, so there's a lot of data to hand-screen, and (b) the tool for applying the log entries hasn't been written yet.
So there's a fair pile of scutwork that needs to be done before I have a clean DB again. Worse, I need to add a good deal of infrastructure to make sure this doesn't happen again, which is going to be a pain in the ass. (Sadly, I think I'm going to have to make the Rolls at least lightly moderated...)