Heisenbug, noun

[jducoeur]

"A computer error that goes away every time you look at it closely, making it difficult to diagnose."

As, for example, spending the evening trying to figure out why your Internet connection has become flaky -- only to realize in the morning that the cablemodem is plugged into the light switch. D'oh...

Comments

[goldsquare.livejournal.com]

I think I first encountered the term Heisenbug in 1988 - I was working on a system that failed every once in a while, and the only clue was that the speaker on the PC would always be on.

The resulting bug was a nightmare to find.

[jducoeur]

My worst-ever bug was while working at Looking Glass, and was kind of an "anti-Heisenbug". It was the legendary "clock slowing down" bug. The symptoms were very simple and clear, but incredibly bizarre: after playing the game, over the course of the next week or so, you'd begin to lose time from the Windows system clock. If left long enough, the system would slowly grind to a near-halt (to the point where even mouse movement became sluggish) -- despite the Windows Task Manager claiming that the CPU wasn't doing *anything*.

I lost over a *month* to that goddamn bug, and never did get Microsoft to admit that it was their fault, but my online research basically came to the conclusion that I'd hit a bug deep inside the OS, specifically in the sound libraries.

I don't remember the full details, but when you started using this particular media library, it would begin a timer in the kernel that would occasionally wake up, scan an internal linked list, pop the first thing from the list and add a new one to the end. Problem was, if you killed the process without properly closing that library (because the program crashed or -- as happened all the time for us -- if you stopped it in the debugger), it would stop *removing* things from the list, but would keep *adding* to it. This only happened a relatively few times per second, but eventually added up to scanning a linear linked list that was *millions* of elements long, several times a second. And of course, since it was in the kernel, the OS didn't think anything at all was happening, but it would begin to miss OS events like mouse movement and clock ticks since it was spending all its time scanning the list.

It was an "anti-Heisenbug" in that, as I eventually figured out, it *only* happened if you were debugging. Fixing it in the shipping product was trivial: I just made sure that we had an outer exception catch, and always shut down that damned Windows library properly. But even if we hadn't fixed it, I suspect it would never have been noticed in the field. It was aggravating to realize that the bug only really showed up when debugging, and that the only solution was, really and truly, to just reboot eff'ing Windows every now and then.

(Of course, this was around the time that we found out that it was physically impossible to run Windows for more than two months before an internal timer would roll over and crash the OS. Far as we could tell, nobody had ever hit this because it was nearly impossible to successfully run Windows for that long...)

[serakit.livejournal.com]

*blink* If it was that buggy, how on earth did Windows manage to keep selling stuff? These days it's big through sheer force of inertia, but back in the day, wouldn't Linux, Windows, and Apple all have been on a reasonably equal footing?

[jducoeur]

Heh -- sorry, but sometimes you do show your youth. This was 1999: Linux barely existed yet, and nobody even believed it was possible for it to be used as a consumer-level OS. Linux (like all Unix derivatives) was used for servers and embedded systems, not consumer boxen. The geeks loved it, but far less than 1% of the population had even heard of it.

As for Apple, it was the usual story: Apple's "my way or the highway" attitude turned a lot of folks off, and the variety of hardware and software available for it was a small fraction of what you could do with Windows. It was basically a comfortable high-end niche player.

So basically, there was no competition to speak of -- indeed, that "sheer force of inertia" was much, much stronger at that point. (*Now*, Windows is in serious trouble, because iOS and Android have badly disrupted the assumptions underlying it.) Everyone know that Windows was crappy (certainly all the programmers did), but it was what the consumers had, so we targeted it. And since all the programs ran on Windows, all the consumers bought it. It was very sweet for Microsoft, for quite a while...

[metahacker.livejournal.com]

It's *still* that buggy and they still keep selling stuff. But they know their target market.

At work we run into a Windows "bug" (they call it a feature) that means that network access is 3x - 20x slower on Windows than it needs to be, as evinced by the speed on the same box running Linux. Normal people don't find it painful enough to jump ship, however.

Admittedly this is much better under Windows 7 than it was under XP, where it was 100x-300x slower.

[eclecticmagpie.livejournal.com]

Unix, at least the version that ran on SUN 3 hardware, had a similar problem, with a continuously increasing log file for which the simple solution was to reboot every couple of months (before the system froze).

Of course, the big difference was that a lot of people knew about this problem, because otherwise you could keep the system up indefinitely.

[goldsquare.livejournal.com]

Mine was a similar OS problem.

OLD HACKER MODE ON
I was working on a multitasking implementation of DOS, and we found that the system would suddenly crash-and-hang the OS on rare occasions. Our original hint was that the sound was always on, and while we weren't ALWAYS typing, we frequently were typing.

Back in the day when programming to DOS on an IBM PC, it was well understood that the BIOS calls were safely re-entrant. Not only were they, but IBM had actually published the code for the original BIOS to allow users to rely on its behavior.

It turns out that this BIOS was not sufficiently re-entrant - it had allowed itself to be called up to two-deep, but if it was called 3 deep, that was a problem. The developer was likely sure that could not happen. But that product I was using included some very smart boards that slotted into the data bus, and indirectly it could cause its own BIOS re-entrant calls.

Also, as it turns out, if you REALLY tried hard, you could make 3 levels of BIOS calls without using an external Board, although why would you?

I wrote a lovely assembly language program that would do just that - make 3 level deep calls into the BIOS. And my Heisenbug became a bug.

Alas: when I called the OS vendor, they sort of couldn't understand what I was talking about. That code had been written by a contractor who was long-gone, and they knew they didn't understand it, and weren't going to fix it. They offered to fly me to their offices for a week to fix it (and would pay me and my employer a bounty to do so), but my employer wouldn't bite.

So, we dumped that OS and went on to another. My 3-deep tool became a standard acceptance test for evaluating OS's.

[jducoeur]

Ouch. Yeah, that's a pretty brutal and very classic Heisenbug. While it's not quite universal, I do usually assume that a Heisenbug involves concurrency in *some* fashion: that's probably true 95% of the time.

I confess to being morbidly curious what they were doing to allow level 2 re-entrancy but not level 3 -- that smells like somebody hacked something horribly...

[goldsquare.livejournal.com]

I didn't get too into that - but from the behavior and a little hackerish use of the old DOS DEBUG commands obscure "reverse assemble" option... They just made a tiny little stack that only went 2 deep, and were sure that was enough.

Remember back in those days (and even now with embedded devices), space cost money and no one used space they didn't have to.

[jducoeur]

True. My very first summer job, working for my father, was to add a printer subsystem for the POS system we were building. We had a total of 16k of EEPROM, not expandable, and that was mostly already full. I pretty much had to invent refactoring from first principles (with some suggestions from Dad, of course), just to clear enough space to get the job done.

Fun times: it was my first experience of "they didn't tell me it was impossible when I got the assignment"...

[cvirtue.livejournal.com]

You two have reminded me how much I liked listening to you and others talk programming after dance practice.

[jducoeur]

Happy to provide entertainment. War stories are fun to pass around, regardless of the context...

[cvirtue.livejournal.com]

It was not only entertainment, but I learned a lot about how programmers interact, which meant I was highly employable as support staff, so it had actual benefits!

Plus 95% of my sweethearts have been techies...

[ilaine-dcmrn.livejournal.com]

You are reminding me of the link() system call bug I ran across years ago. It was insufficiently atomic, resulting sometimes in a caller being told it had won a race it had not.

[jducoeur]

Ouch. Good reminder of why I've become so deeply enamored of the Actor Model for -- well, pretty much anything of scale. Having an architecture where, if you follow the rules (not a trivial caveat) threading issues simply don't arise is a *huge* comfort.

But even there, I'm still dependent on the system libraries not sabotaging things from the get-go...

[metahacker.livejournal.com]

Ooooops.

Often heisenbugs are the result of bogon fields interacting poorly with the hardware and/or software, which can be extra hard to diagnose--even taking into account the laws of quantum chromobogodynamics. Glad this one was easier to track down...