Resolving a Heisenbug

Resolving a Heisenbug can be tricky since by its nature it resists the debugger. Instead we decided to go the unscientific way of throwing a bunch of solutions at the problem, then in true programmer fashion not questioning what fixed Wattsup. 

Part one of the solution was to rewrite the packet handling code within wattsup to fix several line errors and how wattsup was handling mis-sized packets. Originally the code would give up and kill the run if the packet size (packets being what the watts data was sent inside) changed due to the program falling out of sync. Now Wattsup recaptures the mis-sized packet and concatenates it with the following packet (which for the majority of the time will also be mis-sized and follow the first packet in data order).

Part two involved unplugging the meters for a while to clear their caches. While unplugging the meters we checked just in case the physical meters were over heating. One meter did feel warm to the touch which could be a result of overheating or normal heat left over from finishing a workload. A last minute idea when we went back to reconnect all the cables included switching the meters measuring each computers. In addition the work space was also rearranged to provide more air flow around the meters since they previously felt warm to the touch.


Wattsup now runs without problems! What actually fixed the Heisenbug? Probably a combination of all three solutions (rewriting code, switching meters and increasing air circulation). The important question now move toward analyzing all the data that we have finished collecting.

Here are a few graphs from our data that will inspire the future headaches from analysis:

Cannot touch `cupcake': No such file or directory

Yes, calibration script, no cupcakes for you. Instead, victory cupcakes for me because I made it work! Nevermore did I want to dance, cheer and celebrate seeing that line. It meant my wonderful script from last week (touch the log, and test the directory path headache) was finally working. I can actually start looking at a new problem within some other file. Any file. Maybe calib again, so long as I don’t have to go near that block of code for a while. Why? Coding and patience are fragile things.

If that wonderful ssh connected terminal, happens to have a poor internet connection, well, you’re out of luck to debug the script as you test without serious lag. Frustrating but there’s workaround hacks like write the code offline, connect, run the test, look at the bugs, disconnect, repeat. Even better, right as the solution dangles itself to you to solve all problems once and for all, said internet connection DIES.

It’s like playing a game when Windows wants to restart for an update. Windows keeps minimizing your game right as you’re battling the big boss bad guy to bug you “Oh hey do you want to restart now or later for these updates to take effect?” No amount of patience will save you from the rage quit. Write down the genius solution. Take a day off. Pray to the campus dorm internet idols for a better connection & ping rate, then go at that sucker like there is no tomorrow. Code in anger. It gets the job done. Especially with enough lies, like if you finally get it working you never have to touch that code again.