Bug Hunting

Talking with some new colleagues at work, this old programming war story from a decade ago came to mind. My memory on all the details is fuzzy¹ but here's what I recall:

I was working for a web development company doing a lot of local e-commerce. Everything was on FTP, one staging server and one live server. While I was using git for personal projects (even though the computer science department in college didn't teach it!) I tried to evangelize it at work, but people thought it was needlessly complex and time-wasting. 🤷

I did a lot of support tickets and content changes for clients, but had somewhat specialized in a complicated wedding registry script that we often sold to e-commerce clients.

That day I got a familiar tasking: an older customer wanted the additional feature I had programmed for a newer client.

I copied the newer registry file over to the client folder, updated the client-specific IDs in the SQL queries, added the feature, refreshed my browser and clicked around to dev test my work. I logged my time, and sent the URL over to the project manager to say I was done, and moved on to the next support ticket.

My project manager Google Chat'd me: "hey man, this isn't done." Annoyed, I dropped the task I was now working and clicked back over and sure enough the page was broken. I opened up the file, scanned through and made all the changes again. This time I browser tested in both my browser and the project manager's preferred variant, and marked it as done. After lunch I got another message: "Still broken."

Over the next few days this loop continued. I'd fix the bug, it would work for me, I'd send it off to the PM, and he would lightly scold me for not finishing my work. I'd wrap up whatever I was then fixing, take another stab at the changes, and the cycle would continue.

Needless to say, my project manager and I were both getting really annoyed with each other.

I was encountering some kind of bug… but I was having a hard time determining what was breaking. As a junior developer there were a lot of systems at play that I didn't understand: a shared database with live and changing data, CSS and HTML bugs between browsers, and a complex homebrew CMS with no documentation. I didn't yet have the system understanding to determine what system drove what behavior… so I tried brute force for far too long.

Finally I sat down, stopped multitasking, and slowly worked through the problem, instrumenting the solution with print-to-screen logging and comments… until I yelled with victory and anger.

I walked into the next room, unplugged a computer, waited six hours, and the application was fixed permanently.

…

While working on the logging I saw that the file had magically reverted back to the start of the process. Using my FTP client I inspected the file and discovered that the last modified username was from an employee that had been on vacation that week…

I was right, and my project manager was right. I had correctly installed the registry, but my changes had not stayed on the server.

It turns out that the other developer had opened the whole folder in Homesite+ on his work laptop and then left. He had set his editor to autosave his open work every five minutes or so… and therefore his sleeping computer was clobbering my work, usually right around the time I would send it to my Project Manager.

With permission from the boss, I unplugged his laptop, waited for the battery to die, then finished the feature in seconds… I had already written it a hundred times already.

After this horribleness… I learned how to use git-ftp and never lost my work again… even though the company never used git while I was there.