So your developer made a git mistake

… and it went in to production.

I have come across at least three such instances over the past year. The incidents were so similar that I couldn’t help but see a pattern.

Things happened somewhat like this -

An urgent chore

The team steps into a frenzy for rollback. If we used something like Capistrano, rolling-back production to the last working state is easy. Then we want to remove the wrong code from master. This becomes a high priority because we can’t deliver any more code on top of a broken master. We can either add a fix that reverses the wrong change or revert it to an older, healthier commit-id. Adding a new, undo fix may not be an option if the error needs analysis.

Reverting to an older stable point may also not be simple. If the failing code has too many commits or merge commits with master, it becomes even more complicated. It is hard to even pin-point which commit-id we should revert to.

This will be an annoying chore if the team doesn’t have a lot of experience with reverting in git. In these instances, it takes between a few hours to a full day’s work. Its generally the senior-most developer who does it. And all other ==code delivery has to stop while the revert finishes==.

Fault finding

Since the incident causes such a major disruption in the team’s schedule, the manager/product-owner gets involved. And the natural next step is to fix the blame. The developer may have been confident about the code, but could have made an unintentional git mistake, like one of these -

For a non technical person, these details don’t matter. He made some kind of mistake. Because he was not careful, or because he didn’t follow the process.

And the developer should get punished for that. In all three of my instances the developer gets scolded.

Now lets stop there for a moment.

Unfortunately, answers to all these questions were in negative.

So a junior developer makes a mistake for the lack of training, gets scolded, and the only prevention plan for the future is more scolding?

To be objective, the problem is that we all underestimate the risks of git. Cost of mistakes is much lower with the rest of our development stack. Most of our tools and libraries are much more developer friendly than git. So if there is one take-away from these experiences, it is: ==git is risky==

So if you are a junior developer in a company with no processes for git - be very careful. You will make a mistake sometime, and you will get blamed. Learn git as much as possible, and be careful and conservative. Here are some tips:

Play safe:

And if you’re in the position to influence a process, get one adopted. There are many, get any one. Having one is better than not having one. Here’s one I like:

The manager

People make mistakes. And nearly every occurrence of a mistake can be blamed on someone’s carelessness or lack of diligence. Can punishment by scolding prevent future occurrences of a similar problem? Perhaps it can improve the level of caution, and that would help if lack of caution was the real cause of the problem.

From what I claim above, the real reason could have been lack of training. Or a missing process related to git. There should actually be an introspection session after the firefighting finishes. A blame-free analysis on what caused the issue, and what could have been done for the future.

And if you are the manager, also think about the punishment part. Scolding just became a part of your company culture. The word will spread, and people will know. Everyone avoids companies that have such problems in work culture. Find better ways of dealing with this.

Published: 26 February 2016