Skip to main content

org

Solving the one engineer can work on a codebase at once problem

One engineer per project was the ceiling. Using tooling and process to clear bottlenecks.

1 → 12 engineers per release

Situation
One project, one engineer. Having two people work together required a great deal of coordination and communication.
Constraint
Two or more engineers trying to work together could easily override each other's work. Config might be stored in the database and require extensive manual steps for deployment. This slowed down development cycles and forced every engineer to be a full stack expert; there was little room for a junior.
Decision
Switch to git and adopt GitLab. Start using a development workflow with merge requests and timed releases. We ignore vanilla scrum and gitflow in favor of semantic git.
Tradeoff
Dynamic content is no longer in version control. Branching with multiple developers is possible in SVN, but git is easier to manage and has better tooling and developer experience.
Result
Two engineers on a release used to be a struggle; five became routine, and over twelve during crunch periods. Any project could take any number of engineers at any time, and a release no longer waited for the one person with the expertise to be free.

What “one engineer per project” actually cost

The scheduling reality: a release waited on one named person. If they were on another project, on holiday, or gone, the work waited with them. Worse, our JavaScript-focused developers could not work on PHP even when they had availability, because they lacked the depth.

Config lived in the database, so shipping meant a manual checklist someone had to remember or communicate. This was a common problem in Drupal 7, which is where the process was originally developed; deployment was a real problem.

The biggest issue with that platform was that there was no built-in way to stage configuration changes in code. The available solutions were the contributed Features module, update hooks, and copious note-taking. With Drupal 7 there was no good universal system, so we had to develop a sub-process of standards and guidelines to keep everyone on the same page. On that platform the problem was never going to fully go away, and it is important to recognize the real constraints you are working within so they can be worked around.

Every engineer had to be full stack because there was no safe seam to hand a junior. I was brought in as an expert Drupal developer. That method of hiring is unsustainable, and unfortunate when there is a pool of competent engineers and technical producers already at the firm. This workflow allowed the existing QA specialist and technical producer to transfer into engineering, where I continued to mentor them. Our three other JavaScript engineers were able to gain Drupal experience as well. I led the Drupal team, but the JavaScript team adopted the process too and were eventually able to contribute to any project.

In the end we could have the whole department and more working on a single release. Before, we struggled to have two people working on one release; after, we routinely had five people involved in a release or feature, and that number got over twelve during crunch periods.

Why not vanilla gitflow

There is no one-size-fits-all process for a team. When I look at any “best practice” I want to know why it is considered one. Gitflow is not inherently bad — I like the multi-branch model. My objection is that the branches end up disorganized, or worse, stale. That leads to confusion and dead branches lying around, which slows down future cycles.

The bottleneck was in who was tasked with a feature, the branching model, and the deploy steps. Instead of adopting the rigors of scrum, we fell into a purer agile workflow facilitated by our semantic git workflow.

The other thing vanilla gitflow fails at is that people like to see numbers go up. It helps communicate progress to clients, and it shows when a feature is taking a long time. I consider it counterproductive to exist solely in a state of change; better to have each release numbered, and to have those numbers mean something.

Semantic git

The convention I wrote up in semantic git (2016):

  • Stable code is a tag1.2.3. Nothing else claims to be stable.
  • Development happens on a version branch1.2.x.
  • Feature work branches off it and is rebased before merge — 123-checkout-widget. With GitLab it was more convenient to use the issue number rather than the target branch in branch names.
  • Increments follow semver: patch for a change that adds nothing and breaks nothing, minor for a backwards-compatible addition, major for a breaking change.
  • Security releases are patch versions unless they also add features.
  • Features return to the version branch by merge request; releases are tagged and cut on a timed cadence rather than when someone declares them done. Everyone knows when their feature is due, because they know which release it targets.
  • With the adoption of GitLab this became highly integrated into the tooling, but I have used the process on GitHub and Forgejo as well.

What had to change besides the branching model

To solve the one-developer-per-project problem, more than the branching model had to change. Reproducible deployments, standardized local development environments, and standard tooling all had to be adopted by the whole department. Some of this came from the top down; other pieces came from teams adopting the process on their own.

Adopting something like this works much better when it is enforced from the top down. Inevitably, when a new team attempted to adopt the process, they would try to put their own spin on it. Everyone ends up on the same page eventually, but the confusion can be avoided if the process can be dictated.

What it cost, and what it made possible

When a department adopts a process it has to be a net gain, but there is always a cost. One thing we lost was the safety net of keeping user content files in version control. Switching from SVN to git means adapting to some of git’s shortcomings: git is highly optimized for text files, which means it is bad at binary ones. There are solutions, like Git LFS, but there was no compelling reason to keep relying on SVN for file storage backup — there was no automated commit of new files. It really only made local environments easier where files were concerned. Regular backups were already being made at multiple levels, so this was a nicety rather than a safeguard.

If there was a real cost, it was in individual autonomy.

The adopted model let us schedule releases with multiple engineers working on multiple features. Engineers could start to specialize and make use of those specializations. Accessibility, localization, and JavaScript expertise all became easier to integrate. We were able to expand the team beyond the idea that everyone must be an expert at everything, which opened easier paths for junior engineers to develop.