Let's Not Demonize Technical Debt

In my experience working with executives across the insurance industry, I have seen firsthand how technical debt is very much misunderstood, criticized, and demonized. It is primarily believed that technical debt can be avoided. Truth is — it’s inevitable. In many cases, it’s even intentional — used as a tool by high-performing software organizations to enable speed to market. Sounds strange, right?

My other observation is that everyone across the organization has a different understanding of technical debt. Because it’s a metaphor, an abstract concept, the definition varies depending on whom you ask.

Engineers: We really should insert a design and research story in our backlog, if we want to do this the right way.

Business: What do you mean — we now have to fund the refactoring work? Why are we refactoring, again?

Product Managers: Can we discuss some trade-offs? What if we put this off for a few more sprints?

If you cannot define it, chances are that you will not recognize it. And you will certainly fail to manage it.

In this article, I aim to establish a definition of technical debt and then share a few ways in which you can leverage Technical Debt as a tool to your advantage and manage it before it becomes overwhelming and all-consuming.

What exactly is technical debt?

The term was first coined by Ward Cunningham (one of the authors of the Agile Manifesto and inventor of wiki software). I have linked his original article in the references section, and I highly recommend that you check it out later.

Many definitions exist out there, but here is one that I crafted, and I find it very useful for explaining the concept to non-technical folks.

Technical debt results from the shortcuts we take during software development to expedite the value delivery, driving us farther away from an ideal technical design path. At some point, the codebase needs refactoring to move us closer to a good design. This refactoring takes more work later but can be put off for immediate rapid gains.

Metaphorically, it's like acquiring any financial debt — for instance, using debt as leverage for investments (borrow at lower interest and invest in an asset class with a higher rate of return). Or think of a mortgage to buy a property. In both these examples, you use the debt as a financial instrument to accomplish more today than you otherwise could, but you will pay a higher cost later. Until you pay off the debt, you pay interest for a while. In software engineering, the interest is the extra time/ effort you put into pushing more features out on a less-than-ideal design. Let's remember — good design, though an investment upfront pays off via enabling agility (through testability, flexibility, maintainability).

How do we acquire Technical debt?

Depending on the knowledge, experience, and design principles of your engineering team, the technical debt can be acquired either accidentally or intentionally.

To explain the concept further, I will borrow from Martin Fowler's Technical Debt Quadrant and layer in my personal observations across the organizations. There are four main categories of technical debt based on attitude and intention.

Type 1: Reckless /Accidental

Lack of technical leadership maturity and experience can lead engineers to make poor design choices without knowing the potential future fallouts. I have seen this happen at companies where enterprise architecture practice doesn't exist, and they have mostly junior engineers who do not have experience or credentials for software design.

Type 2: Reckless/ Deliberate

Sound design and coding practices cannot be established without architecture governance. Technical debt is thus gradually amassed through a series of quick and dirty solutions causing waxy build-up, which is hard to maintain or evolve. At a certain point, it grows so much that the cost of refactoring the application exceeds the cost of replacing the software.

Type 3: Prudent/ Deliberate

With such debts, you are essentially buying time. This is a classic scenario when experimenting with a new digital product or a new business model where Test, Learn, and Pivot is the motto. The engineering team creates a strategic, intentional architecture; however, it might make design trade-offs in the interest of time. Decisions are made after careful consideration of the consequences.

Type 4: Prudent/ Accidental

This type of debt is inevitable and is acquired by all engineering shops — even the most agile, high-performing ones. Over the course of time, business context changes, functionality grows, underlying technology or infrastructure evolves, and the engineering team knows more now than they did six months or years ago. At a certain point, refactoring the codebase becomes important to drive a good design.

How should we manage this debt before it becomes overwhelmingly huge?

Presumably, the technical debt in your organization is of types 2, 3, and 4 above; you can adopt practices to manage it while also being agile. I am borrowing basic principles from the financial debt management discipline to explain the ways by which engineering teams can be a pro at deploying technical debt as an instrument:

1. Adjust the attitude towards debt: It all starts with the mindsets and attitudes toward debt. Do your teams understand how it is amassed and how it costs your organization?

• Creating a standard definition of technical debt for your context is the first step, followed by education through various relatable examples/ scenarios for teams to learn how reckless design decisions can be avoided.
• Next, establishing guidance, standards, and process structures as a measure to avoid accruing any accidental debts.
• And in case they acquire the debt for a good reason, the attitude should be — "let us pay this off as soon as possible — before the interest exceeds the principal."

2. Not all debts are good debts: Assess if the intentional debt is a good idea. If yes, still shop around for better interest rates. Pressure test your assumptions:

• Take a simple quantitative and qualitative assessment to determine if the technical debt is worth it.
• Ask questions such as: —
  - Is the gain from rapid delivery worth the effort of refactoring later?
  - What is the interest we will be paying?
  - And for how long?
  - Is there another better debt option out there?

3. Know your payback plan: Like a pro, create your payback plan before you take on debt.

• In case of any technical debts acquired intentionally, the concept of Design Pay off Line applies — which refers to the maximum amount of functionality that can be delivered before the technical debt HAS to be paid off. If this pay-off line is not respected, the interest will surpass the principal.
• Engineering teams should have an understanding of how and when the refactoring can potentially be done. The current state of technical health of your product, product roadmap, team capacity, and demand pressures determine how much debt you can accrue.
• Teams should partner with the Product Owners to slot the refactoring features into the product roadmaps and understand the mechanisms through which the debt payoff can be funded.

4. Keep your books updated: At any given time, you should have a full understanding of your investment portfolio and in the case of your software product, read that as a full understanding of your product health vis-a-vis the backlog.

• Be disciplined about recording the debts in your backlog management tool (JIRA or Rally) — just as you record your features and defects.
• Take stock of your backlog regularly; you should review the debt thresholds, product metrics, and overall health dashboard during your monthly product reviews.

5. Communicate how debt accrual is hurting your goals: Obtain the organizational buy-in for the payback.

• Make it easy for the Product Owners as well as Businesses to understand the implications of this debt to your velocity and how it can deteriorate if the debt is not paid off.
• Use simple mathematical equations to calculate the interest you will be paying vs. the cost to pay off the principal via refactoring. Remember that, after all, the debt metaphor was created specifically to explain the concept of payoff to non-tech users who hold the purse strings.

Just as financial debt can be a great way to create strategic leverage, technical debt can enable you to take on strategic bets that you otherwise won't be able to. When debt is acquired intentionally and managed with discipline throughout the product lifecycle, you can achieve more faster.

If we can understand Technical debt, we can define it and hopefully stop demonizing it; rather, let us embrace it as an inevitable consequence of growth over time.

Since you have reached the end of this post, I assume you like my content. If so, click here to subscribe to my newsletter and get exclusive content from me.

References

[1] Know more about Ward Cunningham here

[2] Ward Cunningham Explains Debt Metaphor

[3] Martin Fowler’s Technical Debt Quadrant

[4] Read more about Design Pay off line