Measuring Engineering Capability: Looking Beyond Delivery

3 days ago

8 min read

Organizations have become increasingly sophisticated at measuring software delivery. We can measure lead time, cycle time, throughput, deployment frequency, change failure rate, reliability, predictability, and a growing number of other engineering metrics.

These measurements tell us a great deal about what an engineering organization is delivering and how effectively software moves through the delivery system.

But there is another question that is just as important:

Are we actually becoming better at engineering?

An organization can improve its delivery metrics while simultaneously accumulating technical debt, losing critical knowledge, relying on a small number of specialists, and failing to develop the skills it will need tomorrow. Delivery can look healthy in the short term while engineering capability quietly deteriorates underneath it.

This is why engineering measurement needs another dimension.

Instead of measuring only delivery outcomes, organizations should also measure engineering capability. Capability metrics help answer questions such as:

Do we have the skills required for our technology landscape?
Are engineers continuously developing?
Is knowledge distributed or concentrated in a few individuals?
Can teams adapt when technologies or business priorities change?
Are we creating opportunities for engineers to grow?
Are we actively managing technical debt?
Can our engineering organization sustain its current level of delivery?

The objective isn't to create another performance scorecard for individuals. It is to understand the health and resilience of the engineering capability as a system.

Why Capability Matters

Delivery metrics tell us what happened. Capability metrics help us understand why we are able to deliver and whether we will continue to be able to do so.

Imagine two organizations delivering the same number of features every quarter. The first has:

Broad technical skills.
Strong internal communities.
Well-distributed knowledge.
Continuous learning.
Low dependence on individual specialists.
Engineers actively addressing technical debt.

The second relies heavily on a small number of experienced engineers, has limited cross-skilling, little time for learning, and a growing backlog of technical debt.

Their delivery metrics might look identical today. Their future capability, however, is very different. Capability measurement provides an early warning system for this type of problem.

It helps organizations recognize that engineering health isn't simply about how much software is being delivered. It is also about whether the organization is developing the capability required to continue delivering effectively.

1. Skills Maturity

What it measures: The depth and maturity of engineering skills across technologies, practices, and capabilities. Skills maturity can cover areas such as:

Programming.
Architecture.
Cloud engineering.
Testing and quality engineering.
DevSecOps.
Data engineering.
Observability.
AI engineering.
Security.
Product and domain knowledge.

A simple skills inventory can identify which capabilities exist within an organization and at what level of maturity. For example, organizations might define maturity levels such as:

Awareness → Working Knowledge → Proficient → Advanced → Expert

The important distinction is that this should not become a simplistic certification exercise. Someone completing a course doesn't necessarily mean they can apply the skill effectively in production. A stronger measurement combines self-assessment, peer assessment, practical evidence, project experience, and demonstrated application.

Why it matters

Technology changes quickly. An engineering organization that stops developing its capabilities can become increasingly dependent on legacy skills while the technology landscape moves forward. Skills maturity therefore provides insight into whether the organization is prepared for its current and future technology needs.

How to use it

Create a capability matrix aligned to the organization's technology strategy and periodically assess:

Current capability.
Required capability.
Capability gaps.
Critical skills.
Skills at risk of becoming obsolete.

This turns skills development from an informal activity into something that can be actively managed.

2. Cross-Skilling

What it measures: The extent to which engineers can contribute across multiple technologies, systems, or engineering disciplines. Specialization is valuable. However, excessive specialization creates dependency.

If only one engineer understands a critical service, that person becomes a potential bottleneck. If only two people understand a deployment platform, operational knowledge becomes concentrated. If a team has no experience outside its current technology stack, its ability to adapt becomes limited.

Cross-skilling reduces these dependencies. It can be measured through indicators such as:

Number of engineers capable of supporting critical systems.
Number of technologies each engineer has demonstrated competence in.
Number of people able to perform critical engineering activities.
Percentage of critical systems with more than one capable owner.
Participation in pair programming or knowledge-sharing activities.

The goal isn't to make everyone capable of doing everything. That would be unrealistic and potentially counterproductive. The goal is to ensure that critical capability isn't dependent on a single person or a very small group.

3. Knowledge Distribution

Skills and knowledge aren't necessarily distributed evenly across an organization. Some systems inevitably have experts. The risk emerges when expertise becomes a single point of failure. Knowledge distribution measures how concentrated critical technical knowledge is.

For example, an organization might identify:

Systems with only one experienced engineer.
Components with no secondary ownership.
Technologies understood by only one team.
Architectural decisions known only by their original authors.
Operational procedures that exist only as tribal knowledge.

This can be visualized as a knowledge concentration map.

A particularly useful question is: "If this person left tomorrow, what knowledge would leave with them?"

The objective isn't to diminish the value of experts. Quite the opposite. Experts should be encouraged to multiply their knowledge through mentoring, documentation, pairing, workshops, and communities of practice.

The measurement therefore becomes a mechanism for identifying where knowledge-sharing investment is most needed.

4. Learning Investment

Learning is often treated as something engineers do when they have spare time. That is a dangerous assumption. If learning only happens when delivery pressure is low, it will eventually disappear because there will always be another deadline. Engineering capability requires deliberate investment.

Organizations can measure:

Training hours.
Conference participation.
Technical workshops.
Certifications.
Internal knowledge-sharing sessions.
Pair programming.
Communities of Practice.
Hackathon participation.
Technical experimentation.
Engineering research and exploration.

However, hours alone aren't enough. An organization could spend thousands of hours on training while seeing little improvement in capability. A stronger approach measures both investment and outcomes.

For example:

Learning investment → demonstrated capability → application in engineering work → measurable improvement.

This helps distinguish activity from actual development.

5. Internal Mobility

What it measures: How effectively engineers can move between teams, domains, technologies, or roles within the organization. Internal mobility is an often-overlooked indicator of engineering capability.

A healthy engineering organization creates opportunities for people to grow rather than forcing career development to happen exclusively through external recruitment. Mobility might include:

Engineers moving into new technical domains.
Developers becoming architects.
Engineers moving into platform engineering.
Test engineers expanding into broader quality engineering.
Engineers developing AI or data capabilities.
Technical specialists moving into leadership roles.

A useful measurement isn't simply how many people moved.

It is understanding whether the organization has the capability and structures to enable those transitions. High internal mobility can also help distribute knowledge across teams, reduce organizational silos, and create broader technical perspectives.

6. Technical Debt Ownership

Technical debt is often measured as a backlog. But a backlog alone doesn't tell us whether an organization actually owns its technical debt. A better capability metric asks:

Who understands the debt, who prioritizes it, and who is accountable for addressing it?

Useful measurements can include:

Percentage of technical debt with an identified owner.
Age of technical debt items.
Percentage of debt actively prioritized.
Engineering capacity allocated to remediation.
Critical systems with known unresolved technical risks.
Ratio of new technical debt created versus debt retired.

This creates a more useful view of technical debt. The question becomes not simply: "How much debt do we have?" but: "Are we managing the debt responsibly?"

Technical debt is inevitable in most software systems. Unmanaged technical debt is the real problem.

Capability Is About Resilience

One of the most important reasons to measure engineering capability is resilience. Consider what happens when an organization encounters an unexpected challenge:

A critical engineer leaves.
A technology reaches end-of-life.
A major security vulnerability is discovered.
A new regulatory requirement appears.
The organization adopts AI.
A major platform migration becomes necessary.
Business priorities change dramatically.

Organizations with strong engineering capability can adapt. They have transferable skills, distributed knowledge, learning mechanisms, and engineers capable of operating outside their existing comfort zones. Organizations with highly concentrated knowledge and limited learning capacity may struggle significantly more.

Capability therefore isn't simply an HR concern. It is an engineering resilience concern.

Measuring Capability Without Measuring People

There is an important distinction that organizations need to make. Capability metrics should primarily measure the system, not turn engineers into a collection of scores.

For example, saying: "Team A has a lower skills score than Team B" may create little useful insight. Instead, ask: "Which capabilities are missing from the organization?" Or: "Which critical systems have insufficient knowledge coverage?" Or: "Where are we most exposed if priorities or technologies change?"

This changes the conversation from individual performance to organizational improvement.

Capability metrics should help leaders answer: Where should we invest? and not: Whom should we blame?

Connecting Capability to Delivery

Capability metrics become particularly powerful when combined with the delivery metrics discussed throughout this series. For example, an organization could discover:

High deployment frequency + high change failure rate + low test automation capability

This suggests a very different intervention from simply telling teams to improve quality. Or:

Long lead time + high knowledge concentration + significant dependency on specialist teams

This might indicate organizational constraints rather than development productivity problems. Similarly:

Stable delivery + declining learning investment + increasing technical debt

could indicate an organization that is currently performing well but potentially consuming its future engineering capacity. The value comes from connecting the measurements.

Delivery tells you what is happening. Capability helps explain whether the organization has the means to improve it.

Building an Engineering Capability Dashboard

A capability dashboard could bring these dimensions together:

Skills maturity: % of required capabilities at target maturity
Cross-skilling: % of critical systems with multiple capable engineers
Knowledge distribution: Knowledge concentration across critical systems
Learning investment: Learning hours + demonstrated capability growth
Internal mobility: Internal technical moves and transitions
Technical debt ownership: % of debt with accountable ownership
Critical skills risk: Number of capabilities with insufficient coverage
Capability growth: Change in capability maturity over time

The dashboard should be used to identify trends rather than produce a single "engineering capability score." A single score can hide important differences. An organization might have excellent cloud capability but poor security expertise. It might have strong development skills but weak observability capability.

From Skills Inventory to Capability Strategy

The ultimate purpose of measurement is action. Once capability gaps are visible, organizations can respond through:

Training.
Mentoring.
Pair programming.
Communities of Practice.
Recruitment.
Internal mobility.
Architecture rotations.
Technical workshops.
Hackathons.
Conference participation.
Engineering experimentation.

This creates a continuous cycle:

Measure → Identify gaps → Invest → Apply → Reassess

Over time, this turns engineering development into an intentional organizational capability rather than something left to individual initiative.

AI Makes Capability Measurement Even More Important

The rapid adoption of AI makes this particularly relevant. Organizations are not simply learning another programming language. They are introducing new engineering capabilities around:

AI-assisted development.
Prompt engineering.
AI evaluation.
LLM application architecture.
AI security.
Data engineering.
Model governance.
AI observability.
Human oversight.

Instead of measuring "How many engineers have access to an AI coding assistant?", companies need to instead answer the question: "Do our engineers know how to use AI responsibly and effectively?"

This requires organizations to measure capability rather than tool adoption.

An engineer who can generate thousands of lines of AI-assisted code but cannot evaluate its security, correctness, maintainability, or architectural implications hasn't necessarily increased engineering capability. AI increases the importance of judgment, verification, systems thinking, security awareness, and continuous learning.

Conclusion

For years, organizations have become increasingly sophisticated at measuring software delivery. We can measure how quickly work moves, how frequently we deploy, how many defects escape, how quickly incidents are resolved, and how predictable delivery is. But these metrics only tell part of the story.

The deeper question is whether the organization is building the capability required to continue improving. Skills maturity, cross-skilling, knowledge distribution, learning investment, internal mobility, and technical debt ownership provide another lens through which engineering health can be understood.

They help organizations see risks that traditional delivery metrics may miss: critical knowledge concentrated in individuals, skills that are becoming obsolete, insufficient investment in learning, and technical debt that nobody truly owns. Most importantly, capability measurement changes the conversation.

Instead of asking only: "How much did we deliver?" we begin asking: "What kind of engineering organization are we becoming?"

That may ultimately be the more important measurement. Because a high-performing engineering organization isn't simply one that delivers quickly today.

It is one that continuously develops the people, knowledge, practices, and technical capability required to deliver well tomorrow.

CRAIG RISI