The Contemporary Application of Agile: Scrum Is Failing (Part 1)

This is Part 1 of a two-part series examining why Scrum is no longer doing very well in the context of modern software development. Part 2 will explore alternative agile approaches that can meet the needs of today’s software teams. If you haven’t read the Agile Manifesto in a while, you might want to refresh; it’s referred to frequently.

In 2001, a bunch of really smart software professionals went skiing. Every one of them had their own ideas about why 90% of software projects throughout the 1990s were late, overbudget, or both. One thing was clear: methods that had worked in the 1970s and 1980s were no longer working. So, these professionals gathered to figure out what to do about it. The result was a set of principles we all know as the Agile Manifesto.

The first time I saw it, I thought, “That’s it? A list? It’s a manifesto. Surely there’s some actual paragraphs, or strings of connected sentences, or something.”

Then I read it. There was no need for paragraphs. Anybody who lived through the pain of trying to deliver custom software in the 1990s knew exactly what problem each and every item on that list was meant to solve.

What they didn’t do was develop a methodology for implementing their principles. In the ensuing years, various approaches emerged, and Scrum became the primary means of implementing Agile development. For many years, this worked very well.

Today, ask any software team if they’re an agile development team, and they’ll almost certainly say yes. Ask them if they are using Scrum, and some will say yes, while others look confused because they think you're essentially asking the same question again. Ask them to describe what they are doing, and you’ll hear about sprint planning, standups, retrospectives, story points…classic Scrum, right? Well…not exactly.

Ask them more about what they’re actually doing day-to-day, and it won’t take long to hear them say something along the lines that they’ve “adapted Scrum to fit their needs”, which is all fine and good. Everybody has always done that.

But in today’s world, what you’ll find with many teams is that they are actually doing an often-messy Franken-Scrum that keeps Scrum’s ceremonies and behaviors by name, while unwittingly discarding or violating so many of the original Agile principles. Perhaps it could be called “Scrumban,” referring to the competing agile methodology called “Kanban”, though I think that implies more intentionality than what’s really happening. A lot of teams are engaged in what I like to call ceremonial chaos: going through the motions of Scrum while most of the positive results they produce are due to the dedicated teamwork of people determined to get it done despite the chaos.

The thing is, it’s not the teams' fault. It isn’t a minor problem that can be dismissed as teams “doing Scrum wrong” as we’ve all heard a million times. I will argue that the Scrum way to achieve Agile has become incompatible with the realities of modern software development. The extinction of corporate data-centers, which have evaporated into the clouds, microservice architectures, physically distributed teams across time zones, and the blurring of development and operations (i.e., DevOps) have created an environment where one of Scrum’s key value propositions—predictability through iterative development—has become very difficult to achieve in any meaningful way.

Scrum’s core principles are straightforward: a dedicated team working on a prioritized backlog without external interruptions can achieve a predictable velocity. Knowing your velocity allows you to forecast delivery. Elegant in theory and in practice, it has worked very well to deliver results aligned with the Agile Manifesto for many years. In the past decade or so, natural technological evolutions and inevitable corporate governance have begun to create cracks in the framework that holds Scrum together.

The traditional separation between development and operations – where developers built features while ops teams kept systems running – has been significantly blurred. The DevOps movement, combined with cloud infrastructure and microservice architectures, has created an environment in which developers in many organizations (particularly small- to mid-sized companies) are directly responsible for maintaining production systems and responding to production issues. When something breaks, it goes straight to the dev team rather than being filtered through support tiers and operational procedures.

Developers now must constantly handle sprint work, fix bugs caught by the QA team, and address DevOps/ProdOps issues that arise. This modern-world reality is fundamentally incompatible with Scrum’s model of focused, predictable sprint work.

The Manifesto embraced changing requirements because business priorities shift, market conditions evolve, and customer needs become clearer over time. As they aren’t known in advance, these are expectable changes that can be incorporated into planning for the next sprint, or whenever the Product Owner decides to prioritize them. Production issues tend to be very sporadic; 3 this week, none next week, 2 the week after, etc. You can’t forecast for this. Product management can’t deprioritize a high-priority customer-reported issue; it goes to the front of the line. Half the time, the problem might be in your microservice, or it might be three services downstream. You won’t know until you investigate, and customer-reported issues bounce from team to team until someone figures out which system contains the root cause. Meanwhile, the days of the sprint don’t stop.

Every sprint becomes a game of wondering what set of interruptions will occur today or later this week. Because of the variability in occurrence rates of fires, etc., velocity changes every sprint. Predictability – one of the primary reasons executives like Scrum as a process – becomes less meaningful.

The context switching required in this environment keeps growing as teams are pressured to “do more with less.” Developers are expected to maintain deep focus on complex feature development while simultaneously being available for urgent issues. It’s impossible for a team to maintain a constant pace indefinitely, as the Manifesto prescribes.

Agile’s fourth principle states: “Business people and developers must work together daily throughout the project.” But there’s an implicit assumption here: that someone on the development side is given the role of ensuring technical coherence and maintainability of the codebase.

In my early years as a lead engineer, I wasn’t necessarily expected to have the same point output as other senior engineers; my primary job was to oversee the design of the system, make sure everyone understood what they needed technically, hold impromptu discussions and reviews while development was occurring rather than after, and manage technical debt. I also handled all the fires and production issues that would come up, only involving others when absolutely necessary. Meanwhile, I would usually get a few points done, or sometimes not. If the team was making working software, we were making progress. In this environment, someone was playing the role of advocating for clean, high-quality, maintainable code.

Today’s lead engineers are usually expected to produce at the same rate as other experienced developers, as well as do other things. They’re often looked at as individual contributors first, technical leaders second.

The consequences of this can be quite significant.

Without close technical oversight, system design problems aren’t identified until PR review. At that point, it’s usually too late. The pressure to deliver means suboptimal code gets merged and called “tech debt” to be addressed later, which never comes.

Each developer gets their story and implements it however they see fit. They write helper functions and utilities that already exist elsewhere in the codebase or in proprietary libraries they didn’t know existed. Tried-and-true software design patterns aren’t used. Standard software engineering best practices, such as separation-of-concerns and dependency-injection, are ignored. The result can be fatally unmaintainable systems; anyone reading this has probably seen the “fixing one bug creates another” scenario in a messy legacy codebase. The Manifesto states that continuous attention to technical excellence and good design enhances agility.

Scrum’s ceremonies – sprint planning, daily standups, backlog grooming, retrospectives – were designed to facilitate collaboration, transparency, and continuous improvement.

Standup: the daily standup often turns into analysis sessions about whatever high priority matters are at hand, instead of a quick turn for each and then breakout sessions with whomever is needed.

Backlog Grooming: Stories being discussed here should be for requirements that will be in several sprints downstream of the current one. However, for a variety of (often legitimate) reasons beyond the scope of this article, the typical grooming sessions often are composed of the entire development team sitting and watching the product owner type sometimes vague and often quite loquacious business-language into the online tool. So often, these descriptions are not actual requirements; they’re just business feature descriptions. Developers ask questions, but there’s always more to cover than time available. In the end, from a developer’s perspective, the stories lack the detail necessary for actual implementation: user scenarios aren’t specified, edge cases aren’t explored, and technical constraints aren’t considered. The result is that the developers must track down the missing information when assigned, leading to confusion, especially when it’s time to test according to acceptance criteria. Generally, it takes more than a 1 hour meeting per week to make serious progress on the backlog, but often stakeholders are only available for 1 hour, if even that, having to jump from meeting to meeting. So often the feeling is that the product team is not really a part of the development team; it's usually clear that they want to be, but demands put on various roles in modern times make it impossible.

Sprint Planning: Hopefully the stories as they come up in planning have been seen before by the dev team, but sometimes higher priority stories pop up that have not been seen before. Either way, point estimates are made off-the-cuff by everybody, usually with no time taken to think through the various user scenarios involved or the technical challenges those scenarios might pose. This isn’t estimation; it’s guessing.

Retrospectives: Retrospectives are supposed to drive continuous improvement through team reflection. The positives are usually perfunctory, yet true: good communication, helpful teamwork, etc. The improvements list occasionally has something actionable by the team, and good suggestions lead to solid action items. Very often however, the list chronically the same items every time around which the team has no control over:

• “Dependencies on other teams block us and they aren’t able to do the needful”
• “Production issues and other fires keep interfering with sprint work”
• “Timezone differences with offshore team creates delays in both directions”

Those unsolvable problems are usually the biggest impediments.

The Manifesto emphasizes “customer collaboration over contract negotiation,” but it assumes that the team building the software has the complete autonomy to deliver the value to that customer. Microservice architectures, especially when they aren’t designed correctly, lead to dependencies. Loose ones, but still dependencies. Cross-team communication and prioritization is a primary bottleneck in the microservices era. Often the dependencies are hard to spot early on as well; often a developer will start to work on a story and then realize they need something that doesn’t exist in another service. Often there are workarounds which add complexity, but that work wasn’t planned for in the sprint, which blows up the sprint timeline.

This scenario is quite common. Stories ping-pong between “In Progress” and “Blocked” states. Developers maintain context on multiple partially-complete features, further fragmenting their cognitive resources. Stories routinely get pulled from last sprint to next sprint. The sprint cadence becomes arbitrary when there’s no set of work agreed upon as expected to be done in the sprint. This violates Agile’s seventh principle: “Working software is the primary measure of progress.” The software isn’t working—it’s partially complete, blocked, and scattered across multiple teams’ backlogs.

Many organizations have offshored QA teams to reduce costs, and in these environments, minimal timezone overlap between developers and testers creates an unfortunate impact on the rhythm of sprints.

Let’s say a developer completes a story mid-morning and marks it “Ready for Testing.” At that time, the QA team has already quit for the day, as it is very late for them. When they start their day, they pick up the story to test, realize they have some question or other, and send a query about it to the dev team. The dev formulates an answer and send it, but by then the QA team is done for the day, so another day goes by before that testing occurs. Sometimes this process happens more than once. This is very common, yet few consider it as a problem and it is never accounted for.

Some teams are aware of how Scrum is failing in modern times, and they will call what they do “Scrumban”—a hybrid approach that combines Scrum’s ceremonies with Kanban’s focus on flow and work-in-progress limits. In theory, this should address some of Scrum’s problems in the modern environment.

They keep the sprint ceremonies and cadence (because you can’t tell management that you aren’t doing Scrum), but stories still get blocked by dependencies. Production issues still interrupt flow. QA bottlenecks still exist.

In this case, they’re really just doing Kanban—managing a flow of work items through a series of states until they’re done, which the Scrum elements add overhead without adding much value at all.

The modern reality is that consistent valid predictability is virtually impossible, technical excellence is rarely considered important, low value activities consume time for little more than show. Velocity metrics get reported and tracked despite being mostly just the result of guesswork rather than a useful statistical value.

It is not that teams are doing Scrum wrong. It is that Scrum has become incompatible with how software is actually built in modern technical and business environments.

Yet curiously, everyone continues to insist on keeping Scrum’s ceremonies while ignoring how it has become the enemy of agile principles. There’s a disconnect between how Scrum is supposed to operate and what it actually does in modern environments. Teams are going through the motions, burning hours in ceremonies, tracking metrics that really don’t mean anything useful, and delivering software despite the process rather than because of it.

Agile’s core principles are being ignored every day in service of Scrum’s guidelines, because Scrum’s guidelines used to align very well.

They no longer do.

In Part 2, we’ll explore how teams can efficiently deliver software effectively in the modern cloud-native, microservices, globally-distributed development environment with a different set of Agile guidelines.