Want to Succeed at DevOps?
Don’t be a Hero
Don’t be a Hero
When people ask me what I do for a living, I tell them I help teams to adopt DevOps. Inevitably I have to explain what DevOps is: I say that I help teams to deliver better software, faster, more sustainably.
One of the organizations I have been helping is slowly but steadily adopting DevOps and changing culture. Their application has grown from a small app developed as a hobby to a scalable, cloud-based web application that serves thousands of stores around the globe. They have around several developments teams, a database team and an ops team. They do have automated builds, but their deployment processes are manually orchestrated and automated testing is manually triggered. They are using centralized source control and have quarterly release cycles.
The point is that their scenario is fairly typical: a small application grows rapidly, and as it grows more developers are hired. The developers are split into teams and siloed from database and ops in an effort to optimize. Business frequently makes commitments to customers (who pay the bills, after all), fixing both scope and dates for releases. Features are often added to the backlog right up into the week the release is supposed to go live, and testers seem to never have enough time to run a full set of tests. The teams are constantly fighting fires in production, desperately trying to keep the lights on while they develop too many features in too little time.
Fortunately, some of their leadership realize that this isn’t sustainable. I was initially contracted to help the team automate their releases. However, after spending a few days with them it became apparent that they have some cultural challenges beyond just automation: their branching strategy is outdated. What automation they have is complicated and can only be understood and maintained by one or two engineers on their teams. They’re constantly late on their planned releases and they have high numbers of bugs in production. This organization, like so many others, doesn’t need automation: it needs a DevOps culture.
Heroic Efforts Undermine Culture Change
When discussing their practices, I suggested that they shorten their release cycle. This should lead to more manageable releases, faster feedback and ultimately improved quality. The team was very excited and managed to convince business to move to a 6-week deployment cycle – still fairly long, but better than quarterly. I also walked the team through the benefit of migrating to Git, where topic-branching would help them isolate feature development more effectively. They also realized the need to simplify their automated builds and automate their release processes (including integrating test automation).
Unfortunately, while business loved the idea of halving the release cycle, it also failed to reduce the scope. The development teams are now under pressure to deliver just as much as before, in half the time. And here’s the kicker: they’ve been doing it.
And herein lies the problem: their engineers (developers, ops DBAs and testers) are simply fabulous. They pull late hours, they deal with tight deadlines, they keep the lights on and still manage to somehow deliver. But it’s starting to hurt them.
These heroic efforts from the engineering teams undermine the culture shift expected of business. When business is making unreasonable demands, the team is pushing back. However, they’re not pushing back hard enough, and at the end of the day they appear to be delivering what business wants. Business is dictating both scope and delivery date – and they’re getting what they want.
The engineering teams are struggling to maintain the pace and if nothing changes, they’re on their way to burnout. However, every time they push back on unreasonable demands from business, and then deliver anyway, they undermine the very changes they wish to see.
Donovan Brown, a principle DevOps manager at Microsoft has a fantastic definition of DevOps: is “… the union of people, process and products to deliver value to the end user.” However, while peopleare one of the pillars of DevOps – what does that really mean?
It means that DevOps values a culture where people matter. I was talking to a colleague and he was in a company where utilization was closely monitored. Every time there was pushback that utilization can be a poor metric, the response was, “How can we trust our people to deliver if we don’t monitory their utilization?” One of the 12 principles of The Agile Manifesto states that “Working software is the primary measure of progress.” Most of the teams I work with are passionate about what they do: they don’t need to be told to deliver. Quite often they need to be told to go home and do something that’s not work related! There will always be people that don’t have this passion, and if they won’t change you probably need to think of letting them go. But the majority of folks that I work with can be trusted to do their work.
Engineers Need to Take Responsibility to Help Business Change
DevOps is not about a few mighty heroes swooping in to save the day: it’s a culture where everyone works passionately to deliver value to customers in a sustainable and responsible manner. But what if business is still expecting miracles?
Engineers – developers, ops, DBAs, testers – you need to be having full and frank discussions with business to help them understand what challenges you face and how they can change. Don’t throw up your hands and “just do it”. When did you last take the time to explain what technical debt is? Or how much time context switching can waste? Or how sacrificing testing leads to poor quality, which leads to more time fixing bugs and less time delivering new features?
The Agile and DevOps movements have demanded that Dev and Ops teams change. So too should business: business might pay the bills, but IT is no longer just a support function – IT is the digital heart of most competitive businesses today. If business does not change its culture and adapt and take advantage of this sweeping movement, it will be left behind in the dust. And no-one wants that.
I encourage teams to track what work is planned and what work is later added and changed. When business demands another feature, you can dispassionately and objectively ask them to decide which other feature to push to the next sprint. Every person in the organization needs to be taking responsibility for delivery of value: if business is compromising that because of unreasonable demands, then they need a culture shift. Often that starts with saying no.
I’m not suggesting that engineers should clock out and never pull an all-nighter: I’m suggesting that heroics should be an exception, not a norm.
DevOps Culture Leads to Innovation
This isn’t just about keeping engineering teams happy: teams that are given the time and space to do the right thing end up delivering more, faster, with higher quality. This in turn means less time spent fixing bugs. This in turn gives teams space to innovate. Teams that regularly have to pull heroics to deliver won’t have time to improve. Ultimately that has direct impact on business: either quality issues will drive customers away or the business won’t be able to deliver fast enough and competitors will push the business out of the market. DevOps isn’t a nice-to-have in today’s markets: it’s table stakes, and organizations that don’t change will be left behind.
Take a look at your teams today. Can you spot heroes that regularly swoop in to save the day? That’s actually an indication that you’re not adopting a DevOps culture – or not adopting it widely enough in your organization. DevOps is not a tool or a team or a role. It is a culture that needs to pervades the entire organization, empowering every person to deliver value – and clock out to enjoy life outside of work. We shouldn’t be expecting our teams to choose value or life: it’s possible to have both!