DevOps and Agile in the modern enterprise

Dive Brief:

• The average company has a backlog of planned IT projects going back between three months and one year, according to research published Tuesday. The report, conducted by The Economist Intelligence Unit on behalf of Appian, surveyed 1,002 business and IT decision-makers in nine countries.
• There are many drivers behind the backlog, but at least part of the problem has to do with finding the right data, the report found. Six in ten leaders canceled a digital project due to an inability to access the necessary data.
• "There's immense pressure to change right now in businesses around the world," said Appian founder and CEO Matthew Calkins. "The pandemic, the changing circumstances, the new regulations, the new employee and customer expectations ... No wonder we've got so much need to change our systems that it's more than IT can keep up with."

Dive Insight:

IT backlogs trap companies in a harmful cycle — failing to keep up with market demands and competition, while technologists spend time managing technical debt. 

The existing backlog is, in part, a side effect of IT's role during a time of crisis. Companies relied heavily on technology to survive, at the expense of budgets and continuous change for IT departments.

"While this increased reliance on IT to deliver often has a positive impact on the view of IT within the greater organization, the state of affairs for most operations currently consists of a backlog of stories, spikes and tasks, with dramatic year-over-year (or, quarter-over-quarter) growth rates, that seems to be continually increasing in size," said George Burns, senior consultant of cloud operations at SPR, in an email.

Non-IT workers shoulder part of the IT burden, according to the report. More than half of leaders say business units already do more than IT to procure or develop new applications.

But the shift to non-IT work aimed at the backlog could mean compromising the quality of applications, said Calkins.

One strategy businesses can use to solve the IT backlog is take an Agile approach, iterating toward execution on deferred projects.

"Agile teaches us that small victories and quick failures, which are the direct result of correctly defined requirements and acceptance criteria, will lead to the compounded results necessary to get ahead," said Burns.

Slowly but surely, companies that focus on easy fixes rather than robust solutions can accumulate technical debt while building software — and see their productivity dwindle with every development cycle.

For software engineers, fallout from technical debt is taxing daily bandwidth, as time spent writing new code is instead occupied by fixing what's broken or precariously held together. Six in 10 engineers say tech debt slows pace of development, according to a report published by software company Stepsize. The company surveyed more than 200 engineers, engineering leads and CTOs. 

Tech debt can grow and become difficult to manage, especially in teams that don't tackle it from the start through engineering excellence principles, according to Nicholas Charriere, staff software engineer at autonomous car company Cruise. 

"Once you get a certain amount of tech debt, the rate at which you can build new things, or even maintain something that already exists, crawls to zero," said Charriere. When technical debt mounts, organizations get stuck in a vicious cycle of devoting resources to troubleshooting instead of new iterations. 

And time spent fixing past issues adds up. On average, engineers spend six hours every week dealing with technical debt, according to Stepsize.

Technical debt can accumulate through several factors. The use of too many software languages and frameworks, development team turnover, and the need to accept known defects in order to meet deadlines are three of the top drivers of technical debt, according to a report from OutSystems. The use of outdated software languages or frameworks, bad infrastructure decisions or security defects from evolving threats can also contribute to the backlog.

"You're often dealing with technical debt that you didn't create," said Charriere. "Not only is it hard to work with because it's technical debt ... but also you don't necessarily understand exactly why it came to be, which makes it even harder."

Tech debt impacts teams, customers

Many see technical debt as a necessary evil, a consequence of working on projects with multiple engineers who might have different biases or skill levels, according to Chandra Kalle, head of engineering for LeanTaaS.

If not managed well, technical debt slows down the development process, potentially leading to a bad developer and customer experience, said Kalle. The impacts include features taking longer than expected due to refactoring dependencies, frustrated engineers having to read and rewrite other peoples' code under tight deadlines and annoyed users dealing with buggy features.

More than half of engineers believe technical debt negatively impacts team morale, according to Stepsize.

Mismanaged technical debt can bring consequences to the business side — it isn't merely a technical problem.

"Technical debt can also hinder your ability to solve business needs quickly, as it prevents you from taking advantage of more modern software development capabilities and tooling," said Justin Stone, senior director of Secure DevOps Platforms at Liberty Mutual, in an email. "This can impact the amount of time you spend developing new capabilities and/or the cost of maintaining existing ones."

Organizations aiming to manage technical debt need to introduce high engineering standards, according to Charriere. 

"It has a lot to do with having a high engineering bar, forcing everything that goes in and gets merged in to go through some kind of peer review process, whether that's a code review or design review," said Charriere. "Conversations need to be had, decisions need to be justified and documented."

Tech leaders hoping to steer software development teams for optimal performance should balance communication, collaboration and independence. Engineers also require enough support from other business units to effectively manage technical debt and advance on projects.

Similar to financial debt, technical debt is more likely to become a problem if left unattended, according to Stone.

"Instead, take the time to understand how much you have and what the trade-offs are for having it," said Stone. "The key is to actively manage the trade-offs so you have the most customer value with the least amount of debt."

Maximizing the productivity of software engineers while upholding company communications is a puzzle tech leaders have long sought the answer to. 

Input from software developers suggests one potential pitfall: Meeting overload and frequent interruptions can derail productivity, according to GitHub's Good Day Project. The study polled and tracked the daily work of 40 software engineers over the course of two weeks to find what patterns of work were most indicative of having a "good day" or "bad day" at work.

Going from two to three daily meetings lowered the chances of developers making progress on their goals from 74% to 14%, according to the report. When developers faced minimal or no interruptions, they had an 82% chance of reporting a "good day," but if interrupted the majority of the day those chances fell to 7%.

Heads-down work alone can't set a company up for success. Without sufficient touch points as engineers put lines of code together, teams can build products that don't fulfill their expectations. Leaders who fine-tune productivity in software engineering strike a balance between autonomy and support. Finding the ideal cadence can lead organizations to higher profits.

A network of supporters, from security to compliance and business, should assist software development work, under a framework of joint accountability, according to Chris Condo, principal analyst at Forrester. 

"If engineers are working on a new product, they shouldn't be working in a silo," said Condo. 

But add up planning sessions, check-ins and stand up meetings, and a bloated schedule can reduce the bandwidth engineers have to build and ship code. 

"Touch points are necessary for any collaboration," said Eirini Kalliamvakou, senior researcher at GitHub, in an email. Too many of these touch points "can disrupt our days and our progress." Balancing these instances with the rest of the work can guide operations "to a team’s or company’s benefit."

Obstacles to productivity

Technical debt impedes an organization's ability to advance on its digital goals. Teams must fix existing problems in the tech stack without easing off ongoing modernization efforts.

Engineering teams suffer when faced with the consequences of choosing easy solutions now instead of taking longer and better strategies, said Dave Garrett, chief strategy and growth officer at the Project Management Institute.

"While unavoidable at times due to time constraints, resources and other hurdles, technical debt can have a major impact on businesses," said Garrett in an email. For example, early time investments in design, integration and testing can be a boon for productivity. 

Context-switching is another drag on an engineer's ability to stay productive, according to Condo. This can be a consequence of having engineers work on too many projects at once.

There's a financial motivation to whip software development processes into their best shape. It's tech-savvy companies that are more likely to expect their businesses to grow, MIT found. 

And while there "isn't necessarily a direct link" between efficiency in software engineering and a company's profitability, an efficient team is indicative of a high-performing organization, according to Bola Rotibi, research director, software development CCS Insight.

"That means that the company has invested in people in terms of having the right skills, and maybe they've supported them in having the right environment, and in having the right tools, so that they are as efficient as they can be," said Rotibi. 

How to strike the right balance

If any work lesson from the pandemic will have lasting power, it's the fact that efficient teams can carry on regardless of their location. Teams even found ways to replace synchronous touch points such as calls or meetings with asynchronous communication.

However, teams don't just slip into ideal workflows by accident. Think of software development as a military exercise, said Rotibi: Without proper planning the objectives will fail.

It's critical to articulate what needs to be done and then "partition people out" to tackle their tasks, Rotibi said. Planning lets employees focus on what they need to do, while interacting or collaborating when they need to do so.

As part of the search for balance in a software engineer's day, executives should include some sense of independence, according to Garrett.

"I think you need to give people some autonomy here – the space to do what makes sense, which will likely evolve over time," said Garrett. "At the same time, you should understand and assess what’s working for specific teams and projects and make sure that people are sharing what works. You want to offer insightful guidance, but not dictate solutions." 

Without sufficient support from other business units, projects can stall. In dysfunctional organizations, a business team creates "a giant backlog," then deprioritizes the development team when they need input, said Condo. 

It's one instance where meetings can save, not waste time.

Units such as security, business or compliance need to "be part of the design upfront, and that's the only way you can actually get software developed from the start, correctly, or as close to correct as possible, and meets all the company's best practices before it gets to final stages," said Condo.

While comparing DevOps adoption across sectors, retail leads in its ability to deploy on-demand, according to Puppet's industry report card based on a survey of almost 3,000 respondents for its 2019 DevOps report. The technology sector has the most mature DevOps adoption and is the leader in integrating security into the software delivery pipeline.

In retail, 57% of respondents can deploy to production on demand; 28% are doing so regularly. It's "head and shoulders" above other industries, Alanna Brown, senior director community director and developer relations at Puppet told CIO Dive. In part, its advancement comes from Amazon's revolutionary pressure. Retail also does not face the same regulatory burdens as industries such as financial services or government. 

A key metric of DevOps success is how well companies integrate security. In technology, 35% of companies see security as a shared responsibility across teams, above the cross-industry average of 31%. "If you are doing DevOps well, chances are you are already doing security well," Brown said.

The Puppet report's demographics add context to retail deployment success ahead of the tech industry. Top tech companies are renowned for elite DevOps performance. Amazon, Google and Netflix can deploy "thousands of times per day," according to DevOps Research and Assessment.

Many tech companies surveyed were smaller, which prevents the data from skewing toward the technology sector, according to Brown.

The more-regulated industries, government and financial services and insurance, lag in DevOps implementation. While financial services and insurance have built a DevOps foundation, they are slow to progress to more advanced adoption, according to the report. 

The government sector is the slowest to remediate critical vulnerabilities — only 24% of respondents are able to remediatethem in less than one day, Puppet found. It's a stark contrast to retail, where 53% can remediate critical vulnerabilities in the same time frame. 

Cross-industry differences highlight the burdens of each sector. Some struggle with regulation and bureaucracy; rules mean tech teams have to slow down and follow outlined processes. 

In retail, the biggest burden is competition. To stay relevant, companies are forced to move fast and iterate, breaking down department barriers to optimize systems.

Achieving operational excellence and innovation are critical to the long-term success of any business. 

Innovation, both internal and external, can help a company grow faster, better serve customers, and positively impact revenue. Operational excellence, supported by a strong DevOps culture, can drive the speed of innovation, accelerate time-to-market, enhance productivity, and improve reliability and quality. 

Yet, 76% of IT leaders find it challenging to obtain the right balance between business innovation and operational excellence, primarily due to the increased focus on functional tasks brought about by the COVID-19 pandemic, according to this year's State of the CIO survey. 

Every organization's needs are different, but arguably most CEOs are more focused on driving innovation. They are more likely to earmark capital spending for customer-facing projects that generate revenue or increase competitiveness versus improving the performance of internal or back-office systems. 

CIOs in the survey seem to agree; 68% say that generating revenue is part of their mandate. The challenge for CIOs is to execute on the former without neglecting the latter while staying within budget. 

Here are a few best practices and principles that strategic CIOs can apply to drive innovation and deliver operational excellence at the same time with minimal, incremental increases in investment.

Invest once to solve multiple challenges

When considering investments in innovation or operational excellence, look for ways to do things more efficiently and deliver added customer value at the same time. 

For instance, when rolling out a next-generation product or a set of new technology features, take advantage to refactor the underlying technologies to improve performance, reduce code complexity, or initiate other changes that can make future upgrades easier and faster. Conversely, if investing in improving software or fixing a defect, take the opportunity to look for ways to deliver better and faster or improve workflow for end users.

Addressing both efficiency and customer value with each new initiative requires a little extra planning and a new way of thinking, but it allows CIOs to tackle multiple problems with a single investment of time and money. 

Embrace automation around targeted processes

All organizations have certain expenditure categories, such as compliance and security, that are non-negotiable. In these situations, it makes the most sense to optimize spending through automation and seek out investments that will scale. 

As an example, invest in automated tools to detect and automatically deploy patches or critical software updates instead of manual processes that are expensive and error prone. 

Automating business and/or IT process improvements was among a number of activities that have increased in importance over the past year, according to the survey, and 81% of respondents said they expect it to become more important over the next six to 12 months. Doing so will allow CIOs to focus resources on value-driving activities, rather than squandering limited IT dollars on repetitive "mandatory" tasks.

Beyond that, automation can create operational efficiencies while delivering new capabilities and innovations to the market faster. 

A great example of automation that can drive both outcomes is the move to microservices, or micro-frontend applications. Let's say an organization initially develops a new product or service for a specific geography or market segment, but eventually wants to scale the product globally. Re-usable micro-frontends can help streamline and scale the development process so that it only takes a sprint or two to localize the offering for a particular market or segment rather than the months or years it might require if each market or segment were addressed individually. 

Breaking up large monolithic applications that are difficult to change at scale is critical for faster and safer software deployments.

Take advantage of market shifts or disruptions to invest in IT

Understandably, many organizations tend to retreat and retrench when facing financial uncertainties or similar seismic marketplace shifts. But organizations that embrace these shifts can bounce back more quickly and become more resilient when they have the means to innovate even faster. 

Eighty-two percent of respondents to the CIO survey say they have implemented new technologies, IT strategies and/or methodologies due to the pandemic. Among the top business initiatives driving IT investments, 36% of CIOs now say transforming existing business processes is the top priority, followed by increasing cybersecurity protections (34%) and improving customer experience (33%). This is in contrast to last year's survey, where increasing operational efficiency (37%) was the most frequently mentioned business initiative driving investment.

The pandemic has made businesses realize they can (or must) change the way they operate. The move toward hybrid work environments required new IT systems and collaborative technologies, processes, and approaches. Addressing these new requirements has helped CIOs identify new opportunities for both innovation and operational excellence. 

Implementing advanced technologies like machine learning and natural language processing, and exploring emerging technologies such as IoT, augmented reality, or blockchain are still in focus. Other customer-facing initiatives have become more of a priority. These include enabling better customer experiences and alternatives to face-to-face communications, adapting products and services to meet new demands, and delivering products and services in new ways. 

As the pandemic recedes, CIOs will likely return to familiar priorities like driving business innovation, refreshing business and IT strategy, leading change efforts, and implementing new systems and architecture. 

But hopefully the experience of having to make rapid technological changes out of necessity has permanently changed the way that CIOs approach new tasks, helping them recognize it is possible to drive innovation and operational excellence simultaneously, even with their traditionally tight budgets. 

Shifting from an "either or" to a "both and" approach might cost a few extra dollars and a little more time up front, but the dividends from these incremental investments can include lower future costs, higher revenue, accelerated product and IT development/deployments, and increased efficiency, flexibility, and scalability. The approach is also likely to improve overall satisfaction among customers, the CEO, and other key stakeholders.

Software managers fear draining the batteries on their teams. But they cannot miss deadlines on work that can't slow down.

With his software development team working remotely, Lloyd Closson, CTO at G&A Partners, says one of the things he misses most about the office is hosting whiteboard sessions.

"They're my favorite things to do, to sit down and architect a system in front of a whiteboard," said Closson, in an interview with CIO Dive.   

Nostalgia aside, the sessions helped the team assess new ideas and improve the company's HR outsourcing tools. Amid the pandemic, a whiteboarding feature within Microsoft Teams — the companywide collaboration tool — became an adequate proxy.

Equipped with iPads as virtual canvasses, home-bound team members recently bounced around ideas digitally on how to solve a freshly discovered bug in the company's HR outsourcing platform. 

Pushing the work forward in uncharted territory has made IT execs lean into their collaboration platforms, expanding their use and exploring new features, as agile methodologies help teams iterate in quick bursts with increased communication.

Fresh, challenging context

Three pandemic-induced changes are impacting application development and delivery work, according to a Forrester report:

• Work and life shifted online, pushing home-bound employees to rely solely on digital tools and retooling business processes. 
• Teams became more distributed, creating new patterns for collaboration. 
• Organizations clamored for digital products to oversee business-critical processes such as controlling building access or supporting public service processes under strain.

Development teams that entered the pandemic using modern distributed development tools, including integrated collaboration platforms, are faring better than their laggard counterparts.

For cloud engineering company DigitalOcean, early reliance on collaboration tools eased the transition. Its engineering team was 90% remote, while the rest of the company was roughly 65% to 70% primarily remote.

"When it came time to go 100% remote as a company, that change was relatively painless," said Al Sene, VP of Engineering at DigitalOcean, in an interview with CIO Dive.

Agile methodologies have proven their worth at DigitalOcean, allowing the team to work in batches, deploying smaller snippets of code frequently. If there's an issue, the approach lets the company address it quickly, with minimized impact to the broader infrastructure.

A successor to the traditional waterfall model, the approach helps shorten decision times, letting teams iterate quickly and bring "velocity" to the process, Sene said.

Increased communication allows remote teams to build trust, which makes agile methodologies a good match for distributed teams to operate smoothly. Among the 12 principles of agile are face-to-face conversations between the development team and frequent reflections on how to increase efficiency, according to nonprofit organization Agile Alliance.

The COVID-19 pandemic is driving up agile adoption at the enterprise level. In May, 55% of IT leaders said their company planned to increase the use of agile over the next 12-14 months, according to a survey published by Digital.ai. Momentum for agile adoption has increased over the past 90 days for 43% of companies.

"There's a very clear following of agile practices in how we do all of our projects internally as well as for customers, which means frequently scheduled communication" said Malcolm Ross, VP of product strategy and deputy CTO at Appian, in an interview with CIO Dive.

Following the structures of agile methodology means Appian adheres to 15-minute daily standups, sprint reviews and use case definition sessions as part of its workflow. 

"Agile methodology is not really about doing things faster, it's about frequent communication to the key stakeholders and constituents involved in an application development project, as well as remaining agile to change as different conditions arise," Ross said.

Companies which adhere to a waterfall model amid the pandemic risk devoting resources to projects that won't fulfill the needs of end-users, potentially sinking resources into dysfunctional products.

The urgency of collaboration

Sustaining the pace of work at DigitalOcean has required reliance on the collaboration tools the team adopted well ahead of the pandemic.

"The engineering team makes use of all the tools at their disposal for communication and collaboration," Sene said.

The toolkit at DigitalOcean includes screen sharing, tools for social development and shared development environments for code reviews.  

"It is really a great time in terms of the technology that is available for us to collaborate in these ways," Sene said. "I could only imagine 20 or 30 years ago, would we have been in the middle of the pandemic, how would we have done the collaboration?"

But the deployment of tools for collaboration, even in the midst of challenging circumstances, does not guarantee team-wide adoption.

To ease any type of change into dev team dynamics, a clear understanding of needs and requirements is a critical starting point, according to Closson.

"I think that if you can clearly articulate what you're trying to solve with the new technology, and how the technology solves the problem, that facilitates change much easier," Closson said.

With modern businesses transforming into technology companies, the spotlight falls on software engineering culture. The wrong set-up can break a business as competitors sprint ahead.

"If technology is the core of your business, if technology is something with which you can win or lose the market, you need to have the mastery on that technology," said Amir Arooni, EVP & CIO, Discover Financial Services, speaking Thursday at the Forbes CIO Summit. 

Building an engineering culture to advance innovation isn't a one-and-done effort, but rather a continued cycle of improvement. Knowing when to scrap technology projects, attracting top-flight tech talent and enabling development frameworks such as DevOps helps businesses connect their tech and business aspirations.

"Any of us that grew up with any kind of software engineering background, we like to think of what we do really as an elegant marriage of art and science," said Jen Hartsock, CIO at Baker Hughes, speaking on the same panel.

Over the past five years, Hartsock helped Baker Hughes bring much of its technology talent in-house, a push away from outsourcing and toward building an internal software engineering culture.

"If you're really going to be a master craftsman in software engineering, that requires an investment of time and talent, tightly coupled to that process area, or that customer outcome or solution that you're trying to deliver," said Hartsock.

Here are three strategies to shape engineering culture:

1. 'Radical decommissioning'

In addition to the "build it, run it, love it" mantra embraced by the engineering community, leaders need to understand when a piece of technology has run its course, according to Hartsock.

"When something is born, sometimes we just want to let it live forever," said Hartsock. "On life support, if necessary."

As part of its continuous improvement journey, engineering teams at Baker Hughes shifted toward product thinking and envisioning the full life-cycle of a technology product. This includes deciding when a product has reached the end of its useful life for the organization or its customers," said Hartsock.

The IT environment can be simplified through "radical decommissioning and radical modernization," said Arooni. An inability to embrace these concepts can hold companies back from achieving "speed, quality and craftsmanship" in its engineering practice.

2. Talent playbook

Employers across industries are struggling to attract tech talent, especially in highly competitive subsets of technology. Attracting that high-demand talent must begin with fostering innovation and a sense of autonomy within the company, according to Arooni.

"You're talking about building an environment and collectively creating an environment that keeps engineers happy and engaged in their work," said Arooni. "People do not want to get instructions, they want to have autonomy."

Talent attraction works to bring outside talent into the pipeline. According to Arooni, Discover build talent internally through a number of strategies, including:

• Certifications

• Badges

• Empowering engineers to contribute to open source projects, and

• The Discover Technology Academy, a hub for upskilling built by engineers for engineers.

At Baker Hughes, the strategy on talent attraction shifted to adapt to what incoming talent required.

"We historically had hired people into leadership development programs with the assumption that everyone wanted to be a generalist," said Hartsock. "What we had to do is change so that we actually allowed both generalist career paths, and then also deep technical." 

3. DevOps

Atop the CIO agenda lie big ticket priorities, such as improving productivity, quality or efficiency. 

"If you want to do that, DevOps is not something which is new, but it is the way forward, so I will encourage everyone to start as soon as possible," said Arooni.

Practices and tools can help leaders improve and develop the engineering culture. Discover implemented DevOps, DevSecOps and Site Reliability Engineering as part of its approach to engineering culture.

These methodologies employ the "automation, monitoring and continuous deployment that form the foundation of our collaborative cycle of software development and operations."