This is a guest post from Arthur D. Little. It was written by Mitch Beaumont, Ben Thuriaux, Prashanth Prasad, Chandler Hatton and Colin Davies.

Breakthrough innovation – innovation aimed at delivering disruptive impact, or creating new market spaces or step-changes in product, process or business-model performance – is increasingly important for companies. However, outside of the software industry most organizations, especially those with complex engineered products and longer development lifecycles, struggle to deliver it systematically. This is principally because the agile approach needed to realize breakthroughs is a challenge to the established practices that have served them well. In this article we look at how non-software product-based companies can successfully embrace agile, as well as non-agile, methods in a complementary way.

The challenges of breakthrough innovation

Business executives have always been under pressure to generate growth, and today’s fast-moving and competitive business environment does not make that any easier. Arthur D. Little’s eighth Innovation Excellence Survey revealed that leading companies expect their share of revenue from breakthrough, as opposed to incremental, innovation to double over the next five years. However, achieving breakthroughs is easier said than done: we also found that 88% of business leaders were dissatisfied with their breakthrough innovation performances. They have become increasingly frustrated with the limitations of their current innovation systems on producing significant results.

The underlying issue for these organizations is usually that they are applying a non-optimal innovation approach to realize breakthrough innovation. For the past three decades, most technology-based companies have employed a waterfall (or phase-gate) approach to all of their innovation efforts. In fact, they have made significant investment in the design and adoption of these approaches so they would become rigorous and mechanical. Their fundamental goal has been to minimize variances (i.e., risk) from a well-understood set of requirements and a detailed plan that are both established at the beginning of a development project. As a result, they have created the perfect environment for incremental innovation, reducing cycle times and improving on-time delivery. Unfortunately, this well-honed model is not conducive to breakthrough innovation, in which requirements are rarely set in stone and uncertainty is not only the norm but a vehicle to explore beyond the usual boundaries. And while some companies realized some time ago that they needed to create separate initiatives for breakthrough innovation with some independence from the narrower focus and bureaucracy of their core R&D, all too often the innovation process and the way governance and teams worked were left fundamentally unchanged.

In the meantime, for the past two decades the information technology and software world has been applying its own, highly dynamic innovation model – the agile approach. For some time agile has been applied almost exclusively to software development, and this has borne fruit: the software industry has consistently produced patents at three times the level of the next-most prolific sectors. Today, agile approaches are increasingly being deployed alongside phase-gate processes in engineering and R&D functions outside software, with a very positive result. Arthur D. Little’s research reveals that companies that have successfully added agile methods to their toolboxes, and tailor their innovation approaches by the type of innovation, perform significantly better than those that stick to single, waterfall approaches. (See Figure 1.)

Figure 1:  Different pathways impact innovation success

How can agile methods be applied to product development?

When applying an agile approach to product development the key agile principles remain the same. However, certain elements take on a different twist, as shown in Figure 2.

Figure 2: Summary comparison of key agile elements in software and product development applications

Iterative approach. The heart of the agile approach in product development is the use of a series of rapid, iterative loops, similar to an agile iteration for software. At the early “exploration” stages of the development lifecycle, each loop focuses on answering a key question that is determined to have a high degree of importance and uncertainty, in order to build a progressively clearer picture of the desired solution. Examples of loop objectives include evaluation of technical feasibility, assessment of user experience, and testing of a business model. Through these loops, the team is effectively building the user stories. A key artifact of each typically two- to four-week loop is a prototype used to test the part of the concept in question. Prototypes need to be fast and inexpensive – simple mockups, models, videos and simulations are appropriate. This may entail using tools and methodologies new to the organization, or even collaborating with external design and prototyping firms. Prototypes are shared with a sampling of customers, the key questions tested and the learnings assessed to determine if the team can move on to a new objective for the next loop.

At the later stages of the development cycle, the loops shift their focus to “realization” and develop parts of the solution similar to agile in software. Loops mimic iterations and implement prioritized functionality. However, there are important constraints and considerations which lead some companies to limit their adoption of agile methods to the early-stage activities, at least initially. We discuss this situation later.

Teams. Clear roles and responsibilities and the right balance of authority and accountability are important for team success in an agile product development environment. Teams must be nimble and the individual members comfortable with ambiguity and experimentation. In the product development environment, agile teams are multidisciplinary teams of specialists that expand and contract depending on their current focus. For example, if a team is testing a business-model concept, there might need to be greater involvement of people with experience and knowledge in finance, sales, and behavioral economics. This would be different from the skills needed to do a technology-feasibility loop. To support this model, agile product development teams are often put together with part-time or limited-time (i.e., “tour of duty”) resources. A very small “core” stays constant, and there is a designated team leader throughout the development cycle; however, that role is more analogous to a sports team captain than a hierarchical manager. The most effective teams are those that realize they do not have all the answers and embrace the involvement of external partners or experts in loop activities.

Governance. While governance is not often identified as a key element of agile software development, it is critical within product development. In the agile environment, governance acts less like a go/no-go decision-maker and more like a coach to project teams. Governance also serves to mitigate “organizational antibodies” that try to impede or marginalize breakthrough innovations. Loop reviews done at the end of each loop to assess whether the key question has been addressed are a discussions between project teams and their governance, using poster boards, prototypes and other visual aids to facilitate the conversation. To enable this environment, it is important that an agile governance group is comprised of individuals who can foster a culture of experimentation and learning, a sense of urgency and agility, and a passion for helping teams jump over hurdles (versus governance being the hurdle itself). Sometimes this means the most appropriate governance team members might not be the usual functional leaders, but instead, people with the right mindset and knowledge, regardless of titles, and from different places across the organization. In addition, because governance is relied on for coaching more than for decision-making, it creates an opportunity to include outsiders that have expertise in new and less familiar areas.

Embracing agile as part of a holistic innovation approach

The phase-gate (waterfall) and agile approaches are distinct in their implementation, and generally suited to different innovation objectives when applied in the context of companies with engineered products. We see companies adopting two general approaches when trying to introduce agile into an existing phase-gate process:

• Integrating agile into a single innovation process
• Adding a partly parallel agile path

Integrating agile into a single innovation process typically involves using iterative loops within the existing phase-gate process, but with the overall structure retained as-is. Our experience is that attempting to integrate approaches will sub-optimize at least one of them. Although there could be some minor benefits in terms of speed during the development phase, at best the impact is limited, and at worst it results in frustrated teams and missed market opportunities. With this approach companies may be, in effect, deceiving themselves that they have embraced agile principles, when nothing fundamental has changed.

A better solution is to run phase gate and agile side by side, so an organization can apply the right  approach across an innovation portfolio of both incremental and breakthrough innovation, as shown in Figure 3.

Figure 3:  Different approaches for different types of innovation – adding the agile path

In this model the agile path is the right size to handle the anticipated flow of breakthrough innovation as per a company’s particular innovation strategy, which is usually substantially less volume than the phase-gate path. The dedicated resources are significantly less also, given the dynamic team model highlighted above. The existing phase-gate pathway, in which significant investment has been made to establish and optimize processes, team roles and responsibilities, and governance and tools, is still needed to bring incremental innovation to market. In most companies, these are the key projects that “keep the lights on” and make up the majority of the innovation portfolio.

When setting up an agile path from scratch, one of the key considerations is how the approach links in to existing processes at the downstream end. Typically, agile approaches start to be applied at the front end of the development process. Here there are clear benefits to be enjoyed from enabling concepts with substantial levels of uncertainty to be better understood and explored, before entering the downstream development activities of detailed design, test and launch, in which companies can continue to leverage their significant talents for realization. This is depicted by arrow A in Figure 3. In this situation, the hand-off point between the agile process and the existing phase-gate process needs to be carefully defined and managed, especially if aspects of the detailed design have already been covered in the development of earlier-stage prototypes. Ultimately, many companies may wish to evolve more complete agile development pathways which also include these downstream activities in parallel streams. This way is better able to reap the full benefits of the agile approach and avoids the pitfall of falling back to the phase-gate culture and way of working. For industries in which product development is highly regulated, such as healthcare, approach A may continue to be needed. (See case examples.)

Case Study 1.  Applying agile to develop a breakthrough concept

A company in the healthcare industry was developing a highly innovative decision-support tool for physicians. The envisioned product was based on a new, potentially game-changing technology that was unproven at the desired application. There were many unknowns regarding technical feasibility, physician preferences and user experience.  A series of iterative loops were defined to address several key questions, which were prioritized by their importance and degree of uncertainty. For example, one loop tested how to optimally incorporate the product into the physician workflow. During that loop, product mockups were created and introduced to physicians in a simulated work environment to obtain their feedback. Over the course of several loops, the solution was refined to yield the desired functionality. Then the project was put into the phase-gate process normally used by the company. The result was the successful and timely development and launch of a first-of-a-kind breakthrough product. In addition, by employing the iterative loops early to test technical feasibility and physician acceptance, there was the added benefit of accelerating some of the development and testing activities that would normally have been left to the later phases in a pure phase-gate process.

Case Study 2.  Introducing agile using a pilot approach

A technology product company recently implemented an agile approach for its new-product development process, with an objective to involve customers earlier in the process to improve market success. Company leadership recognized that this would be a departure from their accustomed practices, and decided to take a measured approach to setting up an agile product development capability. They initiated a pilot project in which a three-week “sprint” approach was used to gather customer feedback and refine prototypes. The organization had to adjust to a new way of working, as the iterative approach and lack of phase-gate checklists and known standard deliverables created a degree of discomfort for the management team. While the time spent in the early phase of the project was slightly extended due to the use of multiple sprints, the project moved through the later phases to product launch much faster than was typical, as the key risks and uncertainties had been resolved earlier. By the end of the pilot project, the organization had a high degree of confidence in applying the agile approach more broadly to pursue their breakthrough innovation goals.

Implementing these agile development practices in product-based companies can come with a number of challenges. Figure 4 highlights some of the more commonly observed hurdles that companies can expect to encounter, and how they can be overcome.

Figure 4: Key hurdles and solutions for proper application of agile approaches in product development

Insight for the executive

It is increasingly important for companies to deliver breakthrough innovation to their markets. However, the approaches that have been successful at improving innovation delivery over the last 30 years are holding most organizations back. Senior executives in product- and process-technology companies are already championing agile approaches to improve breakthrough innovation performance. Creating a separate path with agile principles tuned for a product development environment has been shown to be an effective approach, with key success factors as follows:

• Use portfolio management to make a deliberate choice on what innovation management process to use for different projects.
• Initially introduce agile to help nurture breakthrough innovation concepts during the exploratory part of product development (the front end).
• Give executive leadership and sponsorship to the agile teams.
• Bring in experts with experience of agile product development methods to guide the teams and governance introduction of an agile process; building an agile culture takes repetition and time.
• Ensure governance breaks down organizational barriers and takes on more of a coaching than a decision-making role; be open to including experts from outside the organization; ensure that the organization is receptive and there are viable commercialization routes.

This is a guest post from Arthur D. Little. It was written by Mitch Beaumont, Ben Thuriaux, Prashanth Prasad, Chandler Hatton and Colin Davies and was originally published in Arthur D. Little’s corporate magazine: Prism – Edition: 1 / 2017.

About the authors

Mitch Beaumont is a Partner in the San Francisco office of Arthur D. Little and a member of the Technology & Innovation Management Practice.

Ben Thuriaux-Alemán is a Principal in the London office of Arthur D. Little and a member of the Technology & Innovation Management Practice and the Energy Practice.

Prashanth Prasad is a Manager in the San Francisco office of Arthur D. Little and a member of the Technology & Innovation Management Practice and the Healthcare Practice.

Chandler Hatton is a Manager in the Amsterdam office of Arthur D. Little and a member of the Technology & Innovation Management Practice and the Strategy & Organization Practice.

Colin Davies is a former Manager in the London office of Arthur D. Little and now a Commercialization Manager at Queen Mary Innovation Ltd.

Photo Credit: Wonder woman0731 Flickr via Compfight cc