How We Handle Incorporating New Things into Existing Operations

For enterprises, doing new things is almost a daily requirement, whether driven by competition, regulation or curiosity. Here we look at several approaches to tackling release operations.

To choose the right new things to do, and to transition this updated capability into the operational machine of the organisation, people are forced to tackle complexity and unknowns.

There are several interesting approaches we have seen organisations take, all of which have interesting learning points.

1. Make the change/transition super safe

Set up the change to avoid it being all that different. We have seen organisations force changes down a process that rejects anything out of the ordinary and packages it so that it is 95% similar to things that have worked to date. This seems to work well for organisations that are amending via limited code updates that can be simulated in environments that are understood to be almost mimics of live release operations. It has downsides in not being able to handle the widespread changes that are sometimes required. It is also difficult to achieve in circumstances where the reality of live release operations is hard to mimic in tests, simulations and scenarios.

2. Make changes fast but put them into operation selectively

Here, the organisation is not limited to excruciatingly long judgement and testing on whether the change to a new way of doing things will destroy release operations. Rather, the route is to expose a limited number of users in live operations to the new way of doing things.  As part of our research with companies like Facebook, we have seen good practices here. Changes are implemented quickly and exposed to select groups such as internal employees first before then going out to wider customers. In each case, metrics and methods are required to turn up deployment or immediately roll back if a large snag is hit.

3. Planned deployment but extensive control

Here, organisations perform the required testing and quality review to ensure they are releasing changes that should work in live. The addition, however, is to put in place effective critical event control for the transition. This ensures that deployment takes place in an orderly fashion and the inevitable in-flight engineering during the transition period is effectively controlled to coordinate diverse global teams to land the changes successfully.

There is no single silver bullet approach but rather there are interesting aspects of each approach that organisations can learn from to improve their ability to usher in new ways of operating without causing significant issues with the existing operational estate.