Staff Interview: Practising Continuous Integration

When developing software, Continuous Integration is a best practice, supporting the high level of efficiency and safety needed to ensure that your business runs smoothly. In a nutshell, CI is the process of continuously integrating the code being created by developers multiple times daily. The objective is simple: to find and resolve issues quickly, whilst rapidly delivering improved software to the customer.

So how can we understand CI in practice? We spoke to Kieran, our CTO, about how it works at Cutover.

Everybody within business wants to be able to make and deploy products safely. Facebook, for example, has a great motto of “move fast and break things”, which essentially means swiftly making developments in incremental adjustments, which is great. However, this has to be done safely, because there are certain things you just can’t break, particularly in the financial and business product space in which we operate. For example, in the payments space, if a customer is accidentally charged £1million, they’re not going to be happy!

We operate in a tightly regulated industry. In any areas in which you have users with different access rights on the system, you absolutely cannot have things like permissions leaking into other roles or the permissions system breaking down as you deploy new software. This means that, prior to the point of deployment, you need rigorous testing at all levels, from unit all the way through to integration; rigorous checkpoints and rigorous sign-off in order to eliminate that risk.

So how do you achieve that quickly?

In the CI world, by using modern tooling we can write enough automated testing to help achieve this, which is great from a software perspective. We can do quick and incremental deploys to representative hardware and datasets, as opposed to letting everything we’d like to change build up for six to 18 months on local developer environments and then doing very big deploys that, by their nature, have got more change in new features, security patches and bug fixes - and therefore carry more risk. However, you still need to organise all the people and teams that need to be involved in the non-technical aspects of actually coordinating a software release, which is what the Cutover tool helps to do. The organisation of these teams can often be overlooked, with significant reliance being placed upon just a few key individuals who have a fuller knowledge of both the system and relevant communication lines.

CI and Continuous Deployment, or CD, are perhaps easier to implement in a startup because we don’t have the legacy to deal with. If you are a company that has been trading for decades, you may have IT systems that are harder to maintain because they’re using languages and frameworks that are out of vogue, so it’s harder to find people with the expertise needed. It may be the case that you actually have to contact a software vendor to get a software patch applied. By contrast, a startup is likely to be able to have more control over the entirety of their stack.

If we were not practising CI/CD, we’d end up with bottlenecks at some point in the development process. In that situation, you tend to have just one person signing off all the change. That means you end up with a classic linchpin, rather than automated testing which gives us broader coverage and confidence for anyone to be able to run a deploy to a customer instance.

What practical advice would you give for employing continuous integration as an approach?

Make sure that you look at what you can and can’t break. There will be certain pieces that you can change quickly and it’s reasonably okay if you break them. So maybe it’s the application of the CSS sheet to your website - you’d rather it didn’t break, but it’s not going to cause a serious problem or affect anybody if it does. Whereas if you’re working on something like sending out communications to senior stakeholders or your entire customer base, you absolutely cannot get it wrong. It has to work 100% of the time. You have to put these pillars in your product and be constantly aware that those are the things that cannot break without causing a major problem. At the same time, it’s always worth regularly revisiting these pillars as more information becomes available in your growth journey.

So whilst the smaller things can be continuously changed, these pillars are the things you must take longer on?

Yes. Additionally, for the smaller changes that could have unintended consequences, we use something called feature flags in the software, so that we can have software-defined switches that allow us to turn things on and off. This eliminates the problem of not being able to back out a feature that could, for some unforeseen reason, cause a problem in the UAT or Production environments, without having to do another deploy. If something goes wrong, we can just alter the config file for the software and turn off that problematic feature.

Would you say that CI is important in maintaining customer satisfaction?

Absolutely. Software is everywhere now; in my opinion, everything we touch is either being replaced by software or being enhanced with software, so you need to be able to experiment with software quickly to consistently improve things - but at the same time have the safety and assurance that critical items aren’t going to break. Unfortunately, it is very easy to break things in software if you let everybody deploy whenever they feel like it! That means you need to ensure that there is a process in place and great communication within the Development team. That way, everybody knows if and when a new software version or feature is going out, as well as promoting group focus on establishing whether the change will affect one of those key pillars of your product or service, prior to deploying.

What does that process look like at Cutover?

Our team has a ticketing system, through which the entire deployment team gets involved in looking at changes and deciding whether or not they are going to affect any of our key pillars. We are much more hesitant to change anything around scheduling or permissions than we are for perhaps a UI feature. Everybody will be involved in making any changes around permissions, login or any security aspects of our system, whereas for user experience or front-end changes, perhaps only one or two people will be needed to sign that off. We’re much slower on making changes to the things that are core to our product. The fastest we can get a change made for any of these pillar features is a week but we often take a little longer. However, the smaller items we can discuss and deploy in the same day.

At the moment, it’s very helpful that we have the majority of the Development team co-located together, so we can have those discussions very easily across the table. If we were all working remotely or spread across different areas, getting that essential face time could possibly be a little more difficult, so that’s perhaps one of the challenges we’re going to face going forward as we grow.