If there is one thing the software development world loves, it’s following the latest development trends. Call it the “cool factor” or simply a desire to be at the cutting edge of technology, but there is a big rush for many companies to adopt particular development and software processes. One such trend - which admittedly has been around for many years now - is that of Continuous Integration and Continuous Deployment/Continuous Development (CI/CD).
The principles behind CI/CD encourage development teams to deploy code more often in production. The core principle is that big releases are fraught with risk and so focusing on making smaller releases more often, actually constitutes less risk and makes it easier for the code in production to be maintained.
It’s a great idea that does make a lot of sense for teams to adopt. However, if companies are not prepared to make changes to how they operate, then their journey to adopting CI/CD is going to be a difficult one. While there are certainly technical changes that need to be made to the architecture and tooling to facilitate a move to CI/CD, it is as much of a cultural change as it is a technical one. And if companies are not prepared to adopt the following best practices, even with the most well-intentioned technical changes they are likely to struggle in achieving their goal of rapid and continuous delivery.
Why the move to CI/CD and DevOps?
Before we discuss these best practices though,h it’s perhaps best to look at the evolution of CI/CD and what brought the software development world to adopt this approach so widely. And a lot o this has to do with the move to Agile development that was so popular in the early 2000s. The agile mindset lowered the amount of time required to develop new application functionality, encouraging development teams to release early and often. Or at least that was its intention. Certain catastrophic outcomes prevailed as deploying to production would still take weeks or months, and teams failed to place enough emphasis on their requirements and software design nor understood the complexities of software testing and automation. Which led to excessive amounts of scope creep, rework, and the hard toil of regression testing, which only became more necessary and time-consuming.
So, the principles of CI/CD and DevOps emerged to help reduce the gap and get teams to think away from trying to work on big releases and rather adopt smaller release schedules. Coupled with the growing popularity of Cloud Computing in the early 2010s, which allowed teams to deploy directly into a production environment and a new set of development principles emerged to help facilitate teams to do this.
And even if companies weren’t interested in utilizing the cloud, tools like Docker and Kubernetes emerged which allowed a similar VM approach that cloud systems utilized and allowed more companies to explore ways of adopting CI/CD into their deployment processes.
Now that we understand why companies want to make the transition to a DevOps approach and utilize CI/CD principles, it's time to look at the different practices which will help them get there successfully.
1. Shifting Left
As mentioned above, one of the core issues with gel development was the continued need for regression testing. Something which only increases if you’re looking to deploy daily or multiple times a day. So, teams need to revisit the way they were going about their testing and focus on “shifting left”. A process that essentially means moving tests as close to the code as possible, with a greater emphasis on unit tests with a high code coverage percentage and a strong emphasis on automation at every stage in the testing cycle.
With unit tests easier to write, more reliable, and faster to execute than other traditional forms of automation, it allows errors to be caught earlier as tests would fail and prevent the pipeline from building further. This allows most bugs to be caught earlier and is, therefore, a lot cheaper to fix.
Shifting left though is more than just writing unit tests, but needs to also focus on Test-Driven Development (TDD) where these tests are written before the code is written, otherwise, unit tests can easily be retrofitted to pass the existing code. This is best done in conjunction with professional testers who can help analyze the requirements and identify all the required test scenarios to help increase the coverage, along with code coverage tools to make it easier to measure the effectiveness of the testing.
It’s worth noting though that placing more emphasis on Unit tests and other lower-level forms of automation, does not remove the need for other end-to-end test cycles. It simply reduces the number you require and with the code, in theory, being of a higher quality as a result of the testing lower-down, makes your automation of these tests (which take far longer to execute) a lot more reliable.
2. Commit Daily
CI/CD works best when every small code change is put through a form of testing cycle. This prevents untested code from becoming too big and requiring significant fixing to get through its testing phase. And reduces the risk at the point of deployment.
Committing daily to either the main branch or significant development branch also reduces the risk of complex branching and code merging strategies in the development phase.
3. Fix The Broken Builds
Having fewer branches is important in reducing the maintenance effort in the development cycle, but it’s not just about making regular deployments but also ensuring that the pipeline stays green. When there are failures in the build process, these should be fixed immediately and not left to fix later or closer to deployment.
The idea behind CI/CD is that teams should be able to deploy into production frequently and by keeping the code “fixed” at all times, teams are in a position to always deploy their code into production should it be needed. Something especially useful when an urgent coding fix or change needs to be made and allows teams to deploy their main branches knowing that it is in good working order, stable and well-tested.
4. Adapt and Reduce Technical Debt
Every organization has a process for creating and delivering code. This process may change over time due to new technologies, teams, or processes. However, if you want to make a success out of CI/CD, companies need to be prepared to constantly re-evaluate and improve these processes. The software industry is constantly moving forward and changing and teams need to ensure their pipelines grow with it.
Our CI/CD pipelines are a reflection of where teams are in terms of DevOps maturity. Typical delivery responsibilities include provisioning infrastructure, deploying applications, approving deployment changes, quality assurance/testing, and monitoring. Not all of these responsibilities and capabilities may be included in the first iteration of your CI/CD pipeline and will need to be improved as teams are ready for it. Here is no one pipeline template that suits everyone, and so teams need to be able to adapt to find the approach that works best for them and be able to constantly reevaluate how they can improve their CI/Cd process along the way.
Another key important point here is to reduce the amount of technical debt that is incurred by any team. By forgoing moving software and processes over to the latest versions or addressing knowing testing or technical deficiencies, we threaten the health of our deployment pipelines as they may easily become out of date and no longer be as reliable as hoped, or require a lot of effort to fix. With CI/CD all about being ready to deploy into production on a daily basis, having any reliability issues in your pipeline can become a big blocker to achieving this.
5. Define A Release And Rollback Strategy
No matter how thorough your testing or deployment may be, the risk of things going wrong will remain and so we need to be prepared with a process for both releasing software and rolling it back accordingly.
Anytime we release software, we introduce the risk of vulnerabilities, issues, bugs, and non-performant software. There could be any number of reasons for rolling back a deployment or producing a hotfix. It’s important that you define a release strategy that guides teams on how to release for these different reasons. The different types of testing and rules that need to be met in the pipelines, ensure that teams can fix and support production while keeping risk at a minimum.
Release strategies should also include how software needs to be rollbacked as well. Rollback strategies tend to mirror release strategies. These strategies are needed to recall earlier versions of the software when different issues occur and should make it easy for teams to test and deploy when required. Whether you decide to keep the current version of your application running while deploying a new instance is up to a team to decide, though should often be guided by some sort of advice from your release strategy. Just ensure you have a workflow that addresses any downtime and data loss.
The beauty with your release and rollback strategies though is that once they become more mature and refined, they can be automated and built into your pipeline tooling itself, minimizing the need for any form of overhead and allowing teams to make quick and clear decisions based on what the tooling is telling them.
It’s also worth integrating your metrics, monitoring, and alerting solutions into your CI/CD pipeline to operationalize your deployments. Much like how you should leverage unit tests in the earlier stages of a CI/CD pipeline, it’s important to ensure quality after a deployment. Data-driven continuous delivery will allow teams to improve their delivery capabilities and amplify feedback to encourage continuous improvement in DevOps teams.
6. Prioritize Security
With any deployment to production, there is an element of risk. And while your CI/CD pipelines should alleviate a lot of the functional risk by ensuring high test coverage, this is not the only risk that teams need to be aware of. Security is equally critical and it’s important that the CI/CD approach includes a form of security scanning and testing to ensure the software that is deployed doesn’t put an organization at risk from a security perspective.
It goes beyond just running scans on the code though. With CI/CD pipelines often having a route directly to the production environment, it’s important that credentials are carefully managed and that certain rules need to apply as to who can authorize certain deployments. It’s also important to leverage secrets management to ensure sensitive deployment or environment information is properly stored to prevent this information from becoming compromised in any way.
7. Amplify Feedback
Reliability is everyone's responsibility and so it’s important that as an organization you adopt a culture of feedback and improvement. Oftentimes, software delivery or software development issues occur as symptoms caused by anti-patterns and practices. Targeting specific outcomes following a retrospective can help improve and alleviate pain points related to incidents, bugs, and defects. Some incidents are caused by known bugs or defects, so even a setting error or incident error can be helpful for better software delivery. It’s common to freeze or lock production deployments given a history of poor application performance or frequent incidents.
To make this work effectively though you want to ensure that the proper Root Cause Analysis process is followed so that these issues are properly understood between all pirates and that the correct mitigating actions are followed by all parties. Too often companies take their issues too lightly and fail to learn from them, repeating critical mistakes that hinder their ability to deliver reliable software.
Technical difficulties in building CI/CD pipelines
These best practices defined above are crucial in helping companies make a success out of their CI/CD efforts. If companies are not prepared to change their practices to follow these principles, they shouldn’t make a move toward DevOps and Continuous Deployment.
However, there are still technical factors to consider and not every architecture solution is designed to match the quick deployment and scalability that CI/CD offers. Some common difficulties in building CI/CD pipelines include:
Creation and management of deployment scripts
Lack of self-service delivery capabilities
The upgrade, configuration, and management of plugins
Platform stability and scalability
It’s important to ensure that your software architecture can deliver on these elements, while also having engineers trained in these areas so that your company can be the best place to build the technology solutions that can help you take advantage of your desired strategy.
Choose Wisely
At the end of the day, while CI/CD is a great strategy for teams and companies to adopt, it’s not without its challenges. So if you’re a company looking to adopt C/CD into your development approach or have been struggling to make a success out of your existing CI/CD approach, hopefully, the tips in this article can help set you in the right direction.