CI/CD pipeline with Docker and Git branching strategy

Let's dive into managing a CI/CD pipeline using Docker with a focus on Git branching strategy that involves development (DEV), staging (STAGING), and production (LIVE) branches. Our goal is to clarify the role each branch plays in the process, particularly how Docker artifacts are created and managed within this framework.

Docker deployment using Git Flow with multiple branches

The core of our branching strategy involved three branches:

1. DEV: This is where all development work happens. Every new feature, bug fix, or enhancement starts in a feature branch, branched off from DEV. Once the work on a feature branch is complete, reviewed and meets the Definition of Done (DoD), it's merged back into DEV.
2. STAGING: This branch mirrors a pre-production environment. It's where integrated features from DEV get tested as a whole and accepted by stakeholders to ensure they work together seamlessly.
3. MASTER: The branch that reflects the current state of our live environment. Only thoroughly tested and stakeholder-approved changes from STAGING make it to Production.

This was our variation of a Git Flow strategy, we used until now. When you start working with Artifacts this all changes back to a normal Git Flow structure. We still have environments running (Dev, Staging and Prod). The real difference is that code commits get done once instead of getting merged into a branch running on Staging or in Production.

Instead we now have one Branch we called it Main. This is where a runnable artifact gets created after a successful code merge using Docker images. Everything deployed into our Staging or Production environment is a specific version of an earlier created Artifact.

The biggest change is that STAGING and PROD aren’t branches anymore where you merge code into. They have instead became environments that run a specific version of a Docker image. This all has its own challenges, especially when you need to Hotfix. Hotfixing has it’s own post which you can read after this post. Lets break down all of this without introducing hotfixing step by step first. It is comprehensive enough already.

What is an Artifact?

A Docker artifact, often referred to interchangeably with a Docker image, is a lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files. In simple terms, it's like a snapshot of your application and its environment, bundled together, ready to be deployed and run anywhere Docker is installed, ensuring consistency across different environments.

The Importance of DoD When Merging code

A comprehensive and clear Definition of Done (DoD) plays a pivotal role in this new process, even more when merging changes into a single branch that triggers a Docker image which should be runnable and deployable:

• Quality Assurance: The DoD ensures that all changes, including new features, enhancements, and hotfixes merged back into MAIN, meet a consistent quality standard. This includes code quality checks, successful completion of automated tests, and adherence to security best practices.
• Preventing Regression: By applying the DoD to all merges into MAIN, the team can prevent regressions and ensure that new changes do not negatively affect existing functionality. This is crucial for maintaining the integrity and stability of the codebase.
• Streamlining Deployments: A clear and comprehensive DoD facilitates smoother deployments by ensuring that only fully vetted and ready-to-deploy changes are merged into MAIN. This reduces the risk of wasting resources on building Docker images that are bugged or broken, and therefore contributes to more efficient and reliable artifacts.

Creating and Managing Artifacts

The first thing for this proces to work is when a forked branch is merged back into MAIN, this triggers an automated process in our CI/CD pipeline. Here's what happens:

1. Automated Testing: First, automated tests run to ensure the new changes meet our quality standards. This is crucial for maintaining a stable branch.
2. Docker Artifact Creation: Upon successful running al tests, a Docker image (our Artifact) is created from the MAIN branch. This is because the MAIN branch is the single branch that represents the latest state of every DONE development, This is where we generate our artifacts from that can become “shippable” products.
3. Why using MAIN for Docker Artifacts?: Creating artifacts at this stage allows us to catch any integration issue. Since every feature merged into MAIN is going to be considered ready for a release, generating an artifact here ensures that we always have a tested and potentially deployable version of our product at hand.

Transferring from Branches to Environments

In traditional Git workflows, branches like STAGING and PROD are used to represent different stages of the development process, from testing to production. However, in a Docker-centric CI/CD pipeline, the focus shifts from using these branches for code integration to using them as references for deployment environments.

Deploying to STAGING

• Creating Docker Artifacts: Once development is complete and merged into the MAIN branch, a Docker artifact is automatically created. This artifact is a snapshot of your application at that point in time, encapsulating all the changes that were merged.
• Deployment, Not Merging: The STAGING environment's role is to test this Docker artifact in conditions that closely mimic the production environment. Instead of merging code from MAIN to a STAGING branch, you deploy the Docker artifact to a STAGING environment. This means the code itself isn't moving between branches; rather, the compiled and containerized version of your application is being tested.
• Purpose of STAGING: The STAGING environment is essential for comprehensive testing, including performance and security assessments. It's a final check to ensure everything works as expected in a production-like setting without directly affecting your actual users.

Going to Production

• Stakeholder Approval: Once your Docker artifact has been rigorously tested in STAGING and approved by stakeholders, it's ready to move to production. This step is crucial for ensuring that all features, fixes, and updates meet the necessary standards and expectations.
• Deployment to Production: The transition from STAGING to PROD involves deploying the tested Docker artifact to your production environment. This deployment process ensures consistency, as the exact artifact tested in STAGING is what gets deployed to PROD, eliminating any discrepancies between tested and deployed code.

Key takeaways

• No More Branch Merging for Deployment: In this streamlined workflow, STAGING and PROD branches are no longer used for merging code. Instead, they reference specific deployment environments where Docker artifacts are deployed for testing and production.
• Focus on Artifacts: The emphasis is on creating, testing, and deploying Docker artifacts. These artifacts represent stable versions of your application ready for STAGING and, eventually, PROD deployment.
• Consistency and Reliability: By deploying the same Docker artifact from STAGING to PROD, you ensure that what has been tested is exactly what gets released, increasing the reliability and predictability of your deployments.

This approach simplifies the deployment process, reduces the risk of errors, and ensures that every release is thoroughly tested in an environment that mirrors production as closely as possible.

Are multiple Branches still needed?

Given this new workflow, the STAGING and PROD branches in Git branching serve no active purpose anymore and therefore they have became obsolete. They helped track which versions of the code have been deployed to each environment which we have solved by running a specific version of an Artifact.

Conclusion

Incorporating Docker into our Git branching strategy streamlines the development-to-deployment process by ensuring that every merge into MAIN triggers a Docker image with the potential to go to production. The key to this strategy is the creation of Docker artifacts early in the development cycle, allowing for continuous integration and delivery while maintaining high standards of quality and stability.

By understanding the changes to Git branches and the process of creating and deploying Docker artifacts, we can rely on the complexities of CI/CD with confidence, ensuring that all new features and fixes are delivered to users as efficiently and reliably as possible.

Hotfixing

We havent spoken about hotfixing yet. As i said this deserves its own Post. You can read all about Hotfixing and its caveats here.