CI/CD Practices

Continuous integration and continuous delivery (CI/CD) practices define how code changes flow from developer commit through automated verification to production deployment. This reference specifies pipeline stage requirements, environment promotion rules, deployment strategy configurations, and tooling standards for mission-driven organisations operating across diverse infrastructure contexts.

Continuous Integration (CI): The practice of merging code changes to a shared repository frequently, with each merge triggering automated build and test processes that verify the change does not break existing functionality.
Continuous Delivery (CD): The practice of maintaining code in a deployable state at all times, with automated processes capable of releasing any successful build to production through a single manual approval.
Continuous Deployment: An extension of continuous delivery where every change passing automated verification deploys to production without manual intervention.
Pipeline: An automated sequence of stages that transforms source code into deployed software, with each stage performing specific verification or transformation tasks.
Artifact: A versioned, immutable output produced by a build process, such as a container image, compiled binary, or deployment package.

Pipeline architecture

A CI/CD pipeline consists of sequential stages, each with defined inputs, outputs, and success criteria. Every stage must complete successfully before the next stage begins. Failed stages halt the pipeline and notify responsible parties.

+------------------------------------------------------------------+
|                      CI/CD PIPELINE FLOW                         |
+------------------------------------------------------------------+
|                                                                  |
|  +----------+    +----------+    +----------+    +----------+    |
|  |          |    |          |    |          |    |          |    |
|  |  SOURCE  +--->+  BUILD   +--->+  TEST    +--->+ SECURITY |    |
|  |          |    |          |    |          |    |          |    |
|  +----------+    +----------+    +----------+    +----------+    |
|       |              |              |                |           |
|       v              v              v                v           |
|   Checkout       Compile        Unit tests      SAST scan        |
|   Validate       Package        Integration     Dependency       |
|   Lint           Version        Contract        Secret detect    |
|                  Tag                                             |
|                                                                  |
|  +----------+    +----------+    +----------+    +----------+    |
|  |          |    |          |    |          |    |          |    |
|  | ARTIFACT +--->+  DEPLOY  +--->+  VERIFY  +--->+ PROMOTE  |    |
|  |          |    |   DEV    |    |          |    |          |    |
|  +----------+    +----------+    +----------+    +----------+    |
|       |              |              |                |           |
|       v              v              v                v           |
|   Store image    Deploy to      Smoke tests     Gate check       |
|   Sign           development    Health check    Approval         |
|   Scan           Configure      E2E subset      Next env         |
|                                                                  |
+------------------------------------------------------------------+

Figure 1: Pipeline stage sequence from source commit through environment promotion

Stage requirements

Each pipeline stage has mandatory elements that must be present for the pipeline to meet organisational standards.

Source stage

The source stage retrieves code and validates basic integrity before resource-intensive operations begin.

Requirement	Specification	Failure action
Repository checkout	Full clone for release branches, shallow clone (depth 50) for feature branches	Abort pipeline
Commit signature	GPG signature verification for protected branches	Abort pipeline
Branch policy	Merge commits only from approved base branches	Abort pipeline
Lint execution	Language-appropriate linter with zero tolerance for errors	Abort pipeline
Format verification	Code formatting matches project standard	Abort pipeline
Commit message	Follows conventional commit format	Warning only

Source stage timeout: 5 minutes maximum. Stages exceeding this threshold indicate network or repository issues requiring investigation.

Build stage

The build stage compiles source code, resolves dependencies, and produces versioned artifacts.

Requirement	Specification	Failure action
Dependency resolution	Lock file verification, no floating versions in production builds	Abort pipeline
Compilation	Zero warnings policy for production builds	Abort pipeline
Version tagging	Semantic version derived from git tags or commit metadata	Abort pipeline
Build reproducibility	Same inputs produce byte-identical outputs	Warning, investigate
Cache utilisation	Dependency and build caches reduce rebuild time by minimum 60%	Warning only
Build metadata	Embed commit SHA, build timestamp, builder version in artifact	Abort pipeline

Build stage timeout varies by project complexity:

Project type	Maximum build time	Cache expectation
Frontend application	10 minutes	70% cache hit rate
Backend service	15 minutes	65% cache hit rate
Mobile application	25 minutes	50% cache hit rate
Monorepo (affected only)	20 minutes	80% cache hit rate

Test stage

The test stage executes automated verification at multiple levels. Test stage composition reflects the project’s Testing Strategy but must include minimum coverage.

Test type	Minimum requirement	Maximum duration	Failure threshold
Unit tests	80% line coverage for new code	5 minutes	0 failures
Integration tests	All external service boundaries	10 minutes	0 failures
Contract tests	All API consumer contracts	5 minutes	0 failures
Smoke tests	Critical path verification	3 minutes	0 failures

Test parallelisation is mandatory for test suites exceeding 3 minutes sequential execution time. Configure test runners to distribute across available workers with maximum 4x parallelisation factor to prevent resource exhaustion.

Security stage

The security stage performs automated security verification. Implementation details appear in Security and Compliance Testing; this section specifies pipeline integration requirements.

Check type	Tool category	Gate policy	Maximum duration
Static analysis (SAST)	Semgrep, SonarQube, CodeQL	Block on high/critical	10 minutes
Dependency scan	Trivy, Snyk, OWASP Dependency-Check	Block on critical CVE	5 minutes
Secret detection	Gitleaks, TruffleHog	Block on any detection	2 minutes
Container scan	Trivy, Grype	Block on critical CVE	5 minutes
IaC scan	Checkov, tfsec	Block on high severity	3 minutes

Security findings produce machine-readable output in SARIF format for aggregation in security dashboards. Pipeline logs must not contain finding details that could aid attackers; findings route to secure reporting channels.

Artifact stage

The artifact stage stores, signs, and catalogues build outputs for deployment.

Requirement	Specification
Storage location	Organisation-controlled registry (not public registries for production artifacts)
Naming convention	`{project}/{component}:{semver}-{short-sha}`
Signature	Cosign or Notary signature with organisation signing key
Retention	Development: 7 days, Staging: 30 days, Production: 1 year minimum
Metadata	Build provenance attestation in SLSA format
Vulnerability scan	Scan stored artifact, not just build output

Artifact immutability is mandatory. Once published to a registry with a specific tag, that tag must never be overwritten. The latest tag is prohibited for production deployments.

Environment model

Environments provide isolated deployment targets with distinct purposes and access controls. Code promotes through environments sequentially; no environment may be bypassed except under documented emergency procedures.

+--------------------------------------------------------------------+
|                     ENVIRONMENT PROMOTION MODEL                    |
+--------------------------------------------------------------------+
|                                                                    |
|  +----------------+     +----------------+     +----------------+  |
|  |                |     |                |     |                |  |
|  |  DEVELOPMENT   +---->+    STAGING     +---->+  PRODUCTION    |  |
|  |                |     |                |     |                |  |
|  +-------+--------+     +-------+--------+     +-------+--------+  |
|          |                      |                      |           |
|          v                      v                      v           |
|  +----------------+     +----------------+     +----------------+  |
|  | Auto-deploy on |     | Deploy on      |     | Deploy on      |  |
|  | merge to main  |     | staging        |     | production     |  |
|  |                |     | approval       |     | approval       |  |
|  | No approval    |     |                |     |                |  |
|  | required       |     | QA team        |     | Release        |  |
|  |                |     | approval       |     | manager + PO   |  |
|  +----------------+     +----------------+     +----------------+  |
|                                                                    |
|  Duration in env:       Duration in env:       Deployment window:  |
|  Until next merge       Minimum 24 hours       Tue-Thu 09:00-16:00 |
|                         for non-critical       (local time)        |
|                                                                    |
+--------------------------------------------------------------------+

Figure 2: Environment promotion with approval gates and timing constraints

Environment specifications

Characteristic	Development	Staging	Production
Purpose	Integration testing, feature validation	Pre-production verification, UAT	Live service delivery
Data	Synthetic only, reset weekly	Anonymised production subset	Real user data
Scale	10% of production capacity	50% of production capacity	Full capacity with headroom
Availability target	Best effort, business hours	99% during business hours	Per SLA (99.5% minimum)
Access	Development team	Development + QA + selected stakeholders	Operations team only
Deployment frequency	On every merge (10-50/day typical)	1-5 deployments per day	1-10 deployments per week
Rollback time	Immediate, automated	Under 15 minutes	Under 5 minutes
Monitoring	Basic health checks	Full monitoring, alert to dev team	Full monitoring, 24/7 alerting
Secrets	Development secrets, rotated monthly	Staging secrets, rotated monthly	Production secrets, rotated per policy

Environment parity

Production parity reduces deployment failures by ensuring staging behaviour predicts production behaviour. Maintain parity across these dimensions:

Dimension	Parity requirement	Acceptable deviation
Operating system	Identical version	Patch level within 30 days
Runtime version	Identical	None
Database version	Identical major.minor	Patch level within 30 days
Network topology	Matching architecture	Scale differences acceptable
Configuration schema	Identical keys	Environment-specific values expected
Third-party services	Same providers	Sandbox vs production tiers acceptable
Geographic region	Same cloud region preferred	Different region for DR testing acceptable

Deployment strategies

Deployment strategies control how new versions replace running versions. Strategy selection depends on rollback requirements, traffic patterns, and acceptable risk.

+-------------------------------------------------------------+
|                    DEPLOYMENT STRATEGIES                    |
+-------------------------------------------------------------+
|                                                             |
|  1. ROLLING UPDATE (Incremental Replacement)                |
|     (Gradually replace instances one by one)                |
|                                                             |
|   t=0          t=1          t=2          t=3                |
|   +----+       +----+       +----+       +----+             |
|   | v1 |       | v2 |       | v2 |       | v2 |             |
|   | v1 |  -->  | v1 |  -->  | v2 |  -->  | v2 |             |
|   | v1 |       | v1 |       | v1 |       | v2 |             |
|   +----+       +----+       +----+       +----+             |
|   100% v1      33% v2       66% v2       100% v2            |
|                                                             |
+-------------------------------------------------------------+
|                                                             |
|  2. BLUE / GREEN (Instant Switch)                           |
|     (Two identical environments; switch the router)         |
|                                                             |
|   t=0 (Blue Active)           t=1 (Switch to Green)         |
|   +------------+              +------------+                |
|   |  ROUTER    |--+           |  ROUTER    |--+             |
|   +------------+  |           +------------+  |             |
|     |             |             |             |             |
|   +----+       +----+         +----+       +----+           |
|   | v1 |       | v2 |         | v1 |       | v2 |           |
|   |BLUE|       |GRN |         |BLUE|       |GRN |           |
|   +----+       +----+         +----+       +----+           |
|   (Live)       (Idle)         (Idle)       (Live)           |
|                                                             |
+-------------------------------------------------------------+
|                                                             |
|  3. CANARY (Traffic Splitting)                              |
|     (Route small traffic % to new version, then roll out)   |
|                                                             |
|   t=0          t=1 (Test)   t=2 (Expand) t=3                |
|   +----+       +----+       +----+       +----+             |
|   | v1 |       | v2 |       | v2 |       | v2 |             |
|   | v1 |  -->  | v1 |  -->  | v2 |  -->  | v2 |             |
|   | v1 |       | v1 |       | v1 |       | v2 |             |
|   | v1 |       | v1 |       | v1 |       | v2 |             |
|   +----+       +----+       +----+       +----+             |
|   100% v1      25% v2       50% v2       100% v2            |
|                                                             |
+-------------------------------------------------------------+

Figure 3: Traffic distribution patterns for rolling, blue-green, and canary deployments

Strategy specifications

Strategy	Rollback time	Resource overhead	Deployment duration	Best for
Rolling update	2-10 minutes	Minimal (25% surge)	5-15 minutes	Stateless services, frequent deploys
Blue-green	Under 30 seconds	100% (full duplicate)	2-5 minutes	Critical services, instant rollback required
Canary	1-2 minutes	10-50% additional	30 minutes to 4 hours	High-risk changes, gradual validation
Recreate	N/A (requires redeploy)	None	1-5 minutes	Development only, batch jobs

Rolling update configuration

Rolling updates replace instances incrementally while maintaining service availability.

# Kubernetes rolling update specification
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Maximum 5 pods during update (4 + 1)
      maxUnavailable: 0  # Never reduce below 4 available pods
  minReadySeconds: 30    # Pod must be ready 30s before continuing

Parameter	Development	Staging	Production
maxSurge	50%	25%	25%
maxUnavailable	50%	0	0
minReadySeconds	10	30	60
progressDeadlineSeconds	300	600	900

Blue-green configuration

Blue-green deployments maintain two identical environments, with traffic switching between them atomically.

+------------------------------------------------------------------+
|                    BLUE-GREEN ARCHITECTURE                       |
+------------------------------------------------------------------+
|                                                                  |
|                      +----------------+                          |
|                      |   LOAD         |                          |
|                      |   BALANCER     |                          |
|                      +-------+--------+                          |
|                              |                                   |
|              +---------------+---------------+                   |
|              |                               |                   |
|              v                               v                   |
|     +--------+--------+           +----------+------+            |
|     |                 |           |                 |            |
|     |  BLUE (active)  |           |  GREEN (idle)   |            |
|     |  Version 1.4.2  |           |  Version 1.5.0  |            |
|     |                 |           |                 |            |
|     |  +---+ +---+    |           |  +---+ +---+    |            |
|     |  |   | |   |    |           |  |   | |   |    |            |
|     |  +---+ +---+    |           |  +---+ +---+    |            |
|     |  +---+ +---+    |           |  +---+ +---+    |            |
|     |  |   | |   |    |           |  |   | |   |    |            |
|     |  +---+ +---+    |           |  +---+ +---+    |            |
|     |                 |           |                 |            |
|     +-----------------+           +-----------------+            |
|                                                                  |
|     After verification, swap load balancer target to GREEN       |
+------------------------------------------------------------------+

Figure 4: Blue-green environment architecture with load balancer switch

Blue-green swap checklist (automated):

Deploy new version to inactive environment
Run smoke tests against inactive environment (internal endpoint)
Run synthetic transactions against inactive environment
Verify metrics collection functioning
Switch load balancer target group
Monitor error rates for 5 minutes
If error rate exceeds 0.1%, automatic rollback to previous environment

Canary configuration

Canary deployments route a percentage of traffic to new versions, increasing gradually based on success metrics.

Phase	Traffic percentage	Duration	Promotion criteria
Initial	5%	15 minutes	Error rate under 0.1%, latency p99 within 10% of baseline
Expand	25%	30 minutes	Error rate under 0.1%, no alerts triggered
Majority	50%	60 minutes	Error rate under 0.05%, user feedback channels clear
Full	100%	Permanent	Canary version becomes baseline

Automatic rollback triggers when any condition fails. Manual promotion gates may be inserted at any phase for high-risk changes.

Pipeline security

Pipeline infrastructure requires security controls equivalent to production systems, as pipelines have access to secrets, signing keys, and deployment credentials.

Secret management

Secret type	Storage location	Access pattern	Rotation frequency
API keys	External secrets manager (Vault, AWS Secrets Manager)	Injected at runtime, never in logs	90 days
Signing keys	Hardware Security Module or managed KMS	Reference only, never exported	Annual
Database credentials	External secrets manager	Short-lived tokens preferred	30 days
Cloud credentials	Workload identity (no static keys)	Federated authentication	N/A (no static credential)
Registry credentials	External secrets manager	Scoped to specific repositories	90 days

Prohibited practices:

Secrets in repository (including encrypted secrets in git)
Secrets in pipeline configuration files
Secrets passed as command-line arguments (visible in process lists)
Secrets in build logs (mask all secret values)
Long-lived static credentials when workload identity is available

Runner security

CI/CD runners execute arbitrary code from repositories. Runner configuration must limit blast radius.

Runner type	Use case	Security model
Ephemeral container	Standard builds	New container per job, destroyed after
Ephemeral VM	Builds requiring elevated privileges	New VM per job, destroyed after
Persistent runner	Specialised hardware (mobile builds)	Isolated network, restricted repository access
Self-hosted runner	Air-gapped or on-premises requirements	Hardened OS, no internet egress

Runner hardening requirements:

No persistent storage between jobs
No network access to production systems
Limited egress to approved registries and package repositories
Resource quotas preventing denial of service
Execution timeout of 60 minutes maximum

Feature flags and progressive delivery

Feature flags decouple deployment from release, enabling code to reach production before functionality activates for users. Progressive delivery extends this pattern to gradually expose features based on defined criteria.

+-------------------------------------------------------------+
|                   FEATURE FLAG LIFECYCLE                    |
+-------------------------------------------------------------+
|                                                             |
|   DECOUPLING DEPLOYMENT FROM RELEASE                        |
|   (Code is shipped to production, but hidden behind config) |
|                                                             |
|   Step A: Deploy Code         Step B: Toggle Flag           |
|   (Code dormant)              (Feature Live)                |
|                                                             |
|   +--------------+            +--------------+              |
|   |   APP CODE   |            |   APP CODE   |              |
|   | [New Feat]   |            | [New Feat]   |              |
|   +--------------+            +--------------+              |
|          |                           |                      |
|          v                           v                      |
|   +--------------+            +--------------+              |
|   |  FLAG: OFF   |            |  FLAG: ON    |              |
|   +--------------+            +--------------+              |
|          |                           |                      |
|   (User sees Old)             (User sees New)               |
|                                                             |
+-------------------------------------------------------------+
|                                                             |
|  PROGRESSIVE ROLLOUT STRATEGY                               |
|  (Gradually increasing the user impact radius)              |
|                                                             |
|   Day 1 (0%)     Day 2 (10%)    Day 3 (25%)    Day 7 (100%) |
|   +----------+   +----------+   +----------+   +----------+ |
|   |          |   |#         |   |###       |   |##########| |
|   +----------+   +----------+   +----------+   +----------+ |
|   Internal       Staff /        Beta Users     General      |
|   Devs Only      Canary                        Availability |
|                                                             |
+-------------------------------------------------------------+

Figure 5: Feature flag lifecycle from deployment through progressive rollout

Flag specifications

Flag type	Lifespan	Owner	Cleanup deadline
Release flag	Temporary (under 30 days)	Product owner	14 days after 100% rollout
Experiment flag	Temporary (under 90 days)	Product owner	7 days after experiment conclusion
Ops flag	Permanent	Engineering	Annual review
Permission flag	Permanent	Product owner	Annual review
Kill switch	Permanent	Engineering	Never remove

Stale flag policy: flags exceeding their expected lifespan by 30 days trigger automated alerts. Flags exceeding lifespan by 90 days require documented exception or removal.

Flag evaluation

Flag evaluation must not introduce latency or single points of failure.

Evaluation pattern	Latency impact	Availability	Use case
Local evaluation (cached rules)	Under 1ms	Survives flag service outage	Production services
Remote evaluation (API call)	10-50ms	Depends on flag service	Administrative tools
Edge evaluation (CDN)	Under 5ms	High	Frontend applications

Default behaviour when flag service unreachable: flags return their designated default value, which must be the safe/conservative option (typically disabled for new features, enabled for kill switches).

Tooling reference

Pipeline platforms

Platform	Licence	Hosting model	Strengths	Considerations
GitLab CI	MIT (core)	Self-hosted or SaaS	Integrated with source control, comprehensive features	Resource intensive for self-hosting
Gitea Actions	MIT	Self-hosted	Lightweight, GitHub Actions compatible syntax	Smaller ecosystem than GitHub
Woodpecker CI	Apache 2.0	Self-hosted	Simple, container-native, low resource requirements	Limited built-in integrations
Jenkins	MIT	Self-hosted	Extensive plugin ecosystem, mature	High maintenance burden, security track record
GitHub Actions	Proprietary	SaaS	Large marketplace, good documentation	US-headquartered (CLOUD Act), vendor lock-in
Azure DevOps	Proprietary	SaaS or Server	Microsoft ecosystem integration	Licensing complexity, Microsoft account dependency

For organisations prioritising data sovereignty or operating in regions with US data concerns, self-hosted GitLab or Woodpecker CI provide full control over pipeline execution and logs.

Artifact registries

Registry	Licence	Artifact types	Considerations
Harbor	Apache 2.0	Container images, Helm charts	Full-featured, supports replication, vulnerability scanning
Gitea packages	MIT	Containers, npm, PyPI, Maven, NuGet	Integrated with Gitea, lightweight
Nexus Repository	EPL (OSS), Proprietary (Pro)	Universal	Mature, wide format support
Artifactory	Proprietary	Universal	Enterprise features, high cost
GitHub Packages	Proprietary	Containers, npm, Maven, NuGet	Integrated with GitHub, US-hosted
AWS ECR	Proprietary	Container images	AWS integration, regional deployment

Feature flag services

Service	Licence	Hosting model	Considerations
Unleash	Apache 2.0	Self-hosted or managed	Full-featured open source, good SDK coverage
Flagsmith	BSD 3-Clause	Self-hosted or managed	Open source, includes remote config
GrowthBook	MIT	Self-hosted or managed	Combines flags with experimentation
LaunchDarkly	Proprietary	SaaS	Enterprise features, comprehensive SDKs, high cost
Split	Proprietary	SaaS	Strong experimentation focus

Monitoring and observability

Pipeline observability enables identification of bottlenecks, flaky tests, and infrastructure issues before they impact delivery velocity.

Pipeline metrics

Metric	Target	Alert threshold	Measurement point
Lead time (commit to production)	Under 4 hours	Over 24 hours	Deployment timestamp minus commit timestamp
Deployment frequency	Daily or more	Under weekly	Count of production deployments
Change failure rate	Under 5%	Over 15%	Failed deployments divided by total deployments
Mean time to recovery	Under 1 hour	Over 4 hours	Incident duration for deployment-caused outages
Pipeline success rate	Over 95%	Under 85%	Successful runs divided by total runs
Pipeline duration (p95)	Under 15 minutes	Over 30 minutes	End-to-end pipeline execution time
Queue time	Under 2 minutes	Over 10 minutes	Time from trigger to execution start

Build metrics

Metric	Target	Investigation trigger
Build cache hit rate	Over 60%	Under 40% for 3 consecutive days
Test flakiness rate	Under 1%	Over 3% for any test
Security finding resolution time	Under 7 days (critical)	Over 14 days
Artifact size growth	Under 10% month-over-month	Over 25% growth

Dashboards must display these metrics at team, project, and organisation levels. Weekly review of pipeline metrics identifies degradation trends before they impact delivery.