Back to blog
cicddevopsreliabilityplatform-engineering

Why deployments still feel risky despite CI/CD

What to review when a pipeline exists but releases still depend on manual steps, uncertain rollback and low operational confidence.

Why deployments still feel risky despite CI/CD

The team already has GitHub Actions, GitLab CI, Jenkins, CodePipeline or a mixture of tools. The pipeline builds. The tests run. There is a deployment job. On paper, the process looks reasonable.

But an important release comes along and the tension returns: someone validates things outside the pipeline, someone else checks the AWS console, everyone waits for a “safe” time window, rollback is not entirely clear, and nobody wants to touch a database migration at the end of the day.

That fear does not mean the team is bad or the CI/CD tool was the wrong choice. It usually points to something more uncomfortable: part of the journey has been automated, but a deployment process the team trusts has not yet been designed.

The distinction matters. CI/CD is not just about moving code from a repository to production. It is about knowing what is being deployed, how it has been validated, which signals will be watched, and what options are available if the change goes wrong. The goal is not to eliminate all risk—that does not exist in production—but to make it smaller, visible and manageable.

A green pipeline does not answer every question

A passing pipeline can create a false sense of security if it leaves out important operational questions:

  • Which exact version is running in each environment?
  • What has been tested before production?
  • Which configuration changes alongside the code?
  • Who approves the move to production, and on what basis?
  • Which signals show that the deployment is working well?
  • What happens if we need to roll back or move forward with a fix?

The AWS Well-Architected Framework recommends using build and deployment systems, automating integration and deployment, testing changes, and planning for unsuccessful changes. Automation helps, but it needs to connect to validation, observability and recovery capability.

When the pipeline only runs commands while the important decisions live outside it, the release still depends too much on human coordination and tribal knowledge.

Invisible manual steps are part of the system too

One of the first places to look is everything that happens around the pipeline.

Approvals in chat. Manual console changes. Variables updated by hand. Validations only one person knows about. Commands copied from an old document. A staging check nobody can fully explain. A business decision that arrives late and blocks production.

Not every manual intervention is bad. Some approvals make sense because of risk, compliance, commercial coordination or customer impact. The problem arises when those approvals replace clear controls, or when the team cannot distinguish a deliberate pause from inherited friction.

A useful review starts by mapping the real workflow, not the ideal one. From merge to production: what happens, who is involved, which tools are used, which checks are performed, and which steps depend on one particular person. Anything that is not written down or modelled in the pipeline is still part of the release system.

Staging offers little reassurance if it differs where it matters

Many teams have dev, staging and production. Those names alone do not guarantee comparable environments.

Staging may have less traffic, less data, different permissions, different configuration, disabled integrations or manually created infrastructure. Sometimes it is enough to validate a screen. Sometimes it cannot detect the kind of failure that will appear in production.

Staging does not need to match production's scale. For many teams, that would be expensive and unnecessary. But it should be representative in critical behaviours: configuration, permissions, key dependencies, infrastructure versions, deployment paths and checks that affect the release.

Infrastructure as code helps here, but the problem should not become a debate about Terraform or GitOps. The useful question is simpler: if something happens in staging, does it genuinely increase confidence about what will happen in production?

Tests and gates: when green is not convincing

Another common symptom: the pipeline passes, but nobody trusts it.

This can happen because of flaky tests that fail for no clear reason, slow suites the team avoids running, coverage that does not validate critical paths, checks habitually ignored, or approvals given out of fatigue. The result is a pipeline that adds ceremony but not judgement.

The review should not begin with “more tests” in the abstract, but with a concrete question: which real failures do we want to detect before production?

For a service deployed on AWS, that may include application code, configuration, infrastructure, permissions, migrations, version compatibility, container health, application startup or minimum security checks. Not everything has to block the release, but the team should know which controls are informational and which are genuine gates.

A useful gate reduces uncertainty. A gate nobody understands just postpones the fear by a few minutes.

Rollback cannot be a generic phrase

Fear becomes particularly strong when the team cannot see a clear way out.

“If something fails, we'll roll back” sounds good until someone asks how. Are there versioned artefacts? Are releases tagged? Does configuration roll back too? Can the database support running the previous version? What happens to asynchronous tasks, queues, events or protocol changes?

AWS recommends planning for unsuccessful changes and using safe deployment strategies when they fit the architecture: rolling deployments, canary, blue/green, traffic splitting or feature flags, among others. But no strategy fixes a poorly designed release workflow on its own.

Data migrations deserve particular care. Redeploying the previous version may not be enough if the change modified a schema, transformed data or introduced an incompatible format. In those cases, design for backward compatibility, migration phases or a specific remediation plan.

Rollback is not a line in a runbook. It is a capability that is designed, documented and, when the risk warrants it, tested.

Release observability: knowing whether the change is going well

A deployment does not end when the job reports success. It ends when the team has reasonable signals that the system is still behaving as expected.

Useful observability during a release is not about opening lots of graphs. It is about knowing what to look at and which decision to make based on what appears.

Logs, metrics, traces, alarms and dashboards can help if they connect to relevant signals: application errors, latency, saturation, dependency health, queue backlogs, failures in critical tasks or impact on important business flows.

Correlating deployments with production signals also helps. If an alarm fires shortly after a release, the team should be able to see what changed, which version is active and who is aware of it. This is reliability, but also basic operational hygiene.

Ownership: who decides, who executes and who responds

Deployment confidence is not only technical. It is organisational too.

In some teams, product sets the date, engineering prepares the change, DevOps maintains the pipeline, platform looks after infrastructure, and operations receives the alerts. If each group has a different definition of “ready for production”, every release becomes a constant negotiation.

The point is not to blame someone or turn platform engineering into the place where all the chaos ends up. It is to clarify responsibilities:

  • who decides a change is ready;
  • who can pause or cancel a release;
  • who watches the signals after deployment;
  • who leads the response if something fails;
  • who maintains the pipeline, runbooks and gates.

When this is unclear, the pipeline can be technically correct and still create tension.

How to start without redesigning the entire platform

You do not need a huge internal platform to improve deployment confidence. Often, it makes sense to start smaller.

A good first exercise is to bring together the people involved in releases and draw the current workflow. Not the aspirational one: the real one. From merge to production. Then review six areas:

  1. Manual steps and approvals: which provide control and which exist only through inertia.
  2. Environments: which differences between staging and production reduce validation reliability.
  3. Tests and gates: which relevant failures they detect and which slip through.
  4. Rollback or fix-forward: which option exists for different types of change.
  5. Observability: which signals are watched during and after deployment.
  6. Ownership: who decides, executes and responds at each stage.

From there, choosing two or three small improvements is often more useful than redesigning everything: versioning artefacts consistently, documenting rollback criteria, removing a fragile manual step, making staging more representative for a critical dependency, or adding an actionable alarm associated with the release.

Maturity shows when the team stops depending on heroics. Not because deployment is trivial, but because the path is clear.

Useful CI/CD: less YAML, more operational confidence

Having CI/CD is a good starting point. But if every release still depends on invisible manual steps, uncertain rollback, unrepresentative environments and weak signals, adding more YAML will not fix the problem.

A reliable deployment process combines automation, technical judgement and clear ownership. It makes it possible to know what changed, what was validated, how production is observed and what options exist if the change does not go as expected.

If your pipeline exists but every release still requires too much coordination, AstralDeploy can help you review the AWS deployment workflow and turn it into a clearer, more traceable and more controllable process, without promising automatic results.

Sources consulted

AR

Alejandro Rodríguez

Freelance Cloud & DevOps Consultant, Madrid