Diagnosing GitLab CI/CD Failures

A pipeline turns red the moment one command misbehaves, and the fix always starts the same way: read the job log top-down to the first error, then name the failure class it belongs to. A GitLab CI job succeeds only when every command in its script: runs to the end and returns an exit status of 0; the operating system reports 0 for success and any non-zero number for failure, so pytest, pyats, and ansible-lint are ordinary shell commands that fail a job the same way false does, by exiting non-zero when a check does not pass. The runner records that as the line you will learn to look for, ERROR: Job failed: exit code 1. (GitLab CI/CD YAML: script^[1])

Fail-fast: the shell stops at the first non-zero exit

The commands under script: run in order, and the shell stops at the first one that returns non-zero, so everything after that line never executes. That is why a half-finished log is not a mystery: if you listed three test commands and see the output of only the first two, the second failed and short-circuited the rest, and the missing third line is the evidence rather than lost information. before_script: runs before script: and fails by the same rule, so a job that dies before its real work began is pointing at setup, not at your tests. after_script: is the exception; it always runs, on success or failure, and its own exit code never changes the verdict, so it is for teardown and extra diagnostics, never the place a test's pass or fail is decided. (before_script and after_script^[1])

The first error sorts the failure into one of three classes

Because the verdict is a single exit code, diagnosis is finding the command that produced it and reading what it said, top-down and not bottom-up. The last line is almost always the runner's generic ERROR: Job failed; the cause is the first command whose output shows an error, together with the traceback printed just above that final line. (GitLab: view job logs^[2]) The error text on that first failing line puts the failure into exactly one of three families, and the rest of this page takes one family per section:

• Missing dependency (command not found, ModuleNotFoundError): a tool or file the job needed was never delivered into the container.
• Version incompatibility (a SyntaxError on valid code, an ImportError or AttributeError after no code change): the code ran, but on the wrong runtime or library version.
• Failed test (AssertionError, a pyATS Result: FAILED, a lint rule id): the code ran and a check disagreed with it.

Name the class first. It tells you which lever to pull, and it stops you editing automation code that was never the problem.

A job that ran a Python script fine on your workstation can fail the moment it runs in the pipeline, reporting python: command not found or ModuleNotFoundError: No module named 'nornir'. The code did not change; the environment did. Every job runs inside a brand-new container that GitLab builds from the job's image:^[1] keyword, on a runner that shares nothing with your workstation or with the other jobs in the pipeline, so whatever the job needs at runtime has to be put into that container deliberately.

Two error strings account for almost every missing-dependency failure, and each fingerprints a different gap. command not found (for example ansible-playbook: command not found) is the shell reporting that no executable by that name is on the job's PATH: the tool is absent from the base image and was never installed. Do not misread it as a YAML error; invalid YAML fails CI Lint^[1] and the pipeline never starts, so if you are reading a job log at all, the YAML already parsed and the tool, not the syntax, is missing. ModuleNotFoundError: No module named 'nornir' is different: the Python interpreter is present and running, it just cannot import the library. ModuleNotFoundError is a subclass of ImportError, raised when a module cannot be located^[3]; the interpreter arrived through the image, while the library was meant to arrive through an install step that was never written or never listed that package.

Four supply channels, one of them empty

A dependency reaches a job through exactly one of four channels, and a missing-dependency failure is deciding which one was supposed to deliver and came up empty:

• Base image sets the interpreters and command-line tools that exist the instant the job starts (python, git, ansible when the image ships them).
• Install step is a before_script:^[1] block that installs the packages the image lacks; it runs before script: on every job.
• Artifacts are files produced by an earlier job and handed to a later one, but only when the producer declares them (the next section covers this channel).
• Cache reuses download directories to speed repeated jobs, but caching is an optimization and is not guaranteed to always work^[4], so it is never a channel you may depend on.

The first three are channels you may rely on; cache is not. The figure below shows the isolated job drawing from all four, with cache drawn as best-effort.

Minimal images omit tools

The base image sets what already exists in the container, so the smaller the image, the more you install yourself. Minimal images such as python:3.12-slim and alpine ship almost nothing beyond the interpreter, which is exactly why a job on one reports git: command not found or cannot compile a C-extension: git, gcc, and make were never included. You have two levers, mapping to the first two channels: pick a fuller image: (swap python:3.12-slim for python:3.12, the Debian-based default, which already ships git and a build toolchain), or install the one tool you need in before_script:.

run-automation:
  image: python:3.12          # Debian-based: ships git + a build toolchain
  before_script:
    - pip install -r requirements.txt   # installs nornir, ncclient, etc.
  script:
    - ansible-playbook -i inventory.yml deploy.yml

Match the install command to the image's package manager. On a Debian or Ubuntu image you install with apt-get update && apt-get install -y git; on an Alpine image you install with apk add --no-cache git, because Alpine uses apk and the musl C library, not apt-get and glibc. A Debian snippet copied onto Alpine fails with apt-get: command not found, a missing-dependency failure of its own. When the tooling gets heavy, a purpose-built image is cheaper to maintain than a long before_script:.

Files a job creates do not survive into the next job unless the producing job declares them as artifacts. Each job starts from the clean container of the previous section, so a render-configs job that writes configs/router1.cfg leaves nothing behind for a later deploy job, which then fails with a no-such-file error while the automation code is blameless. The supported hand-off has three moving parts: the producer lists the files under artifacts:paths:^[5], GitLab uploads them when the job succeeds, and by default later jobs fetch a copy of all artifacts from jobs in earlier stages^[5] automatically.

render-configs:
  stage: build
  script:
    - python render.py            # writes configs/*.cfg
  artifacts:
    paths:
      - configs/                  # without this line, nothing transfers

deploy:
  stage: deploy
  script:
    - ansible-playbook push_configs.yml   # reads configs/*.cfg

Because deploy is in a later stage, it downloads configs/ with no extra keyword. When the consumer cannot see the file, check the producer first: a missing artifacts:paths: is the usual cause, and a stray dependencies: []^[1] on the consumer, which downloads nothing, is the next.

Cache is an optimization, not a dependency channel

Cache makes repeated jobs faster, and you must never depend on it for correctness. A cache: reuses a directory of downloaded packages (a pip or npm cache) so the next run skips re-downloading, but GitLab is explicit that caching is an optimization and is not guaranteed to always work^[4]: the cache can be absent on a fresh runner, evicted, or simply not populated yet. A job that assumes its dependency is already unpacked in the cache therefore fails intermittently, passing on runners that happen to have it and failing on those that do not. Keep the real supply in a channel you control: runtime tools install in before_script:, files that move between jobs travel as artifacts, and cache sits on top of those two, never replacing them.

	Cache	Artifacts
Purpose	Speed up repeated jobs	Pass files between stages
Guaranteed present	No, best-effort	Yes, when declared
Right use	pip / npm download dirs	build outputs a later job reads
Safe to depend on	Never	Yes

If a design or troubleshooting question offers passing the built files to the next stage via cache, it is wrong by construction: use artifacts to pass intermediate build results between stages, and cache for dependencies you download^[4].

A job that installed and ran cleanly for months can start failing overnight with SyntaxError: invalid syntax on a line nobody touched, or with an AttributeError on a library call that worked yesterday. The code is fine; the environment no longer matches what the code needs. Every version-incompatibility failure has one shape, the code expects one version and the environment supplies another, and the mismatch lives on one of two surfaces. The runtime is the language interpreter the job executes on, fixed by the job's image: tag^[1]; a dependency is a library the job installs into that runtime, fixed by a version specifier in requirements.txt or a lockfile. A transitive dependency is one you never named yourself, pulled in because a package you did name depends on it. The error class names the surface: a SyntaxError on plainly valid code means the interpreter is too old to parse the syntax (runtime), while an ImportError, AttributeError, or TypeError on a call that worked before means an installed package resolved to a different version (dependency).

Runtime mismatch: align the image tag

The runner image sets the language runtime, so a job can only run code the image's interpreter understands. Structural pattern matching, the match/case statement, was added in Python 3.10^[6]; point a job at the python:3.8 image and it fails at parse time with SyntaxError: invalid syntax on the match line, before a single statement runs. Confirm which image actually ran by reading the CI_JOB_IMAGE predefined variable^[7] rather than assuming the pipeline default, then point image: at a tag whose runtime is new enough:

test:
  image: python:3.12        # was python:3.8 - now match/case parses
  script:
    - python -m pytest

Prefer an explicit version tag over a floating one. A bare python resolves to latest and python:3 tracks the newest 3.x, so either can move under you on the next pull and turn the runtime itself into drift, the same class of bug the dependency surface has.

Dependency drift: pin and lock

An unpinned dependency is a build that reinstalls itself differently over time. Write ansible with no version in requirements.txt and each run resolves it to that day's latest release; the moment a new major ships a breaking change, a pipeline that changed nothing goes red. Pin the exact version with pip's == operator^[8] so every run reinstalls the same release:

# requirements.txt - drift-prone
ansible

# requirements.txt - reproducible
ansible==9.2.0

A single == pin fixes the package you named, but its own transitive dependencies can still drift. A committed lockfile closes that gap: requirements.txt with hashes, poetry.lock, or Pipfile.lock records the exact version of every package in the resolved graph, so CI installs precisely what was tested. When a green pipeline turns red with no relevant code change, resist editing the code: reinstall from the committed lockfile and re-run, and if the failure disappears an unpinned transitive update caused it and the fix is a pin, not a rewrite. Then read the offending tool's changelog to confirm the breaking change and pick the version to hold.

A failed test is the one class where the code actually ran, and a check judged it wrong: pytest prints an AssertionError, pyATS reports Result: FAILED, and ansible-lint names a rule id. That shared trait, the code ran and something disagreed with it, is exactly what separates this class from a missing dependency or a version mismatch, where the code could not run at all. The fix lives in the code or the test's expected value, read straight from the log.

Three tools show up constantly on 350-901 automation pipelines, each with its own failure fingerprint:

Tool	What a failure prints	Where the fix lives
pytest	AssertionError[3], FAILED test_x::test_y	The code under test, or the expected value in the assertion
pyATS / aetest	Result: FAILED, a section Reason: and Genie diff	The device-facing code, or the expected state in the test
ansible-lint	A rule id with file:line (e.g. yaml[indentation])	The playbook, or an explicit rule waiver

pytest raises Python's built-in AssertionError when an assert fails, and its assertion rewriting prints the compared values, so you see assert 3 == 4 with both sides expanded rather than a bare failure; the short test summary info block lists each FAILED test_file.py::test_name. pyATS and its aetest engine report per-section results instead of raising: a section that calls self.failed(reason=...), or whose Genie diff finds a mismatch, is marked Result: FAILED, and the reason plus the expected-versus-actual device-state diff prints with that section. (Cisco pyATS documentation^[9]) ansible-lint prints each violation as a rule id with the file and line it fired on, for example yaml[indentation], and exits non-zero when any rule at or above its failure threshold matches. (Ansible documentation^[10])

allow_failure downgrades a gate to an advisory

allow_failure: true lets a job exit non-zero without failing the pipeline; the pipeline shows an orange warning and keeps going. When it is absent or false, which is the default, a failed test blocks every later stage. That is how you separate a blocking gate, such as unit tests that must pass, from an advisory check you are still cleaning up. It downgrades the failure to a warning; it does not hide the job, and it is never a fix for a real failing test. (GitLab CI/CD YAML: allow_failure^[1])

On the 350-901 exam a failure item usually hands you a short job-log excerpt and four candidate fixes, and the excerpt is the whole question: the error text on the first failing line decides the answer, and the trap is a plausible option that ignores it. Read the symptom, name the class from the first three sections, then pick the option that fills that specific class.

Stem shows	Failure class	Right fix
command not found	Missing dependency	Fuller image:, or install in before_script:
ModuleNotFoundError	Missing dependency	pip install -r requirements.txt in before_script:
Later job: file not found	Missing dependency (artifacts)	artifacts:paths: on the producing job
SyntaxError on valid match/case	Version incompatibility	Set image: to python:3.10 or newer
Passed for months, now red, no code change	Version incompatibility	Pin the version and install from the lockfile
AssertionError / Result: FAILED / lint id	Failed test	Fix the code or the expected value

Learn the distractors by name, because the same few recur:

• Fixing the YAML indentation is a distractor whenever a job log is shown at all: invalid YAML fails CI Lint and no job runs, so a shell error like command not found proves the YAML already parsed.
• Adding the built files to cache: so the next stage can read them inverts the cache-versus-artifacts rule; files that must reach a later job go in artifacts:paths:, because cache is best-effort and cannot be depended on.
• Rewriting the playbook or the script targets code that was never the problem when the log shows ModuleNotFoundError or a SyntaxError: the code is fine and its dependency or runtime is wrong.
• Retrying the job, or clearing the cache, does not fix drift. A retry reinstalls the same unpinned version, and clearing the cache can discard the last-good resolved set, so a green-to-red run with no code change is answered by a pin or a lockfile, never a retry.
• allow_failure: true is never a fix for a failing test; it downgrades a real regression to a warning so the pipeline reports green.

One more reflex worth rehearsing: a job that fails, then passes on a plain re-run of the same commit with nothing changed, is flaky infrastructure, not any of the three classes. Re-run it and, separately, make the test deterministic; the wrong move is to paper over a reproducible failure by retrying until it happens to pass.