Skip to content

CI/CD Tiered Pipeline

The GE CI/CD pipeline uses a 3-tier execution model to balance speed on every push against thoroughness on merge requests and nightly runs. Each tier is a superset of the previous: STANDARD includes all FAST jobs, FULL includes all STANDARD jobs.

Tier Overview

Tier Trigger Approx. Duration Job Count Purpose
FAST Every push to any branch ~2 minutes 10 Immediate feedback. Blocks obvious failures fast.
STANDARD MR open/update + push to main ~5 minutes 25 Full gate before merge. Includes build, sign, and staging deploy.
FULL Nightly at 02:00 CEST + manual trigger ~20 minutes 31 Complete validation including mutation, E2E, DAST, and property tests.

FAST Tier — Every Push (~2 min, 10 jobs)

These jobs run on every push to any branch. They are fast, deterministic, and provide immediate signal.

Job What it does
lint:python ruff zero-tolerance linting
lint:secrets gitleaks secret detection (445-finding baseline; blocks NEW secrets only)
build:backend Verifies 4 core Python modules import without error
test:unit 553 unit tests via pytest
tdd:oracle-check Verifies TDD test files do not import from src/ (oracle independence)
security:bandit Python SAST — CRITICAL/HIGH severity blocks
security:semgrep TypeScript SAST + 8 custom LLM anti-pattern rules
security:dependency-scan Python dependency vulnerability audit via safety
test:integration Real PostgreSQL 15 + Redis 6381 service containers
types:typescript tsc --noEmit --strict across all TypeScript

All 10 jobs are blocking. A failure on any FAST job blocks further CI execution on that commit.

STANDARD Tier — MR + main (~5 min, 25 jobs)

Runs on merge request open/update and on every push to main. Includes all FAST jobs plus:

Job What it does
types:python pyright --strict type checking across all Python modules
lint:deadcode knip (TypeScript) + vulture (Python) dead code detection
iac:checkov Checkov scan of k8s manifests and Dockerfiles
iac:kubesec Kubesec security scoring across all k8s YAML
test:e2e Playwright E2E suite — 4 parallel workers in CI
test:adversarial AST forbidden-call scan + 100-input fuzz test
test:reconciliation Jasper: TDD-phase suite vs post-implementation comparison
test:contract OpenAPI contract verification
build:image kaniko container build → push to GitLab Container Registry
sign:image cosign signing with SLSA Level 3 attestation
sbom:generate Syft CycloneDX SBOM — attached to container image
deploy:staging ArgoCD API-triggered sync to staging namespace
verify:health HTTP health checks, SSL verification, error rate < 0.1%
merge:gate Reads JUnit/SARIF artifacts; computes release readiness score (threshold: 70/100)
deps:node npm audit + Trivy filesystem scan

All 25 jobs are blocking. The following jobs were previously soft-failures (allow_failure: true) and have been hardened as of 2026-04-02:

  • types:python
  • lint:deadcode
  • iac:checkov
  • iac:kubesec
  • test:e2e
  • verify:health

FULL Tier — Nightly + Manual (~20 min, 31 jobs)

Runs nightly at 02:00 CEST via GitLab scheduled pipeline and on manual trigger. Includes all STANDARD jobs plus:

Job What it does
mutation:typescript Full Stryker mutation run across all TypeScript — blocking
mutation:python Full mutmut mutation run across all Python
test:property:python Hypothesis property-based tests (max_examples=1000)
test:property:typescript fast-check property-based tests (numRuns=1000)
dast:zap OWASP ZAP baseline scan against staging
dast:nuclei Nuclei template scan against staging

The FULL tier also includes the manual production deployment gate:

Job What it does
deploy:production Manual trigger required — human-in-the-loop gate before production promote

Note on mutation testing: On STANDARD tier runs, test:mutation runs in Stryker incremental mode — only files changed in the MR are mutated. The mutation:typescript job in the FULL tier runs the complete Stryker suite against all TypeScript. This distinction keeps STANDARD tier fast while ensuring full mutation coverage nightly.

How to Trigger a Full Run Manually

Via GitLab UI

  1. Navigate to your project in GitLab (gitlab.ge.internal).
  2. Go to Build → Pipelines.
  3. Click Run pipeline.
  4. Set the variable CI_TIER to full.
  5. Click Run pipeline.

The pipeline will execute all 31 jobs including mutation testing, property-based testing, DAST, and the production deploy gate.

Via GitLab API

curl --request POST \
  --form "token=$CI_JOB_TOKEN" \
  --form "ref=main" \
  --form "variables[CI_TIER]=full" \
  "https://gitlab.ge.internal/api/v4/projects/$PROJECT_ID/trigger/pipeline"

Replace $PROJECT_ID with your numeric project ID (visible in the GitLab project settings page).

Via git push with CI variable

git push -o ci.variable="CI_TIER=full"

This works from any branch and forces the FULL tier for that specific push only.

Nightly Schedule

The FULL tier runs automatically at 02:00 CEST every night via a GitLab scheduled pipeline. This ensures:

  • Full mutation testing runs on all code, not just changed files.
  • DAST scans run against the current staging environment.
  • Property-based tests run with high iteration counts (1000 examples/runs).
  • Any regressions introduced during the day are caught before business hours.

Results from the nightly FULL run are visible in Build → Pipelines with the scheduled source label.

Tier Selection Logic

The pipeline determines the tier from the following precedence:

  1. If CI_TIER=full variable is set → FULL tier.
  2. If CI_PIPELINE_SOURCE=schedule → FULL tier.
  3. If the ref is main or the pipeline source is merge_request_event → STANDARD tier.
  4. Otherwise → FAST tier.

allow_failure Status

The following table summarises all jobs where allow_failure is relevant. All other jobs default to blocking (allow_failure: false).

Job Tier allow_failure Notes
types:python STANDARD false Hardened 2026-04-02, was true
lint:deadcode STANDARD false Hardened 2026-04-02, was true
iac:checkov STANDARD false Hardened 2026-04-02, was true
iac:kubesec STANDARD false Hardened 2026-04-02, was true
test:e2e STANDARD false Hardened 2026-04-02, was true
mutation:typescript FULL false Hardened 2026-04-02, was true
verify:health STANDARD false Hardened 2026-04-02, was true
deploy:production FULL — Manual trigger, not a failure gate

There are no remaining allow_failure: true entries on enforcement stages.