Immutable Deployment Packages¶

Last Updated: 2026-01-29 Status: Active Maintained by: GE Infrastructure Team

Overview¶

Immutable deployment packages provide a reliable, auditable, and reproducible method for deploying client workloads to Kubernetes. Each package is a self-contained directory containing all manifests, metadata, and verification tools needed for deployment.

Key Benefits: - Immutability: Packages cannot be modified after creation - Reproducibility: Same package produces identical deployments - Auditability: Full traceability from source to deployment - Safety: Checksum verification prevents corruption - Rollback: Previous packages can be redeployed instantly

Package Structure¶

Every immutable package follows this standard directory layout:

acme-corp-v1.2.3/
├── MANIFEST.json          # Package metadata (JSON)
├── manifests/
│   └── all.yaml           # Complete K8s manifests
├── images.txt             # Container image references with digests
├── secrets.env.enc        # Secrets reference (NOT actual secrets)
├── deploy.sh             # Deployment script
└── checksum.sha256       # SHA256 checksums for all files

File Descriptions¶

File	Purpose	Format
`MANIFEST.json`	Package metadata, version info, creation timestamp	JSON
`manifests/all.yaml`	All Kubernetes resources in a single YAML file	YAML
`images.txt`	Container images with SHA256 digests for verification	Plain text
`secrets.env.enc`	Reference to secrets (Vault paths), NOT actual values	Plain text
`deploy.sh`	Executable deployment script with safety checks	Bash script
`checksum.sha256`	SHA256 hashes of all package files	Plain text

Package Naming Convention¶

Packages are named using the pattern:

{client-name}-{version}/

Examples: - acme-corp-v1.2.3/ - bigcorp-v2.0.0/ - test-client-v1.0.0-rc1/

Versioning: - Must follow semantic versioning (semver) - Format: vMAJOR.MINOR.PATCH or vMAJOR.MINOR.PATCH-prerelease - Examples: v1.0.0, v2.1.3, v1.0.0-beta1

MANIFEST.json Schema¶

The MANIFEST.json file contains package metadata and serves as the authoritative source of package information.

Schema Definition¶

{
  "client": "string",           // Client name (lowercase, alphanumeric)
  "version": "string",          // Semantic version (vX.Y.Z)
  "namespace": "string",        // Target Kubernetes namespace
  "created": "string",          // ISO 8601 timestamp
  "created_by": "string",       // Package generation tool
  "source": "string",           // Source directory path
  "files": {
    "manifests": "string",      // Path to manifests (relative)
    "images": "string",         // Path to images list (relative)
    "secrets_ref": "string",    // Path to secrets reference (relative)
    "deploy_script": "string"   // Path to deploy script (relative)
  }
}

Example MANIFEST.json¶

{
  "client": "acme-corp",
  "version": "v1.2.3",
  "namespace": "sh-acme-corp",
  "created": "2026-01-29T10:15:30+01:00",
  "created_by": "package-client.sh",
  "source": "/home/claude/ge-bootstrap/k8s/clients/acme-corp",
  "files": {
    "manifests": "manifests/all.yaml",
    "images": "images.txt",
    "secrets_ref": "secrets.env.enc",
    "deploy_script": "deploy.sh"
  }
}

Field Descriptions¶

Field	Description	Required	Example
`client`	Unique client identifier	Yes	`acme-corp`
`version`	Semver version string	Yes	`v1.2.3`
`namespace`	K8s namespace for deployment	Yes	`sh-acme-corp`
`created`	Package creation timestamp (ISO 8601)	Yes	`2026-01-29T10:15:30+01:00`
`created_by`	Tool or user that created package	Yes	`package-client.sh`
`source`	Original source directory	Yes	`/k8s/clients/acme-corp`
`files.manifests`	Relative path to manifests	Yes	`manifests/all.yaml`
`files.images`	Relative path to image list	Yes	`images.txt`
`files.secrets_ref`	Relative path to secrets reference	Yes	`secrets.env.enc`
`files.deploy_script`	Relative path to deploy script	Yes	`deploy.sh`

Validation¶

The MANIFEST.json must: - Be valid JSON - Contain all required fields - Use semver format for version - Reference files that exist in the package

Validation Command:

# Check JSON syntax
jq empty MANIFEST.json

# Extract and verify fields
jq -r '.client, .version, .namespace' MANIFEST.json

Image Digest Pinning¶

Why Pin Image Digests¶

Container image tags (like nginx:alpine) are mutable and can change over time. Digest pinning ensures:

Immutability: SHA256 digest guarantees exact image version
Security: Prevents supply chain attacks via tag hijacking
Reproducibility: Same digest always pulls identical image
Auditability: Know exactly which image was deployed

Digest Format¶

Without Digest (Mutable):

nginx:alpine

With Digest (Immutable):

nginx@sha256:a1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcd

With Tag and Digest (Best Practice):

nginx:alpine@sha256:a1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcd

images.txt Format¶

The images.txt file contains one image reference per line, with SHA256 digests:

nginx@sha256:4c0fdaa8b6341bfdeca5f18f7837462c80cff4a26c2fc39f59f4aca48f60d3a0
redis:7-alpine@sha256:8b6cf2f4d2c2ba6d8d1e6d8f5c3e2f4d2c2ba6d8d1e6d8f5c3e2f4d2c2ba6d8d

Generation Process: 1. Extract image references from manifests 2. Pull images to local Docker daemon 3. Inspect images to get RepoDigests 4. Write digests to images.txt

Automatic Digest Resolution¶

The package-client.sh script automatically resolves digests:

# Extract images from manifests
grep -E "^\s+image:" manifests/all.yaml | sed 's/.*image:\s*//'

# Resolve digest for each image
docker inspect --format='{{index .RepoDigests 0}}' nginx:alpine

Output:

nginx@sha256:4c0fdaa8b6341bfdeca5f18f7837462c80cff4a26c2fc39f59f4aca48f60d3a0

Manual Digest Lookup¶

If you need to find a digest manually:

# Pull image
docker pull nginx:alpine

# Get digest
docker inspect nginx:alpine --format='{{index .RepoDigests 0}}'

# Or use crane (Google's container tool)
crane digest nginx:alpine

Checksum Verification¶

Purpose¶

The checksum.sha256 file ensures package integrity by storing SHA256 hashes of all files. This detects:

File corruption during transfer
Unauthorized modifications
Incomplete package copies
Bit rot on storage media

Checksum File Format¶

a1b2c3d4e5f6...  ./MANIFEST.json
f6e5d4c3b2a1...  ./manifests/all.yaml
9876543210ab...  ./images.txt
fedcba098765...  ./secrets.env.enc
1234567890cd...  ./deploy.sh

Format:

<sha256-hash>  <filepath>

Generation¶

Generated during package creation:

cd acme-corp-v1.2.3/
find . -type f -not -name "checksum.sha256" -exec sha256sum {} \; > checksum.sha256

Important: The checksum file itself is NOT included in checksums.

Verification¶

Automatic Verification:

cd acme-corp-v1.2.3/
sha256sum -c checksum.sha256

Output (Success):

./MANIFEST.json: OK
./manifests/all.yaml: OK
./images.txt: OK
./secrets.env.enc: OK
./deploy.sh: OK

Output (Failure):

./manifests/all.yaml: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

Quiet Mode (Scripts):

sha256sum -c checksum.sha256 --quiet
echo $?  # 0 = success, non-zero = failure

Checksum in Deployment¶

The deploy.sh script verifies checksums before deployment:

#!/bin/bash
# Verify package integrity
if ! (cd "$PACKAGE_DIR" && sha256sum -c checksum.sha256 --quiet 2>/dev/null); then
    echo "ERROR: Package integrity check failed!"
    exit 1
fi

Secrets Reference Format¶

Critical: Secrets Are NOT Stored in Packages¶

IMPORTANT: Packages contain references to secrets, NOT actual secret values.

Why: - Security: Secrets never leave Vault - Auditability: Packages can be stored in Git - Flexibility: Secrets can be rotated without regenerating packages - Compliance: Meets security requirements for secret management

secrets.env.enc Format¶

# Secrets Reference for acme-corp v1.2.3
# ============================================
# This file contains references to secrets, not actual values.
# Actual secrets must be retrieved from Vault during deployment.
#
# Generated: 2026-01-29T10:15:30+01:00
# Namespace: sh-acme-corp
#
# Required secrets:
#   - ge-secrets (from Vault path: secret/clients/acme-corp)
#
# Vault paths:
VAULT_SECRET_PATH=secret/data/clients/acme-corp
VAULT_NAMESPACE=sh-acme-corp

# Secret keys expected:
# - redis-password
# - anthropic-api-key (if applicable)
# - database-url (if applicable)
# - api-token (if applicable)
#
# To deploy, ensure Vault is unsealed and accessible.

Secret Retrieval During Deployment¶

Secrets must be retrieved from Vault and created in K8s before deployment:

Step 1: Access Vault

export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN="<root-token>"

# Or use kubectl port-forward
kubectl port-forward -n ge-system svc/vault 8200:8200 &

Step 2: Retrieve Secrets

# Read secrets from Vault
vault kv get secret/clients/acme-corp

# Example output:
# Key                Value
# ---                -----
# redis-password     s3cr3tp@ssw0rd
# api-token          abc123def456

Step 3: Create K8s Secret

kubectl create secret generic ge-secrets \
  -n sh-acme-corp \
  --from-literal=redis-password="s3cr3tp@ssw0rd" \
  --from-literal=api-token="abc123def456"

Step 4: Deploy Package

./deploy-package.sh --package acme-corp-v1.2.3

Automated Secret Sync¶

For production, use a secret sync tool:

Option 1: Vault Agent Injector

apiVersion: v1
kind: Pod
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "acme-corp"
    vault.hashicorp.com/agent-inject-secret-config: "secret/data/clients/acme-corp"

Option 2: External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: acme-corp-secrets
  namespace: sh-acme-corp
spec:
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: ge-secrets
  data:
    - secretKey: redis-password
      remoteRef:
        key: secret/clients/acme-corp
        property: redis-password

Package Generation Workflow¶

Using package-client.sh¶

Script: /home/claude/ge-bootstrap/tools/package-client.sh

Basic Usage¶

cd /home/claude/ge-bootstrap/tools

# Generate package
./package-client.sh --client acme-corp --version v1.2.3

# Custom output directory
./package-client.sh \
  --client acme-corp \
  --version v1.2.3 \
  --output /tmp/packages

Step-by-Step Process¶

Step 1: Validate Inputs - Check client directory exists in /k8s/clients/ - Validate version format (semver) - Check for existing package with same version

Step 2: Generate Manifests

# Using kustomize
kustomize build /k8s/clients/acme-corp > manifests/all.yaml

# Or using kubectl
kubectl kustomize /k8s/clients/acme-corp > manifests/all.yaml

Step 3: Extract Images

# Find all image references
grep -E "^\s+image:" manifests/all.yaml | \
  sed 's/.*image:\s*//' | \
  sort -u > images.txt.raw

# Resolve digests
while read -r image; do
  digest=$(docker inspect --format='{{index .RepoDigests 0}}' "$image")
  echo "$digest" >> images.txt
done < images.txt.raw

Step 4: Generate Secrets Reference

cat > secrets.env.enc << EOF
# Secrets Reference for acme-corp v1.2.3
VAULT_SECRET_PATH=secret/data/clients/acme-corp
VAULT_NAMESPACE=sh-acme-corp
EOF

Step 5: Create MANIFEST.json

cat > MANIFEST.json << EOF
{
  "client": "acme-corp",
  "version": "v1.2.3",
  "namespace": "sh-acme-corp",
  "created": "$(date -Iseconds)",
  "created_by": "package-client.sh",
  "source": "/k8s/clients/acme-corp",
  "files": {
    "manifests": "manifests/all.yaml",
    "images": "images.txt",
    "secrets_ref": "secrets.env.enc",
    "deploy_script": "deploy.sh"
  }
}
EOF

Step 6: Generate deploy.sh

cat > deploy.sh << 'EOF'
#!/bin/bash
# Deployment script for acme-corp v1.2.3
# (script content auto-generated)
EOF
chmod +x deploy.sh

Step 7: Generate Checksums

find . -type f -not -name "checksum.sha256" -exec sha256sum {} \; > checksum.sha256

Example Output¶

==========================================
Generating package: acme-corp-v1.2.3
==========================================
[INFO] Building manifests from kustomize...
[OK] Manifests generated
[INFO] Extracting container images...
[INFO] Resolving image digests...
[OK] Extracted 3 unique images
[INFO] Generating secrets reference...
[OK] Secrets reference generated
[INFO] Generating deployment script...
[OK] Deployment script generated
[OK] Package generated: /home/claude/ge-bootstrap/packages/acme-corp-v1.2.3

Package contents:
total 24
-rw-r--r-- 1 claude claude  356 Jan 29 10:15 MANIFEST.json
-rw-r--r-- 1 claude claude  487 Jan 29 10:15 checksum.sha256
-rwxr-xr-x 1 claude claude 2134 Jan 29 10:15 deploy.sh
-rw-r--r-- 1 claude claude  245 Jan 29 10:15 images.txt
drwxr-xr-x 2 claude claude 4096 Jan 29 10:15 manifests
-rw-r--r-- 1 claude claude  423 Jan 29 10:15 secrets.env.enc

[INFO] Manifest size: 237 lines
[INFO] Images: 3 unique
[OK] To deploy: /home/claude/ge-bootstrap/packages/acme-corp-v1.2.3/deploy.sh
[OK] To verify: ./verify-package.sh /home/claude/ge-bootstrap/packages/acme-corp-v1.2.3

Package Storage¶

Default Location:

/home/claude/ge-bootstrap/packages/
├── acme-corp-v1.0.0/
├── acme-corp-v1.1.0/
├── acme-corp-v1.2.3/
└── bigcorp-v2.0.0/

Best Practices: - Keep at least 3 most recent versions - Archive old packages to backup storage - Never modify packages after creation - Use Git LFS for package version control (optional)

Package Verification¶

Using verify-package.sh¶

Script: /home/claude/ge-bootstrap/tools/verify-package.sh

Basic Usage¶

cd /home/claude/ge-bootstrap/tools

# Verify package
./verify-package.sh ../packages/acme-corp-v1.2.3

Verification Steps¶

1. Check Required Files

[PASS] MANIFEST.json exists
[PASS] manifests/all.yaml exists
[PASS] images.txt exists
[PASS] secrets.env.enc exists
[PASS] deploy.sh exists
[PASS] checksum.sha256 exists

2. Verify Checksums

[INFO] Verifying checksums...
[PASS] All checksums valid

3. Validate MANIFEST.json

[PASS] MANIFEST.json is valid JSON
[INFO]   Client: acme-corp
[INFO]   Version: v1.2.3
[INFO]   Namespace: sh-acme-corp

4. Validate Kubernetes Manifests

[PASS] YAML syntax valid
[PASS] Manifests validate with kubectl
[INFO]   Resources: 6

5. Check Image Digests

[INFO]   Images listed: 3
[PASS] All images have pinned digests

6. Validate Deploy Script

[PASS] deploy.sh is executable
[PASS] deploy.sh has valid bash syntax

Example Output (Success)¶

==========================================
Verifying package: acme-corp-v1.2.3
==========================================
[PASS] MANIFEST.json exists
[PASS] manifests/all.yaml exists
[PASS] images.txt exists
[PASS] secrets.env.enc exists
[PASS] deploy.sh exists
[PASS] checksum.sha256 exists

[INFO] Verifying checksums...
[PASS] All checksums valid

[INFO] Validating MANIFEST.json...
[PASS] MANIFEST.json is valid JSON
[INFO]   Client: acme-corp
[INFO]   Version: v1.2.3
[INFO]   Namespace: sh-acme-corp

[INFO] Validating Kubernetes manifests...
[PASS] Manifests validate with kubectl
[INFO]   Resources: 6

[INFO] Validating images...
[INFO]   Images listed: 3
[PASS] All images have pinned digests

[INFO] Validating deploy script...
[PASS] deploy.sh is executable
[PASS] deploy.sh has valid bash syntax

==========================================
[OK] Package verification PASSED
[INFO] Package is ready for deployment
==========================================

Example Output (Failure)¶

==========================================
Verifying package: acme-corp-v1.2.3
==========================================
[PASS] MANIFEST.json exists
[PASS] manifests/all.yaml exists
[FAIL] images.txt missing
[PASS] secrets.env.enc exists
[PASS] deploy.sh exists
[PASS] checksum.sha256 exists

[INFO] Verifying checksums...
[FAIL] Checksum verification failed
./images.txt: FAILED

==========================================
[ERROR] Package verification FAILED with 2 error(s)
==========================================

Deployment Procedure¶

Using deploy-package.sh¶

Script: /home/claude/ge-bootstrap/tools/deploy-package.sh

Pre-Deployment Checklist¶

[ ] Package has been verified with verify-package.sh
[ ] Secrets exist in Vault
[ ] Kubernetes cluster is healthy
[ ] Namespace exists or will be created
[ ] Backup of current deployment taken (if updating)
[ ] Deployment window scheduled (if production)

Basic Deployment¶

cd /home/claude/ge-bootstrap/tools

# Deploy to current context
./deploy-package.sh --package ../packages/acme-corp-v1.2.3

# Deploy to specific context
./deploy-package.sh \
  --package ../packages/acme-corp-v1.2.3 \
  --context upcloud-prod

# Dry run (preview changes)
./deploy-package.sh \
  --package ../packages/acme-corp-v1.2.3 \
  --dry-run

Deployment Process¶

Step 1: Verify Package

[INFO] Verifying package integrity...
[OK] Package verification passed

Step 2: Check Cluster Health

[INFO] Checking cluster health...
[OK] Cluster is reachable
[INFO] Nodes ready: 3

Step 3: Capture Current State

[INFO] Capturing current deployment state...
[INFO] State captured to: /tmp/deploy-state-sh-acme-corp-20260129101530.yaml

Step 4: Confirmation Prompt

==========================================
Deploying: acme-corp v1.2.3
Namespace: sh-acme-corp
Context: default
==========================================

Proceed with deployment? (y/N)

Step 5: Apply Manifests

[INFO] Executing deployment...
namespace/sh-acme-corp configured
deployment.apps/web configured
service/web unchanged
ingress.networking.k8s.io/web configured
networkpolicy.networking.k8s.io/client-isolation unchanged

Step 6: Wait for Rollout

[INFO] Waiting for deployment to stabilize...
deployment.apps/web condition met
[OK] Deployment complete!

Step 7: Show Status

[INFO] Current status:
NAME                  READY   STATUS    RESTARTS   AGE
pod/web-7d4f8b-xyz    1/1     Running   0          30s

Advanced Options¶

Skip Verification:

# NOT RECOMMENDED for production
./deploy-package.sh \
  --package ../packages/acme-corp-v1.2.3 \
  --skip-verify

Force Deployment (No Confirmation):

./deploy-package.sh \
  --package ../packages/acme-corp-v1.2.3 \
  --force

Deployment with Context and Dry Run:

./deploy-package.sh \
  --package ../packages/acme-corp-v1.2.3 \
  --context staging \
  --dry-run

Using Package's deploy.sh Directly¶

Each package includes its own deploy.sh script:

cd /home/claude/ge-bootstrap/packages/acme-corp-v1.2.3

# Deploy
./deploy.sh

# Dry run
./deploy.sh --dry-run

# Specific context
./deploy.sh --context upcloud-prod

Rollback from Packages¶

Why Packages Enable Easy Rollback¶

Previous versions are immutable and preserved
No need to regenerate manifests from Git
Instant rollback to any previous version
Verified packages guarantee consistent state

Rollback Procedure¶

Step 1: Identify Previous Version

ls -lt /home/claude/ge-bootstrap/packages/ | grep acme-corp

# Output:
# acme-corp-v1.2.3/  (current - broken)
# acme-corp-v1.2.2/  (previous - working)
# acme-corp-v1.2.1/

Step 2: Deploy Previous Package

./deploy-package.sh --package ../packages/acme-corp-v1.2.2

Step 3: Verify Rollback

kubectl get deployment web -n sh-acme-corp -o jsonpath='{.metadata.annotations.deployment\.version}'
# Should show: v1.2.2

kubectl get pods -n sh-acme-corp
# Should show: Running with older image

Using kubectl rollout undo¶

For quick rollback without packages:

# Rollback to previous version
kubectl rollout undo deployment/web -n sh-acme-corp

# Rollback to specific revision
kubectl rollout undo deployment/web -n sh-acme-corp --to-revision=3

# Check rollout history
kubectl rollout history deployment/web -n sh-acme-corp

Rollback Best Practices¶

Keep Recent Packages
Maintain at least 3 most recent versions
Archive older versions to backup storage
Test Rollback in Staging
Verify rollback process works
Ensure previous package is functional

Document Rollback Reason

# Add annotation to deployment
kubectl annotate deployment web -n sh-acme-corp \
  rollback.reason="Database migration failed" \
  rollback.from-version="v1.2.3" \
  rollback.to-version="v1.2.2"

Monitor After Rollback

kubectl get pods -n sh-acme-corp -w
kubectl logs -n sh-acme-corp -l app=web --tail=50 -f

Package Versioning Strategy¶

Semantic Versioning (Semver)¶

All packages MUST follow semantic versioning:

vMAJOR.MINOR.PATCH[-PRERELEASE]

Examples: - v1.0.0 - Initial release - v1.0.1 - Patch (bug fix) - v1.1.0 - Minor (new feature, backward compatible) - v2.0.0 - Major (breaking changes) - v1.0.0-beta1 - Pre-release

Version Increment Guidelines¶

Change Type	Example	Version Bump
Bug fix, security patch	Fix nginx config	PATCH (v1.0.0 → v1.0.1)
New feature, backward compatible	Add health endpoint	MINOR (v1.0.1 → v1.1.0)
Breaking change, major refactor	Change API format	MAJOR (v1.1.0 → v2.0.0)
Pre-release testing	Beta release	PRERELEASE (v1.0.0-beta1)

Version Naming Conventions¶

Production Releases:

v1.0.0
v1.1.0
v2.0.0

Pre-releases:

v1.0.0-alpha1   (Early testing)
v1.0.0-beta1    (Feature complete)
v1.0.0-rc1      (Release candidate)

Development:

v1.0.0-dev      (Development build)
v1.0.0-snapshot (Nightly build)

Version Validation¶

The package generation script validates versions:

# Valid versions
./package-client.sh --client acme-corp --version v1.2.3
./package-client.sh --client acme-corp --version v2.0.0-rc1

# Invalid versions (will fail)
./package-client.sh --client acme-corp --version 1.2.3      # Missing 'v'
./package-client.sh --client acme-corp --version v1.2       # Missing patch
./package-client.sh --client acme-corp --version latest     # Not semver

Version Tracking¶

Track versions in a registry:

# /home/claude/ge-bootstrap/config/clients.yaml
clients:
  - name: acme-corp
    namespace: sh-acme-corp
    packages:
      - version: v1.0.0
        deployed: 2026-01-15T10:00:00Z
        status: superseded
      - version: v1.1.0
        deployed: 2026-01-20T14:30:00Z
        status: superseded
      - version: v1.2.3
        deployed: 2026-01-29T10:15:30Z
        status: active

Troubleshooting¶

Package Generation Failures¶

Issue: Client directory not found

[ERROR] Client directory not found: /k8s/clients/acme-corp
[ERROR] Create client first with: create-client.sh --name acme-corp --type <type>

Solution:

# Create client first
/home/claude/ge-bootstrap/tools/create-client.sh \
  --type shared \
  --name acme-corp \
  --resources small

Issue: Invalid version format

[ERROR] Invalid version format: 1.2.3
[ERROR] Expected format: v1.2.3 or v1.2.3-rc1

Solution:

# Add 'v' prefix
./package-client.sh --client acme-corp --version v1.2.3

Issue: Image digest not resolved

[WARN] Could not resolve digest for: custom-app:latest (using tag)

Solution:

# Pull image first
docker pull custom-app:latest

# Or specify registry
docker pull registry.example.com/custom-app:latest

# Then regenerate package
./package-client.sh --client acme-corp --version v1.2.3

Package Verification Failures¶

Issue: Checksum mismatch

[FAIL] ./manifests/all.yaml: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

Solution:

# Package was modified - regenerate checksums
cd acme-corp-v1.2.3/
find . -type f -not -name "checksum.sha256" -exec sha256sum {} \; > checksum.sha256

# Or regenerate entire package
rm -rf acme-corp-v1.2.3/
./package-client.sh --client acme-corp --version v1.2.3

Issue: Missing files

[FAIL] images.txt missing

Solution:

# Package is incomplete - regenerate
rm -rf acme-corp-v1.2.3/
./package-client.sh --client acme-corp --version v1.2.3

Issue: Invalid YAML

[FAIL] Manifests failed kubectl validation

Solution:

# Check manifest syntax
kubectl apply --dry-run=client -f acme-corp-v1.2.3/manifests/all.yaml

# Fix source files in /k8s/clients/acme-corp/
# Then regenerate package
./package-client.sh --client acme-corp --version v1.2.3

Deployment Failures¶

Issue: Package verification failed

[ERROR] Package integrity check failed!

Solution:

# Verify manually
./verify-package.sh acme-corp-v1.2.3

# Regenerate if corrupted
./package-client.sh --client acme-corp --version v1.2.3

Issue: Cluster not reachable

[ERROR] Cannot connect to Kubernetes cluster

Solution:

# Check cluster
kubectl cluster-info

# Check context
kubectl config current-context

# Specify correct context
./deploy-package.sh \
  --package acme-corp-v1.2.3 \
  --context upcloud-prod

Issue: Deployment timeout

[WARN] Deployment may still be progressing

Solution:

# Check pod status
kubectl get pods -n sh-acme-corp

# Check events
kubectl get events -n sh-acme-corp --sort-by='.lastTimestamp'

# Check logs
kubectl logs -n sh-acme-corp -l app=web --tail=50

# Common causes:
# - Image pull errors
# - Resource limits too low
# - Missing secrets
# - Health check failures

Issue: Secrets missing

Error: secret "ge-secrets" not found

Solution:

# Create secrets from Vault
vault kv get secret/clients/acme-corp

kubectl create secret generic ge-secrets \
  -n sh-acme-corp \
  --from-literal=redis-password="..." \
  --from-literal=api-token="..."

# Redeploy
./deploy-package.sh --package acme-corp-v1.2.3

Architecture Overview - Hosting architecture
Client Onboarding - Creating clients
Zero-Downtime Deployments - Deployment strategies
Platform Startup - Platform management

Best Practices¶

Package Management¶

Version Control
Store packages in dedicated repository or object storage
Use Git LFS for large packages
Never commit packages to main code repository
Retention Policy
Keep 3 most recent versions online
Archive older versions to cold storage
Delete packages older than 90 days (except major versions)
Security
Scan images for vulnerabilities before packaging
Never include actual secrets
Sign packages with GPG (optional)
Testing
Test packages in staging before production
Verify rollback works before deploying new version
Automate verification in CI/CD pipeline

Deployment¶

Pre-Deployment
Always run verify-package.sh first
Check cluster health
Ensure secrets exist
Take backup of current state
During Deployment
Monitor pod rollout status
Watch application logs
Check metrics and health endpoints
Be ready to rollback
Post-Deployment
Verify application functionality
Check resource usage
Monitor for errors
Document deployment in changelog

This documentation is maintained by the GE Infrastructure Team. For updates or questions, contact the infrastructure lead or create an issue in the ge-ops repository.

Immutable Deployment Packages¶

Overview¶

Table of Contents¶

Package Structure¶

File Descriptions¶

Package Naming Convention¶

MANIFEST.json Schema¶

Schema Definition¶

Example MANIFEST.json¶

Field Descriptions¶

Validation¶

Image Digest Pinning¶

Why Pin Image Digests¶

Digest Format¶

images.txt Format¶

Automatic Digest Resolution¶

Manual Digest Lookup¶

Checksum Verification¶

Purpose¶

Checksum File Format¶

Generation¶

Verification¶

Checksum in Deployment¶

Secrets Reference Format¶

Critical: Secrets Are NOT Stored in Packages¶

secrets.env.enc Format¶

Secret Retrieval During Deployment¶

Automated Secret Sync¶

Package Generation Workflow¶

Using package-client.sh¶

Basic Usage¶

Step-by-Step Process¶

Example Output¶

Package Storage¶

Package Verification¶

Using verify-package.sh¶

Basic Usage¶

Verification Steps¶

Example Output (Success)¶

Example Output (Failure)¶

Deployment Procedure¶

Using deploy-package.sh¶

Pre-Deployment Checklist¶

Basic Deployment¶

Deployment Process¶

Advanced Options¶

Using Package's deploy.sh Directly¶

Rollback from Packages¶

Why Packages Enable Easy Rollback¶

Rollback Procedure¶

Using kubectl rollout undo¶

Rollback Best Practices¶

Package Versioning Strategy¶

Semantic Versioning (Semver)¶

Version Increment Guidelines¶

Version Naming Conventions¶

Version Validation¶

Version Tracking¶

Troubleshooting¶

Package Generation Failures¶

Package Verification Failures¶

Deployment Failures¶

Related Documentation¶

Best Practices¶

Package Management¶

Deployment¶