#go #kubernetes #prometheus #observability #oci

k8s-container-monitor: Prometheus Exporter for Websites and OCI Artifacts

17 May 2026

A Go rewrite of a Python/Flask monitoring tool that exposes Prometheus metrics for website uptime and OCI artifact presence. The rewrite drops subprocess shell calls in favour of native oras-go/v2, shrinks the image from 130 MB to 25 MB, and adds goroutine-based concurrency.

The problem

k8s-container-monitor monitors two things: whether configured websites are up, and whether OCI image digests listed in a catalog repository are present in your registries. The original implementation was Python/Flask. It worked, but had three problems that became harder to ignore over time.

Shell injection. Registry authentication used subprocess.run with shell=True:

subprocess.run(
    f'echo "{token}" | oras login --username {user} --password-stdin {registry}',
    shell=True,
)

Any variable in that string — token, user, registry — gets executed verbatim by the shell. $(...) in a token, a space in a username, a semicolon in a registry hostname: all of them reach the shell unescaped.

Image size. python:slim weighs in at roughly 130 MB. For an app whose entire job is polling URLs and calling a registry API, that’s a lot of runtime to carry around.

Subprocess oras dependency. The oras CLI had to be installed in the image and called via shell on every check cycle. Brittle, slow, and the root cause of the shell injection surface.

The Go rewrite fixes all three.

Architecture

The Go version has a straightforward structure. Three background goroutines start at launch and run until the process exits, each writing to a shared Prometheus registry. Two collectors cover the core monitoring work (website uptime, OCI artifact presence); the third samples the process’s own memory for runtime telemetry. A single HTTP handler serves /metrics and /health.

  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
  │  Web Monitor     │  │ Artifact Checker  │  │ Memory Collector │
  │  (30s poll)      │  │  (60s poll)       │  │  (1s sample)     │
  └────────┬─────────┘  └────────┬──────────┘  └────────┬─────────┘
           │                     │                       │
           └─────────────────────┼───────────────────────┘
                                 ▼
                     ┌───────────────────────┐
                     │  Prometheus Registry  │
                     └───────────┬───────────┘
                                 ▼
                     GET /metrics   GET /health

Each goroutine is independent — the web monitor doesn’t wait for the artifact checker. Shutdown is handled via context cancellation: each goroutine blocks on select { case <-ctx.Done(): } and returns cleanly when the context is cancelled.

Key decisions

Go over Python. Three concrete wins: (1) oras-go/v2 replaces subprocess oras calls natively — no shell, no injection surface; (2) a distroless final image is 25 MB vs 130 MB Python slim; (3) goroutines with context cancellation are simpler to reason about than Python daemon threads.

oras-go/v2. The Go OCI client library maps cleanly to the operations the app needs: login, resolve digest, check existence. Replacing the subprocess calls was straightforward and the result is faster and more reliable.

Distroless base image. No shell, no package manager, no debug tooling. This reduces the attack surface as well as the size — a class of exploits that require a shell to work simply can’t execute.

“catalog” not “allowlist”. The original Python code used internal project names throughout. catalog better describes the intent (the OCI Distribution Spec uses catalog terminology) and works as a public, generic repo.

Learnings

Shell injection is easy to miss. The Python code had shell=True + an f-string token and it looked completely normal. The problem only becomes visible when you ask “what if this string contains a semicolon?” Auditing subprocess calls for shell=True is now on the checklist.

oras-go/v2 is less intimidating than it looks. The OCI registry interaction is three operations: authenticate, resolve digest, check existence. The library has a clean API for all three. Fear of “complex Go OCI code” was not warranted.

Distroless is a free win. The size reduction gets the headline but the reduced attack surface is the better reason. Switch the FROM line and fix any RUN instructions that rely on a shell — that’s it.

Registry config keys can’t have dots. The app uppercases config keys to build env var names (registry.prod → REGISTRY.PROD). Dots aren’t valid in environment variable names. Use underscores in registry keys (registry_prod, prod, staging) and document this clearly — it isn’t obvious until a pod fails to start with a cryptic env var rejection.

select { case <-ctx.Done(): } is the idiom. Goroutine shutdown via context cancellation is the standard Go pattern. Once you’ve written it once it becomes muscle memory.

Tutorial

Configuration

The app reads a YAML config file. Path defaults to configure/config.yaml, overridable with the CONFIG_PATH environment variable.

monitoring:
  websites:
    hosts:
      - example.com          # polled every 30s — records status code + response time
      - status.example.com
  artifacts:
    catalog:
      url: git/your-catalog-repo   # git repo with an oci-images/ directory
    registry:
      prod:
        host: yourregistry.example.com
      staging:
        host: yourregistry-staging.example.com

Secrets are passed as environment variables — never put tokens in the config file.

Variable	Description
`CATALOG_TOKEN`	Git token for cloning the artifact catalog repo
`PROD_USER`	Registry username (`PROD` = uppercased key from config)
`PROD_TOKEN`	Registry password or token
`STAGING_USER`	Registry username for staging
`STAGING_TOKEN`	Registry password or token for staging

Registry key naming: keys must not contain dots. The app uppercases the key to build the env var name — my.registry becomes MY.REGISTRY, which is not a valid env var name. Use underscores: my_registry → MY_REGISTRY_USER / MY_REGISTRY_TOKEN.

The catalog repo must have an oci-images/ directory containing *.yaml files in this format:

allowed:
  - digest: sha256:abc123...
    image: myapp/backend
    registry: yourregistry.example.com
    tag: v1.2.3

Duplicate digests across files are deduplicated automatically.

Local Docker run

# Build the image
just docker-build

# Run with a local config file
docker run --rm \
  -v $(pwd)/configure/config.yaml:/config.yaml \
  -e CONFIG_PATH=/config.yaml \
  -e CATALOG_TOKEN=your-token \
  -e PROD_USER=your-user \
  -e PROD_TOKEN=your-token \
  -p 8000:8000 \
  container-monitor-go:local

With the container running, check metrics:

curl -s http://localhost:8000/metrics | grep -E 'status_code|loading_time|missing'

Example output:

status_code{host="example.com"} 200
loading_time{host="example.com"} 0.087
number_missing_artifacts{registry="prod"} 0

Kubernetes via Helm (Colima)

The Helm charts live in k8s-container-monitor-helm. The justfile drives the full local workflow.

# 1. Start Colima with k3s
just colima-start

# 2. Configure insecure registry — run once, then restart Colima
just colima-configure
colima stop && just colima-start

# 3. Build the app image and push to the in-cluster registry
just build
just install-registry
just push

# 4. Deploy the monitor
just install-monitor

# 5. Check it's working
just check-metrics

What this sets up:

  ┌────────────────────────────────────────────────────────────┐
  │  Colima VM (k3s)                                           │
  │                                                            │
  │  namespace: registry                                       │
  │  ┌──────────────────────┐                                  │
  │  │  docker-registry     │ ← registry.registry.svc.cluster  │
  │  │  (or zot for prod)   │   .local:5000                    │
  │  └──────────────────────┘                                  │
  │                                                            │
  │  namespace: monitoring                                     │
  │  ┌──────────────────────┐                                  │
  │  │  container-monitor   │ → GET /metrics on :8000          │
  │  │  (3 goroutines)      │                                  │
  │  └──────────────────────┘                                  │
  └────────────────────────────────────────────────────────────┘

Switch to zot for production by passing charts/registry/values-prod.yaml. No template changes needed — registry.type: zot in values is enough.

Metrics reference

All metrics are exposed on GET /metrics in Prometheus text format.

Metric	Type	Labels	Description
`status_code`	Gauge	`host`	HTTP status code of last check
`loading_time`	Gauge	`host`	Response time in seconds of last check
`missing_artifacts`	Counter	`registry`, `image`, `digest`	Artifacts in catalog not found in registry
`number_missing_artifacts`	Gauge	`registry`	Count of missing artifacts per registry
`memory_usage_bytes`	Gauge	`hostname`	Process memory obtained from OS
`http_active_requests`	Gauge	—	Requests currently being served
`http_requests_total`	Counter	`status`, `path`, `method`	Total HTTP requests to this service

/health returns {"status":"ok"} and is used as the Kubernetes liveness probe.

What’s next

Grafana dashboard for missing_artifacts and status_code — the metrics are there, the dashboard isn’t built yet.
Context timeout on artifact resolver — resolveArtifact is currently unbounded. A slow or unreachable registry blocks the goroutine for the full check cycle. A 10s context timeout per digest would contain the blast radius.
Decommission the Python original — the Go version is running in the cluster. The Python app at k8s-container-monitor is kept for reference and will be archived once the Go version has been stable for a few weeks.