#zabbix #kubernetes #go #observability #k3d

zabbix-agent-sim: Simulating 300 Zabbix Agents on Kubernetes

18 May 2026

A Kubernetes-based Zabbix agent simulator: one image that auto-registers as a Zabbix host, scales to 300 pods with a single command, and cleans up after itself on shutdown. This is the story of building it, what broke at scale, and how the fixes worked.

The problem

Testing Zabbix configuration, templates, and server limits requires monitored hosts. In production that’s fine — the hosts already exist. For pre-production work, template development, or stress-testing the server itself, you need hosts that don’t exist yet.

The options aren’t great:

Spin up VMs: slow, expensive, requires cleanup.
Use staging hosts: ties up real infrastructure, can’t scale past what you have.
Simulate them: fast, cheap, disposable, scales to whatever Kubernetes allows.

zabbix-agent-sim is the third option. Each pod runs a real Zabbix Agent 2 and a Go metrics server that reads actual /proc data. The agent auto-registers with Zabbix using the pod name as hostname. Scale to 300 with one command.

It started as a way to verify a template change wouldn’t break anything at scale. It ended up teaching me exactly where Zabbix breaks.

Architecture

Each pod runs two processes managed by entrypoint.sh. zabbix_agent2 handles the actual monitoring connection; metrics-server is a Go binary that serves real system data from /proc.

Pod
├── zabbix_agent2        Active mode → pushes to zabbix-zabbix-server.zabbix.svc:10051
│                        Hostname    = pod name (injected via fieldRef: metadata.name)
│                        HostMetadata = "zabbix-agent-sim" (triggers auto-registration)
│                        Passive: Server=0.0.0.0/0, port 10050 (availability check)
└── metrics-server       HTTP on :8080
                         GET /health  → {"status":"ok","hostname":"<pod-name>"}
                         GET /metrics → cpu_usage_percent, memory_used_mb, disk_*, uptime_seconds
                         Sources: /proc/stat, /proc/meminfo, /proc/uptime, syscall.Statfs

entrypoint.sh generates the Zabbix Agent 2 config at runtime from environment variables. It guards the hostname with : "${HOSTNAME:?HOSTNAME must be set}" — if the fieldRef injection is missing from the deployment manifest, the pod fails immediately with a clear error rather than registering with a blank hostname and silently polluting the host list.

A liveness probe on GET /health :8080 catches metrics-server crashes. zabbix_agent2 runs as PID 1 and doesn’t monitor background processes — without the probe, a crashed metrics-server goes undetected and the pod keeps reporting healthy.

Multi-stage Dockerfile: Go binary compiled in golang:1.22-alpine, copied into alpine:3.19 alongside zabbix-agent2. The image is loaded into k3d with k3d image import — no registry needed.

One-time Zabbix setup

Two configuration steps are required before any pods will auto-register. These are one-time operations per Zabbix instance.

Port-forward to the Zabbix web UI:

kubectl port-forward svc/zabbix-zabbix-web 8888:80 -n zabbix

Step 1: Disable auto-registration encryption

Open http://localhost:8888/zabbix.php?action=autoreg.edit → set encryption to No encryption → Update.

The direct URL is intentional — the menu path for this setting moved in Zabbix 7.0 and isn’t where the documentation says it is.

Step 2: Create an auto-registration action

Open http://localhost:8888/actionconf.php?eventsource=2 → Create action:

Field	Value
Name	`Register simulated agents`
Condition	Host metadata contains `zabbix-agent-sim`
Operations	Add host · Add to host group `Simulated Agents` · Link template `Linux by Zabbix agent active`

New pods register as Zabbix hosts within approximately 60 seconds of starting.

Tutorial: Deploy the simulator

Prerequisites

k3d cluster named zabbix running
Zabbix stack deployed in the zabbix namespace
One-time setup from the previous section completed
jq installed on the host (used by the deregistration script)

Create the API credentials Secret

The preStop deregistration hook authenticates with the Zabbix API to delete the pod’s host on shutdown. Create the Secret before deploying — if it’s missing, the hook fails silently and pods leave stale hosts behind.

kubectl create secret generic zabbix-api-credentials \
  --from-literal=username=Admin \
  --from-literal=password=zabbix \
  --namespace=zabbix

Update the password if the Zabbix Admin password has been changed from the default.

Deploy

make all        # build Go binary + Docker image → k3d image import → kubectl apply
make status     # show running pods + live /metrics from the first pod

make all runs three steps: build (compiles the Go binary and builds the Docker image), import (loads the image into the k3d cluster without a registry), and deploy (applies the Kubernetes manifests).

Scale and verify

make scale N=5

Wait approximately 60 seconds, then open Monitoring → Hosts in Zabbix. Five hosts named after the pod names should appear, with Linux by Zabbix agent active template applied.

To check metrics directly from a pod:

kubectl exec -n zabbix <pod-name> -- wget -qO- http://localhost:8080/metrics

Scale testing

With the simulator running at small scale, the obvious next step is to push it. Here’s what happened.

Small scale works fine. One pod registers in ~60 seconds. Five, twenty, fifty — they all register, data flows, templates apply, triggers evaluate. Everything looks normal.

The scale ceiling isn’t what you’d expect. k3d runs k3s with a default cap of 110 pods per node. A three-node k3d cluster has room for roughly 330 pods once system pods are accounted for. That’s the limit — not the subnet size (/24 gives 762 IPs) and not available memory. Running make scale N=300 gets close to that ceiling.

At 300 hosts, the Zabbix server crashes.

CrashLoopBackOff

Logs:

__zbx_shmem_realloc(): out of memory

The config syncer runs out of shared memory. The default CacheSize=8M handles roughly 50 hosts. 300 hosts with the Linux by Zabbix agent active template need approximately 100KB of cache each — that’s 30MB minimum, and the default is 8MB.

The fix is one line in the Helm values:

zabbixServer:
  extraEnv:
    ZBX_CACHESIZE: 128M

128MB covers approximately 1,300 hosts. After upgrading, trigger a rollout — the old pod keeps running otherwise:

kubectl rollout restart deployment/zabbix-zabbix-server -n zabbix

Scale-down doesn’t clean up Zabbix hosts. This was the other discovery. Scale 5→3 and the two terminated pods disappear from Kubernetes immediately — but they stay in Zabbix, showing “Not available.” After several test runs at 100–300 pods, the host list contained 603 orphan entries from previous tests. A one-shot kubectl exec script bulk-deleted them. A permanent fix was needed.

The fix is a lifecycle.preStop hook. Before Kubernetes terminates the pod, deregister.sh runs:

Reads ZABBIX_USER / ZABBIX_PASSWORD from the zabbix-api-credentials Secret
Calls user.login to get an auth token
Calls host.get to find the host by $HOSTNAME
Calls host.delete with the host ID

jq handles the JSON parsing — added to the Alpine image because shell-based JSON parsing with grep and awk was unreliable against the Zabbix API response format.

Verified: scale 5→3 and two hosts disappear from Zabbix within seconds.

Tutorial: Stress testing and tuning

Scale up incrementally

make scale N=20
make scale N=100
make scale N=300

After each step, verify the Zabbix server is still running:

kubectl get pods -n zabbix -l app=zabbix-server

Fix CacheSize if the server crashes

If the server pod enters CrashLoopBackOff, check the logs:

kubectl logs -n zabbix deployment/zabbix-zabbix-server --previous | grep shmem

If you see __zbx_shmem_realloc(): out of memory, add ZBX_CACHESIZE: 128M to the Zabbix server Helm values and upgrade with an extended timeout:

helm upgrade zabbix zabbix/zabbix \
  -f helmfiles/zabbix/values.yaml \
  --timeout 10m \
  --namespace zabbix
kubectl rollout restart deployment/zabbix-zabbix-server -n zabbix

The 10-minute timeout is necessary when the server is crash-looping — the default 5-minute timeout expires before the upgrade completes.

Verify auto-deregistration

make scale N=5
# wait ~60s for all 5 to appear in Zabbix
make scale N=3
# within ~30 seconds, 2 hosts disappear from Zabbix

Scale to zero when done

make scale N=0

All pods terminate, all hosts deregister via the preStop hook. The Zabbix host list returns to zero.

Gotchas

These are specific to k3d/Colima. A real cluster won’t have the serverlb or Colima issues, but the Helm timeout and rollout gotchas apply anywhere.

Gotcha	What happened	Fix
Scale-down leaves stale Zabbix hosts	Pods terminate but hosts stay in Zabbix showing “Not available”. 603 orphans accumulated across test runs.	Deploy with `lifecycle.preStop` hook. Create `zabbix-api-credentials` Secret before scaling.
k3d serverlb OOM at 300 pods	`k3d-zabbix-serverlb` container exits 137 (OOM) under 300-pod load	`docker start k3d-zabbix-serverlb && k3d kubeconfig write zabbix`
`make restart` doesn’t cleanly restart k3d	Uses Colima stop/start which freezes k3d containers without stopping the cluster properly	`k3d cluster stop zabbix && k3d cluster start zabbix`
Helm upgrade timeout during CrashLoop	Default 5m timeout not enough when server is crash-looping	Run `helm upgrade --timeout 10m` directly instead of `make zabbix-install`
Old Zabbix server pod keeps running after Helm upgrade	Upgrade succeeds but doesn’t trigger a rollout automatically	`kubectl rollout restart deployment/zabbix-zabbix-server -n zabbix`
Zabbix API `"auth"` field in 7.x	Zabbix 7.0 JSON-RPC uses a token from `user.login` in the `"auth"` field. Some older 6.x docs show different patterns.	Use `"auth": "<token>"` in every subsequent request body after `user.login`.

What’s next

readinessProbe — currently omitted. At scale, the metrics-server startup time may need a grace period before the pod accepts traffic from the availability check. Adding it once the scale behaviour at 100+ pods is better understood.
Prometheus scrape of /metrics — the Go metrics server exposes real /proc data on port 8080. No ServiceMonitor or scrape config exists yet. The data is there.
Use against a production Zabbix backup — the simulator is useful for validating template behaviour before upgrades, not just stress testing. Clone the production database, spin up an isolated Zabbix instance, register 50 simulated hosts, and test the upgrade path before touching production.