Zabbix 7.0 on Kubernetes: Building a Local Test Stack with k3d and Helm

The zabbix-community/helm-zabbix chart deploys a complete Zabbix 7.0 stack on Kubernetes in one command. Getting it to a state where nothing produces spurious errors takes a few more steps. This post covers the full deployment, what broke, and how to reproduce it cleanly.

The Problem

Production Zabbix works well for monitoring real hosts. Pre-production work is harder: template development needs disposable hosts, server config tuning needs a stack you can break, and scale testing needs a controlled environment where you control how many agents connect.

Two specific goals drove this setup:

1. A local Zabbix environment for template and configuration development. Test changes here before touching production.
2. A scale testing platform. The load testing work in the zabbix-agent-sim post required a local stack to drive agents against.

The approach: a k3d cluster running the full Zabbix stack via zabbix-community/helm-zabbix. One Helm chart installs server, web frontend, proxy, PostgreSQL, and agents together.

The Stack

The chart deploys six components into a zabbix namespace:

Component	Notes
PostgreSQL 16	Bundled subchart, 5 Gi PVC
Zabbix Server	Single replica
Zabbix Web	Single replica (nginx)
Zabbix Proxy	Active mode, registers as zabbix-proxy
Zabbix Agent (sidecar)	Runs inside the server pod — required for the default “Zabbix server” host
Zabbix Agent (DaemonSet)	Runs on every node for node-level monitoring

graph LR
    Web[Zabbix Web] --> Server[Zabbix Server]
    Server --> PG[(PostgreSQL 16)]
    Proxy[Zabbix Proxy] --> Server
    AgentSidecar[Agent Sidecar] --> Server
    AgentDS[Agent DaemonSet] --> Server

What Broke

Three separate problems surfaced during the initial deployment, each requiring a different fix.

Beat 1: Received empty response from Zabbix Agent

After the first make zabbix-install, the default “Zabbix server” host showed a red availability indicator in the frontend with Received empty response from Zabbix Agent. The agent pod ran. The port-forward worked. The frontend loaded fine.

The root cause sat in the database. The chart initialises the “Zabbix server” host’s agent interface with port 10052 (the HA port) instead of port 10050 (the agent port). Checking the interface table in PostgreSQL confirmed it:

kubectl exec -n zabbix zabbix-postgresql-0 -- \
  psql -U zabbix -d zabbix -c \
  "SELECT hostid, port FROM interface WHERE type=1;"

The result showed 10052. No agent listens on 10052 in this deployment.

Beat 2: Still broken after the port fix

Patching the port to 10050 fixed the database record but the error persisted. The default “Zabbix server” host checks zabbix-zabbix-server.zabbix.svc (the server’s own ClusterIP service). For that check to pass, an agent must run at that address.

The chart’s runAsSidecar option runs the agent inside the server pod, which means it answers on zabbix-zabbix-server.zabbix.svc:10050. Without runAsSidecar: true in values.yaml, the sidecar does not start, the service has no agent behind it, and the availability check fails.

Beat 3: kubectl stops responding under load

After scaling the agent simulator to several hundred pods, kubectl commands started timing out with TLS errors. Running docker ps -a showed k3d-zabbix-serverlb had stopped with exit code 137 (out of memory). The k3d load balancer container sits in front of the cluster’s API server. When it exits, kubectl cannot reach the cluster.

The Fixes

Port bug — auto-patched by the Makefile. The zabbix-install Makefile target runs a psql UPDATE after every install or upgrade, setting the interface port to 10050. No manual action needed on any fresh deployment.

Agent sidecar — one values.yaml line. Add runAsSidecar: true to zabbixAgent in helmfiles/zabbix/values.yaml and re-run make zabbix-install. The agent starts inside the server pod and answers on zabbix-zabbix-server.zabbix.svc:10050. The default “Zabbix server” host then reports green.

OOM serverlb — two-command recovery. When k3d-zabbix-serverlb dies:

docker start k3d-zabbix-serverlb
k3d kubeconfig write zabbix

For a full cluster restart (not just the load balancer), use k3d cluster stop/start rather than the Makefile make restart target:

k3d cluster stop zabbix
k3d cluster start zabbix

Bonus: rollout restart after extraEnv changes. Helm upgrade does not always restart the server pod when only environment variables change. After any extraEnv update (such as raising ZBX_CACHESIZE), force the restart:

kubectl rollout restart deployment/zabbix-zabbix-server -n zabbix

Quick Start

Prerequisites: k3d, Helm, Colima (or any local Docker runtime), kubectl

1. Clone the repo and start the cluster

git clone https://github.com/Sifungurux/k8s-colima-cluster
cd k8s-colima-cluster
CLUSTER=zabbix make start

2. Create helmfiles/zabbix/secrets.yaml

postgresAccess:
  password: "choose-a-password"

This file is gitignored. A template lives at helmfiles/zabbix/secrets.yaml.example.

3. Install the stack

CLUSTER=zabbix make zabbix-install

This installs the Helm chart, waits for pods to be ready, and auto-patches the agent interface port. First install takes 2-3 minutes for PostgreSQL to initialise.

4. Check status

CLUSTER=zabbix make zabbix-status

Expected: all pods in Running state, services listed with ClusterIP addresses.

5. Access the frontend

kubectl port-forward svc/zabbix-zabbix-web 8888:80 -n zabbix

Open http://localhost:8888, credentials: Admin / zabbix.

Configuration

Configuration lives in two files under helmfiles/zabbix/:

helmfiles/zabbix/
├── values.yaml      # committed — chart config
└── secrets.yaml     # gitignored — DB password only

secrets.yaml (minimum required):

postgresAccess:
  password: "your-password-here"

Key values.yaml settings:

# Zabbix version
zabbixImageTag: ubuntu-7.0.16

# Server — raise CacheSize for scale testing (default 8M crashes at ~50 hosts)
zabbixServer:
  extraEnv:
    - name: ZBX_CACHESIZE
      value: "128M"

# Agent — sidecar required for default "Zabbix server" host check
zabbixAgent:
  enabled: true
  runAsSidecar: true
  runAsDaemonSet: true
  ZBX_SERVER_HOST: 0.0.0.0/0

# Proxy — active mode
zabbixProxy:
  enabled: true
  ZBX_PROXYMODE: 0
  ZBX_HOSTNAME: zabbix-proxy
  ZBX_SERVER_HOST: zabbix-zabbix-server
  ZBX_SERVER_PORT: 10051

ZBX_CACHESIZE: 128M covers approximately 1,300 hosts using the Linux template. The default 8M exhausts shared memory at around 50 hosts and crashes the server with __zbx_shmem_realloc(): out of memory (config cache).

Gotcha Reference

Received empty response from Zabbix Agent on fresh install Cause: the chart writes port 10052 (HA port) to the agent interface instead of 10050. Fix: make zabbix-install auto-patches this. To verify: kubectl exec -n zabbix zabbix-postgresql-0 -- psql -U zabbix -d zabbix -c "SELECT port FROM interface WHERE type=1;" (should return 10050).

Same error after port fix Cause: runAsSidecar: true missing from values.yaml. The default “Zabbix server” host checks zabbix-zabbix-server.zabbix.svc; no agent answers without the sidecar. Fix: add runAsSidecar: true under zabbixAgent in values.yaml, re-run make zabbix-install.

kubectl times out with TLS errors Cause: k3d-zabbix-serverlb OOM-exited. Check with docker ps -a. Fix: docker start k3d-zabbix-serverlb && k3d kubeconfig write zabbix. Full restart: k3d cluster stop zabbix && k3d cluster start zabbix.

extraEnv change has no effect after Helm upgrade Cause: Helm upgrade does not restart the server pod when only env vars change. Fix: kubectl rollout restart deployment/zabbix-zabbix-server -n zabbix.

proxy 'zabbix-proxy' not found spam in server logs Cause: Zabbix has no auto-registration for proxies. The proxy pod connects but the server rejects it until you register it manually. Fix: in the Zabbix frontend, go to Administration, then Proxies, then Create proxy. Set Name to zabbix-proxy and Mode to Active.

Scale Testing

With the stack stable (green availability checks, no log spam, proxy registered), the next question is whether it holds under load. With default settings, it does not. CacheSize=8M exhausts shared memory at around 50 monitored hosts and at 300 hosts the server crashes with __zbx_shmem_realloc(): out of memory (config cache).

The fix (ZBX_CACHESIZE: 128M) is already in values.yaml above. For the full story of scaling to 300 Kubernetes pods, what the crash looked like, and the preStop deregistration hook that keeps the host list clean, see the follow-on post: zabbix-agent-sim: Simulating 300 Zabbix Agents on Kubernetes.