Context
What looked like an autoscaling project quickly became a question about reliability and operational behavior: when is a job really done, and which metric should KEDA watch?
Approach
The comparison puts an always-on Go HTTP service for classic MinIO events next to a Go worker on k3s. MinIO publishes serverless events to NATS JetStream; KEDA watches consumer lag and scales the worker.
System
- MinIO classic webhook → Go HTTP service as the always-on baseline
- MinIO event → NATS JetStream stream and pull consumer
- KEDA: scale workers from 0 to 10 replicas using JetStream lag
- Go worker + cwebp: normalize images and produce WebP
- MinIO + PostgreSQL: store output, write metadata, then delete the raw object
- Prometheus + Grafana: observe behavior
Engineering decisions
- ACK only after the object upload, database insert and raw-object deletion succeed.
- Scale KEDA from JetStream consumer lag, not the number of objects in the raw bucket.
- Compare scale-to-zero under load with the always-on HTTP baseline, observing both through Prometheus and Grafana.
What did not work
Scale-to-zero is not the whole problem. Scale-down, message acknowledgement, worker failures and a fair comparison with always-on determine whether the system behaves reliably.
Result
The worker normalizes the image, produces WebP, uploads it to MinIO, writes a database row and deletes the raw object. Only then does it acknowledge the NATS message.
What stayed with me
A worker is not done when its function returns. What matters is whether every downstream effect is complete before the message is acknowledged.