We were evaluating a Loki-based logging stack on Kubernetes for a new environment, pushing real data through it end to end rather than trusting the dashboards. Every pod Running. Zero restarts. Every dashboard green. When we sat down to validate the storage layer, confirming the data was actually landing rather than just that the pods were healthy, the object store had zero log entries for the entire run.
This is a write-up of two bugs found during that evaluation. They were independent and had unrelated fixes, but they shared one shape: at the layer we were watching, a tool reported success while the real work failed at a layer we weren't. Helm rendered a clean template, and the config it produced did nothing. Pods ran without restarting, and the writes they were supposed to make never landed. The first bug eventually crash-looped, so it announced itself in the end. The second never did. That second one is where the title lands: zero surface errors, zero objects, for the entire run. In distributed systems, "no error" and "working" are separate claims, and treating them as one is how a stack passes every surface check while its actual job silently fails.
Part 1: Failures with no vocabulary to report themselves
The first bug was not algorithmically hard. It came down to a single line of config in the wrong place.
A Helm key that rendered clean and did nothing. S3 credentials for Loki went into a Secret, referenced under loki.extraEnvFrom. helm template rendered without complaint. The install succeeded. Then the pod crash-looped: no EC2 IMDS role found. The AWS SDK had walked its entire credential chain (environment variables, then config files, then instance metadata, the IMDS lookup) and found nothing, because the key belonged under global:, not loki:. Helm does not validate unrecognized keys; it accepts them and silently drops them. Valid YAML in the wrong place renders perfectly and does nothing.
Part 2: The same failure mode, at greater scale
The question driving the next phase of testing was simple: was Loki actually storing anything? It turned into exactly the same class of bug, this time invisible at every layer that is supposed to catch it.
Surface state: all three loki-write pods Running, zero restarts, no CrashLoopBackOff, nothing in kubectl get pods to act on. Nothing about the deployment looked like it needed attention.
What grepping the logs found: each pod was steadily emitting the same error, on the order of a thousand-plus occurrences every couple of hours: PutObject returned 501 NotImplemented, AWS chunked encoding not supported. The status code mattered. A 403 would point at credentials and a 404 at a missing bucket, but 501 means the server understood the request and does not implement that request shape. That points at the client's format, not at config or permissions.
Confirming it was real data loss, not backlog: listing the target bucket returned an empty prefix list, and the write path had never once succeeded across the entire period it had been running. The write PVCs (the local disks that buffer data before upload) sat at 0.1% used. If chunks were queuing locally waiting for storage to recover, disk usage would climb. It was not climbing. Writes were retried roughly eight times per chunk and then discarded. Active loss, not backlog: every chunk the write path produced was rejected and dropped, and the bucket held zero objects at the end of the run.
Root cause: grafana/loki:3.6.11 bundles an AWS SDK version that unconditionally wraps uploads in checksummed aws-chunked encoding. Real AWS S3 supports this; the OCI S3-compatible endpoint behind our bucket does not. We had already set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on these pods as a general precaution against this class of SDK default, and verified it against the aws CLI, which respects that variable. It made zero difference here: Loki's embedded Go client does not read it. Two components sharing a symptom and an error string, with unrelated fixes.
The path forward: rather than keep chasing environment variables against a client that was not respecting them, we switched to Loki's Thanos-based object store client, which is built on minio-go instead of the AWS SDK. Enabling it (loki.storage.use_thanos_objstore: true, plus an object_store block for endpoint, region, and TLS) meant reckoning with two schema differences from the legacy s3.endpoint config: the new block wants a bare hostname with no scheme prefix, and it drops s3ForcePathStyle entirely, with bucket_name auto-populated from the existing bucketNames. That client simply does not default into the same checksum and chunked-encoding path, and the errors stopped.
The verification, not the fix, is the lesson
Two bugs, one underlying failure: a component silently does the wrong thing while every surface signal (pod status, exit codes, existing environment variables) shows green. Each fix was specific and, in hindsight, small. The expensive part is the time to notice. Here it ran for roughly 36 hours, and over that window the write path produced an estimated 300-plus objects that should have landed in the bucket; every one was rejected and dropped, and the objsect count stayed at zero. Until a write is confirmed in storage, the loss runs at line rate and nothing on the dashboard shows it. The transferable part is the checklist that finds these before they cost days:
A clean helm template or terraform plan is not evidence a key landed. Grep the rendered output for the exact field you expect.
A CLI tool succeeding against a backend proves that the CLI's code path works, not that your application's embedded client does. Different HTTP client, different defaults, different bugs.
"Waiting" and "stuck" look identical from kubectl get. The only way to tell them apart is knowing, precisely, the condition being waited on.
For anything storage-related, the only proof of a successful write is an object count on the other end. Application logs, SDK exit codes, and CLI smoke tests all stop one layer short of that.
None of this is exotic. It is the difference between infrastructure that appears correct and infrastructure that has been made to prove it.