- Root cause: live Application kept spec.source; Argo ignored observability chart - Add jsonpatch-multisource.json + argocd/README.md migration steps - Grafana: disable subchart ingress; add templates/ingress-grafana-onelab.yaml Made-with: Cursor
2.3 KiB
Observability (Loki / Promtail / Grafana)
The umbrella chart under gitops/observability/ deploys:
- Loki — log storage (SingleBinary, filesystem PVC, 7-day retention by default).
- Promtail — DaemonSet: Kubernetes pod logs (
/var/log/pods) plus OneLab file logs from the same host path the app chart uses (/opt/onelab/logsby default). - Grafana — explore logs; datasource points at this release’s Loki gateway.
It is synced by the same Argo CD Application as the OneLab chart (gitops/argocd/application.yaml): second sources entry, Helm release name onelab-obs (so services are like onelab-obs-loki-gateway).
First-time setup
- Change the Grafana admin password in
gitops/observability/values.yaml(grafana.adminPassword) or switch toadmin.existingSecretper the upstream Grafana chart. - Align host paths — if you change
persistence.hostPath.logsfor OneLab, updatepromtail.extraVolumes/extraVolumeMountsin the samevalues.yamlso Promtail still reads the shared log directory. - Multi-node — with
hostPathlogs, each node only sees its own files; Promtail runs on every node, so you still get coverage when pods move.
Access Grafana
An Ingress named grafana-onelab is created by the umbrella chart (templates/ingress-grafana-onelab.yaml), Traefik + cert-manager, matching the OneLab web UI pattern in gitops/values/k3s-example.yaml:
- Host:
grafana.k8s.selair.it— editgrafanaOnelabIngressandgrafana.ini.serveringitops/observability/values.yamltogether. - TLS Secret:
grafana-tls-k8s-selair(cert-manager withletsencrypt-prod).
Point DNS at your ingress, sync the app, then open https://<grafana-host>/ (user admin until you change values).
For debugging without DNS:
kubectl -n onelab port-forward svc/onelab-obs-grafana 3000:80
Upgrading chart dependencies
From gitops/observability/:
helm dependency update
Commit updated Chart.lock and charts/*.tgz if you want Argo to render without calling remote Helm repos at sync time.
OneLab logs.path
The OneLab chart now sets onelab.logs.path: "/logs" in the generated configuration so application file logs match the /logs volume mount (see Enterprise guide §7.2).