gitops: observability stack (Loki/Promtail/Grafana), Grafana Ingress, Argo multi-source

- Add gitops/observability umbrella chart with vendored Helm deps
- Grafana Ingress: Traefik, letsencrypt-prod, grafana.k8s.selair.it + root_url
- Argo Application: spec.sources (onelab + onelab-obs)
- OneLab: configuration secret override, compliance/LDAP values, logs.path /logs
- Docs: OBSERVABILITY, BOOTSTRAP, README, instance-overrides example

Made-with: Cursor
This commit is contained in:
timotheereausanofi
2026-03-20 11:10:06 +01:00
parent 9cb1b10d6c
commit b91c35c410
18 changed files with 360 additions and 14 deletions

View File

@@ -0,0 +1,44 @@
# Observability (Loki / Promtail / Grafana)
The umbrella chart under [`gitops/observability/`](../observability/) deploys:
- **Loki** — log storage (SingleBinary, filesystem PVC, 7-day retention by default).
- **Promtail** — DaemonSet: Kubernetes pod logs (`/var/log/pods`) plus **OneLab file logs** from the same host path the app chart uses (`/opt/onelab/logs` by default).
- **Grafana** — explore logs; datasource points at this releases Loki gateway.
It is synced by the **same** Argo CD Application as the OneLab chart ([`gitops/argocd/application.yaml`](../argocd/application.yaml)): second `sources` entry, Helm release name **`onelab-obs`** (so services are like `onelab-obs-loki-gateway`).
## First-time setup
1. **Change the Grafana admin password** in [`gitops/observability/values.yaml`](../observability/values.yaml) (`grafana.adminPassword`) or switch to `admin.existingSecret` per the upstream Grafana chart.
2. **Align host paths** — if you change `persistence.hostPath.logs` for OneLab, update `promtail.extraVolumes` / `extraVolumeMounts` in the same `values.yaml` so Promtail still reads the shared log directory.
3. **Multi-node** — with `hostPath` logs, each node only sees its own files; Promtail runs on every node, so you still get coverage when pods move.
## Access Grafana
An **Ingress** is enabled by default (Traefik + cert-manager), matching the OneLab web UI pattern in `gitops/values/k3s-example.yaml`:
- Host: **`grafana.k8s.selair.it`** (edit in `gitops/observability/values.yaml` alongside `grafana.ini.server` `domain` / `root_url`).
- TLS Secret: **`grafana-tls-k8s-selair`** (cert-manager with `letsencrypt-prod`).
Point DNS at your ingress, sync the app, then open `https://<grafana-host>/` (user `admin` until you change values).
For debugging without DNS:
```bash
kubectl -n onelab port-forward svc/onelab-obs-grafana 3000:80
```
## Upgrading chart dependencies
From `gitops/observability/`:
```bash
helm dependency update
```
Commit updated `Chart.lock` and `charts/*.tgz` if you want Argo to render without calling remote Helm repos at sync time.
## OneLab `logs.path`
The OneLab chart now sets `onelab.logs.path: "/logs"` in the generated configuration so application file logs match the `/logs` volume mount (see Enterprise guide §7.2).