gitops: observability stack (Loki/Promtail/Grafana), Grafana Ingress, Argo multi-source
- Add gitops/observability umbrella chart with vendored Helm deps - Grafana Ingress: Traefik, letsencrypt-prod, grafana.k8s.selair.it + root_url - Argo Application: spec.sources (onelab + onelab-obs) - OneLab: configuration secret override, compliance/LDAP values, logs.path /logs - Docs: OBSERVABILITY, BOOTSTRAP, README, instance-overrides example Made-with: Cursor
This commit is contained in:
@@ -47,3 +47,9 @@ kubectl apply -f gitops/argocd/application.yaml
|
||||
## 3. RabbitMQ TLS
|
||||
|
||||
Secret `onelab-rabbit-tls` must exist before RabbitMQ starts (created once from `app/rabbit/ssl/` or your own PEMs).
|
||||
|
||||
## 4. Argo CD version + observability stack
|
||||
|
||||
`gitops/argocd/application.yaml` uses **`spec.sources`** (two Helm charts in one Application). Use **Argo CD 2.6 or newer**.
|
||||
|
||||
The second source installs Loki/Promtail/Grafana from `gitops/observability/` (`releaseName: onelab-obs`). Set a strong **`grafana.adminPassword`** in `gitops/observability/values.yaml` before production. Details: [OBSERVABILITY.md](OBSERVABILITY.md).
|
||||
|
||||
44
gitops/docs/OBSERVABILITY.md
Normal file
44
gitops/docs/OBSERVABILITY.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Observability (Loki / Promtail / Grafana)
|
||||
|
||||
The umbrella chart under [`gitops/observability/`](../observability/) deploys:
|
||||
|
||||
- **Loki** — log storage (SingleBinary, filesystem PVC, 7-day retention by default).
|
||||
- **Promtail** — DaemonSet: Kubernetes pod logs (`/var/log/pods`) plus **OneLab file logs** from the same host path the app chart uses (`/opt/onelab/logs` by default).
|
||||
- **Grafana** — explore logs; datasource points at this release’s Loki gateway.
|
||||
|
||||
It is synced by the **same** Argo CD Application as the OneLab chart ([`gitops/argocd/application.yaml`](../argocd/application.yaml)): second `sources` entry, Helm release name **`onelab-obs`** (so services are like `onelab-obs-loki-gateway`).
|
||||
|
||||
## First-time setup
|
||||
|
||||
1. **Change the Grafana admin password** in [`gitops/observability/values.yaml`](../observability/values.yaml) (`grafana.adminPassword`) or switch to `admin.existingSecret` per the upstream Grafana chart.
|
||||
2. **Align host paths** — if you change `persistence.hostPath.logs` for OneLab, update `promtail.extraVolumes` / `extraVolumeMounts` in the same `values.yaml` so Promtail still reads the shared log directory.
|
||||
3. **Multi-node** — with `hostPath` logs, each node only sees its own files; Promtail runs on every node, so you still get coverage when pods move.
|
||||
|
||||
## Access Grafana
|
||||
|
||||
An **Ingress** is enabled by default (Traefik + cert-manager), matching the OneLab web UI pattern in `gitops/values/k3s-example.yaml`:
|
||||
|
||||
- Host: **`grafana.k8s.selair.it`** (edit in `gitops/observability/values.yaml` alongside `grafana.ini.server` `domain` / `root_url`).
|
||||
- TLS Secret: **`grafana-tls-k8s-selair`** (cert-manager with `letsencrypt-prod`).
|
||||
|
||||
Point DNS at your ingress, sync the app, then open `https://<grafana-host>/` (user `admin` until you change values).
|
||||
|
||||
For debugging without DNS:
|
||||
|
||||
```bash
|
||||
kubectl -n onelab port-forward svc/onelab-obs-grafana 3000:80
|
||||
```
|
||||
|
||||
## Upgrading chart dependencies
|
||||
|
||||
From `gitops/observability/`:
|
||||
|
||||
```bash
|
||||
helm dependency update
|
||||
```
|
||||
|
||||
Commit updated `Chart.lock` and `charts/*.tgz` if you want Argo to render without calling remote Helm repos at sync time.
|
||||
|
||||
## OneLab `logs.path`
|
||||
|
||||
The OneLab chart now sets `onelab.logs.path: "/logs"` in the generated configuration so application file logs match the `/logs` volume mount (see Enterprise guide §7.2).
|
||||
Reference in New Issue
Block a user