gitops: observability stack (Loki/Promtail/Grafana), Grafana Ingress, Argo multi-source

- Add gitops/observability umbrella chart with vendored Helm deps - Grafana Ingress: Traefik, letsencrypt-prod, grafana.k8s.selair.it + root_url - Argo Application: spec.sources (onelab + onelab-obs) - OneLab: configuration secret override, compliance/LDAP values, logs.path /logs - Docs: OBSERVABILITY, BOOTSTRAP, README, instance-overrides example Made-with: Cursor
2026-03-20 11:10:06 +01:00
parent 9cb1b10d6c
commit b91c35c410
18 changed files with 360 additions and 14 deletions
--- a/gitops/README.md
+++ b/gitops/README.md
@@ -8,7 +8,8 @@ This directory holds the **Helm chart** that replaces `docker stack deploy` from
 |------|---------|
 | `charts/onelab` | Helm chart (StatefulSets, Deployments, Services, ConfigMaps, Secrets) |
 | `values/*.yaml` | Environment-specific overrides (non-secret defaults; use sealed/external secrets for prod) |
-| `argocd/application.yaml` | Example `Application` — set `repoURL` / `targetRevision` to your remote |
+| `argocd/application.yaml` | `Application` (multi-source): OneLab chart + [`observability/`](observability/) (Loki/Promtail/Grafana) |
+| `observability/` | Umbrella Helm chart for log aggregation (same Argo app, release `onelab-obs`) |

 ## Prerequisites

@@ -36,8 +37,14 @@ helm upgrade --install onelab . -n onelab --create-namespace \
 2. Edit `argocd/application.yaml`: `repoURL`, `targetRevision`, and values file as needed.
 3. `kubectl apply -f gitops/argocd/application.yaml` (from a machine with a working kubeconfig).

+The Application uses **`spec.sources`** (Argo CD 2.6+): source 1 is the OneLab chart (`releaseName: onelab`), source 2 is [`observability/`](observability/) (`releaseName: onelab-obs`). Both deploy to namespace **`onelab`**.
+
 Sync waves order Postgres → Redis/Rabbit/config → application pods.

+### Logs / Grafana
+
+See [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md). Change `grafana.adminPassword` in `observability/values.yaml` before relying on it in production.
+
 ## kubectl / credentials

 If `kubectl` reports *You must be logged in*, refresh your kubeconfig (e.g. copy `/etc/rancher/k3s/k3s.yaml` from the server or re-run your auth plugin) before applying manifests.
@@ -50,6 +57,15 @@ See [docs/BOOTSTRAP.md](docs/BOOTSTRAP.md) for Argo CD access to `git.luneski.fr

 Helm 3.19 may return empty content for `.Files.Get` on Windows; this chart uses `fromYaml (.Files.AsConfig)` as a workaround so packaged files still render correctly.

+## Application configuration (`configurations.yml`)
+
+Do **not** need to edit `app/configurations.yml` in Git for Kubernetes. The chart builds `configurations.yml` from `charts/onelab/files/configurations.gotmpl` and stores it in Secret **`onelab-configurations`** (mounted by app pods and `ldap-worker`).
+
+1. **Values (recommended)** — set `onelab.compliance.enabled`, `onelab.ldap.enabled`, and related fields. See `values/instance-overrides.example.yaml`. Point Helm/Argo at an extra values file for your site (Argo: add another path under `spec.source.helm.valueFiles`, relative to the chart directory).
+2. **Bring your own Secret** — set `configuration.existingSecretName` to a Secret you manage (SealedSecrets, External Secrets, `kubectl create secret ... --from-file=configurations.yml=...`). The chart will **not** create `onelab-configurations` in that case; the Secret must contain key **`configurations.yml`**.
+
+A **ConfigMap** alone is fine if you mount it yourself, but this chart expects a **Secret** for the config file (same as Swarm-style sensitivity). LDAP TLS file paths in values are container paths; mount PEMs with extra volumes on `ldap-worker` if you use them.
+
 ## Ingress (web UI)

 Enable `ingress.enabled` and set `ingress.host` (and optional TLS). Traffic is sent to Service **`revproxy`** (internal nginx). On k3s, `ingress.className: traefik` matches the default controller.
--- a/gitops/argocd/application.yaml
+++ b/gitops/argocd/application.yaml
@@ -1,4 +1,5 @@
-# Syncs chart from Git; ensure Argo CD can clone repoURL (add credentials in Argo if private).
+# Syncs OneLab app + observability (Loki/Promtail/Grafana) into namespace onelab.
+# Requires Argo CD 2.6+ (spec.sources). Ensure repoURL matches your remote.
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
@@ -8,13 +9,21 @@ metadata:
    - resources-finalizer.argocd.argoproj.io
 spec:
  project: default
-  source:
-    repoURL: https://git.luneski.fr/luneski/onelab-k8s.git
-    targetRevision: main
-    path: gitops/charts/onelab
-    helm:
-      valueFiles:
-        - ../../values/k3s-example.yaml
+  sources:
+    - repoURL: https://git.luneski.fr/luneski/onelab-k8s.git
+      targetRevision: main
+      path: gitops/charts/onelab
+      helm:
+        releaseName: onelab
+        valueFiles:
+          - ../../values/k3s-example.yaml
+    - repoURL: https://git.luneski.fr/luneski/onelab-k8s.git
+      targetRevision: main
+      path: gitops/observability
+      helm:
+        releaseName: onelab-obs
+        valueFiles:
+          - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: onelab
--- a/gitops/charts/onelab/files/configurations.gotmpl
+++ b/gitops/charts/onelab/files/configurations.gotmpl
@@ -2,6 +2,7 @@
 onelab:
  domain: {{ .Values.onelab.domain | quote }}
  logs:
+    path: "/logs"
    level: info
  assets:
    purge: 1d
@@ -41,6 +42,15 @@ onelab:
      remember_me: true
    lab:
      creation_policy: many
+{{- if .Values.onelab.compliance.enabled }}
+    compliance:
+      require_electronic_signature: {{ .Values.onelab.compliance.requireElectronicSignature }}
+      execution_operator_restriction_policy: {{ .Values.onelab.compliance.executionOperatorRestrictionPolicy | quote }}
+      execution_admin_expert_restriction_policy: {{ .Values.onelab.compliance.executionAdminExpertRestrictionPolicy | quote }}
+      prevent_csv_import: {{ .Values.onelab.compliance.preventCsvImport }}
+      prevent_manual_metadata_edit: {{ .Values.onelab.compliance.preventManualMetadataEdit }}
+      device_restart: {{ .Values.onelab.compliance.deviceRestart }}
+{{- end }}
    signup: false
 {{- if .Values.onelab.intercom.appid }}
    intercom:
@@ -56,7 +66,39 @@ onelab:
      maxtries: 3
      timeout: 60
  ldap:
-    enabled: {{ .Values.features.ldapWorker }}
+    enabled: {{ if or .Values.onelab.ldap.enabled .Values.features.ldapWorker }}true{{ else }}false{{ end }}
+{{- if or .Values.onelab.ldap.enabled .Values.features.ldapWorker }}
+    {{- if .Values.onelab.ldap.timeout }}
+    timeout: {{ .Values.onelab.ldap.timeout | int }}
+    {{- end }}
+    {{- if .Values.onelab.ldap.encryption }}
+    encryption: {{ .Values.onelab.ldap.encryption | quote }}
+    {{- end }}
+    {{- if .Values.onelab.ldap.policy }}
+    policy: {{ .Values.onelab.ldap.policy | quote }}
+    {{- end }}
+    {{- if kindIs "bool" .Values.onelab.ldap.verifyCertificates }}
+    verify_certificates: {{ .Values.onelab.ldap.verifyCertificates }}
+    {{- end }}
+    {{- if or .Values.onelab.ldap.tlsCaPath .Values.onelab.ldap.tlsCertPath .Values.onelab.ldap.tlsKeyPath .Values.onelab.ldap.tlsCiphers .Values.onelab.ldap.tlsSslVersion }}
+    tls:
+      {{- if .Values.onelab.ldap.tlsCaPath }}
+      ca: {{ .Values.onelab.ldap.tlsCaPath | quote }}
+      {{- end }}
+      {{- if .Values.onelab.ldap.tlsCertPath }}
+      cert: {{ .Values.onelab.ldap.tlsCertPath | quote }}
+      {{- end }}
+      {{- if .Values.onelab.ldap.tlsKeyPath }}
+      key: {{ .Values.onelab.ldap.tlsKeyPath | quote }}
+      {{- end }}
+      {{- if .Values.onelab.ldap.tlsCiphers }}
+      ciphers: {{ .Values.onelab.ldap.tlsCiphers | quote }}
+      {{- end }}
+      {{- if .Values.onelab.ldap.tlsSslVersion }}
+      ssl_version: {{ .Values.onelab.ldap.tlsSslVersion | quote }}
+      {{- end }}
+    {{- end }}
+{{- end }}
  services:
    db:
      host: db
--- a/gitops/charts/onelab/templates/_helpers.tpl
+++ b/gitops/charts/onelab/templates/_helpers.tpl
@@ -31,3 +31,7 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
 {{- $entry := dict "username" $user "password" $pass "auth" $auth -}}
 {{- dict "auths" (dict $server $entry) | toJson -}}
 {{- end }}
+
+{{- define "onelab.configurationSecretName" -}}
+{{- .Values.configuration.existingSecretName | default "onelab-configurations" }}
+{{- end }}
--- a/gitops/charts/onelab/templates/deployment-optional-workers.yaml
+++ b/gitops/charts/onelab/templates/deployment-optional-workers.yaml
@@ -1,5 +1,5 @@
 {{- $root := . }}
-{{- if .Values.features.ldapWorker }}
+{{- if or .Values.onelab.ldap.enabled .Values.features.ldapWorker }}
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -43,7 +43,7 @@ spec:
      volumes:
        - name: configurations
          secret:
-            secretName: onelab-configurations
+            secretName: {{ include "onelab.configurationSecretName" $root }}
        {{- if eq $root.Values.persistence.mode "hostPath" }}
        - name: logs
          hostPath:
@@ -98,7 +98,7 @@ spec:
      volumes:
        - name: configurations
          secret:
-            secretName: onelab-configurations
+            secretName: {{ include "onelab.configurationSecretName" $root }}
        {{- if eq $root.Values.persistence.mode "hostPath" }}
        - name: logs
          hostPath:
--- a/gitops/charts/onelab/templates/secret-configurations.yaml
+++ b/gitops/charts/onelab/templates/secret-configurations.yaml
@@ -1,3 +1,4 @@
+{{- if not .Values.configuration.existingSecretName }}
 {{- $cfg := fromYaml (.Files.AsConfig) }}
 apiVersion: v1
 kind: Secret
@@ -11,3 +12,4 @@ type: Opaque
 stringData:
  configurations.yml: |
 {{- tpl (index $cfg "configurations.gotmpl") . | nindent 4 }}
+{{- end }}
--- a/gitops/charts/onelab/templates/workloads.yaml
+++ b/gitops/charts/onelab/templates/workloads.yaml
@@ -76,7 +76,7 @@ spec:
        {{- if .config }}
        - name: configurations
          secret:
-            secretName: onelab-configurations
+            secretName: {{ include "onelab.configurationSecretName" $root }}
        {{- end }}
        {{- if eq $root.Values.persistence.mode "hostPath" }}
        {{- if has "logs" .mounts }}
--- a/gitops/charts/onelab/values.yaml
+++ b/gitops/charts/onelab/values.yaml
@@ -3,6 +3,11 @@
 nameOverride: ""
 fullnameOverride: ""

+# If non-empty, workloads mount this Secret instead of chart-generated onelab-configurations.
+# Secret must contain key `configurations.yml`. Chart will NOT create onelab-configurations.
+configuration:
+  existingSecretName: ""
+
 images:
  registry: hub.andrewalliance.com/releases
  tag: "1.27.0"
@@ -71,11 +76,32 @@ onelab:
    authTokenKey: "TokenAuthPlaceholder"
    monitoringToken: "TokenMonitoringPlaceholder"
    rabbitToken: "TokenRabbitPlaceholder"
+  # Mirrors app/configurations.yml params.compliance (enable without editing app/).
+  compliance:
+    enabled: false
+    requireElectronicSignature: true
+    executionOperatorRestrictionPolicy: "reviewed"
+    executionAdminExpertRestrictionPolicy: "reviewed"
+    preventCsvImport: true
+    preventManualMetadataEdit: true
+    deviceRestart: true
+  # Set enabled: true to turn on LDAP in configurations.yml and deploy ldap-worker (or use features.ldapWorker).
+  ldap:
+    enabled: false
+    timeout: ""
+    encryption: ""
+    policy: ""
+    tlsCaPath: ""
+    tlsCertPath: ""
+    tlsKeyPath: ""
+    tlsCiphers: ""
+    tlsSslVersion: ""
  intercom:
    appid: "zxvgsagz"
    secret: "QUw2jEV8utIpe9DeYjOqBjhBY9VxjXddKUCISUNu"

 features:
+  # Deprecated for LDAP: prefer onelab.ldap.enabled (either enables ldap-worker + ldap.enabled in config).
  ldapWorker: false
  mailerWorker: false

--- a/gitops/docs/BOOTSTRAP.md
+++ b/gitops/docs/BOOTSTRAP.md
@@ -47,3 +47,9 @@ kubectl apply -f gitops/argocd/application.yaml
 ## 3. RabbitMQ TLS

 Secret `onelab-rabbit-tls` must exist before RabbitMQ starts (created once from `app/rabbit/ssl/` or your own PEMs).
+
+## 4. Argo CD version + observability stack
+
+`gitops/argocd/application.yaml` uses **`spec.sources`** (two Helm charts in one Application). Use **Argo CD 2.6 or newer**.
+
+The second source installs Loki/Promtail/Grafana from `gitops/observability/` (`releaseName: onelab-obs`). Set a strong **`grafana.adminPassword`** in `gitops/observability/values.yaml` before production. Details: [OBSERVABILITY.md](OBSERVABILITY.md).
--- a/gitops/docs/OBSERVABILITY.md
+++ b/gitops/docs/OBSERVABILITY.md
@@ -0,0 +1,44 @@
+# Observability (Loki / Promtail / Grafana)
+
+The umbrella chart under [`gitops/observability/`](../observability/) deploys:
+
+- **Loki** — log storage (SingleBinary, filesystem PVC, 7-day retention by default).
+- **Promtail** — DaemonSet: Kubernetes pod logs (`/var/log/pods`) plus **OneLab file logs** from the same host path the app chart uses (`/opt/onelab/logs` by default).
+- **Grafana** — explore logs; datasource points at this release’s Loki gateway.
+
+It is synced by the **same** Argo CD Application as the OneLab chart ([`gitops/argocd/application.yaml`](../argocd/application.yaml)): second `sources` entry, Helm release name **`onelab-obs`** (so services are like `onelab-obs-loki-gateway`).
+
+## First-time setup
+
+1. **Change the Grafana admin password** in [`gitops/observability/values.yaml`](../observability/values.yaml) (`grafana.adminPassword`) or switch to `admin.existingSecret` per the upstream Grafana chart.
+2. **Align host paths** — if you change `persistence.hostPath.logs` for OneLab, update `promtail.extraVolumes` / `extraVolumeMounts` in the same `values.yaml` so Promtail still reads the shared log directory.
+3. **Multi-node** — with `hostPath` logs, each node only sees its own files; Promtail runs on every node, so you still get coverage when pods move.
+
+## Access Grafana
+
+An **Ingress** is enabled by default (Traefik + cert-manager), matching the OneLab web UI pattern in `gitops/values/k3s-example.yaml`:
+
+- Host: **`grafana.k8s.selair.it`** (edit in `gitops/observability/values.yaml` alongside `grafana.ini.server` `domain` / `root_url`).
+- TLS Secret: **`grafana-tls-k8s-selair`** (cert-manager with `letsencrypt-prod`).
+
+Point DNS at your ingress, sync the app, then open `https://<grafana-host>/` (user `admin` until you change values).
+
+For debugging without DNS:
+
+```bash
+kubectl -n onelab port-forward svc/onelab-obs-grafana 3000:80
+```
+
+## Upgrading chart dependencies
+
+From `gitops/observability/`:
+
+```bash
+helm dependency update
+```
+
+Commit updated `Chart.lock` and `charts/*.tgz` if you want Argo to render without calling remote Helm repos at sync time.
+
+## OneLab `logs.path`
+
+The OneLab chart now sets `onelab.logs.path: "/logs"` in the generated configuration so application file logs match the `/logs` volume mount (see Enterprise guide §7.2).
--- a/gitops/observability/Chart.lock
+++ b/gitops/observability/Chart.lock
@@ -0,0 +1,12 @@
+dependencies:
+- name: loki
+  repository: https://grafana.github.io/helm-charts
+  version: 6.55.0
+- name: promtail
+  repository: https://grafana.github.io/helm-charts
+  version: 6.17.1
+- name: grafana
+  repository: https://grafana.github.io/helm-charts
+  version: 10.5.15
+digest: sha256:5b34192a8db9d940587777fbc62a13503c21217da814308654ce73fca2ed5d56
+generated: "2026-03-20T11:06:47.9376325+01:00"
--- a/gitops/observability/Chart.yaml
+++ b/gitops/observability/Chart.yaml
@@ -0,0 +1,16 @@
+apiVersion: v2
+name: onelab-observability
+description: Loki + Promtail + Grafana for OneLab (same Argo Application as app chart via multi-source).
+type: application
+version: 0.1.0
+appVersion: "1.0"
+dependencies:
+  - name: loki
+    version: 6.55.0
+    repository: https://grafana.github.io/helm-charts
+  - name: promtail
+    version: 6.17.1
+    repository: https://grafana.github.io/helm-charts
+  - name: grafana
+    version: 10.5.15
+    repository: https://grafana.github.io/helm-charts
--- a/gitops/observability/README.md
+++ b/gitops/observability/README.md
@@ -0,0 +1,7 @@
+# OneLab observability (Helm umbrella)
+
+Loki + Promtail + Grafana dependencies are pinned in `Chart.lock`; packaged charts live in `charts/*.tgz`.
+
+Deployed by Argo CD as the second `sources` entry in `gitops/argocd/application.yaml` with **`releaseName: onelab-obs`**.
+
+See [../docs/OBSERVABILITY.md](../docs/OBSERVABILITY.md) for operations and security notes.
--- a/gitops/observability/charts/grafana-10.5.15.tgz
+++ b/gitops/observability/charts/grafana-10.5.15.tgz
--- a/gitops/observability/charts/loki-6.55.0.tgz
+++ b/gitops/observability/charts/loki-6.55.0.tgz
--- a/gitops/observability/charts/promtail-6.17.1.tgz
+++ b/gitops/observability/charts/promtail-6.17.1.tgz
--- a/gitops/observability/values.yaml
+++ b/gitops/observability/values.yaml
@@ -0,0 +1,128 @@
+# Umbrella chart: Loki (SingleBinary + filesystem) + Promtail + Grafana.
+# Keep hostPath below in sync with persistence.hostPath.logs in gitops/values/k3s-example.yaml.
+
+loki:
+  deploymentMode: SingleBinary
+  loki:
+    auth_enabled: false
+    commonConfig:
+      replication_factor: 1
+    storage:
+      type: filesystem
+    schemaConfig:
+      configs:
+        - from: "2024-04-01"
+          store: tsdb
+          object_store: filesystem
+          schema: v13
+          index:
+            prefix: loki_index_
+            period: 24h
+    limits_config:
+      retention_period: 168h
+      ingestion_rate_mb: 16
+      ingestion_burst_size_mb: 32
+  singleBinary:
+    replicas: 1
+    persistence:
+      enabled: true
+      size: 10Gi
+  backend:
+    replicas: 0
+  read:
+    replicas: 0
+  write:
+    replicas: 0
+  ingester:
+    replicas: 0
+  querier:
+    replicas: 0
+  queryFrontend:
+    replicas: 0
+  queryScheduler:
+    replicas: 0
+  distributor:
+    replicas: 0
+  compactor:
+    replicas: 0
+  indexGateway:
+    replicas: 0
+  bloomCompactor:
+    replicas: 0
+  bloomGateway:
+    replicas: 0
+  ruler:
+    replicas: 0
+  minio:
+    enabled: false
+  lokiCanary:
+    enabled: false
+  test:
+    enabled: false
+  chunksCache:
+    enabled: false
+  resultsCache:
+    enabled: false
+
+promtail:
+  config:
+    clients:
+      - url: http://{{ .Release.Name }}-loki-gateway.{{ .Release.Namespace }}.svc.cluster.local/loki/api/v1/push
+    snippets:
+      extraScrapeConfigs: |
+        - job_name: onelab-host-log-files
+          static_configs:
+            - targets:
+                - localhost
+              labels:
+                job: onelab-files
+                __path__: /onelab-host-logs/**/*
+  extraVolumes:
+    - name: onelab-host-logs
+      hostPath:
+        path: /opt/onelab/logs
+        type: DirectoryOrCreate
+  extraVolumeMounts:
+    - name: onelab-host-logs
+      mountPath: /onelab-host-logs
+      readOnly: true
+
+# Grafana Ingress: align host/TLS with gitops/values/k3s-example.yaml ingress (Traefik + cert-manager).
+grafana:
+  adminUser: admin
+  adminPassword: changeme
+  persistence:
+    enabled: true
+    size: 2Gi
+  service:
+    type: ClusterIP
+  # Required when served behind Ingress (redirects, OAuth callbacks).
+  grafana.ini:
+    server:
+      domain: grafana.k8s.selair.it
+      root_url: https://grafana.k8s.selair.it/
+  ingress:
+    enabled: true
+    ingressClassName: traefik
+    annotations:
+      cert-manager.io/cluster-issuer: letsencrypt-prod
+    hosts:
+      - grafana.k8s.selair.it
+    path: /
+    pathType: Prefix
+    tls:
+      - secretName: grafana-tls-k8s-selair
+        hosts:
+          - grafana.k8s.selair.it
+  datasources:
+    datasources.yaml:
+      apiVersion: 1
+      datasources:
+        - name: Loki
+          type: loki
+          uid: loki
+          url: http://{{ .Release.Name }}-loki-gateway.{{ .Release.Namespace }}.svc.cluster.local
+          access: proxy
+          isDefault: true
+          jsonData:
+            maxLines: 1000
--- a/gitops/values/instance-overrides.example.yaml
+++ b/gitops/values/instance-overrides.example.yaml
@@ -0,0 +1,34 @@
+# Copy to a private file (e.g. gitops/values/private-k3s.yaml, gitignored) or merge into your env values.
+# Reference from Helm:  -f ../../values/k3s-example.yaml -f ../../values/private-k3s.yaml
+# Argo CD: add a second entry under helm.valueFiles (paths relative to chart path).
+
+onelab:
+  compliance:
+    enabled: true
+    # Optional tweaks (defaults match chart values.yaml):
+    # requireElectronicSignature: true
+    # executionOperatorRestrictionPolicy: "reviewed"
+    # executionAdminExpertRestrictionPolicy: "reviewed"
+    # preventCsvImport: true
+    # preventManualMetadataEdit: true
+    # deviceRestart: true
+
+  ldap:
+    enabled: true
+    # timeout: 30
+    # encryption: "start_tls"
+    # policy: "your-policy"
+    # verifyCertificates: true
+    # Paths inside the ldap-worker container (mount certs via extraVolumes if needed):
+    # tlsCaPath: "/ldap/ca.crt"
+    # tlsCertPath: "/ldap/client.crt"
+    # tlsKeyPath: "/ldap/client.key"
+    # tlsCiphers: ""
+    # tlsSslVersion: ""
+
+# Alternative: supply the full YAML yourself (no Helm templating of compliance/LDAP blocks).
+# 1. kubectl create secret generic onelab-configurations-custom -n onelab \
+#      --from-file=configurations.yml=./my-configurations.yml
+# 2. Set in values:
+# configuration:
+#   existingSecretName: onelab-configurations-custom