OneLab Kubernetes GitOps (Argo CD)
- Helm charts: onelab app + observability (Loki/Promtail/Grafana) - Values under gitops/values/ with public-safe placeholders - Argo CD Application (spec.sources, 2.6+) Made-with: Cursor
This commit is contained in:
timotheereausanofi
223
gitops/README.md
Normal file
223
gitops/README.md
Normal file
@@ -0,0 +1,223 @@
|
||||
# OneLab GitOps (Argo CD)
|
||||
|
||||
This directory is the **declarative source** for OneLab on Kubernetes. [Argo CD](https://argo-cd.readthedocs.io/) applies **two Helm-based sources** from Git (Argo invokes Helm; you do not use Helm CLI as the primary install path for the cluster).
|
||||
|
||||
This repository is **GitOps-only**: everything required to deploy OneLab on Kubernetes lives under **`gitops/`**.
|
||||
|
||||
## What gets deployed
|
||||
|
||||
| Helm release | Chart path | Namespace | Role |
|
||||
|--------------|------------|-----------|------|
|
||||
| `onelab` | [`charts/onelab`](charts/onelab) | `onelab` | Application workloads, Postgres, Redis, RabbitMQ, revproxy, ingress hooks |
|
||||
| `onelab-obs` | [`observability/`](observability/) | `onelab` | Loki, Promtail, Grafana (logs) |
|
||||
|
||||
Both are wired by a single Argo CD [`Application`](argocd/application.yaml) using **`spec.sources`** (requires **Argo CD 2.6+**).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph git [Git repository]
|
||||
V[gitops/values]
|
||||
C[charts/onelab]
|
||||
O[observability]
|
||||
end
|
||||
subgraph argo [Argo CD]
|
||||
App[Application onelab]
|
||||
end
|
||||
subgraph cluster [Kubernetes cluster]
|
||||
NS[namespace onelab]
|
||||
end
|
||||
V --> App
|
||||
C --> App
|
||||
O --> App
|
||||
App --> NS
|
||||
```
|
||||
|
||||
1. You commit changes under **`gitops/values/`** (and optionally edit `repoURL` / `targetRevision` in the Application).
|
||||
2. Argo reconciles: Helm renders `onelab` + `onelab-obs` into namespace **`onelab`**.
|
||||
3. Sync waves order StatefulSets and app Deployments (Postgres → Redis/Rabbit/config → apps).
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| [`charts/onelab`](charts/onelab) | OneLab Helm chart — **Argo source 1** |
|
||||
| [`values/`](values/) | **Operator entry point**: [`values/env-example.yaml`](values/env-example.yaml), [`values/observability.yaml`](values/observability.yaml), templates — see [`values/README.md`](values/README.md) |
|
||||
| [`observability/`](observability/) | Loki / Promtail / Grafana umbrella chart — **Argo source 2** (`releaseName: onelab-obs`) |
|
||||
| [`argocd/application.yaml`](argocd/application.yaml) | `Application` manifest (`spec.sources`, destination namespace `onelab`) |
|
||||
| [`argocd/jsonpatch-multisource.json`](argocd/jsonpatch-multisource.json) | One-time JSON patch if a live `Application` still has legacy `spec.source` only |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Kubernetes** (e.g. k3s) with a default **StorageClass** for Postgres/Rabbit/Loki/Grafana PVCs (e.g. `local-path`).
|
||||
2. **Container images** from `hub.andrewalliance.com` (or your mirror): registry credentials via Helm values or a pre-created `docker-registry` Secret — see [Private registry credentials](#private-registry-credentials).
|
||||
3. **RabbitMQ TLS** before RabbitMQ starts: Kubernetes Secret `onelab-rabbit-tls`, or `rabbitmq.tls.embed` in private values — see [RabbitMQ TLS](#rabbitmq-tls).
|
||||
4. **Host paths** when using `persistence.mode: hostPath`: `/opt/onelab/data` and `/opt/onelab/logs` on nodes that run those pods (aligned with Promtail in [`values/observability.yaml`](values/observability.yaml)), or use RWX storage for multi-node.
|
||||
|
||||
## Configuration (single place to edit)
|
||||
|
||||
- **OneLab app / ingress / DB / tokens / registry**: [`values/env-example.yaml`](values/env-example.yaml) — copy patterns, replace **`REPLACE_*`** placeholders, or overlay a gitignored file; see [`values/secrets.example.yaml`](values/secrets.example.yaml) and [`values/README.md`](values/README.md).
|
||||
- **Observability**: [`values/observability.yaml`](values/observability.yaml) — Grafana password (**`REPLACE_GRAFANA_ADMIN_PASSWORD`**), ingress host, Promtail log hostPath.
|
||||
- **Argo Git coordinates**: [`argocd/application.yaml`](argocd/application.yaml) — set `repoURL`, `targetRevision`, and `helm.valueFiles` if you add e.g. `secrets.local.yaml`.
|
||||
|
||||
The chart default [`charts/onelab/values.yaml`](charts/onelab/values.yaml) holds non-secret structure and safe placeholders only; **do not rely on it for production secrets**.
|
||||
|
||||
## Bootstrap checklist
|
||||
|
||||
1. Fork or clone this repository and push it to a Git remote your cluster can reach.
|
||||
2. Edit [`argocd/application.yaml`](argocd/application.yaml): **`repoURL`** → your remote, **`targetRevision`** → your branch/tag.
|
||||
3. Edit [`values/env-example.yaml`](values/env-example.yaml): DNS hostnames, TLS secret names, cert-manager issuer, and all **`REPLACE_*`** values (or use a gitignored overlay).
|
||||
4. Edit [`values/observability.yaml`](values/observability.yaml): Grafana host/password aligned with your DNS and TLS.
|
||||
5. If the Git repo is **private**, register it in Argo CD (see below).
|
||||
6. Ensure **RabbitMQ TLS** Secret exists (or use embedded TLS in private values).
|
||||
7. Apply the Application:
|
||||
|
||||
```bash
|
||||
kubectl apply -f gitops/argocd/application.yaml
|
||||
```
|
||||
|
||||
8. In Argo UI or CLI, confirm sync to namespace **`onelab`** and fix any `ImagePullBackOff` / TLS issues using the sections below.
|
||||
|
||||
**Single controller:** Use **only** this Argo CD `Application` for `onelab` / `onelab-obs`. Do not manage the same namespace with a parallel **Helm CLI** release.
|
||||
|
||||
## Private registry credentials
|
||||
|
||||
Set `registry.username` / `registry.password` in values (prefer a **gitignored** file merged last), or create the Secret manually:
|
||||
|
||||
```bash
|
||||
kubectl create secret docker-registry hub-andrewalliance -n onelab \
|
||||
--docker-server=hub.andrewalliance.com \
|
||||
--docker-username='YOUR_USER' \
|
||||
--docker-password='YOUR_PASSWORD'
|
||||
```
|
||||
|
||||
…then set `registry.createPullSecret: false` and keep `imagePullSecrets: [{ name: hub-andrewalliance }]` in values.
|
||||
|
||||
### StatefulSet pods still get `401 Unauthorized` / `ImagePullBackOff` after enabling registry auth
|
||||
|
||||
If `db-0` / `rabbitmq-0` were created **before** `imagePullSecrets` existed, delete those pods once so they pick up the new Pod spec:
|
||||
|
||||
```bash
|
||||
kubectl delete pod -n onelab db-0 rabbitmq-0
|
||||
```
|
||||
|
||||
## Argo CD private Git repository
|
||||
|
||||
If the Application shows `authentication required: Unauthorized`, register the repo (use a deploy token or PAT with read access):
|
||||
|
||||
```bash
|
||||
argocd repo add https://github.com/YOUR_ORG/YOUR_REPO.git \
|
||||
--username git \
|
||||
--password YOUR_TOKEN
|
||||
```
|
||||
|
||||
## RabbitMQ TLS
|
||||
|
||||
Secret **`onelab-rabbit-tls`** must exist in namespace **`onelab`** before RabbitMQ starts (keys: typically `tls.crt`, `tls.key`, and optionally chain). **PEM material is not shipped in this GitOps tree** — generate certs or use your PKI, then:
|
||||
|
||||
```bash
|
||||
kubectl create secret tls onelab-rabbit-tls -n onelab \
|
||||
--cert=path/to/fullchain.pem \
|
||||
--key=path/to/key.pem
|
||||
```
|
||||
|
||||
Alternatively, set `rabbitmq.tls.embed: true` and supply `rabbitmq.tls.crt` / `rabbitmq.tls.key` / `rabbitmq.tls.fullchain` via a **private** values file (never commit real keys).
|
||||
|
||||
## Deploy with Argo CD (details)
|
||||
|
||||
**Requirements:** Argo CD **2.6+** (`spec.sources`).
|
||||
|
||||
Each entry under `spec.sources` has its own `helm.releaseName` and `helm.valueFiles` (paths are **relative to that source’s `path`**):
|
||||
|
||||
- Source `gitops/charts/onelab` → e.g. `../../values/env-example.yaml`, optionally `../../values/secrets.local.yaml`
|
||||
- Source `gitops/observability` → e.g. `../../values/observability.yaml`
|
||||
|
||||
### Migrating `spec.source` → `spec.sources`
|
||||
|
||||
If the `onelab` `Application` was created earlier with **`spec.source` only**, a plain `kubectl apply` may **not** remove `spec.source`, and Argo will not reconcile the observability chart.
|
||||
|
||||
Check:
|
||||
|
||||
```bash
|
||||
kubectl get application onelab -n argocd -o jsonpath='{.spec.source}{"\n"}{.spec.sources}{"\n"}'
|
||||
```
|
||||
|
||||
If `source` is set and `sources` is empty, patch once (edit `repoURL` in the patch file to match your remote):
|
||||
|
||||
```bash
|
||||
kubectl patch application onelab -n argocd --type json --patch-file gitops/argocd/jsonpatch-multisource.json
|
||||
```
|
||||
|
||||
## Observability (Loki / Promtail / Grafana)
|
||||
|
||||
The umbrella chart under [`observability/`](observability/) deploys:
|
||||
|
||||
- **Loki** — log storage (SingleBinary, filesystem PVC; retention from [`values/observability.yaml`](values/observability.yaml)).
|
||||
- **Promtail** — DaemonSet: Kubernetes pod logs plus **OneLab file logs** from the host path configured in [`values/observability.yaml`](values/observability.yaml) (keep in sync with OneLab `persistence.hostPath.logs`).
|
||||
- **Grafana** — Explore; datasource points at this release’s Loki gateway.
|
||||
|
||||
### First-time setup
|
||||
|
||||
1. Set a strong **`grafana.adminPassword`** in [`values/observability.yaml`](values/observability.yaml) (or use `admin.existingSecret` per the upstream Grafana chart).
|
||||
2. Align **Grafana ingress host** with `grafana.ini.server.domain` / `root_url` in the same file.
|
||||
3. **Multi-node** — with `hostPath` logs, each node only sees its own files; Promtail runs on every node.
|
||||
|
||||
### OneLab-only ingestion
|
||||
|
||||
Promtail **`extraRelabelConfigs`** keep only pods in namespace **`onelab`**. Host file logs are tagged **`namespace: onelab`**, **`component: host-logs`**.
|
||||
|
||||
### Dashboard: **OneLab logs**
|
||||
|
||||
Grafana sidecar loads the dashboard from `observability/dashboards/onelab-logs.json` (`uid` `onelab-logs`).
|
||||
|
||||
#### Grafana pod: `init-chown-data` CrashLoopBackOff
|
||||
|
||||
This repo sets **`grafana.initChownData.enabled: false`** with **`fsGroup: 472`** for Pod Security–friendly clusters. If Grafana cannot write to the PVC, delete the Grafana PVC once after changing values or adjust Pod Security for namespace `onelab`.
|
||||
|
||||
### Access Grafana
|
||||
|
||||
Ingress **`grafana-onelab`** is defined in `observability/templates/ingress-grafana-onelab.yaml`. Defaults use example hosts in [`values/observability.yaml`](values/observability.yaml) (`grafana.onelab.example.com`); change to your DNS and TLS Secret name.
|
||||
|
||||
```bash
|
||||
kubectl -n onelab port-forward svc/onelab-obs-grafana 3000:80
|
||||
```
|
||||
|
||||
### Maintainers: vendored chart dependencies
|
||||
|
||||
From `gitops/observability/`:
|
||||
|
||||
```bash
|
||||
helm dependency update
|
||||
```
|
||||
|
||||
Commit updated `Chart.lock` and `charts/*.tgz` so Argo can render without live Helm repo access at sync time.
|
||||
|
||||
## Application configuration (`configurations.yml`)
|
||||
|
||||
For Kubernetes you do **not** need a separate `configurations.yml` in Git. The OneLab chart renders it from [`charts/onelab/files/configurations.gotmpl`](charts/onelab/files/configurations.gotmpl) into Secret **`onelab-configurations`**.
|
||||
|
||||
1. **Values (recommended)** — set `onelab.compliance`, `onelab.ldap`, etc. See [`values/instance-overrides.example.yaml`](values/instance-overrides.example.yaml) and add paths under **`spec.sources[].helm.valueFiles`** for the `gitops/charts/onelab` source.
|
||||
2. **Bring your own Secret** — set `configuration.existingSecretName`; the Secret must contain key **`configurations.yml`**.
|
||||
|
||||
## Ingress (web UI)
|
||||
|
||||
Set `ingress.enabled`, `ingress.host`, and optional TLS in [`values/env-example.yaml`](values/env-example.yaml). Traffic goes to Service **`revproxy`**. On k3s, `ingress.className: traefik` matches the default controller. For cert-manager, set `ingress.certManager.clusterIssuer` and TLS secret name; DNS for `ingress.host` must resolve before ACME completes.
|
||||
|
||||
## Security notes for public repositories
|
||||
|
||||
- Never commit real passwords, tokens, or TLS private keys. Use **`REPLACE_*`** in tracked files and a **`*.local.yaml`** overlay (ignored at repo root) for secrets.
|
||||
- If this repo ever contained real credentials, **rotate** them after sanitizing Git history or publishing a clean fork.
|
||||
|
||||
## kubectl / credentials
|
||||
|
||||
If `kubectl` reports *You must be logged in*, refresh your kubeconfig before applying manifests.
|
||||
|
||||
## Developer note (local render)
|
||||
|
||||
Running **`helm template` on Windows** against some paths can return empty `.Files.Get` content; the OneLab chart uses `fromYaml (.Files.AsConfig)` where needed. **Argo CD runs on Linux** and renders the same charts in-cluster.
|
||||
|
||||
## Not covered by this chart
|
||||
|
||||
- **Edge proxy** in front of the cluster — use **Ingress** + `revproxy` and optional cert-manager.
|
||||
- **Non-Kubernetes** install paths — not included; use Kubernetes Secrets or external secret operators as needed.
|
||||
Reference in New Issue
Block a user