onelab-k8s-1.27/gitops

OneLab GitOps (Argo CD)

This directory is the declarative source for OneLab on Kubernetes. Argo CD applies two Helm-based sources from Git (Argo invokes Helm; you do not use Helm CLI as the primary install path for the cluster).

This repository is GitOps-only: everything required to deploy OneLab on Kubernetes lives under gitops/.

What gets deployed

Helm release	Chart path	Namespace	Role
onelab	charts/onelab	onelab	Application workloads, Postgres, Redis, RabbitMQ, revproxy, ingress hooks
onelab-obs	observability/	onelab	Loki, Promtail, Grafana (logs)

Both are wired by a single Argo CD Application using spec.sources (requires Argo CD 2.6+).

Architecture

flowchart LR
  subgraph git [Git repository]
    V[gitops/values]
    C[charts/onelab]
    O[observability]
  end
  subgraph argo [Argo CD]
    App[Application onelab]
  end
  subgraph cluster [Kubernetes cluster]
    NS[namespace onelab]
  end
  V --> App
  C --> App
  O --> App
  App --> NS

1. You commit changes under gitops/values/ (and optionally edit repoURL / targetRevision in the Application).
2. Argo reconciles: Helm renders onelab + onelab-obs into namespace onelab.
3. Sync waves order StatefulSets and app Deployments (Postgres → Redis/Rabbit/config → apps).

Layout

Path	Purpose
charts/onelab	OneLab Helm chart — Argo source 1
values/	Operator entry point: values/env-example.yaml, values/observability.yaml, templates — see values/README.md
observability/	Loki / Promtail / Grafana umbrella chart — Argo source 2 (releaseName: onelab-obs)
argocd/application.yaml	Application manifest (spec.sources, destination namespace onelab)
argocd/jsonpatch-multisource.json	One-time JSON patch if a live Application still has legacy spec.source only

Prerequisites

1. Kubernetes (e.g. k3s) with a default StorageClass for Postgres/Rabbit/Loki/Grafana PVCs (e.g. local-path).
2. Container images from hub.andrewalliance.com (or your mirror): registry credentials via Helm values or a pre-created docker-registry Secret — see Private registry credentials.
3. RabbitMQ TLS before RabbitMQ starts: Kubernetes Secret onelab-rabbit-tls, or rabbitmq.tls.embed in private values — see RabbitMQ TLS.
4. Host paths when using persistence.mode: hostPath: /opt/onelab/data and /opt/onelab/logs on nodes that run those pods (aligned with Promtail in values/observability.yaml), or use RWX storage for multi-node.

Configuration (single place to edit)

• OneLab app / ingress / DB / tokens / registry: values/env-example.yaml — copy patterns, replace REPLACE_* placeholders, or overlay a gitignored file; see values/secrets.example.yaml and values/README.md.
• Observability: values/observability.yaml — Grafana password (REPLACE_GRAFANA_ADMIN_PASSWORD), ingress host, Promtail log hostPath.
• Argo Git coordinates: argocd/application.yaml — set repoURL, targetRevision, and helm.valueFiles if you add e.g. secrets.local.yaml.

The chart default charts/onelab/values.yaml holds non-secret structure and safe placeholders only; do not rely on it for production secrets.

Bootstrap checklist

1. Fork or clone this repository and push it to a Git remote your cluster can reach.

2. Edit argocd/application.yaml: repoURL → your remote, targetRevision → your branch/tag.

3. Edit values/env-example.yaml: DNS hostnames, TLS secret names, cert-manager issuer, and all REPLACE_* values (or use a gitignored overlay).

4. Edit values/observability.yaml: Grafana host/password aligned with your DNS and TLS.

5. If the Git repo is private, register it in Argo CD (see below).

6. Ensure RabbitMQ TLS Secret exists (or use embedded TLS in private values).

7. Apply the Application:

   kubectl apply -f gitops/argocd/application.yaml
   

8. In Argo UI or CLI, confirm sync to namespace onelab and fix any ImagePullBackOff / TLS issues using the sections below.

Single controller: Use only this Argo CD Application for onelab / onelab-obs. Do not manage the same namespace with a parallel Helm CLI release.

Private registry credentials

Set registry.username / registry.password in values (prefer a gitignored file merged last), or create the Secret manually:

kubectl create secret docker-registry hub-andrewalliance -n onelab \
  --docker-server=hub.andrewalliance.com \
  --docker-username='YOUR_USER' \
  --docker-password='YOUR_PASSWORD'

…then set registry.createPullSecret: false and keep imagePullSecrets: [{ name: hub-andrewalliance }] in values.

StatefulSet pods still get 401 Unauthorized / ImagePullBackOff after enabling registry auth

If db-0 / rabbitmq-0 were created before imagePullSecrets existed, delete those pods once so they pick up the new Pod spec:

kubectl delete pod -n onelab db-0 rabbitmq-0

Argo CD private Git repository

If the Application shows authentication required: Unauthorized, register the repo (use a deploy token or PAT with read access):

argocd repo add https://github.com/YOUR_ORG/YOUR_REPO.git \
  --username git \
  --password YOUR_TOKEN

RabbitMQ TLS

Secret onelab-rabbit-tls must exist in namespace onelab before RabbitMQ starts (keys: typically tls.crt, tls.key, and optionally chain). PEM material is not shipped in this GitOps tree — generate certs or use your PKI, then:

kubectl create secret tls onelab-rabbit-tls -n onelab \
  --cert=path/to/fullchain.pem \
  --key=path/to/key.pem

Alternatively, set rabbitmq.tls.embed: true and supply rabbitmq.tls.crt / rabbitmq.tls.key / rabbitmq.tls.fullchain via a private values file (never commit real keys).

Deploy with Argo CD (details)

Requirements: Argo CD 2.6+ (spec.sources).

Each entry under spec.sources has its own helm.releaseName and helm.valueFiles (paths are relative to that source’s path):

• Source gitops/charts/onelab → e.g. ../../values/env-example.yaml, optionally ../../values/secrets.local.yaml
• Source gitops/observability → e.g. ../../values/observability.yaml

Migrating spec.source → spec.sources

If the onelab Application was created earlier with spec.source only, a plain kubectl apply may not remove spec.source, and Argo will not reconcile the observability chart.

Check:

kubectl get application onelab -n argocd -o jsonpath='{.spec.source}{"\n"}{.spec.sources}{"\n"}'

If source is set and sources is empty, patch once (edit repoURL in the patch file to match your remote):

kubectl patch application onelab -n argocd --type json --patch-file gitops/argocd/jsonpatch-multisource.json

Observability (Loki / Promtail / Grafana)

The umbrella chart under observability/ deploys:

• Loki — log storage (SingleBinary, filesystem PVC; retention from values/observability.yaml).
• Promtail — DaemonSet: Kubernetes pod logs plus OneLab file logs from the host path configured in values/observability.yaml (keep in sync with OneLab persistence.hostPath.logs).
• Grafana — Explore; datasource points at this release’s Loki gateway.

First-time setup

1. Set a strong grafana.adminPassword in values/observability.yaml (or use admin.existingSecret per the upstream Grafana chart).
2. Align Grafana ingress host with grafana.ini.server.domain / root_url in the same file.
3. Multi-node — with hostPath logs, each node only sees its own files; Promtail runs on every node.

OneLab-only ingestion

Promtail extraRelabelConfigs keep only pods in namespace onelab. Host file logs are tagged namespace: onelab, component: host-logs.

Dashboard: OneLab logs

Grafana sidecar loads the dashboard from observability/dashboards/onelab-logs.json (uid onelab-logs).

Grafana pod: init-chown-data CrashLoopBackOff

This repo sets grafana.initChownData.enabled: false with fsGroup: 472 for Pod Security–friendly clusters. If Grafana cannot write to the PVC, delete the Grafana PVC once after changing values or adjust Pod Security for namespace onelab.

Access Grafana

Ingress grafana-onelab is defined in observability/templates/ingress-grafana-onelab.yaml. Defaults use example hosts in values/observability.yaml (grafana.onelab.example.com); change to your DNS and TLS Secret name.

kubectl -n onelab port-forward svc/onelab-obs-grafana 3000:80

Maintainers: vendored chart dependencies

From gitops/observability/:

helm dependency update

Commit updated Chart.lock and charts/*.tgz so Argo can render without live Helm repo access at sync time.

Application configuration (configurations.yml)

For Kubernetes you do not need a separate configurations.yml in Git. The OneLab chart renders it from charts/onelab/files/configurations.gotmpl into Secret onelab-configurations.

1. Values (recommended) — set onelab.compliance, onelab.ldap, etc. See values/instance-overrides.example.yaml and add paths under spec.sources[].helm.valueFiles for the gitops/charts/onelab source.
2. Bring your own Secret — set configuration.existingSecretName; the Secret must contain key configurations.yml.

Ingress (web UI)

Set ingress.enabled, ingress.host, and optional TLS in values/env-example.yaml. Traffic goes to Service revproxy. On k3s, ingress.className: traefik matches the default controller. For cert-manager, set ingress.certManager.clusterIssuer and TLS secret name; DNS for ingress.host must resolve before ACME completes.

Security notes for public repositories

• Never commit real passwords, tokens, or TLS private keys. Use REPLACE_* in tracked files and a *.local.yaml overlay (ignored at repo root) for secrets.
• If this repo ever contained real credentials, rotate them after sanitizing Git history or publishing a clean fork.

kubectl / credentials

If kubectl reports You must be logged in, refresh your kubeconfig before applying manifests.

Developer note (local render)

Running helm template on Windows against some paths can return empty .Files.Get content; the OneLab chart uses fromYaml (.Files.AsConfig) where needed. Argo CD runs on Linux and renders the same charts in-cluster.

Not covered by this chart

• Edge proxy in front of the cluster — use Ingress + revproxy and optional cert-manager.
• Non-Kubernetes install paths — not included; use Kubernetes Secrets or external secret operators as needed.