title: Fix cluster-apps kustomization - duplicate system-upgrade namespace

description: cluster-apps failing: duplicate namespace system-upgrade. Fix: Remove duplicate namespace.yaml references in kubernetes/apps/system-upgrade/kustomization.yaml

status: open

issue_type: bug

created_by: Laur IVAN

labels: flux kustomize system-upgrade


title: Watch cluster rollout

description: Watch the rollout of the cluster to ensure all pods are starting correctly

acceptance_criteria:
- Command `kubectl get pods --all-namespaces --watch` is running
- All pods are observed rolling out
- Pods reach Running/Ready state

issue_type: task

labels: bootstrap verification


title: Configure GitHub webhook for Flux

description: Configure GitHub webhook to send push events to Flux for automatic reconciliation on git push

acceptance_criteria:
- Command `kubectl -n flux-system get receiver github-webhook --output=jsonpath='{.status.webhookPath}'` returns webhook path
- Full webhook URL is constructed with format: https://flux-webhook.${cloudflare_domain}/hook/{path}
- Webhook is added to GitHub repository settings
- Webhook payload URL is set correctly
- Content type is set to application/json
- Secret token from github-push-token.txt is configured
- Events are set to "Just the push event"
- Webhook is saved and active

labels: configuration flux github


title: Fix network configuration conflicts (etcd + routes)

description:
Multiple network configuration issues on the cluster nodes:

**Issue 1: etcd Peer URL Conflict**
Node esxi-2cu-8g-01 (10.0.0.146) has duplicate peer URLs in etcd (10.0.0.128 and 10.0.0.146), causing "Peer URLs already exists" error. Node is currently unreachable.

**Issue 2: Network Route Conflict**
Nodes are showing route conflict errors:
```
error adding route: netlink receive: file exists
gateway: 10.0.0.129
```

This is because nodes were previously configured with `/24` subnet and gateway `10.0.0.1`, but now configured with `/27` subnet and gateway `10.0.0.129`. Old routes persist.

**Root Cause:**
Configuration changed from:
- Old: 10.0.0.0/24, gateway 10.0.0.1
- New: 10.0.0.128/27, gateway 10.0.0.129

**Solution:**
1. Reset ALL nodes to clear old network config
2. Re-apply Talos configuration
3. Bootstrap cluster fresh

Commands:
```bash
# Reset each node
talosctl -n 10.0.0.145 reset --graceful=false --reboot
talosctl -n 10.0.0.146 reset --graceful=false --reboot
talosctl -n 10.0.0.147 reset --graceful=false --reboot

# Wait for nodes to boot into maintenance mode, then:
task bootstrap:talos
```

acceptance_criteria:
- Member ceeb52e03fde8032 is removed from etcd cluster
- Node 10.0.0.146 is reset and reconfigured
- Node rejoins etcd cluster with correct peer URL
- `talosctl etcd members` shows only one peer URL per member
- All three nodes are healthy in etcd cluster

notes:
**Recommended Fix: Full Cluster Reset (Option 1)**

All nodes are currently offline. Once nodes are back online, execute:

```bash
# Reset all nodes to maintenance mode
talosctl -n 10.0.0.145 reset --graceful=false --reboot --insecure
talosctl -n 10.0.0.146 reset --graceful=false --reboot --insecure
talosctl -n 10.0.0.147 reset --graceful=false --reboot --insecure

# Wait for nodes to boot into maintenance mode (~2-3 min)
# Verify with: nmap -Pn -n -p 50000 10.0.0.145-147 -vv

# Re-bootstrap
task bootstrap:talos
task bootstrap:apps
```

This is the cleanest approach to clear all lingering network config and etcd state issues. Estimated time: ~15 minutes total.

status: closed

close_reason: Fixed - etcd cluster healthy with 3 members, each with single peer URL. No route conflicts. All cluster health checks passed.

labels: etcd talos urgent


title: Push talhelper encrypted secret to git

description: After installing Talos, commit and push the talhelper encrypted secret to the repository

acceptance_criteria:
- Changes are staged with `git add -A`
- Commit is created with message "chore: add talhelper encrypted secret :lock:"
- Changes are pushed to remote repository

close_reason: Successfully staged, committed, and pushed talhelper encrypted secret to git repository

labels: bootstrap git


title: Verify Flux status and resources

description: Check the status of Flux and verify all Flux resources are up-to-date and in a ready state

acceptance_criteria:
- Command `flux check` passes all checks
- Command `flux get sources git flux-system` shows ready state
- Command `flux get ks -A` shows all kustomizations ready
- Command `flux get hr -A` shows all helm releases ready

close_reason: Verified - Flux check passed. All controllers ready (helm, kustomize, notification, source). GitRepository synced. All Kustomizations applied successfully.

labels: flux verification


title: Fix volsync MutatingAdmissionPolicy API version

description:
Kustomization storage-system/volsync is failing with error:

```
MutatingAdmissionPolicy/storage-system/volsync-mover-jitter dry-run failed: no matches for kind "MutatingAdmissionPolicy" in version "admissionregistration.k8s.io/v1beta1"
```

**Root Cause:**
MutatingAdmissionPolicy API does not exist in Kubernetes 1.34. Only `admissionregistration.k8s.io/v1` is available, which only includes MutatingWebhookConfiguration.

The MutatingAdmissionPolicy feature was experimental and appears to have been removed or never graduated to stable.

**What the policy does:**
Adds a jitter initContainer to volsync jobs to randomize start times (sleep 0-30 seconds). This is optional functionality.

**Fix:**
Remove or comment out the mutating-admission-policy.yaml file from kubernetes/apps/storage-system/volsync/app/kustomization.yaml since this feature is not available in K8s 1.34 and is non-critical.

notes:
**Verified in K8s 1.35.0:**
Issue still exists. MutatingAdmissionPolicy API is not available in Kubernetes 1.35.0.

Only `admissionregistration.k8s.io/v1` exists, which includes:
- MutatingWebhookConfiguration
- ValidatingWebhookConfiguration

MutatingAdmissionPolicy/MutatingAdmissionPolicyBinding are not available.

The fix remains the same: remove or comment out the mutating-admission-policy.yaml file.

close_reason: Fixed - MutatingAdmissionPolicy feature gate enabled in K8s 1.35. API now available (admissionregistration.k8s.io/v1beta1). Volsync kustomization reconciled successfully.

labels: api-version storage volsync


title: Tidy up repository (remove templates)

description: Clean up the repository by removing the templates directory and templating-related files to eliminate clutter and resolve Renovate warnings

acceptance_criteria:
- Command `task template:tidy` completes successfully
- Templates directory is removed
- Templating-related files are cleaned up
- Changes are committed with message "chore: tidy up :broom:"
- Changes are pushed to git

labels: cleanup git


title: Bootstrap cluster applications (cilium, coredns, spegel, flux)

description: Install cilium, coredns, spegel, flux and sync the cluster to the repository state

acceptance_criteria:
- Command `task bootstrap:apps` completes successfully
- Cilium is installed
- CoreDNS is installed
- Spegel is installed
- Flux is installed
- Cluster is synced to repository state

close_reason: Successfully installed cilium, coredns, spegel, cert-manager, flux-operator. Flux-instance is reconciling (timeout is normal). All nodes are Ready.

labels: apps bootstrap


title: Finish monitoring system setup

description: Uncomment the grafana and kube-prometheus-stack resources in kubernetes/apps/monitoring-system/kustomization.yaml to enable the full monitoring stack with Grafana dashboards and Prometheus metrics collection

labels: grafana monitoring prometheus


title: Publish Kubernetes schemas locally

description: Set up CronJob to publish K8s schemas locally. Reference: https://github.com/bjw-s-labs/home-ops/tree/main/kubernetes/apps/jobs/publish-k8s-schemas

labels: cronjob schemas validation


title: Verify DNS resolution for echo subdomain

description: Check that DNS resolution works for the echo subdomain and resolves to the Cloudflare gateway address

acceptance_criteria:
- Command `dig @${cluster_dns_gateway_addr} echo.${cloudflare_domain}` resolves successfully
- DNS resolves to ${cloudflare_gateway_addr}
- DNS resolution is working correctly

close_reason: Verified - DNS resolution working. echo.laurivan.com resolves to 10.0.0.158 (envoy-external gateway) via k8s-gateway

labels: dns verification


title: Verify TCP connectivity to gateways

description: Check TCP connectivity to both the internal and external gateways on port 443

acceptance_criteria:
- Command `nmap -Pn -n -p 443 ${cluster_gateway_addr} ${cloudflare_gateway_addr} -vv` succeeds
- Port 443 is open on both internal and external gateways
- TCP connectivity is confirmed

labels: network verification


title: Fix tuppr HelmRelease timeout

description: tuppr HelmRelease stuck in 'InProgress' status, causing health check timeout after 5m. Related to homelab-oqx (ServiceAccount API version bug in chart v0.0.52)

labels: helm system-upgrade tuppr


title: Verify Cilium status

description: Verify that Cilium is installed and running correctly

acceptance_criteria:
- Command `cilium status` runs successfully
- Cilium reports healthy status
- All Cilium components are operational

close_reason: Verified - Cilium OK, Operator OK, 3/3 DaemonSet ready, 1/1 Operator ready, 29/29 cluster pods managed

labels: cilium verification


title: Fix tuppr HelmRelease - invalid ServiceAccount API version

description:
Tuppr HelmRelease is failing with error:

```
Helm install failed: resource mapping not found for name: "tuppr-talosconfig" namespace: "system-upgrade" from "": no matches for kind "ServiceAccount" in version "talos.dev/v1alpha1"
ensure CRDs are installed first
```

The tuppr chart is trying to create a ServiceAccount with apiVersion `talos.dev/v1alpha1`, which is invalid. ServiceAccount should use `v1` API version.

This appears to be a bug in the tuppr chart itself (version 0.0.52). The chart is incorrectly using a Talos-specific API version for a standard Kubernetes ServiceAccount resource.

Possible fixes:
1. Wait for upstream chart fix
2. Use a different version of tuppr
3. Apply a patch to fix the ServiceAccount apiVersion
4. Disable tuppr if not critical

close_reason: Closed via update

labels: oci-repository system-upgrade tuppr


title: Verify wildcard Certificate status

description: Check the status of the wildcard Certificate in the network namespace

acceptance_criteria:
- Command `kubectl -n network describe certificates` runs successfully
- Certificate status shows Ready condition
- Certificate is valid and not expired

labels: certificates verification


title: Install homepage dashboard

description: Create the homepage application manifests (helmrelease, ocirepository, kustomization) in kubernetes/apps/default/homepage/app/ directory and configure the ks.yaml to deploy it

labels: dashboard deployment homepage


title: Install home assistant for home automation

description:
Create home assistant application manifests (helmrelease, ocirepository, kustomization) in kubernetes/apps/default/home-assistant/app/ directory and configure deployment.

Note: Ensure the application has network access to the IoT VLAN where most smart home devices are located. This may require configuring network policies or multus CNI for VLAN access.

labels: automation home-assistant iot networking