k8s daytwo operation ideas


  • watch for cluster.yaml updates & reflect addons annotations from cluster.yaml to argocd cluster yaml secrets
  • watch argocd repo certs and check for expire, re-adding automatically
  • tanzu watch for service associated with cluster to appear & annotate with fqdn automatically (in order to add dns entry for each cluster kubeapi)
  • watch for certificate authority expiration and update ‘ca-bundle’ stored in vault

Idea: repo-manager


In a world of containers, developers need to have multiple linux repos mirrored on prem for use when building or modifying images.


Similar to how cert-manager works with providers to extend functionality, repo-manager can be extended to mirror additional linux distros.


repo-manager provides an operator-type controller which watches for the CRD type ‘mirror.aarr.xyz’ & manages for each repo mirror:

  • the deployment of a pod for mirroring
  • a pvc for each pod
  • the increasing of the pvc size as needed
  • ingress configuration to reach each repo using a subpath http(s)://mirror.<fqdn>/<path>
  • a status using kubectl via displaying relevant CRDs
  • mapping of a ca-bundle
  • repo-specific settings


  • an overall web interface
  • settings which apply to all repo mirrors

ingress: cert-manager, letsencrypt, and travisloyd.xyz -> www.travisloyd.xyz

It seemed for awhile that the popular web browsers would automatically redirect travisloyd.xyz to www.travisloyd.xyz if travisloyd.xyz didn’t work.  But, after awhile that no longer seemed to happen.  So, let’s do this right.

Here’s an ingress to perform the redirect from travisloyd.xyz to www.travisloyd.xyz:

apiVersion: networking.k8s.io/v1
kind: Ingress
    #nginx.ingress.kubernetes.io/proxy-body-size: 1000m
    nginx.ingress.kubernetes.io/server-snippet: |
      return 301 https://www.travisloyd.xyz$request_uri;
  name: ingress-redirect
  ingressClassName: nginx
  - host: travisloyd.xyz
  - hosts:
    - travisloyd.xyz
    secretName: travisloyd.xyz-tls

But what about an automatic certificate via letsencrypt? Do we need it? Yes, otherwise https://travisloyd.xyz displays an invalid certificate before performing the redirect. But, we can’t just add the annotations for cert-manager to this redirect because the call back from lets encrypt will not verify correctly with the redirect. Instead, we need an ingress specifically for handling the letsencrypt callback:

apiVersion: networking.k8s.io/v1
kind: Ingress
    cert-manager.io/issuer: cluster-letsencrypt-issuer
    cert-manager.io/issuer-kind: ClusterIssuer
    #nginx.ingress.kubernetes.io/proxy-body-size: 1000m
  name: ingress-redirect-letsencrypt
  ingressClassName: nginx
  - host: travisloyd.xyz
      - backend:
            name: exp-wordpress-xyz-travisloyd-www
              name: http
        path: /.well-known
        pathType: Prefix
  - hosts:
    - travisloyd.xyz
    secretName: travisloyd.xyz-tls

Perfect, now when the certs expire they’ll be renewed automatically via letsencrypt.

(ulimit) Heads up for anyone following my videos to install kubernetes on redhat

Redhat has an infinite value for ulimit by default which kubernetes inherits via the container service being used, this can result in some pods maxing out cpu and memory (such as haproxy and rabbitmq). For containerd the following fix solved the issue:

# sed -i 's/LimitNOFILE=infinity/LimitNOFILE=65535/' /usr/lib/systemd/system/containerd.service
# systemctl daemon-reload
# systemctl restart containerd
# k delete deployment <asdf>

homelab: planning next incarnation

Thinking about redeploying my homelab from scratch, perhaps switching from xenserver back to vmware. I’d like to start out with external-secrets and have all secrets in a vault right from the beginning, also curious what a 100% open source, 100% kubernetes environment would look like. Maybe two networks, one 100% kubernetes, and a 2nd for windows client systems. Here’s the k8s plan so far:

- manual setup of seed cluster
  - helm install argocd
  - argocd install clusterapi/crossplane/etc...
- seed-argocd deploy non-production cluster using vcluster or clusterapi/crossplane/etc...
  - deploy metallb & configure loadbalancer ip range (can we automate this w/ cluster deploy?)
  - add cluster to seed-argocd instance
- seed-argocd deploy production cluster using vcluster or clusterapi/crossplane/etc...
  - deploy metallb & configure loadbalancer ip range (can we automate this w/ cluster deploy?)
  - add cluster to seed-argocd instance
- seed-argocd deploy argocd to production cluster (k-prod)

- argocd configure storageclass
- argocd deploy hashicorp vault
  - configure as certificate authority
  - configure as keyvault
- argocd deploy external-secrets
  - configure to use keyvault
  - add secret 'ca-bundle.crt': public certificate authority certificate in DER format
  - *from now on all secrets to get values via external-secrets
- argocd deploy cert-manager
  - configure to use hashicorp vault as certificate authority
- argocd deploy pihole
  - configure dns1 & dns2
- argocd deploy external-dns
  - configure to use pihole as dns
- update with annotations to use external-dns & cert-manager:
  - argocd
  - vault
  - pihole
  - *from now on all ingress yaml to include annotations for external-dns & cert-manager
    - recommended: have annotations from the beginning, at this point they will start working
- argocd deploy keycloak
  - configure realm: create or import from backup
  - add secret 'default_oidc_client_secret': secret part of oidc client/secret
  - configure a user account (or configure federation via AD, openldap, etc...)
- deploy all other apps
  - oidc client_secret should come from external-secrets in all apps configured with oidc
    - this might require an init container for some apps

- pvc storage for all clusters
- block storage can be used for vm disks (making for easy hotswap)
- upgrade to 2 10gb ports on each host system

wdc: (kubevirt in theory but think i'll stick w/ a vm)
- domain controller
- user management
- dhcp
- wds
- wsus using dev sqlserver & data stored on e drive