When experimenting with something like ceph, installing it, making changes, uninstalling and reinstalling … you will find that more advanced apps tend to implement finalizers making a full uninstall rather challenging.  More complex apps tend to have an uninstaller script for just this reason.  When lacking such a script though, here is a generic script which can take care of a lot, or all of the clean up work:

** Note, you are entering a danger zone. **

#!/bin/bash

if [ "$1" == "" ]
then
  echo "Syntax:"
  echo ""
  echo "$0 <searchstr>"

  exit 1
fi

CRDS=`kubectl get crd --no-headers | grep $1 | cut -d ' ' -f 1`

array_crds=( $CRDS )
for crd in "${array_crds[@]}"
do
  echo ""
  echo $crd
  RESOURCES=`kubectl get $crd --no-headers | grep $1 | cut -d ' ' -f 1`
  array_resources=( $RESOURCES )
  for next in "${array_resources[@]}"
  do
    echo $next
    kubectl patch $crd $next -p '{"metadata":{"finalizers":null}}' --type=merge
    kubectl delete $crd $next
  done
done

Thinking again about daytwo operations (controllers watching for cluster events):

daytwo-argocd-register-controller

• watches for cluster.yaml (tanzu, clusterapi, etc…) and registers clusters with argocd automatically once they are in ‘ready’ state
• syncs ‘addons’ labels from cluster.yaml to argocd cluster secrets to auto install addons, including pinniped-concierge, and pinniped-www

daytwo-pinniped-register-controller

• generates a pinniped kubeconfig if new cluster
• adds to git repo if new cluster, removes from git repo if cluster decommissioned
• regenerates configmap used by pinniped-www

daytwo-external-dns-register-controller

• watch for service associated with cluster to appear & annotate with fqdn, goal: to add a dns entry for each cluster kubeapi

Or perhaps, if desired, the event could trigger automation somewhere else:

daytwo-cluster-event-controller

• callback to jenkins
• callback to awx
• callback to vmware-aria
• etc …

Kubernetes implements OIDC via arguments on the API Server such as –oidc-issuer-url, etc. This works great, however a common utility to access kubernetes in this way kubelogin isn’t so great in that an open web browser closed in the wrong way can cause the authentication to fail. So what’s a better solution?

Pinniped allows the configuring of OIDC and login on a running kubernetes cluster. This is great in that you don’t have to restart the cluster to get OIDC to work, and sometimes you just don’t have access to configure OIDC via the API Server.

Pinniped can be configured and deployed in the usual way that you deploy addons to your clusters. (see argocd – getting started, and argocd – getting started – addons)

Whether using the native OIDC solution or using pinniped, a kubeconfig must be created which can be used with kubectl to access the clusters. Pinniped has a utility to create the kubeconfig, but how to get to the end user? There have been various methods to accomplish this:

• the user submits a ticket, through automation you send them the pinniped kubeconfig via email
• all pinniped kubeconfig files are made available via a git repo all clients have access to
• (here’s our idea) make the pinniped kubeconfig available via a website, so we don’t need to grant git repo access in order to deliver the kubeconfig

My strategy has been to use argocd, applicationsets, and a single container running on all clusters as a means to distribute the pinniped kubeconfig to all interested parties.

* Note: pinniped kubeconfigs contain no secrets, they do not need to be treated as secrets, other than as related to a potential denial of service attack, because the user will still need to login and access is granted via groups.

Here’s the implementation:

• argocd applicationsets allow the use cluster name to be used as a variable, this means we can define the url we want in each cluster to be: https://pinniped.<cluster>.<domain>
• the applicationset deploys a container running php which uses a script to determine the clustername. The php site displays some information to help the user get started with pinniped and a link to the pinniped kubeconfig
• a configmap is mounted using a volume to the container and contains all pinniped kubeconfig files, this works because the url generated in the previous step includes the name of the cluster and so points to the correct pinniped kubeconfig
• because this solution is deployed via argocd, when a new cluster is deployed and a pinniped kubeconfig is created & added to a git repo, a script re-generates the configmap from the git repo, all pinniped websites are updated automatically, we just have to add a step to update the pinniped configmap as part of the new cluster deployment process

(video: todo)

• manually store the ca-bundle in a vault, such as azure vault or hashicorp vault
• configure external secrets addon to use the vault
• configure ca-bundle as volume mount using secret provided by external secrets in each pod that needs it

If anyone has a better idea feel free to share.