(bash) script to fully clean up a kubernetes app

When experimenting with something like ceph, installing it, making changes, uninstalling and reinstalling … you will find that more advanced apps tend to implement finalizers making a full uninstall rather challenging.  More complex apps tend to have an uninstaller script for just this reason.  When lacking such a script though, here is a generic script which can take care of a lot, or all of the clean up work:

** Note, you are entering a danger zone. **

#!/bin/bash

if [ "$1" == "" ]
then
  echo "Syntax:"
  echo ""
  echo "$0 <searchstr>"

  exit 1
fi

CRDS=`kubectl get crd --no-headers | grep $1 | cut -d ' ' -f 1`

array_crds=( $CRDS )
for crd in "${array_crds[@]}"
do
  echo ""
  echo $crd
  RESOURCES=`kubectl get $crd --no-headers | grep $1 | cut -d ' ' -f 1`
  array_resources=( $RESOURCES )
  for next in "${array_resources[@]}"
  do
    echo $next
    kubectl patch $crd $next -p '{"metadata":{"finalizers":null}}' --type=merge
    kubectl delete $crd $next
  done
done

(gitops) argocd phoenix configuration: clusterapi with vcluster provider

Standardized git repo layouts helps to keep deployments consistent and clean:

k-argocd
- /appofapps/clusters/application.yaml
- /apps
  - /argocd-seed/update.sh
  - /argocd/applicationset.yaml
  - /clusterapi/applicationset.yaml
  - /daytwo/applicationset.yaml
- /projects
  - /addons.yaml
  - /developer.yaml
  - /devsecops.yaml

k-argocd-addons
- /apps
  - /adcs-issuer-system/applicationset.yaml
  - /adcs-issuer-system/base/Chart.yaml
  - /cert-manager/applicationset.yaml
  - /external-dns/applicationset.yaml
  - /external-dns-root/applicationset.yaml
  - /fluent-bit/applicationset.yaml
  - /kasten/applicationset.yaml
  - /nginx-ingress/applicationset.yaml
  - /metrics-server/applicationset.yaml
  - /pinniped-concierge/applicationset.yaml
  - /prometheus/applicationset.yaml

k-argocd-clusters
- /clusters
  - /vc-non.yaml
  - /vc-prod.yaml

k-vc-non
- /appofapps
  - /namespaces/application.yaml
- /apps
  - /example/applicationset.yaml
  - /example/base/Chart.yaml
- /namespaces
  - /example/namespace.yaml
  - /example/resourcequota.yaml
  - /example/servicemesh.yaml

k-vc-prod
- /appofapps
  - /namespaces/application.yaml
- /apps
  - /example/applicationset.yaml
  - /example/base/Chart.yaml
- /namespaces
  - /example/namespace.yaml
  - /example/resourcequota.yaml
  - /example/servicemesh.yaml

daytwo automates several steps needed when first deploying clusters:

  • register cluster with argocd, also adds annotation allowing applications to target by cluster name
  • copy labels from cluster yaml to argocd secret, useful for deploying addons
  • generates pinniped kubeconfig, allows for initial access without needing admin kubeconfig
  • registers as a kasten secondary cluster, (if kasten is being used)

Scripts / pipelines are needed to:

  • provision / decommission a cluster
    • adjust cluster resources
  • add / remove a namespace
    • adjust namespace resource quota
    • grant developers access to namespaces

(idea) restarting a pod via webapi

At the end of a pipeline it can be nice to restart a pod.  In production this occurs via gitops automatically, but in dev it can be nice to not have to wait for git to sync.  Though there are multiple ways to do this common ways are:

  1. just put the cluster kubeconfig in a secret in the pipeline and use that to restart the pod (a little too powerful)
  2. create a service account, acquire its token and place in pipeline (safest, but takes some work for each app)

What about a solution similar to reloader?  Reloader watches for changes in configmaps and secrets which have a certain annotation and if a they change reloader restarts things.  We could just use reloader & make a change to a configmap in order to trigger the reload.

However, what about creating a controller which listens for a webapi call asking for a deployment to be restarted.  Then a pipeline could call the appropriate url to get things restarted.  By deploying via argocd using an applicationset and using a url convention based on the clustername then all development clusters could be enabled to used this method in their pipelines, consumers would only need to annotate their deployments/statefulset/etc …