Deploying via git using argocd or flux makes disaster recovery fairly straightforward.

Using gitops means you can delete a kubernetes cluster, spin up a new one,  and have everything deployed back out in minutes.  But what about recovering the pvcs used before?

If you are using an infrastructure which implements csi, then you are able to allocate pvcs using storage managed outside of the cluster.  And, it turns out, reattaching to those pvcs is possible but you have to plan ahead.

Instead of writing yaml to spin up a pvc automatically, create the pv and pvc using manually set values.  Or spin up the pvcs automatically and then go back and modify the yaml to set recoverable values.  The howto is right up top in the csi documentation: https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/

Similarly, it is common for applications to spin up with randomly set admin passwords and such.  However, imagine a recovery scenario where a new cluster is stood up, you don’t want a new password stood up.  Use a vault with a password and reference the vault.

These two steps do add a little work, it’s the idea of taking a little more time to do things right, and in a production environment you want this.

Infrastructure side solution: https://velero.io/

Todo:  Create a video deleting a cluster and recovering all apps with a new cluster, recovering pvcs also (without any extra work on the recovery side).

New location is here:

https://github.com/lknite/lknite-docker-wireguard-centos-8-stream

more kubernetes magic

Traditionally Shinto shrines are rebuilt exactly the same next to the old shrine every so many years.  The old shrine is removed and when the time comes it will be rebuilt again.

Something similar can apply to home environments.  Recently I nuked everything and rebuilt from the ground up.  Something I’ve always done after 6 months or a year, for security reasons and to ensure I am always getting the fastest performance from infrastructure.

Such reinstalling is a natural fit for kubernetes.  There are several methods for spinning up a cluster, and after that just by the nature of kubernetes being yaml files it is easy to spin up the services you had running before, and watch them self register in the new dns and self generate certificates with the new active directory certificate authority.  Amazing.   Kubernetes is truly, a work of art.

As I take the deep dive into kubernetes what I’m finding is, though definitely a container management system, it can also been seen as a controller yaml processing engine.  Let me explain.

Kubernetes understands what a deployment is, and what a service is, these are defined as yaml and loaded.  Deployments and services can be seen as controllers which understand those types of objects defined in yaml.

What is interesting about this is that we can implement our own controllers.  For example, I could implement a controller that understands how to manage a tic-tac-toe game.  That controller could also implement an ai that knows how to play the game.  In the same way you can edit a deployment you could edit the game and the kubernetes infrastructure could respond to the change.  Or, a move could be another type recognized by the game controller, so you could create a move associated with a game in the same way you can create a service associated with a deployment.

You can imagine doing a ‘k get games’ and seeing the games being played listed out.  As well as ‘k describe game a123’ to get the details and status of the game.

Seems I’m not the only one who has started thinking down this line.  A quick Google search reveals agones.

This is fascinating and gives me a lot of ideas on how I might reimplement my list processing server & generic game server, within the kubernetes framework.

https://lknight-gh.github.io/helm-charts

My first helm chart, a fun milestone.  Used it to install my new docker container uploaded to quay.io this morning.

Nice feeling to give back to the open source community.

Now to automate:
* watch for wireguard updates & release an updated docker image
* watch for a centos-8-stream update & release an updated docker image
* watch for a helm chart update & update what is necessary for those changes to be seen

But first, time to investigate and implement longhorn.

https://lknight-gh.github.io/helm-charts