Like rebuilding a Shinto shrine

Traditionally Shinto shrines are rebuilt exactly the same next to the old shrine every so many years.  The old shrine is removed and when the time comes it will be rebuilt again.

Something similar can apply to home environments.  Recently I nuked everything and rebuilt from the ground up.  Something I’ve always done after 6 months or a year, for security reasons and to ensure I am always getting the fastest performance from infrastructure.

Such reinstalling is a natural fit for kubernetes.  There are several methods for spinning up a cluster, and after that just by the nature of kubernetes being yaml files it is easy to spin up the services you had running before, and watch them self register in the new dns and self generate certificates with the new active directory certificate authority.  Amazing.   Kubernetes is truly, a work of art.

What is Kubernetes really?

As I take the deep dive into kubernetes what I’m finding is, though definitely a container management system, it can also been seen as a controller yaml processing engine.  Let me explain.

Kubernetes understands what a deployment is, and what a service is, these are defined as yaml and loaded.  Deployments and services can be seen as controllers which understand those types of objects defined in yaml.

What is interesting about this is that we can implement our own controllers.  For example, I could implement a controller that understands how to manage a tic-tac-toe game.  That controller could also implement an ai that knows how to play the game.  In the same way you can edit a deployment you could edit the game and the kubernetes infrastructure could respond to the change.  Or, a move could be another type recognized by the game controller, so you could create a move associated with a game in the same way you can create a service associated with a deployment.

You can imagine doing a ‘k get games’ and seeing the games being played listed out.  As well as ‘k describe game a123’ to get the details and status of the game.

Seems I’m not the only one who has started thinking down this line.  A quick Google search reveals agones.

This is fascinating and gives me a lot of ideas on how I might reimplement my list processing server & generic game server, within the kubernetes framework.

New helm chart: wireguard-centos-8-stream

My first helm chart, a fun milestone.  Used it to install my new docker container uploaded to this morning.

Nice feeling to give back to the open source community.

Now to automate:
* watch for wireguard updates & release an updated docker image
* watch for a centos-8-stream update & release an updated docker image
* watch for a helm chart update & update what is necessary for those changes to be seen

But first, time to investigate and implement longhorn.

New container: docker-wireguard-centos-8-stream

A wireguard container built for centos-8-stream which takes advantage of the scripts from the linuxserver docker-wireguard project.


LinuxServer docker-wireguard project:

To use simply replace the docker-wireguard image with:

Note: Initial startup may take quite awhile, 4 minutes +, if the wireguard module is being recompiled. Be sure to use a volume for the modules folder to avoid having to recompile.

Kubernetes, a hacker’s paradise (the good kind of hacker)

The world of Kubernetes is exactly why I got into computers back in the day, always something new to learn, the fun of problem solving, and being rewarded with new capabilities.


Just initially getting up to speed with an udemy class, setting up my first clusters was so fun and rewarding then, learning about metallb, nginx-ingress, metrics, cert-manager, adding an nfs provisioner, and later a cifs share.  Setting up ip ranges for the clusters and reinstalling everything, then automating the process.

Along the way you are moving over all your existing services and discover helm exists and the services you want are already out there and easily installed… only to discover hey, this project may be somewhat bleeding edge and I can contribute code, already giving back to the community… how cool!

Each time a vm is shutdown and resources are recovered, victory!


Such a bummer when you have a service that doesn’t want to run in kube.  So far only one, wireguard not wanting to run on top of a centos 8 stream based cluster… but I can drive a solution for us.

It is a bit all consuming, I’d like to get back to developing projects, and moving them into my new clusters.  It’s almost time though, almost time, feeling so empowered with all the possibilities, exciting!!!

Update: I got wireguard working. Working to get that shared out with the open source community. Is there anything kubernetes can’t do?

A separate IP range for each local kubernetes cluster


With kubernetes comes load balancing, and with load balancing comes a need for a range of ip addresses dedicated to load balancing. Besides load balancing, just for keeping things organized with multiple clusters separate ip ranges can be nice. This article will let you create this type of multi-cluster setup each with their own ip range:


Create private virtual switch

Using your preferred virtualization technology create a private virtual switch, here we create using powershell on an Hyper-v server:

New-VMSwitch -SwitchName “k-dev” -SwitchType Private

Create vms to use with new cluster

k-dev-mhaproxy server192.168.100.10
k-dev-m01controlplane master 01192.168.100.11
k-dev-m02controlplane master 02192.168.100.12
k-dev-n01worker node 01192.168.100.21
k-dev-n02worker node 02192.168.100.22

Configure gateway system

  • Add second nic attached to lan network. Use a static ip, in this setup we will use
  • Enable ip forwarding:
    • sudo echo “net.ipv4.ip_forward=1” >> /etc/sysctl.conf
    • sudo sysctl -p
  • Disable default gateways by modifying /etc/sysconfig/network-scripts/ifcfg-eth# and commenting out the GATEWAY values (CentOS, if not CentOS use distribution specific method)
  • Add a default route which uses the local lan default gateway (use distribution specific method to make this permanent, my lan gateway is, adjust if yours is different)
    • route add default gw metric 25
  • Add a route which uses the local lan for traffic specific to the local lan, this is required to avoid an asymmetric routing issue. Packets headed to the private lan from the local lan may not use the same path without it (if you are seeing a 60 second timeout with all tcp connections this is why):
    • route add -net netmask gw metric 25
  • Note, the route commands above have a metric of 25, this needs to be lower than all other route metrics so they are used first so if your metrics are lower then adjust accordingly.

Configure VMs on private network

ip addrUse as specified above (or similar)
netmaskIf using as specified above then
Use the ip of your normal lan dns. In this setup all vms on the private lan
can communicate with all systems on your local lan and vice versa via the
gatewayThe ip of the VM with two nics on the private lan, if using the ips above then
this will be

Configure your lan router with a static route

This part is tricky in that I cannot tell you how to configure your router. In most cases even the most basic router that you might be using to connect your local lan to the Internet will have an option to configure a static route.

You’ll most likely have two options:

  • Configure a single static route to the whole subnet such as:
    • via
  • Configure a single static route for each ip address such as:
    • netmask via
    • netmask via
    • netmask via
    • netmask via
    • netmask via

Configure DNS

Add an A record for each host in DNS with its static ip used on the private ip range.

Adjust subnet used by router

Depending on your router, (Netgear R6220 does not require this, RAX20 does, etc…), you may need to adjust the subnet of your router. The ip of a router is often with subnet mask, since you are now using for your private lan it may be necessary to adjust the subnet mask to (or similar) to accommodate the 100 range.

You might need this step if your private network can reach the lan, the lan can reach your private lan, but your private lan cannot reach through the router out to the internet.

You are done!

Now you should be able to reach all systems on the private lan via systems on your local lan as well as the other way around, as well as Internet access. You should be able to join your private lan systems to the domain you are running on your local lan, if you have one, and log in without a password using kerberos, if you have that setup. Enjoy!


The article you just read is a bit rare on the Internet. I suspect this is because most folks who would like to add a private lan in this way to be fully connected to an existing ip range lack the experience with route to do so, and so you may be pushing your limits. If things are not working for you try not to be too hard on yourself and:

  • disable the firewall & selinux on the gateway system till you get things working
  • use traceroute or tracepath from both the client system on your local lan and from the vm on the private lan and ensure they are walking the same path
  • if you used dhcp to load your vms ensure there is no remaining dhcp lease, if there is it will keep altering your static ip record in dns until it expires

Good Luck!

Kubernetes? A comic to get started …

Kubernetes Comic
(It’s Greek for Helmsman)

I went through the steps provided in that comic and they don’t work as is.  Figured things out, use these updated steps instead if you want to go through the example:

# use to set a default zone
gcloud init --console-only

gcloud container clusters create myCluster
#kubectl run app --image
kubectl create deployment app --image
kubectl scale deployment app --replicas 3
#kubectl expose deployment app --port 80 --type=LoadBalancer
kubectl expose deployment app --port 80 --type=LoadBalancer --target-port 8080
kubectl get service app
curl  (use the external ip shown in previous step)
#kubectl set image deployment app
kubectl set image deployment app

# bonus, here's an autoscale example
kubectl autoscale deployment app --cpu-percent=80 --min=1 --max=5