Author Archive: travis

(idea) restarting a pod via webapi

Posted on by

At the end of a pipeline it can be nice to restart a pod.  In production this occurs via gitops automatically, but in dev it can be nice to not have to wait for git to sync.  Though there are multiple ways to do this common ways are:

  1. just put the cluster kubeconfig in a secret in the pipeline and use that to restart the pod (a little too powerful)
  2. create a service account, acquire its token and place in pipeline (safest, but takes some work for each app)

What about a solution similar to reloader?  Reloader watches for changes in configmaps and secrets which have a certain annotation and if a they change reloader restarts things.  We could just use reloader & make a change to a configmap in order to trigger the reload.

However, what about creating a controller which listens for a webapi call asking for a deployment to be restarted.  Then a pipeline could call the appropriate url to get things restarted.  By deploying via argocd using an applicationset and using a url convention based on the clustername then all development clusters could be enabled to used this method in their pipelines, consumers would only need to annotate their deployments/statefulset/etc …

(bash) way to implement an idle processing loop

Posted on by

Bash is what it is, often a quick solution to get something done, once things start to become too advance probably should be writing in a higher level language.

With that said, here’s using tcp via bash to implement an idle processing loop:

#!/bin/bash

# Start tail command in a separate process and track PID so we can take action later
# (send output to this shell's stdout)
tail /var/logs/* > /proc/$$/fd/1 &
PID_TAIL=$!

# Idle processing loop
while (true); do

  # Perform work while 
  if <test condition>; then
  
    # Stop the tail process
    disown $PID_TAIL;
    kill $PID_TAIL;

    # Exit idle processing loop
    break;
  fi

  # avoiding maxing cpu
  sleep 1

done

The trick here is ‘$$’ is the current shell’s PID, the /proc/$$/fd/1 is a tcp device which represents the shell’s stdout. By sending the tail command to this device as a background process with ‘&’ we still see the tail output on our console. Yet, the bash script is actually in a loop able to do whatever it wants while the tail is running. When the script sees some condition it cares about it can stop the tail process and exit, or just keep working until someone presses Ctrl-c.

For more information see: https://www.xmodulo.com/tcp-udp-socket-bash-shell.html

flutter webapp, securely calling a backend

Posted on by

Just thinking outloud,

Since a flutter webapp is all running in the client browser, it is not possible to access a backend which requires credentials in some commonly used methods safely.

  • Loading credentials via environment variables, in the way containers commonly do, isn’t safe because the .env file containing the environment variables can be browsed directly. https://github.com/java-james/flutter_dotenv/issues/74
  • Even if you are able to somehow get the credentials into the app, if they are credentials you don’t want the user to know, they can be exposed via dev tools … as everything is living in the client browser.

So how to connect to a web service backend from flutter?

You have to use an in-between backend, here are some options:

  • Implement a webapi which has methods created just for the flutter app
  • Implement a webapi with the intention of just passing the request along the backend and adding a header with the needed token, but also checking the request to be sure its only the type of request we want to allow.
  • Create an ingress passthrough which adds an appropriate token header and then calls the backend, careful though, does the token give the user too much access?

Note, this in-between webapi must be reachable from the client web browser, so it most likely must be protected, OIDC is a good option. Using the same OIDC parameters on both the flutter webapp and the in-between webapi will let an OIDC token gathered up via the webapp be passed along to the webapi without an additional login.

daytwo is almost ready for beta testing (argocd-daytwo)

Posted on by

Ability to use FQDN to access workload clusters, rather than ip address

Posted on by

Doing my part to try and help gitops take over the world.

By targeting applications to install on a kubernetes cluster by fqdn, rather than ip address, it becomes possible to delete a cluster and stand up a new cluster, and watch as all the applications reinstall automatically to the new cluster via gitops using something like argocd.

Feature request submitted (vcenter clusterapi provider):
https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/issues/2098

If implemented, this would enable all consumers of vmware vcenter / tanzu to better implement gitops.

Tempting to drop everything and be the one to implement the code and submit a pull request, just to get the credit.  Would feel good to know you’ve helped so many people.

Creating webapps w/ flutter & deploying to kubernetes

Posted on by

In the past I’ve written guis in many languages, but found my passion more in backend server apis.  Along this line REST, swagger, OIDC, and websockets have been a dream come true.

Then one day I discovered flutter, created by google, it is cross platform compiling natively on both Android and iPhone.

Flutter changed everything for me, it made me love creating guis, it felt like the first real solution to creating a gui with everything else prior being trail blazer projects helping to figure out everything needed to one day lead to the creation of flutter.

Turns out it also can be used to create windows & linux apps as well, with some exceptions.  On a mobile device you get a free database with your application, so if you deploy on windows / linux you have to solve the database on your own.

Today I noticed that flutter can also be used to create a webapp.  I’m thinking to create a webapp, place it inside a container and deploy it to kubernetes along with a database into the same namespace, making for a similar situation to how mobile apps get a database.  So much potential, this is exciting.

kubernetes daytwo controllers

Posted on by

Thinking again about daytwo operations (controllers watching for cluster events):

daytwo-argocd-register-controller

  • watches for cluster.yaml (tanzu, clusterapi, etc…) and registers clusters with argocd automatically once they are in ‘ready’ state
  • syncs ‘addons’ labels from cluster.yaml to argocd cluster secrets to auto install addons, including pinniped-concierge, and pinniped-www

daytwo-pinniped-register-controller

  • generates a pinniped kubeconfig if new cluster
  • adds to git repo if new cluster, removes from git repo if cluster decommissioned
  • regenerates configmap used by pinniped-www

daytwo-external-dns-register-controller

  • watch for service associated with cluster to appear & annotate with fqdn, goal: to add a dns entry for each cluster kubeapi

Or perhaps, if desired, the event could trigger automation somewhere else:

daytwo-cluster-event-controller

  • callback to jenkins
  • callback to awx
  • callback to vmware-aria
  • etc …

argocd & pinniped

Posted on by

Kubernetes implements OIDC via arguments on the API Server such as –oidc-issuer-url, etc. This works great, however a common utility to access kubernetes in this way kubelogin isn’t so great in that an open web browser closed in the wrong way can cause the authentication to fail. So what’s a better solution?

Pinniped allows the configuring of OIDC and login on a running kubernetes cluster. This is great in that you don’t have to restart the cluster to get OIDC to work, and sometimes you just don’t have access to configure OIDC via the API Server.

Pinniped can be configured and deployed in the usual way that you deploy addons to your clusters. (see argocd – getting started, and argocd – getting started – addons)

Whether using the native OIDC solution or using pinniped, a kubeconfig must be created which can be used with kubectl to access the clusters. Pinniped has a utility to create the kubeconfig, but how to get to the end user? There have been various methods to accomplish this:

  • the user submits a ticket, through automation you send them the pinniped kubeconfig via email
  • all pinniped kubeconfig files are made available via a git repo all clients have access to
  • (here’s our idea) make the pinniped kubeconfig available via a website, so we don’t need to grant git repo access in order to deliver the kubeconfig

My strategy has been to use argocd, applicationsets, and a single container running on all clusters as a means to distribute the pinniped kubeconfig to all interested parties.

* Note: pinniped kubeconfigs contain no secrets, they do not need to be treated as secrets, other than as related to a potential denial of service attack, because the user will still need to login and access is granted via groups.

Here’s the implementation:

  • argocd applicationsets allow the use cluster name to be used as a variable, this means we can define the url we want in each cluster to be: https://pinniped.<cluster>.<domain>
  • the applicationset deploys a container running php which uses a script to determine the clustername. The php site displays some information to help the user get started with pinniped and a link to the pinniped kubeconfig
  • a configmap is mounted using a volume to the container and contains all pinniped kubeconfig files, this works because the url generated in the previous step includes the name of the cluster and so points to the correct pinniped kubeconfig
  • because this solution is deployed via argocd, when a new cluster is deployed and a pinniped kubeconfig is created & added to a git repo, a script re-generates the configmap from the git repo, all pinniped websites are updated automatically, we just have to add a step to update the pinniped configmap as part of the new cluster deployment process

(video: todo)

What is the best way to handle expiring certificate authority?

Posted on by
So far, the best solution I’ve found:
  • manually store the ca-bundle in a vault, such as azure vault or hashicorp vault
  • configure external secrets addon to use the vault
  • configure ca-bundle as volume mount using secret provided by external secrets in each pod that needs it
Now when the ca expires you only have to update the ca-bundle in a single location in the vault, argocd will take care of restarting the pods when the secret updates.

If anyone has a better idea feel free to share.

k8s daytwo operation ideas

Posted on by

controllers:

  • watch for cluster.yaml updates & reflect addons annotations from cluster.yaml to argocd cluster yaml secrets
  • watch argocd repo certs and check for expire, re-adding automatically
  • tanzu watch for service associated with cluster to appear & annotate with fqdn automatically (in order to add dns entry for each cluster kubeapi)
  • watch for certificate authority expiration and update ‘ca-bundle’ stored in vault