See previous posting for more details on the Teamtask / List algorithm.
Quick summary: The algorithm has been called List because it can be used in the most basic use case, but frequently desired use case of needing to perform an action on multiple items, given a list of 100 hostnames for example, ping each one and see if it responds, track status of whether the hostname has been processed, so that if the script has to be restarted we don’t have to repeat work. Turns out though, that if the implementation successfully implements a mutex, such as by the correct use of a database, multiple clients can help to process the list resulting in a well orchestrated distributed processing engine, (hence, Teamtask).
List has been implemented most recently as a webapi covering all the use cases one would expect, and placing this into a container as a microservice along with a database and respective helm chart is the next step. But what about beyond that?
List works by breaking up a list of items into blocks for processing, by selecting an appropriate size the server will not carry a high cpu load or be busy. Additionally a main List implementation can be used to hand out very large blocks, which secondary List servers can consume and then break up into smaller chunks for their clients, further reducing the load on the main server. A misconfigured Job though, say of blocksize one, with millions of items to process and hundreds of clients could produce some peak resource consumption unnecessarily. With this in mind, what if we were to use a controller in Kubernetes itsself?
Kubernetes at its core is a controller engine. Controllers recognize yaml defined objects such as deployments, services, and ingresses. Kubernetes is used around a certain set of controllers related to container management but really, we can create controllers for almost anything. We could create a controller that knows how to play Tic-Tac-Toe for example, defining a game with a current state in a yaml. A controller recognizing the yaml could see that the state indicates a move needs to be made and could make a move and update the status of the object / yaml. One could see how such a system could be used to coordinate many games and through the nature of kubernetes, everything would scale, in the same way kubernetes manages the state of hundreds of deployments it could manage the state of multiple instances of a game.
As an excuse to write a kubernetes controller just for fun we could implement List. A yaml with an api could define a List to process. The controller could then create a block, or a few blocks, for clients to process. The original List could define how many items to process, the size of a block to use, who has permissions to work on the blocks, etc … The idea of a “client” could also be a controller within kubernetes. This could be implemented in a similar way like how argocd recognizes an applicationset and upon processing creates one or more applications, which it also knows how to process. By using kubernetes we could use its database and not have to deploy our own. Potentially, resources could get out of hand if misconfigured, so safety checks would need to be put in place, but with our merging algorithm as described before the “behind the scenes kubernetes database” use should and cpu use should minimal on the list management side.
Such a controller would get all the benefits of using kubernetes, we could take advantage of built in error checking and status of the type we see with pods and scaling. Such an implementation would lead to some fun investigations, how exactly does kubernetes manage all the pods it manages, is it checking them one at a time or all at one, a few at a time. Whatever algorithm kubernetes uses to manage pods would be the same algorithm used to manage the list blocks. Probably there are some built in limits to keep things sane, and perhaps we could take advantage of those.
Maybe controllers to process blocks wouldn’t bring any benefit, perhaps it would be better to just implement server side as a controller and clients could be run as Kubernetes Jobs, or just deployments setup to scale as desired, perhaps within resource quotas. Still, in either case, it might make sense to define a Block type which upon processing would get an index & size added. The Block could show as pending, in the same way an Ingress does while waiting for a loadbalancer ip, consumers could wait for the status to change and then work on the block. Upon completion the status could be advanced to ‘completed’ when finished or something similar to communicate to the controller that the block is done.
How cool would it be do to do a ‘kubectl -n <namespace> get blocks’ and get the blocks currently being worked on displayed in the familiar kubectl style with current status?
$ k get blocks -o wide
NAME READY STATUS RESTARTS INDEX SIZE
primenumber-578b4958fc-cvtbm 1/1 Ready 2 0 1000
primenumber-578b4958fc-segcfs 1/1 Ready 0 1001 1000
primenumber-578b4958fc-wersw 0/1 Pending 0 Pending Pending
If the List implementation were implemented as a controller within Kubernetes, we could still use it outside of the kubernetes cluster without having to implement a webapi because kubernetes itself can be accessed and used via a webapi, no kubectl required. Sweet!!! (course, we might not want users to access the kubeapi directly, wrap that api!)