Category Archives: Kubernetes
(gge) first attempt at creating a crd (CustomResourceDefinition)
It was pretty quick to get something up and working. With time it should be possible to improve the CRDs by adding required fields and restrict certain fields to only allow specific values. Here’s how things are looking after this evenings investigations:
Redesigning gge (generic game engine), multi-player game lifecycle management, using kubernetes (for ai clients)
Am planning a controller to process custom resource definitions which hold all data. By using this method a separate standalone database will not be needed, and the solution can scale as much as desired by increasing resources of the cluster itself. Thousands of games? Millions of games? And if there isn’t really a limit, what’s performance going to be like? These scaling answers must be investigated.
Besides the above, webapis, REST, OIDC, and ingresses mapped to appropriate paths will provide the infrastructure. Looking forward to experimenting more with scaling and high-availability methods provided by kubernetes.
Sure would be nice to work on open-source 100% of the time, wonder if there might be a way to make that happen …
Initial thoughts …
use cases:
- player 0 is always game controller (can do things with items)
- 2 players are 1,2
- 4 players are 1,2,3,4
- 2 player games
- 4 player games
- invite rejected
- invite accepted
- start game once enough players have joined
- send notice:
- game has started
- plays which have occured
- player finished their turn (if not real time strategy)
- if play was not allowed
maybe not:
- define a unit as unique?
crds:
---
boardgame.gge.aarr.xyz
invite.gge.aarr.xyz
event.gge.aarr.xyz ?
tic-tac-toe.game.gge.aarr.xyz
tic-tac-toe.play.gge.aarr.xyz (a controller for each game to process just that game)
reversi.game.gge.aarr.xyz
reversi.play.gge.aarr.xyz (a controller for each game to process just that game)
tournament.gge.aarr.xyz
- single-elimination.tournament.gge.aarr.xyz
- double-elimination.tournament.gge.aarr.xyz
[crd: boardgame.gge.aarr.xyz]
---
short: string
long: string
enabled: bool
path: string
[tic-tac-toe.boardgame.gge.aarr.xyz] (example instance)
---
short: "tic-tac-toe"
long: "Tic-Tac-Toe"
enabled: true
path: "/api/game/tic-tac-toe"
[crd: invite.gge.aarr.xyz]
---
game: string
user: string
[tic-tac-toe_<timestamp>.invite.gge.aarr.xyz] (example instance)
---
game: tic-tac-toe_<timestamp>
user: asdf <user being invited>
[crd: tic-tac-toe.game.gge.aarr.xyz]
---
owner: string
description: tic-tac-toe
is_invite_only: bool
is_public_view: bool
turn_type: string
allowed_num_players:
- <list>
invites:
- <list>{user: aaa, status: accepted}
- <list>{user: bbb, status: accepted}
grids:
- {id: 0, width: 3, height: 3}
players:
- <list>{id: 1, user: aaa}
- <list>{id: 2, user: bbb}
units: (available units in crd)
- id: 0
desc: string
ascii: string of length 1
imgsrc: <url>string
players:
- 1 (list of players allowed to use this unit)
actions:
- add (list of actions which may be performed)
attrs:
- (optional) list of name/value attributes
plays:
- <list>
[tic-tac-toe_<timestamp>.game.gge.aarr.xyz] (example instance)
---
owner: tloyd
is_invite_only: true
is_public_view: true
turn_type: round_robin
allowed_num_players:
- 2
invites:
- {user: aaa, status: accepted}
- {user: bbb, status: accepted}
players:
- {id: 1, user: aaa}
- {id: 2, user: bbb}
items: (units in play)
- {id: 0, unit: <id>, grid: 0, y: 1, x: 1, level: 0, player: 0}
- {id: 1, unit: <id>, grid: 0, y: 0, x: 0, level: 0, player: 1}
plays:
- id: 0
player: 1
plays:
- {id: 0, action: add, grid: 0, y: 1, x: 1, level: 0}
- id: 1
player: 2
plays:
- {id: 0, action: add, grid: 0, y: 0, x: 0, level: 0}
[tic-tac-toe.play.gge.aarr.xyz]
---
game: string
player: <id>
plays:
- <list>{id: 0, action: add, grid: 0, y: 1, x: 1, level: 0, unit: <id>}
[tic-tac-toe_<timestamp>.tic-tac-toe.play.gge.aarr.xyz] (example instance)
---
game: tic-tac-toe_<timestamp>
player: 1
plays:
- <list>{id: 0, action: add, grid: 0, y: 1, x: 1, level: 0, unit: <id>}
ingress: cert-manager, letsencrypt, and travisloyd.xyz -> www.travisloyd.xyz
It seemed for awhile that the popular web browsers would automatically redirect travisloyd.xyz to www.travisloyd.xyz if travisloyd.xyz didn’t work. But, after awhile that no longer seemed to happen. So, let’s do this right.
Here’s an ingress to perform the redirect from travisloyd.xyz to www.travisloyd.xyz:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
#nginx.ingress.kubernetes.io/proxy-body-size: 1000m
nginx.ingress.kubernetes.io/server-snippet: |
return 301 https://www.travisloyd.xyz$request_uri;
name: ingress-redirect
spec:
ingressClassName: nginx
rules:
- host: travisloyd.xyz
tls:
- hosts:
- travisloyd.xyz
secretName: travisloyd.xyz-tls
But what about an automatic certificate via letsencrypt? Do we need it? Yes, otherwise https://travisloyd.xyz displays an invalid certificate before performing the redirect. But, we can’t just add the annotations for cert-manager to this redirect because the call back from lets encrypt will not verify correctly with the redirect. Instead, we need an ingress specifically for handling the letsencrypt callback:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/issuer: cluster-letsencrypt-issuer
cert-manager.io/issuer-kind: ClusterIssuer
#nginx.ingress.kubernetes.io/proxy-body-size: 1000m
name: ingress-redirect-letsencrypt
spec:
ingressClassName: nginx
rules:
- host: travisloyd.xyz
http:
paths:
- backend:
service:
name: exp-wordpress-xyz-travisloyd-www
port:
name: http
path: /.well-known
pathType: Prefix
tls:
- hosts:
- travisloyd.xyz
secretName: travisloyd.xyz-tls
Perfect, now when the certs expire they’ll be renewed automatically via letsencrypt.
(ulimit) Heads up for anyone following my videos to install kubernetes on redhat
Redhat has an infinite value for ulimit by default which kubernetes inherits via the container service being used, this can result in some pods maxing out cpu and memory (such as haproxy and rabbitmq). For containerd the following fix solved the issue:
# sed -i 's/LimitNOFILE=infinity/LimitNOFILE=65535/' /usr/lib/systemd/system/containerd.service
# systemctl daemon-reload
# systemctl restart containerd
# k delete deployment <asdf>
homelab: planning next incarnation
Thinking about redeploying my homelab from scratch, perhaps switching from xenserver back to vmware. I’d like to start out with external-secrets and have all secrets in a vault right from the beginning, also curious what a 100% open source, 100% kubernetes environment would look like. Maybe two networks, one 100% kubernetes, and a 2nd for windows client systems. Here’s the k8s plan so far:
k-seed:
- manual setup of seed cluster
- helm install argocd
- argocd install clusterapi/crossplane/etc...
- seed-argocd deploy non-production cluster using vcluster or clusterapi/crossplane/etc...
- deploy metallb & configure loadbalancer ip range (can we automate this w/ cluster deploy?)
- add cluster to seed-argocd instance
- seed-argocd deploy production cluster using vcluster or clusterapi/crossplane/etc...
- deploy metallb & configure loadbalancer ip range (can we automate this w/ cluster deploy?)
- add cluster to seed-argocd instance
- seed-argocd deploy argocd to production cluster (k-prod)
k-prod:
- argocd configure storageclass
- argocd deploy hashicorp vault
- configure as certificate authority
- configure as keyvault
- argocd deploy external-secrets
- configure to use keyvault
- add secret 'ca-bundle.crt': public certificate authority certificate in DER format
- *from now on all secrets to get values via external-secrets
- argocd deploy cert-manager
- configure to use hashicorp vault as certificate authority
- argocd deploy pihole
- configure dns1 & dns2
- argocd deploy external-dns
- configure to use pihole as dns
- update with annotations to use external-dns & cert-manager:
- argocd
- vault
- pihole
- *from now on all ingress yaml to include annotations for external-dns & cert-manager
- recommended: have annotations from the beginning, at this point they will start working
- argocd deploy keycloak
- configure realm: create or import from backup
- add secret 'default_oidc_client_secret': secret part of oidc client/secret
- configure a user account (or configure federation via AD, openldap, etc...)
- deploy all other apps
- oidc client_secret should come from external-secrets in all apps configured with oidc
- this might require an init container for some apps
k-ceph:
- pvc storage for all clusters
- block storage can be used for vm disks (making for easy hotswap)
- upgrade to 2 10gb ports on each host system
wdc: (kubevirt in theory but think i'll stick w/ a vm)
- domain controller
- user management
- dhcp
- wds
- wsus using dev sqlserver & data stored on e drive
OIDC for Contributors
When you put in a feature request to add OIDC to your favorite project, feel free to include a link to this video.
Was thinking about writing a lunch voting app for work
Something similar to https://lunch.pink but specific for work, so running in a container with OIDC setup for coworkers. Coworker then could create a lunch event public or private, invite folks, folks could respond with interest and vote for their favorite restaurants, or just keep their same favorite restaurants from last time.
As a webapi could create integrations into Teams or a mobile app.
Could be a good excuse to write a controller and some custom resource definitions. Some initial ideas captured:
---
New-Restaurant
Add new restaurant to favorites list
Get-Restaurant
Get listing of preferred restaurants
Set-Restaurant
Enable/disable restaurant as favorite to include with voting
Remove-Restaurant
Remove restaurant from favorites list
---
New-Event
Create new event
-Description ""
-IsViewable <true>
-IsInviteOnly <false>
-Timeout <when_voting_ends, default 5 minutes>
Get-Event
Show all events if admin, otherwise only show public events, or events you've been invited to
Set-Event
Adjust attributes 'IsViewable', 'IsInviteOnly', and reopen voting.
Remove-Event
Remove event
---
New-Invite
Create new invitation to event
- only owner can create invitation
- invite id is a guid
-Username <username> // <optional> send through teams, or email, etc...
-EventId
Get-Invite
View current/expired invitations
Set-Invite
Accept / Reject an invitation
Remove-Invite
Withdrawl an invite