With kubernetes comes load balancing, and with load balancing comes a need for a range of ip addresses dedicated to load balancing. Besides load balancing, just for keeping things organized with multiple clusters separate ip ranges can be nice. This article will let you create this type of multi-cluster setup each with their own ip range:
Create private virtual switch
Using your preferred virtualization technology create a private virtual switch, here we create using powershell on an Hyper-v server:
New-VMSwitch -SwitchName “k-dev” -SwitchType Private
Create vms to use with new cluster
|controlplane master 01
|controlplane master 02
|worker node 01
|worker node 02
Configure gateway system
- Add second nic attached to lan network. Use a static ip, in this setup we will use 192.168.1.100.
- Enable ip forwarding:
- sudo echo “net.ipv4.ip_forward=1” >> /etc/sysctl.conf
- sudo sysctl -p
- Disable default gateways by modifying /etc/sysconfig/network-scripts/ifcfg-eth# and commenting out the GATEWAY values (CentOS, if not CentOS use distribution specific method)
- Add a default route which uses the local lan default gateway (use distribution specific method to make this permanent, my lan gateway is 192.168.1.1, adjust if yours is different)
- route add default gw 192.168.1.1 metric 25
- Add a route which uses the local lan for traffic specific to the local lan, this is required to avoid an asymmetric routing issue. Packets headed to the private lan from the local lan may not use the same path without it (if you are seeing a 60 second timeout with all tcp connections this is why):
- route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.1 metric 25
- Note, the route commands above have a metric of 25, this needs to be lower than all other route metrics so they are used first so if your metrics are lower then adjust accordingly.
Configure VMs on private network
|Use as specified above (or similar)
|If using as specified above then 255.255.255.0
|Use the ip of your normal lan dns. In this setup all vms on the private lan
can communicate with all systems on your local lan and vice versa via the
|The ip of the VM with two nics on the private lan, if using the ips above then
this will be 192.168.100.10
Configure your lan router with a static route
This part is tricky in that I cannot tell you how to configure your router. In most cases even the most basic router that you might be using to connect your local lan to the Internet will have an option to configure a static route.
You’ll most likely have two options:
- Configure a single static route to the whole subnet such as:
- 192.168.100.0/24 via 192.168.1.100
- Configure a single static route for each ip address such as:
- 192.168.100.10 netmask 255.255.255.0 via 192.168.1.100
- 192.168.100.11 netmask 255.255.255.0 via 192.168.1.100
- 192.168.100.12 netmask 255.255.255.0 via 192.168.1.100
- 192.168.100.21 netmask 255.255.255.0 via 192.168.1.100
- 192.168.100.22 netmask 255.255.255.0 via 192.168.1.100
Add an A record for each host in DNS with its static ip used on the private ip range.
Adjust subnet used by router
Depending on your router, (Netgear R6220 does not require this, RAX20 does, etc…), you may need to adjust the subnet of your router. The ip of a router is often 192.168.0.1 with subnet mask 255.255.255.0, since you are now using 192.168.100.0/24 for your private lan it may be necessary to adjust the subnet mask to 255.255.0.0 (or similar) to accommodate the 100 range.
You might need this step if your private network can reach the lan, the lan can reach your private lan, but your private lan cannot reach through the router out to the internet.
You are done!
Now you should be able to reach all systems on the private lan via systems on your local lan as well as the other way around, as well as Internet access. You should be able to join your private lan systems to the domain you are running on your local lan, if you have one, and log in without a password using kerberos, if you have that setup. Enjoy!
The article you just read is a bit rare on the Internet. I suspect this is because most folks who would like to add a private lan in this way to be fully connected to an existing ip range lack the experience with route to do so, and so you may be pushing your limits. If things are not working for you try not to be too hard on yourself and:
- disable the firewall & selinux on the gateway system till you get things working
- use traceroute or tracepath from both the client system on your local lan and from the vm on the private lan and ensure they are walking the same path
- if you used dhcp to load your vms ensure there is no remaining dhcp lease, if there is it will keep altering your static ip record in dns until it expires