I run Luzme, an ebook search system, which runs 24×7 on about 30 virtual machines in 7 countries on 6 continents.
This article explains how I setup Kubernetes from scratch on a set of VMs from Bytemark, to improve the way I use the infrastructure behind Luzme, using the new container-based technologies, such as Docker and Kubernetes, to provide a better, more maintainable, more scalable system
Luzme uses a diverse stack of technologies, including Percona (a MySQL-compatible database), Firebase (real-time NoSQL), Redis, Celery, Django, Solr, AngularJS, Python.
The majority of the system runs on VMs provided by the excellent Bytemark, a UK hosting company, with a few other VMs running in other countries where necessary.
I develop and support this system by myself. So I need the system to just work. I need it to be self-healing, where possible. And I would really like it to scale gracefully so that the next time I find Luzme on the front page of CNET or Lifehacker, the system scales to cope, rather than falling whimpering to its knees.
I liked the idea of using Kubernetes and Docker to containerise some aspects of this infrastructure. But although the documentation at the time made it very easy to get a single node up and running, it proved extremely hard to get a multi-node system up and running.
So this is my documentation of what I needed to do, to help others avoid the pain of discovery. Where possible, I have included listings of the network configuration at each stage, since this was the cause of almost all the problems I encountered.
Please note that this was done in January 2016; current documentation and procedures may be different now.
Getting Started
I followed this documentation.
The first step was to create 2 Ubuntu Trusty nodes, one master and one worker. I did this using the usual Bytemark setup process.
To add more worker nodes, I would just replicate the procedure for the first one.
Note that the documentation says: “Please install Docker 1.6.2 or Docker 1.7.1.”
Setup the Master node
So I created a standard Bytemark VM, using their base configuration of 1 GB memory, 25 GB disk space, choosing Ubuntu 14.04 LTS (Trusty) as the OS to install.
I saved the root password.
I prefer to use less powerful accounts where possible so configured the machine to let me in on a ssh key.
# mkdir ~/.ssh && chmod 700 ~/.ssh # touch .ssh/authorized_keys # chmod 600 .ssh/authorized_keys
I then copied my public key over to the new machine, adding it to .ssh/authorised_keys
Remove IPv6
The cause of many of my problems were due to a confusion in the install script between different results returned when asking programmatically for IP address or hostname.
Bytemark VMs come as standard with both IPv6 and IPv4.
I didn’t have sufficient time to understand why these different results were being returned, there was no obvious reason why that was so. So I decided to just disable IPv6.
Not ideal, and with more time, I would have investigated this further.
# nano /etc/sysctl.conf
and add these lines to sysctl.conf file
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
# sysctl -p
Install Docker
# apt-get install docker.io
Note that this package DOES NOT HAVE the name docker in ubuntu.
So now my network config looked like this.
# ifconfig -a docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99 inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:11 inet addr:46.43.2.130 Bcast:46.43.2.255 Mask:255.255.255.0 inet6 addr: 2001:41c9:1:41f::130/64 Scope:Global inet6 addr: fe80::fcff:ff:fe00:5511/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10428 errors:0 dropped:0 overruns:0 frame:0 TX packets:3801 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:9825303 (9.8 MB) TX bytes:402822 (402.8 KB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Clone Kubernetes Code (v1.1.4)
$ git clone https://github.com/kubernetes/kubernetes.git $ git checkout v1.1.4 $ cd kubernetes/docs/getting-started-guides/docker-multinode
You will need the kubectl program which is not part of that repo.
So download that using the link in the kubernetes documentation, and put it in /usr/local/bin/.
Check Network Config
If you run these 5 commands now,
# ifconfig -a # route # docker ps # docker -H unix:///var/run/docker-bootstrap.sock ps # kubectl get nodes
you should now get output similar to this.
# ifconfig -a docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99 inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:11 inet addr:46.43.2.130 Bcast:46.43.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:93398 errors:0 dropped:0 overruns:0 frame:0 TX packets:43871 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:131469041 (131.4 MB) TX bytes:3516199 (3.5 MB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) # route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0 46.43.2.0 * 255.255.255.0 U 0 0 0 eth0 172.17.0.0 * 255.255.0.0 U 0 0 0 docker0 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES # docker -H unix:///var/run/docker-bootstrap.sock ps FATA[0000] Get http:///var/run/docker-bootstrap.sock/v1.18/containers/json: dial unix /var/run/docker-bootstrap.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS? # kubectl get nodes error: couldn't read version from server: Get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused
Which tells us this:
- we have a docker interface in ifconfig
- we have a docker route in the routing table
- the standard docker daemon is running…
- … but it doesn’t have any live containers
- the 2nd docker daemon needed by kubernetes is not yet running…
- … and neither is the kubernetes server
MASTER
So now we can install the kubernetes master node
cd kubernetes/docs/getting-started-guides/docker-multinode ./master.sh
Whereupon the result of those 5 commands now looks like this
and now we have
# ifconfig -a docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99 inet addr:10.1.55.1 Bcast:0.0.0.0 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:11 inet addr:46.43.2.130 Bcast:46.43.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:155374 errors:0 dropped:0 overruns:0 frame:0 TX packets:54960 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:217671551 (217.6 MB) TX bytes:4489883 (4.4 MB) flannel.1 Link encap:Ethernet HWaddr de:93:0f:c2:96:8f inet addr:10.1.55.0 Bcast:0.0.0.0 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:2404 errors:0 dropped:0 overruns:0 frame:0 TX packets:2404 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:456749 (456.7 KB) TX bytes:456749 (456.7 KB) # route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0 10.1.0.0 * 255.255.0.0 U 0 0 0 flannel.1 10.1.55.0 * 255.255.255.0 U 0 0 0 docker0 46.43.2.0 * 255.255.255.0 U 0 0 0 eth0 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1479e15ed371 gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube schedule About a minute ago Up About a minute k8s_scheduler.2b8ee744_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_61e81a02 2d17874be25e gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube apiserve About a minute ago Up About a minute k8s_apiserver.a94f0183_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_c6a20e6c 226f11219fbd gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube controll About a minute ago Up About a minute k8s_controller-manager.19f4ee5e_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_9ccccf86 b00a7288243d gcr.io/google_containers/pause:0.8.0 "/pause" About a minute ago Up About a minute k8s_POD.e4cc795_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_9a8bd96c 2a825a132d52 gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube proxy -- About a minute ago Up About a minute hopeful_davinci ea52c28a9ac9 gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube kubelet About a minute ago Up About a minute ecstatic_nobel # docker -H unix:///var/run/docker-bootstrap.sock ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6a523dd5f0bd quay.io/coreos/flannel:0.5.0 "/opt/bin/flanneld - 2 minutes ago Up 2 minutes tender_kowalevski d036c4f56cbd gcr.io/google_containers/etcd:2.0.12 "/usr/local/bin/etcd 3 minutes ago Up 3 minutes romantic_mccarthy # kubectl get nodes NAME LABELS STATUS 127.0.0.1 kubernetes.io/hostname=127.0.0.1 Ready
WORKER
Add the IP address of the worker node to the firewall on the master.
Follow the same initial setup as before, and it should look similar to this.
root@kub2:~# ifconfig -a docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99 inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:12 inet addr:46.43.2.131 Bcast:46.43.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:94645 errors:0 dropped:0 overruns:0 frame:0 TX packets:35207 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:130735811 (130.7 MB) TX bytes:2631768 (2.6 MB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) root@kub2:~# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0 46.43.2.0 * 255.255.255.0 U 0 0 0 eth0 172.17.0.0 * 255.255.0.0 U 0 0 0 docker0 root@kub2:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Edit and Run the Worker Install Code
There’s a line in the worker.sh file from the kubernetes repo which causes the IPv4/IPv6 confusion I mentioned earlier.
So edit worker.sh to remove that overrider line. Then run the work installation script.
export MASTER_IP=<public IP on master> ./worker.sh
So now on the worker node, we should have this:
# ifconfig -adocker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99 inet addr:10.1.76.1 Bcast:0.0.0.0 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:12 inet addr:46.43.2.131 Bcast:46.43.2.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:153405 errors:0 dropped:0 overruns:0 frame:0 TX packets:44774 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:211357174 (211.3 MB) TX bytes:3458177 (3.4 MB) flannel.1 Link encap:Ethernet HWaddr 9e:80:4b:9b:9e:75 inet addr:10.1.76.0 Bcast:0.0.0.0 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:100 (100.0 B) TX bytes:100 (100.0 B) # route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0 10.1.0.0 * 255.255.0.0 U 0 0 0 flannel.1 10.1.76.0 * 255.255.255.0 U 0 0 0 docker0 46.43.2.0 * 255.255.255.0 U 0 0 0 eth0 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5b520d7b12e8 gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube proxy -- About a minute ago Up About a minute modest_mclean cfb440bf9a5a gcr.io/google_containers/hyperkube:v1.0.3 "/hyperkube kubelet About a minute ago Up About a minute goofy_babbage # docker -H unix:///var/run/docker-bootstrap.sock ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 569fc709ad84 quay.io/coreos/flannel:0.5.0 "/opt/bin/flanneld - 3 minutes ago Up 3 minutes serene_hoover # kubectl get nodes NAME LABELS STATUS 127.0.0.1 kubernetes.io/hostname=127.0.0.1 Ready 46.43.2.131 kubernetes.io/hostname=46.43.2.131 Ready
FINISHED
And that’s it done. I wrote some ansible scripts to do this, which meant that once I’d figured all this out, it was possible to install a new node in a very short time.
But it took me about a month of digging through the Kubernetes install to get it working.
So I hope this helps someone!