How I Installed Kubernetes From Scratch on a Bytemark VM
I run Luzme, an ebook search system, which runs 24x7 on about 30 virtual machines in 7 countries on 6 continents.
This article explains how I setup Kubernetes from scratch on a set of VMs from Bytemark, to improve the way I use the infrastructure behind Luzme, using the new container-based technologies, such as Docker and Kubernetes, to provide a better, more maintainable, more scalable system Luzme uses a diverse stack of technologies, including Percona (a MySQL-compatible database), Firebase (real-time NoSQL), Redis, Celery, Django, Solr, AngularJS, Python.
The majority of the system runs on VMs provided by the excellent Bytemark, a UK hosting company, with a few other VMs running in other countries where necessary.
I develop and support this system by myself. So I need the system to just work. I need it to be self-healing, where possible. And I would really like it to scale gracefully so that the next time I find Luzme on the front page of CNET or Lifehacker, the system scales to cope, rather than falling whimpering to its knees.
I liked the idea of using Kubernetes and Docker to containerise some aspects of this infrastructure. But although the documentation at the time made it very easy to get a single node up and running, it proved extremely hard to get a multi-node system up and running.
So this is my documentation of what I needed to do, to help others avoid the pain of discovery. Where possible, I have included listings of the network configuration at each stage, since this was the cause of almost all the problems I encountered.
Please note that this was done in January 2016; current documentation and procedures may be different now.
Getting Started
I followed this documentation.
http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html
The first step was to create 2 Ubuntu Trusty nodes, one master and one worker. I did this using the usual Bytemark setup process.
To add more worker nodes, I would just replicate the procedure for the first one.
Note that the documentation says: “Please install Docker 1.6.2 or Docker 1.7.1.”
Setup the Master node
So I created a standard Bytemark VM, using their base configuration of 1 GB memory, 25 GB disk space, choosing Ubuntu 14.04 LTS (Trusty) as the OS to install.
I saved the root password.
I prefer to use less powerful accounts where possible so configured the machine to let me in on a ssh key.
# mkdir ~/.ssh && chmod 700 ~/.ssh
# touch .ssh/authorized_keys
# chmod 600 .ssh/authorized_keys
I then copied my public key over to the new machine, adding it to .ssh/authorised_keys
Remove IPv6
The cause of many of my problems were due to a confusion in the install script between different results returned when asking programmatically for IP address or hostname.
Bytemark VMs come as standard with both IPv6 and IPv4.
I didn’t have sufficient time to understand why these different results were being returned, there was no obvious reason why that was so. So I decided to just disable IPv6.
Not ideal, and with more time, I would have investigated this further.
<code># nano /etc/sysctl.conf</code>
and add these lines to sysctl.conf file
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
# sysctl -p
Install Docker
# apt-get install docker.io
Note that this package DOES NOT HAVE the name docker in ubuntu.
So now my network config looked like this.
# ifconfig -a
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:11
inet addr:46.43.2.130 Bcast:46.43.2.255 Mask:255.255.255.0
inet6 addr: 2001:41c9:1:41f::130/64 Scope:Global
inet6 addr: fe80::fcff:ff:fe00:5511/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10428 errors:0 dropped:0 overruns:0 frame:0
TX packets:3801 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9825303 (9.8 MB) TX bytes:402822 (402.8 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Clone Kubernetes Code (v1.1.4)
$ git clone <a href="https://github.com/kubernetes/kubernetes.git">https://github.com/kubernetes/kubernetes.git</a>
$ git checkout v1.1.4
$ cd kubernetes/docs/getting-started-guides/docker-multinode
You will need the kubectl program which is not part of that repo.
So download that using the link in the kubernetes documentation, and put it in /usr/local/bin/.
Check Network Config
If you run these 5 commands now,
# ifconfig -a
# route
# docker ps
# docker -H unix:///var/run/docker-bootstrap.sock ps
# kubectl get nodes
you should now get output similar to this.
# ifconfig -a
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:11
inet addr:46.43.2.130 Bcast:46.43.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:93398 errors:0 dropped:0 overruns:0 frame:0
TX packets:43871 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:131469041 (131.4 MB) TX bytes:3516199 (3.5 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0
46.43.2.0 * 255.255.255.0 U 0 0 0 eth0
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# docker -H unix:///var/run/docker-bootstrap.sock ps
FATA[0000] Get http:///var/run/docker-bootstrap.sock/v1.18/containers/json: dial unix /var/run/docker-bootstrap.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?
# kubectl get nodes
error: couldn't read version from server: Get <a href="http://localhost:8080/api">http://localhost:8080/api</a>: dial tcp 127.0.0.1:8080: connection refused
Which tells us this:
* we have a docker interface in ifconfig
* we have a docker route in the routing table
* the standard docker daemon is running...
* ... but it doesn't have any live containers
* the 2nd docker daemon needed by kubernetes is not yet running...
* ... and neither is the kubernetes server
MASTER
So now we can install the kubernetes master node
cd kubernetes/docs/getting-started-guides/docker-multinode
./master.sh
Whereupon the result of those 5 commands now looks like this
and now we have
# ifconfig -a
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:10.1.55.1 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:11
inet addr:46.43.2.130 Bcast:46.43.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:155374 errors:0 dropped:0 overruns:0 frame:0
TX packets:54960 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:217671551 (217.6 MB) TX bytes:4489883 (4.4 MB)
flannel.1 Link encap:Ethernet HWaddr de:93:0f:c2:96:8f
inet addr:10.1.55.0 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2404 errors:0 dropped:0 overruns:0 frame:0
TX packets:2404 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:456749 (456.7 KB) TX bytes:456749 (456.7 KB)
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0
10.1.0.0 * 255.255.0.0 U 0 0 0 flannel.1
10.1.55.0 * 255.255.255.0 U 0 0 0 docker0
46.43.2.0 * 255.255.255.0 U 0 0 0 eth0
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1479e15ed371 <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube schedule About a minute ago Up About a minute k8s_scheduler.2b8ee744_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_61e81a02
2d17874be25e <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube apiserve About a minute ago Up About a minute k8s_apiserver.a94f0183_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_c6a20e6c
226f11219fbd <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube controll About a minute ago Up About a minute k8s_controller-manager.19f4ee5e_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_9ccccf86
b00a7288243d <a href="http://gcr.io/google_containers/pause:0.8.0">gcr.io/google_containers/pause:0.8.0</a> "/pause" About a minute ago Up About a minute k8s_POD.e4cc795_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_9a8bd96c
2a825a132d52 <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube proxy -- About a minute ago Up About a minute hopeful_davinci
ea52c28a9ac9 <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube kubelet About a minute ago Up About a minute ecstatic_nobel
# docker -H unix:///var/run/docker-bootstrap.sock ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6a523dd5f0bd <a href="http://quay.io/coreos/flannel:0.5.0">quay.io/coreos/flannel:0.5.0</a> "/opt/bin/flanneld - 2 minutes ago Up 2 minutes tender_kowalevski
d036c4f56cbd <a href="http://gcr.io/google_containers/etcd:2.0.12">gcr.io/google_containers/etcd:2.0.12</a> "/usr/local/bin/etcd 3 minutes ago Up 3 minutes romantic_mccarthy
# kubectl get nodes
NAME LABELS STATUS
127.0.0.1 <a href="http://kubernetes.io/hostname=127.0.0.1">kubernetes.io/hostname=127.0.0.1</a> Ready
WORKER
Add the IP address of the worker node to the firewall on the master.
Follow the same initial setup as before, and it should look similar to this.
root@kub2:~# ifconfig -a
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:12
inet addr:46.43.2.131 Bcast:46.43.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:94645 errors:0 dropped:0 overruns:0 frame:0
TX packets:35207 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:130735811 (130.7 MB) TX bytes:2631768 (2.6 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
root@kub2:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0
46.43.2.0 * 255.255.255.0 U 0 0 0 eth0
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0
root@kub2:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Edit and Run the Worker Install Code
There’s a line in the worker.sh file from the kubernetes repo which causes the IPv4/IPv6 confusion I mentioned earlier.
So edit worker.sh to remove that overrider line. Then run the work installation script.
export MASTER_IP=<public IP on master>
./worker.sh
So now on the worker node, we should have this:
# ifconfig -adocker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:10.1.76.1 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fe:ff:00:00:55:12
inet addr:46.43.2.131 Bcast:46.43.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:153405 errors:0 dropped:0 overruns:0 frame:0
TX packets:44774 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:211357174 (211.3 MB) TX bytes:3458177 (3.4 MB)
flannel.1 Link encap:Ethernet HWaddr 9e:80:4b:9b:9e:75
inet addr:10.1.76.0 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:100 (100.0 B) TX bytes:100 (100.0 B)
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 46-43-2-1.no-re 0.0.0.0 UG 0 0 0 eth0
10.1.0.0 * 255.255.0.0 U 0 0 0 flannel.1
10.1.76.0 * 255.255.255.0 U 0 0 0 docker0
46.43.2.0 * 255.255.255.0 U 0 0 0 eth0
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5b520d7b12e8 <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube proxy -- About a minute ago Up About a minute modest_mclean
cfb440bf9a5a <a href="http://gcr.io/google_containers/hyperkube:v1.0.3">gcr.io/google_containers/hyperkube:v1.0.3</a> "/hyperkube kubelet About a minute ago Up About a minute goofy_babbage
# docker -H unix:///var/run/docker-bootstrap.sock ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
569fc709ad84 <a href="http://quay.io/coreos/flannel:0.5.0">quay.io/coreos/flannel:0.5.0</a> "/opt/bin/flanneld - 3 minutes ago Up 3 minutes serene_hoover
# kubectl get nodes
NAME LABELS STATUS
127.0.0.1 <a href="http://kubernetes.io/hostname=127.0.0.1">kubernetes.io/hostname=127.0.0.1</a> Ready
46.43.2.131 <a href="http://kubernetes.io/hostname=46.43.2.131">kubernetes.io/hostname=46.43.2.131</a> Ready
FINISHED
And that’s it done. I wrote some ansible scripts to do this, which meant that once I’d figured all this out, it was possible to install a new node in a very short time.
But it took me about a month of digging through the Kubernetes install to get it working.
So I hope this helps someone!