How I Installed Kubernetes From Scratch on a Bytemark VM

I run Luzme, an ebook search system, which runs 24×7 on about 30 virtual machines in 7 countries on 6 continents.

This article explains how I setup Kubernetes from scratch on a set of VMs from Bytemark, to improve the way I use the infrastructure behind Luzme, using the new container-based technologies, such as Docker and Kubernetes,  to provide a better, more maintainable, more scalable system

Luzme uses a diverse stack of technologies, including Percona (a MySQL-compatible database), Firebase (real-time NoSQL), Redis, Celery, Django, Solr, AngularJS, Python.

The majority of the system runs on VMs provided by the excellent Bytemark, a UK hosting company, with a few other VMs running in other countries where necessary.

I develop and support this system by myself. So I need the system to just work. I need it to be self-healing, where possible. And I would really like it to scale gracefully so that the next time I find Luzme on the front page of CNET or Lifehacker, the system scales to cope, rather than falling whimpering to its knees.

I liked the idea of using Kubernetes and Docker to containerise some aspects of this infrastructure. But although the documentation at the time made it very easy to get a single node up and running, it proved extremely hard to get a multi-node system up and running.

So this is my documentation of what I needed to do, to help others avoid the pain of discovery. Where possible, I have included listings of the network configuration at each stage, since this was the cause of almost all the problems I encountered.

Please note that this was done in January 2016; current documentation and procedures may be different now.

Getting Started

I followed this documentation.
The first step was to create 2 Ubuntu Trusty nodes, one master and one worker. I did this using the usual Bytemark setup process.
To add more worker nodes, I would just replicate the procedure for the first one.
Note that the documentation says: “Please install Docker 1.6.2 or Docker 1.7.1.”

Setup the Master node

So I created a standard Bytemark VM, using their base configuration of 1 GB memory, 25 GB disk space, choosing Ubuntu 14.04 LTS (Trusty) as the OS to install.
I saved the root password.
I prefer to use less powerful accounts where possible so configured the machine to let me in on a ssh key.
# mkdir ~/.ssh && chmod 700 ~/.ssh
# touch .ssh/authorized_keys
# chmod 600 .ssh/authorized_keys
I then copied my public key over to the new machine, adding it to .ssh/authorised_keys

Remove IPv6

The cause of many of my problems were due to a confusion in the install script between different results returned when asking programmatically for IP address or hostname.

Bytemark VMs come as standard with both IPv6 and IPv4.

I didn’t have sufficient time to understand why these different results were being returned, there was no obvious reason why that was so. So I decided to just disable IPv6.

Not ideal, and with more time, I would have investigated this further.

# nano /etc/sysctl.conf

and add these lines to sysctl.conf file

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

# sysctl -p

Install Docker

# apt-get install docker.io
Note that this package DOES NOT HAVE the name docker in ubuntu.
So now my network config looked like this.
# ifconfig -a

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr fe:ff:00:00:55:11 
          inet addr:46.43.2.130  Bcast:46.43.2.255  Mask:255.255.255.0
          inet6 addr: 2001:41c9:1:41f::130/64 Scope:Global
          inet6 addr: fe80::fcff:ff:fe00:5511/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10428 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3801 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:9825303 (9.8 MB)  TX bytes:402822 (402.8 KB)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Clone Kubernetes Code (v1.1.4)

$ git clone https://github.com/kubernetes/kubernetes.git

$ git checkout v1.1.4

$ cd kubernetes/docs/getting-started-guides/docker-multinode
You will need the kubectl program which is not part of that repo.
So download that using the link in the kubernetes documentation, and put it in  /usr/local/bin/.

Check Network Config

If you run these 5 commands now,
# ifconfig -a

# route

# docker ps

# docker -H unix:///var/run/docker-bootstrap.sock ps

# kubectl get nodes
you should now get output similar to this.
# ifconfig -a
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr fe:ff:00:00:55:11 
          inet addr:46.43.2.130  Bcast:46.43.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:93398 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43871 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:131469041 (131.4 MB)  TX bytes:3516199 (3.5 MB)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         46-43-2-1.no-re 0.0.0.0         UG    0      0        0 eth0

46.43.2.0       *               255.255.255.0   U     0      0        0 eth0

172.17.0.0      *               255.255.0.0     U     0      0        0 docker0


# docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES


# docker -H unix:///var/run/docker-bootstrap.sock ps

FATA[0000] Get http:///var/run/docker-bootstrap.sock/v1.18/containers/json: dial unix /var/run/docker-bootstrap.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS? 


# kubectl get nodes

error: couldn't read version from server: Get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused
Which tells us this:
  • we have a docker interface in ifconfig
  • we have a docker route in the routing table
  • the standard docker daemon is running…
  • … but it doesn’t have any live containers
  • the 2nd docker daemon needed by kubernetes is not yet running…
  • … and neither is the kubernetes server

MASTER

So now we can install the kubernetes master node
cd kubernetes/docs/getting-started-guides/docker-multinode

./master.sh
Whereupon the result of those 5 commands now looks like this
and now we have
# ifconfig -a

          docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet addr:10.1.55.1  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr fe:ff:00:00:55:11 
          inet addr:46.43.2.130  Bcast:46.43.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:155374 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54960 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:217671551 (217.6 MB)  TX bytes:4489883 (4.4 MB)

flannel.1 Link encap:Ethernet  HWaddr de:93:0f:c2:96:8f 
          inet addr:10.1.55.0  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2404 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2404 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0

          RX bytes:456749 (456.7 KB)  TX bytes:456749 (456.7 KB)




# route

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         46-43-2-1.no-re 0.0.0.0         UG    0      0        0 eth0
10.1.0.0        *               255.255.0.0     U     0      0        0 flannel.1
10.1.55.0       *               255.255.255.0   U     0      0        0 docker0

46.43.2.0       *               255.255.255.0   U     0      0        0 eth0

# docker ps
CONTAINER ID        IMAGE                                       COMMAND                CREATED              STATUS              PORTS               NAMES
1479e15ed371        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube schedule   About a minute ago   Up About a minute                       k8s_scheduler.2b8ee744_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_61e81a02           
2d17874be25e        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube apiserve   About a minute ago   Up About a minute                       k8s_apiserver.a94f0183_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_c6a20e6c           
226f11219fbd        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube controll   About a minute ago   Up About a minute                       k8s_controller-manager.19f4ee5e_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_9ccccf86  
b00a7288243d        gcr.io/google_containers/pause:0.8.0        "/pause"               About a minute ago   Up About a minute                       k8s_POD.e4cc795_k8s-master-127.0.0.1_default_82b47e8581e171a8c3f38c284b8a0579_9a8bd96c                  
2a825a132d52        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube proxy --   About a minute ago   Up About a minute                       hopeful_davinci                                                                                         

ea52c28a9ac9        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube kubelet    About a minute ago   Up About a minute                       ecstatic_nobel                

                                                                 

# docker -H unix:///var/run/docker-bootstrap.sock ps
CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS               NAMES

6a523dd5f0bd        quay.io/coreos/flannel:0.5.0           "/opt/bin/flanneld -   2 minutes ago       Up 2 minutes                            tender_kowalevski   

d036c4f56cbd        gcr.io/google_containers/etcd:2.0.12   "/usr/local/bin/etcd   3 minutes ago       Up 3 minutes                            romantic_mccarthy 


# kubectl get nodes
NAME        LABELS                             STATUS

127.0.0.1   kubernetes.io/hostname=127.0.0.1   Ready

WORKER

Add the IP address of the worker node to the firewall on the master.

Follow the same initial setup as before, and it should look similar to this.

root@kub2:~# ifconfig -a
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr fe:ff:00:00:55:12 
          inet addr:46.43.2.131  Bcast:46.43.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:94645 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35207 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:130735811 (130.7 MB)  TX bytes:2631768 (2.6 MB)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0 

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)




root@kub2:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         46-43-2-1.no-re 0.0.0.0         UG    0      0        0 eth0

46.43.2.0       *               255.255.255.0   U     0      0        0 eth0

172.17.0.0      *               255.255.0.0     U     0      0        0 docker0



root@kub2:~# docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Edit and Run the Worker Install Code

There’s a line in the worker.sh file from the kubernetes repo which causes the IPv4/IPv6 confusion I mentioned earlier.
So edit worker.sh to remove that overrider line. Then run the work installation script.
export MASTER_IP=<public IP on master>

./worker.sh
So now on the worker node, we should have this:
# ifconfig -adocker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet addr:10.1.76.1  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr fe:ff:00:00:55:12 
          inet addr:46.43.2.131  Bcast:46.43.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:153405 errors:0 dropped:0 overruns:0 frame:0
          TX packets:44774 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:211357174 (211.3 MB)  TX bytes:3458177 (3.4 MB)

flannel.1 Link encap:Ethernet  HWaddr 9e:80:4b:9b:9e:75 
          inet addr:10.1.76.0  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0 

          RX bytes:100 (100.0 B)  TX bytes:100 (100.0 B)




# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         46-43-2-1.no-re 0.0.0.0         UG    0      0        0 eth0
10.1.0.0        *               255.255.0.0     U     0      0        0 flannel.1
10.1.76.0       *               255.255.255.0   U     0      0        0 docker0

46.43.2.0       *               255.255.255.0   U     0      0        0 eth0




# docker ps
CONTAINER ID        IMAGE                                       COMMAND                CREATED              STATUS              PORTS               NAMES
5b520d7b12e8        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube proxy --   About a minute ago   Up About a minute                       modest_mclean      

cfb440bf9a5a        gcr.io/google_containers/hyperkube:v1.0.3   "/hyperkube kubelet    About a minute ago   Up About a minute                       goofy_babbage 




# docker -H unix:///var/run/docker-bootstrap.sock ps
CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS              PORTS               NAMES

569fc709ad84        quay.io/coreos/flannel:0.5.0   "/opt/bin/flanneld -   3 minutes ago       Up 3 minutes                            serene_hoover     




# kubectl get nodes
NAME          LABELS                               STATUS
127.0.0.1     kubernetes.io/hostname=127.0.0.1     Ready

46.43.2.131   kubernetes.io/hostname=46.43.2.131   Ready

FINISHED

And that’s it done. I wrote some ansible scripts to do this, which meant that once I’d figured all this out, it was possible to install a new node in a very short time.

But it took me about a month of digging through the Kubernetes install to get it working.

So I hope this helps someone!

 

Leave a Reply

Your email address will not be published. Required fields are marked *