kubespray/README.md

323 lines
9.4 KiB
Markdown
Raw Normal View History

2016-01-08 07:20:31 +08:00
[![Build Status](https://travis-ci.org/ansibl8s/setup-kubernetes.svg)](https://travis-ci.org/ansibl8s/setup-kubernetes)
2015-10-04 04:49:48 +08:00
kubernetes-ansible
========
2016-01-15 16:29:28 +08:00
This project allows to
- Install and configure a **Multi-Master/HA kubernetes** cluster.
2016-01-15 17:43:17 +08:00
- Choose the **network plugin** to be used within the cluster
2016-01-15 16:29:28 +08:00
- A **set of roles** in order to install applications over the k8s cluster
- A **flexible method** which helps to create new roles for apps.
2015-10-04 04:49:48 +08:00
2016-01-09 17:45:50 +08:00
Linux distributions tested:
* **Debian** Wheezy, Jessie
* **Ubuntu** 14.10, 15.04, 15.10
* **Fedora** 23
* **CentOS** 7 (Currently with flannel only)
2015-10-04 16:55:52 +08:00
### Requirements
2016-01-09 17:45:50 +08:00
* The target servers must have **access to the Internet** in order to pull docker imaqes.
2015-12-13 23:59:22 +08:00
* The firewalls are not managed, you'll need to implement your own rules the way you used to.
2016-01-15 16:29:28 +08:00
in order to avoid any issue during deployment you should **disable your firewall**
* **Copy your ssh keys** to all the servers part of your inventory.
* **Ansible v1.9.x/v2.x and python-netaddr**
2016-01-15 17:39:34 +08:00
* Base knowledge on Ansible. Please refer to [Ansible documentation](http://www.ansible.com/how-ansible-works)
2015-10-04 16:55:52 +08:00
### Components
2016-01-18 04:31:38 +08:00
* [kubernetes](https://github.com/kubernetes/kubernetes/releases) v1.1.4
* [etcd](https://github.com/coreos/etcd/releases) v2.2.4
2016-01-18 04:31:38 +08:00
* [calicoctl](https://github.com/projectcalico/calico-docker/releases) v0.14.0
* [flanneld](https://github.com/coreos/flannel/releases) v0.5.5
2015-12-11 18:53:42 +08:00
* [docker](https://www.docker.com/) v1.9.1
2015-10-04 16:55:52 +08:00
2015-12-13 02:32:18 +08:00
Quickstart
-------------------------
The following steps will quickly setup a kubernetes cluster with default configuration.
2015-12-13 23:59:22 +08:00
These defaults are good for tests purposes.
2015-12-13 02:32:18 +08:00
Edit the inventory according to the number of servers
```
[kube-master]
10.115.99.31
2015-12-14 17:39:13 +08:00
[etcd]
10.115.99.31
10.115.99.32
2015-12-15 22:27:12 +08:00
10.115.99.33
2015-12-14 17:39:13 +08:00
2015-12-13 02:32:18 +08:00
[kube-node]
10.115.99.32
10.115.99.33
[k8s-cluster:children]
kube-node
kube-master
```
Run the playbook
```
2015-12-18 20:21:22 +08:00
ansible-playbook -i inventory/inventory.cfg cluster.yml -u root
2015-12-13 02:32:18 +08:00
```
2015-12-13 23:59:22 +08:00
You can jump directly to "*Available apps, installation procedure*"
2015-10-04 04:49:48 +08:00
Ansible
-------------------------
2015-10-04 16:55:52 +08:00
### Variables
2015-12-18 20:21:22 +08:00
The main variables to change are located in the directory ```inventory/group_vars/all.yml```.
2015-10-04 16:55:52 +08:00
2015-10-27 22:42:46 +08:00
### Inventory
Below is an example of an inventory.
Note : The bgp vars local_as and peers are not mandatory if the var **'peer_with_router'** is set to false
By default this variable is set to false and therefore all the nodes are configure in **'node-mesh'** mode.
2015-10-28 17:49:09 +08:00
In node-mesh mode the nodes peers with all the nodes in order to exchange routes.
2015-10-27 22:42:46 +08:00
```
2015-12-18 23:40:58 +08:00
2015-10-27 22:42:46 +08:00
[kube-master]
2015-12-18 23:40:58 +08:00
node1 ansible_ssh_host=10.99.0.26
node2 ansible_ssh_host=10.99.0.27
2015-10-27 22:42:46 +08:00
2015-12-14 17:39:13 +08:00
[etcd]
2015-12-18 23:40:58 +08:00
node1 ansible_ssh_host=10.99.0.26
node2 ansible_ssh_host=10.99.0.27
node3 ansible_ssh_host=10.99.0.4
2015-12-14 17:39:13 +08:00
2015-10-27 22:42:46 +08:00
[kube-node]
2015-12-18 23:40:58 +08:00
node2 ansible_ssh_host=10.99.0.27
node3 ansible_ssh_host=10.99.0.4
node4 ansible_ssh_host=10.99.0.5
node5 ansible_ssh_host=10.99.0.36
node6 ansible_ssh_host=10.99.0.37
2015-10-27 22:42:46 +08:00
2015-11-23 01:25:36 +08:00
[paris]
2015-12-18 23:40:58 +08:00
node1 ansible_ssh_host=10.99.0.26
node3 ansible_ssh_host=10.99.0.4 local_as=xxxxxxxx
node4 ansible_ssh_host=10.99.0.5 local_as=xxxxxxxx
[new-york]
node2 ansible_ssh_host=10.99.0.27
node5 ansible_ssh_host=10.99.0.36 local_as=xxxxxxxx
node6 ansible_ssh_host=10.99.0.37 local_as=xxxxxxxx
2015-10-27 22:42:46 +08:00
[k8s-cluster:children]
kube-node
kube-master
```
2015-10-04 16:55:52 +08:00
### Playbook
```
---
- hosts: downloader
sudo: no
roles:
- { role: download, tags: download }
2015-12-13 23:59:22 +08:00
- hosts: k8s-cluster
roles:
2016-01-05 00:00:40 +08:00
- { role: kubernetes/preinstall, tags: preinstall }
2015-10-04 16:55:52 +08:00
- { role: docker, tags: docker }
2016-01-05 00:00:40 +08:00
- { role: kubernetes/node, tags: node }
- { role: etcd, tags: etcd }
2015-10-04 16:55:52 +08:00
- { role: dnsmasq, tags: dnsmasq }
2015-12-13 23:59:22 +08:00
- { role: network_plugin, tags: ['calico', 'flannel', 'network'] }
2015-10-04 16:55:52 +08:00
- hosts: kube-master
roles:
- { role: kubernetes/master, tags: master }
```
### Run
2015-10-04 04:49:48 +08:00
It is possible to define variables for different environments.
For instance, in order to deploy the cluster on 'dev' environment run the following command.
```
2015-12-18 20:21:22 +08:00
ansible-playbook -i inventory/dev/inventory.cfg cluster.yml -u root
2015-10-04 04:49:48 +08:00
```
Kubernetes
-------------------------
2015-12-13 23:59:22 +08:00
### Multi master notes
2016-01-15 16:29:28 +08:00
* You can choose where to install the master components. If you want your master node to act both as master (api,scheduler,controller) and node (e.g. accept workloads, create pods ...),
2015-12-13 23:59:22 +08:00
the server address has to be present on both groups 'kube-master' and 'kube-node'.
* Almost all kubernetes components are running into pods except *kubelet*. These pods are managed by kubelet which ensure they're always running
2015-12-18 20:21:22 +08:00
* For safety reasons, you should have at least two master nodes and 3 etcd servers
2015-12-18 20:32:03 +08:00
* Kube-proxy doesn't support multiple apiservers on startup ([Issue 18174](https://github.com/kubernetes/kubernetes/issues/18174)). An external loadbalancer needs to be configured.
2016-01-15 16:29:28 +08:00
In order to do so, some variables have to be used '**loadbalancer_apiserver**' and '**apiserver_loadbalancer_domain_name**'
2015-10-04 16:55:52 +08:00
### Network Overlay
2015-10-27 22:42:46 +08:00
You can choose between 2 network plugins. Only one must be chosen.
2015-10-05 03:38:34 +08:00
2015-12-18 20:32:03 +08:00
* **flannel**: gre/vxlan (layer 2) networking. ([official docs](https://github.com/coreos/flannel))
2015-10-05 03:38:34 +08:00
2015-12-18 20:32:03 +08:00
* **calico**: bgp (layer 3) networking. ([official docs](http://docs.projectcalico.org/en/0.13/))
2015-10-05 03:25:09 +08:00
2015-10-27 22:42:46 +08:00
The choice is defined with the variable '**kube_network_plugin**'
2015-10-04 16:55:52 +08:00
### Expose a service
2015-10-05 03:38:34 +08:00
There are several loadbalancing solutions.
2015-12-18 20:32:03 +08:00
The one i found suitable for kubernetes are [Vulcand](http://vulcand.io/) and [Haproxy](http://www.haproxy.org/)
2015-10-04 16:55:52 +08:00
2015-10-05 03:59:09 +08:00
My cluster is working with haproxy and kubernetes services are configured with the loadbalancing type '**nodePort**'.
2015-10-04 16:55:52 +08:00
eg: each node opens the same tcp port and forwards the traffic to the target pod wherever it is located.
Then Haproxy can be configured to request kubernetes's api in order to loadbalance on the proper tcp port on the nodes.
2015-10-04 04:49:48 +08:00
2015-12-18 20:32:03 +08:00
Please refer to the proper kubernetes documentation on [Services](https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/user-guide/services.md)
2015-10-04 04:49:48 +08:00
2015-10-05 03:25:09 +08:00
### Check cluster status
#### Kubernetes components
2015-10-05 03:59:09 +08:00
* Check the status of the processes
```
2015-12-18 20:21:22 +08:00
systemctl status kubelet
2015-10-05 03:59:09 +08:00
```
* Check the logs
```
2015-12-18 20:21:22 +08:00
journalctl -ae -u kubelet
2015-10-05 03:59:09 +08:00
```
* Check the NAT rules
```
iptables -nLv -t nat
```
2015-12-18 20:21:22 +08:00
For the master nodes you'll have to see the docker logs for the apiserver
```
docker logs [apiserver docker id]
```
2015-10-05 03:25:09 +08:00
2015-10-20 16:18:30 +08:00
### Available apps, installation procedure
There are two ways of installing new apps
#### Ansible galaxy
2015-10-14 17:42:45 +08:00
Additionnal apps can be installed with ```ansible-galaxy```.
2015-10-19 04:21:08 +08:00
ou'll need to edit the file '*requirements.yml*' in order to chose needed apps.
2015-10-14 17:42:45 +08:00
The list of available apps are available [there](https://github.com/ansibl8s)
2015-10-05 17:27:13 +08:00
For instance it is **strongly recommanded** to install a dns server which resolves kubernetes service names.
2016-01-15 16:29:28 +08:00
In order to use this role you'll need the following entries in the file '*requirements.yml*'
2015-12-18 20:32:03 +08:00
Please refer to the [k8s-kubedns readme](https://github.com/ansibl8s/k8s-kubedns) for additionnal info.
2015-10-11 15:48:58 +08:00
```
2015-10-14 17:42:45 +08:00
- src: https://github.com/ansibl8s/k8s-common.git
path: roles/apps
# version: v1.0
- src: https://github.com/ansibl8s/k8s-kubedns.git
2015-10-14 17:42:45 +08:00
path: roles/apps
# version: v1.0
2015-10-11 15:48:58 +08:00
```
2015-10-14 17:42:45 +08:00
**Note**: the role common is required by all the apps and provides the tasks and libraries needed.
2015-10-19 04:21:08 +08:00
And empty the apps directory
```
rm -rf roles/apps/*
```
Then download the roles with ansible-galaxy
```
ansible-galaxy install -r requirements.yml
```
Finally update the playbook ```apps.yml``` with the chosen roles, and run it
2015-10-11 15:48:58 +08:00
```
...
- hosts: kube-master
roles:
- { role: apps/k8s-kubedns, tags: ['kubedns', 'apps'] }
2015-10-11 15:48:58 +08:00
...
```
```
2016-01-08 23:04:17 +08:00
ansible-playbook -i inventory/inventory.cfg apps.yml -u root
```
2016-01-15 16:29:28 +08:00
#### Git submodules
Alternatively the roles can be installed as git submodules.
That way is easier if you want to do some changes and commit them.
2015-10-05 17:27:13 +08:00
2016-01-15 17:44:54 +08:00
### Networking
#### Calico
2015-10-05 03:25:09 +08:00
Check if the calico-node container is running
```
docker ps | grep calico
```
The **calicoctl** command allows to check the status of the network workloads.
* Check the status of Calico nodes
```
calicoctl status
```
* Show the configured network subnet for containers
```
calicoctl pool show
```
2015-10-05 03:59:09 +08:00
* Show the workloads (ip addresses of containers and their located)
2015-10-05 03:25:09 +08:00
```
calicoctl endpoint show --detail
```
2016-01-09 17:45:50 +08:00
#### Flannel
2015-10-05 03:25:09 +08:00
2016-01-09 17:45:50 +08:00
* Flannel configuration file should have been created there
```
cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.233.0.0/18
FLANNEL_SUBNET=10.233.16.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
```
* Check if the network interface has been created
```
ip a show dev flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether e2:f3:a7:0f:bf:cb brd ff:ff:ff:ff:ff:ff
inet 10.233.16.0/18 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::e0f3:a7ff:fe0f:bfcb/64 scope link
valid_lft forever preferred_lft forever
```
* Docker must be configured with a bridge ip in the flannel subnet.
```
ps aux | grep docker
root 20196 1.7 2.7 1260616 56840 ? Ssl 10:18 0:07 /usr/bin/docker daemon --bip=10.233.16.1/24 --mtu=1450
```
* Try to run a container and check its ip address
```
kubectl run test --image=busybox --command -- tail -f /dev/null
replicationcontroller "test" created
kubectl describe po test-34ozs | grep ^IP
IP: 10.233.16.2
```
```
kubectl exec test-34ozs -- ip a show dev eth0
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 02:42:0a:e9:2b:03 brd ff:ff:ff:ff:ff:ff
inet 10.233.16.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:aff:fee9:2b03/64 scope link tentative flags 08
valid_lft forever preferred_lft forever
```
Congrats ! now you can walk through [kubernetes basics](http://kubernetes.io/v1.1/basicstutorials.html)