kubespray/README.md

241 lines
6.5 KiB
Markdown
Raw Normal View History

2015-10-04 04:49:48 +08:00
kubernetes-ansible
========
2015-10-27 22:42:46 +08:00
Install and configure a kubernetes cluster including network plugin and optionnal addons.
2015-10-04 04:49:48 +08:00
Based on [CiscoCloud](https://github.com/CiscoCloud/kubernetes-ansible) work.
2015-10-04 16:55:52 +08:00
### Requirements
2015-10-05 17:27:13 +08:00
Tested on **Debian Jessie** and **Ubuntu** (14.10, 15.04, 15.10).
2015-10-05 17:30:34 +08:00
The target servers must have access to the Internet in order to pull docker imaqes.
2015-10-05 17:27:13 +08:00
The firewalls are not managed, you'll need to implement your own rules the way you used to.
2015-10-04 04:49:48 +08:00
2015-10-04 16:55:52 +08:00
Ansible v1.9.x
### Components
* [kubernetes](https://github.com/kubernetes/kubernetes/releases) v1.1.2
* [etcd](https://github.com/coreos/etcd/releases) v2.2.2
* [calicoctl](https://github.com/projectcalico/calico-docker/releases) v0.11.0
* [flanneld](https://github.com/coreos/flannel/releases) v0.5.5
2015-10-28 17:49:09 +08:00
* [docker](https://www.docker.com/) v1.8.3
2015-10-04 16:55:52 +08:00
2015-10-04 04:49:48 +08:00
Ansible
-------------------------
2015-10-04 16:55:52 +08:00
### Download binaries
A role allows to download required binaries. They will be stored in a directory defined by the variable
2015-10-05 03:25:09 +08:00
**'local_release_dir'** (by default /tmp).
Please ensure that you have enough disk space there (about **1G**).
2015-10-04 16:55:52 +08:00
2015-10-05 03:25:09 +08:00
**Note**: Whenever you'll need to change the version of a software, you'll have to erase the content of this directory.
2015-10-04 16:55:52 +08:00
### Variables
2015-10-05 03:25:09 +08:00
The main variables to change are located in the directory ```environments/[env_name]/group_vars/k8s-cluster.yml```.
2015-10-04 16:55:52 +08:00
2015-10-27 22:42:46 +08:00
### Inventory
Below is an example of an inventory.
Note : The bgp vars local_as and peers are not mandatory if the var **'peer_with_router'** is set to false
By default this variable is set to false and therefore all the nodes are configure in **'node-mesh'** mode.
2015-10-28 17:49:09 +08:00
In node-mesh mode the nodes peers with all the nodes in order to exchange routes.
2015-10-27 22:42:46 +08:00
```
[downloader]
10.99.0.26
[kube-master]
10.99.0.26
2015-10-27 22:42:46 +08:00
[etcd]
10.99.0.26
[kube-node]
10.99.0.4
10.99.0.5
10.99.0.36
10.99.0.37
2015-11-23 01:25:36 +08:00
[paris]
10.99.0.26
10.99.0.4 local_as=xxxxxxxx
10.99.0.5 local_as=xxxxxxxx
2015-10-27 22:42:46 +08:00
2015-11-23 01:25:36 +08:00
[usa]
10.99.0.36 local_as=xxxxxxxx
10.99.0.37 local_as=xxxxxxxx
2015-10-27 22:42:46 +08:00
[k8s-cluster:children]
kube-node
kube-master
2015-11-23 01:25:36 +08:00
[paris:vars]
2015-10-27 22:42:46 +08:00
peers=[{"router_id": "10.99.0.2", "as": "65xxx"}, {"router_id": "10.99.0.3", "as": "65xxx"}]
2015-11-23 01:25:36 +08:00
[usa:vars]
2015-10-27 22:42:46 +08:00
peers=[{"router_id": "10.99.0.34", "as": "65xxx"}, {"router_id": "10.99.0.35", "as": "65xxx"}]
```
2015-10-04 16:55:52 +08:00
### Playbook
```
---
- hosts: downloader
sudo: no
roles:
- { role: download, tags: download }
- hosts: k8s-cluster
roles:
- { role: etcd, tags: etcd }
- { role: docker, tags: docker }
2015-10-27 22:42:46 +08:00
- { role: network_plugin, tags: ['calico', 'flannel', 'network'] }
2015-10-04 16:55:52 +08:00
- { role: dnsmasq, tags: dnsmasq }
2015-10-04 04:49:48 +08:00
2015-10-04 16:55:52 +08:00
- hosts: kube-master
roles:
- { role: kubernetes/master, tags: master }
- hosts: kube-node
roles:
- { role: kubernetes/node, tags: node }
```
### Run
2015-10-04 04:49:48 +08:00
It is possible to define variables for different environments.
For instance, in order to deploy the cluster on 'dev' environment run the following command.
```
2015-10-05 17:30:34 +08:00
ansible-playbook -i environments/dev/inventory cluster.yml -u root
2015-10-04 04:49:48 +08:00
```
Kubernetes
-------------------------
2015-10-04 16:55:52 +08:00
### Network Overlay
2015-10-27 22:42:46 +08:00
You can choose between 2 network plugins. Only one must be chosen.
2015-10-05 03:38:34 +08:00
* **flannel**: gre/vxlan (layer 2) networking. ([official docs]('https://github.com/coreos/flannel'))
* **calico**: bgp (layer 3) networking. ([official docs]('http://docs.projectcalico.org/en/0.13/'))
2015-10-05 03:25:09 +08:00
2015-10-27 22:42:46 +08:00
The choice is defined with the variable '**kube_network_plugin**'
2015-10-04 16:55:52 +08:00
### Expose a service
2015-10-05 03:38:34 +08:00
There are several loadbalancing solutions.
The ones i found suitable for kubernetes are [Vulcand]('http://vulcand.io/') and [Haproxy]('http://www.haproxy.org/')
2015-10-04 16:55:52 +08:00
2015-10-05 03:59:09 +08:00
My cluster is working with haproxy and kubernetes services are configured with the loadbalancing type '**nodePort**'.
2015-10-04 16:55:52 +08:00
eg: each node opens the same tcp port and forwards the traffic to the target pod wherever it is located.
Then Haproxy can be configured to request kubernetes's api in order to loadbalance on the proper tcp port on the nodes.
2015-10-04 04:49:48 +08:00
2015-10-04 16:55:52 +08:00
Please refer to the proper kubernetes documentation on [Services]('https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/user-guide/services.md')
2015-10-04 04:49:48 +08:00
2015-10-05 03:25:09 +08:00
### Check cluster status
#### Kubernetes components
2015-10-05 03:59:09 +08:00
Master processes : kube-apiserver, kube-scheduler, kube-controller, kube-proxy
Nodes processes : kubelet, kube-proxy, [calico-node|flanneld]
* Check the status of the processes
```
systemctl status [process_name]
```
* Check the logs
```
journalctl -ae -u [process_name]
```
* Check the NAT rules
```
iptables -nLv -t nat
```
2015-10-05 03:25:09 +08:00
2015-10-20 16:18:30 +08:00
### Available apps, installation procedure
There are two ways of installing new apps
#### Ansible galaxy
2015-10-14 17:42:45 +08:00
Additionnal apps can be installed with ```ansible-galaxy```.
2015-10-19 04:21:08 +08:00
ou'll need to edit the file '*requirements.yml*' in order to chose needed apps.
2015-10-14 17:42:45 +08:00
The list of available apps are available [there](https://github.com/ansibl8s)
2015-10-05 17:27:13 +08:00
For instance it is **strongly recommanded** to install a dns server which resolves kubernetes service names.
2015-10-14 17:42:45 +08:00
In order to use this role you'll need the following entries in the file '*requirements.yml*'
Please refer to the [k8s-kubdns readme](https://github.com/ansibl8s/k8s-kubedns) for additionnal info.
2015-10-11 15:48:58 +08:00
```
2015-10-14 17:42:45 +08:00
- src: https://github.com/ansibl8s/k8s-common.git
path: roles/apps
# version: v1.0
- src: https://github.com/ansibl8s/k8s-kubedns.git
2015-10-14 17:42:45 +08:00
path: roles/apps
# version: v1.0
2015-10-11 15:48:58 +08:00
```
2015-10-14 17:42:45 +08:00
**Note**: the role common is required by all the apps and provides the tasks and libraries needed.
2015-10-19 04:21:08 +08:00
And empty the apps directory
```
rm -rf roles/apps/*
```
Then download the roles with ansible-galaxy
```
ansible-galaxy install -r requirements.yml
```
2015-10-20 16:18:30 +08:00
#### Git submodules
Alternatively the roles can be installed as git submodules.
That way is easier if you want to do some changes and commit them.
You can list available submodules with the following command:
```
grep path .gitmodules | sed 's/.*= //'
```
In order to install the dns addon you'll need to follow these steps
2015-10-20 16:18:30 +08:00
```
git submodule init roles/apps/k8s-common roles/apps/k8s-kubedns
git submodule update
```
Finally update the playbook ```apps.yml``` with the chosen roles, and run it
2015-10-11 15:48:58 +08:00
```
...
- hosts: kube-master
roles:
- { role: apps/k8s-kubedns, tags: ['kubedns', 'apps'] }
2015-10-11 15:48:58 +08:00
...
```
```
ansible-playbook -i environments/dev/inventory apps.yml -u root
```
2015-10-05 17:27:13 +08:00
2015-10-05 03:25:09 +08:00
#### Calico networking
Check if the calico-node container is running
```
docker ps | grep calico
```
The **calicoctl** command allows to check the status of the network workloads.
* Check the status of Calico nodes
```
calicoctl status
```
* Show the configured network subnet for containers
```
calicoctl pool show
```
2015-10-05 03:59:09 +08:00
* Show the workloads (ip addresses of containers and their located)
2015-10-05 03:25:09 +08:00
```
calicoctl endpoint show --detail
```
#### Flannel networking
Congrats ! now you can walk through [kubernetes basics](http://kubernetes.io/v1.1/basicstutorials.html)