add documentation
Smaine Kahlouch
parent
ee7ac22f0d
commit
585102ee20
25
README.md
25
README.md
|
|
@ -1,25 +0,0 @@
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
##Deploy a production ready kubernetes cluster
|
|
||||||
|
|
||||||
- Can be deployed on **AWS, GCE, OpenStack or Baremetal**
|
|
||||||
- **High available** cluster
|
|
||||||
- **Composable** (Choice of the network plugin for instance)
|
|
||||||
- Support most popular **Linux distributions**
|
|
||||||
- **Continuous integration tests**
|
|
||||||
|
|
||||||
|
|
||||||
To deploy the cluster you can use :
|
|
||||||
|
|
||||||
* [**kargo-cli**](https://github.com/kubespray/kargo-cli)
|
|
||||||
* **vagrant** by simply running `vagrant up`
|
|
||||||
* **Ansible** usual commands
|
|
||||||
|
|
||||||
A complete **documentation** can be found [**here**](https://docs.kubespray.io)
|
|
||||||
|
|
||||||
If you have questions, you can [invite yourself](https://slack.kubespray.io/) to **chat** with us on Slack! [](https://kubespray.slack.com)
|
|
||||||
|
|
||||||
[](https://travis-ci.org/kubespray/kargo) </br>
|
|
||||||
CI tests sponsored by Google (GCE), and [teuto.net](https://teuto.net/) for OpenStack.
|
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
Ansible variables
|
||||||
|
===============
|
||||||
|
|
||||||
|
|
||||||
|
Inventory
|
||||||
|
-------------
|
||||||
|
The inventory is composed of 3 groups:
|
||||||
|
|
||||||
|
* **kube-node** : list of kubernetes nodes where the pods will run.
|
||||||
|
* **kube-master** : list of servers where kubernetes master components (apiserver, scheduler, controller) will run.
|
||||||
|
Note: if you want the server to act both as master and node the server must be defined on both groups _kube-master_ and _kube-node_
|
||||||
|
* **etcd**: list of server to compose the etcd server. you should have at least 3 servers for failover purposes.
|
||||||
|
|
||||||
|
Below is a complete inventory example:
|
||||||
|
|
||||||
|
```
|
||||||
|
## Configure 'ip' variable to bind kubernetes services on a
|
||||||
|
## different ip than the default iface
|
||||||
|
node1 ansible_ssh_host=95.54.0.12 # ip=10.3.0.1
|
||||||
|
node2 ansible_ssh_host=95.54.0.13 # ip=10.3.0.2
|
||||||
|
node3 ansible_ssh_host=95.54.0.14 # ip=10.3.0.3
|
||||||
|
node4 ansible_ssh_host=95.54.0.15 # ip=10.3.0.4
|
||||||
|
node5 ansible_ssh_host=95.54.0.16 # ip=10.3.0.5
|
||||||
|
node6 ansible_ssh_host=95.54.0.17 # ip=10.3.0.6
|
||||||
|
|
||||||
|
[kube-master]
|
||||||
|
node1
|
||||||
|
node2
|
||||||
|
|
||||||
|
[etcd]
|
||||||
|
node1
|
||||||
|
node2
|
||||||
|
node3
|
||||||
|
|
||||||
|
[kube-node]
|
||||||
|
node2
|
||||||
|
node3
|
||||||
|
node4
|
||||||
|
node5
|
||||||
|
node6
|
||||||
|
|
||||||
|
[k8s-cluster:children]
|
||||||
|
kube-node
|
||||||
|
kube-master
|
||||||
|
etcd
|
||||||
|
```
|
||||||
|
|
||||||
|
Group vars
|
||||||
|
--------------
|
||||||
|
The main variables to change are located in the directory ```inventory/group_vars/all.yml```.
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
Calico
|
||||||
|
===========
|
||||||
|
|
||||||
|
Check if the calico-node container is running
|
||||||
|
|
||||||
|
```
|
||||||
|
docker ps | grep calico
|
||||||
|
```
|
||||||
|
|
||||||
|
The **calicoctl** command allows to check the status of the network workloads.
|
||||||
|
* Check the status of Calico nodes
|
||||||
|
|
||||||
|
```
|
||||||
|
calicoctl status
|
||||||
|
```
|
||||||
|
|
||||||
|
* Show the configured network subnet for containers
|
||||||
|
|
||||||
|
```
|
||||||
|
calicoctl pool show
|
||||||
|
```
|
||||||
|
|
||||||
|
* Show the workloads (ip addresses of containers and their located)
|
||||||
|
|
||||||
|
```
|
||||||
|
calicoctl endpoint show --detail
|
||||||
|
```
|
||||||
|
|
||||||
|
##### Optionnal : BGP Peering with border routers
|
||||||
|
|
||||||
|
In some cases you may want to route the pods subnet and so NAT is not needed on the nodes.
|
||||||
|
For instance if you have a cluster spread on different locations and you want your pods to talk each other no matter where they are located.
|
||||||
|
The following variables need to be set:
|
||||||
|
`peer_with_router` to enable the peering with the datacenter's border router (default value: false).
|
||||||
|
you'll need to edit the inventory and add a and a hostvar `local_as` by node.
|
||||||
|
|
||||||
|
```
|
||||||
|
node1 ansible_ssh_host=95.54.0.12 local_as=xxxxxx
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,22 @@
|
||||||
|
Cloud providers
|
||||||
|
==============
|
||||||
|
|
||||||
|
#### Provisioning
|
||||||
|
|
||||||
|
You can use kargo-cli to start new instances on cloud providers
|
||||||
|
here's an example
|
||||||
|
```
|
||||||
|
kargo [aws|gce] --nodes 2 --etcd 3 --cluster-name test-smana
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Deploy kubernetes
|
||||||
|
|
||||||
|
With kargo-cli
|
||||||
|
```
|
||||||
|
kargo deploy [--aws|--gce] -u admin
|
||||||
|
```
|
||||||
|
|
||||||
|
Or ansible-playbook command
|
||||||
|
```
|
||||||
|
ansible-playbook -u smana -e ansible_ssh_user=admin -e cloud_provider=[aws|gce] -b --become-user=root -i inventory/single.cfg cluster.yml
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
CoreOS bootstrap
|
||||||
|
===============
|
||||||
|
|
||||||
|
Example with **kargo-cli**:
|
||||||
|
|
||||||
|
```
|
||||||
|
kargo deploy --gce --coreos
|
||||||
|
```
|
||||||
|
|
||||||
|
Or with Ansible:
|
||||||
|
|
||||||
|
Before running the cluster playbook you must satisfy the following requirements:
|
||||||
|
|
||||||
|
* On each CoreOS nodes a writable directory **/opt/bin** (~400M disk space)
|
||||||
|
|
||||||
|
* Uncomment the variable **ansible\_python\_interpreter** in the file `inventory/group_vars/all.yml`
|
||||||
|
|
||||||
|
* run the Python bootstrap playbook
|
||||||
|
|
||||||
|
```
|
||||||
|
ansible-playbook -u smana -e ansible_ssh_user=smana -b --become-user=root -i inventory/inventory.cfg coreos-bootstrap.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Then you can proceed to [cluster deployment](#run-deployment)
|
||||||
|
|
@ -0,0 +1,75 @@
|
||||||
|
|
||||||
|
|
||||||
|
##Deploy a production ready kubernetes cluster
|
||||||
|
|
||||||
|
If you have questions, you can [invite yourself](https://slack.kubespray.io/) to **chat** with us on Slack! [](https://kubespray.slack.com)
|
||||||
|
|
||||||
|
- Can be deployed on **AWS, GCE, OpenStack or Baremetal**
|
||||||
|
- **High available** cluster
|
||||||
|
- **Composable** (Choice of the network plugin for instance)
|
||||||
|
- Support most popular **Linux distributions**
|
||||||
|
- **Continuous integration tests**
|
||||||
|
|
||||||
|
|
||||||
|
To deploy the cluster you can use :
|
||||||
|
|
||||||
|
[**kargo-cli**](https://github.com/kubespray/kargo-cli) <br>
|
||||||
|
**Ansible** usual commands <br>
|
||||||
|
**vagrant** by simply running `vagrant up` (for tests purposes) <br>
|
||||||
|
|
||||||
|
|
||||||
|
* [Requirements](#requirements)
|
||||||
|
* [Getting started](docs/getting-started.md)
|
||||||
|
* [Vagrant install](docs/vagrant.md)
|
||||||
|
* [CoreOS bootstrap](docs/coreos.md)
|
||||||
|
* [Ansible variables](docs/ansible.md)
|
||||||
|
* [Cloud providers](docs/cloud.md)
|
||||||
|
* [Openstack](docs/openstack.md)
|
||||||
|
* [Network plugins](#network-plugins)
|
||||||
|
|
||||||
|
Supported Linux distributions
|
||||||
|
===============
|
||||||
|
|
||||||
|
* **CoreOS**
|
||||||
|
* **Debian** Wheezy, Jessie
|
||||||
|
* **Ubuntu** 14.10, 15.04, 15.10, 16.04
|
||||||
|
* **Fedora** 23
|
||||||
|
* **CentOS/RHEL** 7
|
||||||
|
|
||||||
|
Versions
|
||||||
|
--------------
|
||||||
|
|
||||||
|
[kubernetes](https://github.com/kubernetes/kubernetes/releases) v1.3.0 <br>
|
||||||
|
[etcd](https://github.com/coreos/etcd/releases) v3.0.1 <br>
|
||||||
|
[calicoctl](https://github.com/projectcalico/calico-docker/releases) v0.20.0 <br>
|
||||||
|
[flanneld](https://github.com/coreos/flannel/releases) v0.5.5 <br>
|
||||||
|
[weave](http://weave.works/) v1.5.0 <br>
|
||||||
|
[docker](https://www.docker.com/) v1.10.3 <br>
|
||||||
|
|
||||||
|
|
||||||
|
Requirements
|
||||||
|
--------------
|
||||||
|
|
||||||
|
* The target servers must have **access to the Internet** in order to pull docker images.
|
||||||
|
* The **firewalls are not managed**, you'll need to implement your own rules the way you used to.
|
||||||
|
in order to avoid any issue during deployment you should disable your firewall
|
||||||
|
* **Copy your ssh keys** to all the servers part of your inventory.
|
||||||
|
* **Ansible v2.x and python-netaddr**
|
||||||
|
|
||||||
|
|
||||||
|
## Network plugins
|
||||||
|
You can choose between 3 network plugins. (default: `flannel` with vxlan backend)
|
||||||
|
|
||||||
|
* [**flannel**](docs/flannel.md): gre/vxlan (layer 2) networking.
|
||||||
|
|
||||||
|
* [**calico**](docs/calico.md): bgp (layer 3) networking.
|
||||||
|
|
||||||
|
* **weave**: Weave is a lightweight container overlay network that doesn't require an external K/V database cluster. <br>
|
||||||
|
(Please refer to `weave` [troubleshooting documentation](http://docs.weave.works/weave/latest_release/troubleshooting.html))
|
||||||
|
|
||||||
|
The choice is defined with the variable `kube_network_plugin`
|
||||||
|
|
||||||
|
|
||||||
|
[](https://travis-ci.org/kubespray/kargo) </br>
|
||||||
|
CI tests sponsored by Google (GCE), and [teuto.net](https://teuto.net/) for OpenStack.
|
||||||
|
|
||||||
|
|
@ -0,0 +1,51 @@
|
||||||
|
Flannel
|
||||||
|
==============
|
||||||
|
|
||||||
|
* Flannel configuration file should have been created there
|
||||||
|
|
||||||
|
```
|
||||||
|
cat /run/flannel/subnet.env
|
||||||
|
FLANNEL_NETWORK=10.233.0.0/18
|
||||||
|
FLANNEL_SUBNET=10.233.16.1/24
|
||||||
|
FLANNEL_MTU=1450
|
||||||
|
FLANNEL_IPMASQ=false
|
||||||
|
```
|
||||||
|
|
||||||
|
* Check if the network interface has been created
|
||||||
|
|
||||||
|
```
|
||||||
|
ip a show dev flannel.1
|
||||||
|
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
|
||||||
|
link/ether e2:f3:a7:0f:bf:cb brd ff:ff:ff:ff:ff:ff
|
||||||
|
inet 10.233.16.0/18 scope global flannel.1
|
||||||
|
valid_lft forever preferred_lft forever
|
||||||
|
inet6 fe80::e0f3:a7ff:fe0f:bfcb/64 scope link
|
||||||
|
valid_lft forever preferred_lft forever
|
||||||
|
```
|
||||||
|
|
||||||
|
* Docker must be configured with a bridge ip in the flannel subnet.
|
||||||
|
|
||||||
|
```
|
||||||
|
ps aux | grep docker
|
||||||
|
root 20196 1.7 2.7 1260616 56840 ? Ssl 10:18 0:07 /usr/bin/docker daemon --bip=10.233.16.1/24 --mtu=1450
|
||||||
|
```
|
||||||
|
|
||||||
|
* Try to run a container and check its ip address
|
||||||
|
|
||||||
|
```
|
||||||
|
kubectl run test --image=busybox --command -- tail -f /dev/null
|
||||||
|
replicationcontroller "test" created
|
||||||
|
|
||||||
|
kubectl describe po test-34ozs | grep ^IP
|
||||||
|
IP: 10.233.16.2
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
kubectl exec test-34ozs -- ip a show dev eth0
|
||||||
|
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
|
||||||
|
link/ether 02:42:0a:e9:2b:03 brd ff:ff:ff:ff:ff:ff
|
||||||
|
inet 10.233.16.2/24 scope global eth0
|
||||||
|
valid_lft forever preferred_lft forever
|
||||||
|
inet6 fe80::42:aff:fee9:2b03/64 scope link tentative flags 08
|
||||||
|
valid_lft forever preferred_lft forever
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,19 @@
|
||||||
|
Getting started
|
||||||
|
===============
|
||||||
|
|
||||||
|
The easiest way to run the deployement is to use the **kargo-cli** tool.
|
||||||
|
A complete documentation can be found in its [github repository](https://github.com/kubespray/kargo-cli).
|
||||||
|
|
||||||
|
Here is a simple example on AWS:
|
||||||
|
|
||||||
|
* Create instances and generate the inventory
|
||||||
|
|
||||||
|
```
|
||||||
|
kargo aws --instances 3
|
||||||
|
```
|
||||||
|
|
||||||
|
* Run the deployment
|
||||||
|
|
||||||
|
```
|
||||||
|
kargo deploy --aws -u centos -n calico
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
OpenStack
|
||||||
|
===============
|
||||||
|
|
||||||
|
To deploy kubespray on [OpenStack](https://www.openstack.org/) uncomment the `cloud_provider` option in `group_vars/all.yml` and set it to `'openstack'`.
|
||||||
|
|
||||||
|
After that make sure to source in your OpenStack credentials like you would do when using `nova-client` by using `source path/to/your/openstack-rc`.
|
||||||
|
|
||||||
|
The next step is to make sure the hostnames in your `inventory` file are identical to your instance names in OpenStack.
|
||||||
|
Otherwise [cinder](https://wiki.openstack.org/wiki/Cinder) won't work as expected.
|
||||||
|
|
||||||
|
Unless you are using calico you can now run the playbook.
|
||||||
|
|
||||||
|
**Additional step needed when using calico:**
|
||||||
|
|
||||||
|
Calico does not encapsulate all packages with the hosts ip addresses. Instead the packages will be routed with the PODs ip addresses directly.
|
||||||
|
OpenStack will filter and drop all packages from ips it does not know to prevent spoofing.
|
||||||
|
|
||||||
|
In order to make calico work on OpenStack you will need to tell OpenStack to allow calicos packages by allowing the network it uses.
|
||||||
|
|
||||||
|
First you will need the ids of your OpenStack instances that will run kubernetes:
|
||||||
|
|
||||||
|
nova list --tenant Your-Tenant
|
||||||
|
+--------------------------------------+--------+----------------------------------+--------+-------------+
|
||||||
|
| ID | Name | Tenant ID | Status | Power State |
|
||||||
|
+--------------------------------------+--------+----------------------------------+--------+-------------+
|
||||||
|
| e1f48aad-df96-4bce-bf61-62ae12bf3f95 | k8s-1 | fba478440cb2444a9e5cf03717eb5d6f | ACTIVE | Running |
|
||||||
|
| 725cd548-6ea3-426b-baaa-e7306d3c8052 | k8s-2 | fba478440cb2444a9e5cf03717eb5d6f | ACTIVE | Running |
|
||||||
|
|
||||||
|
Then you can use the instance ids to find the connected [neutron](https://wiki.openstack.org/wiki/Neutron) ports:
|
||||||
|
|
||||||
|
neutron port-list -c id -c device_id
|
||||||
|
+--------------------------------------+--------------------------------------+
|
||||||
|
| id | device_id |
|
||||||
|
+--------------------------------------+--------------------------------------+
|
||||||
|
| 5662a4e0-e646-47f0-bf88-d80fbd2d99ef | e1f48aad-df96-4bce-bf61-62ae12bf3f95 |
|
||||||
|
| e5ae2045-a1e1-4e99-9aac-4353889449a7 | 725cd548-6ea3-426b-baaa-e7306d3c8052 |
|
||||||
|
|
||||||
|
Given the port ids on the left, you can set the `allowed_address_pairs` in neutron:
|
||||||
|
|
||||||
|
# allow kube_service_addresses network
|
||||||
|
neutron port-update 5662a4e0-e646-47f0-bf88-d80fbd2d99ef --allowed_address_pairs list=true type=dict ip_address=10.233.0.0/18
|
||||||
|
neutron port-update e5ae2045-a1e1-4e99-9aac-4353889449a7 --allowed_address_pairs list=true type=dict ip_address=10.233.0.0/18
|
||||||
|
|
||||||
|
# allow kube_pods_subnet network
|
||||||
|
neutron port-update 5662a4e0-e646-47f0-bf88-d80fbd2d99ef --allowed_address_pairs list=true type=dict ip_address=10.233.64.0/18
|
||||||
|
neutron port-update e5ae2045-a1e1-4e99-9aac-4353889449a7 --allowed_address_pairs list=true type=dict ip_address=10.233.64.0/18
|
||||||
|
|
||||||
|
Now you can finally run the playbook.
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
Vagrant Install
|
||||||
|
=================
|
||||||
|
|
||||||
|
Assuming you have Vagrant (1.8+) installed with virtualbox (it may work
|
||||||
|
with vmware, but is untested) you should be able to launch a 3 node
|
||||||
|
Kubernetes cluster by simply running `$ vagrant up`.<br />
|
||||||
|
|
||||||
|
This will spin up 3 VMs and install kubernetes on them. Once they are
|
||||||
|
completed you can connect to any of them by running <br />
|
||||||
|
`$ vagrant ssh k8s-0[1..3]`.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ vagrant up
|
||||||
|
Bringing machine 'k8s-01' up with 'virtualbox' provider...
|
||||||
|
Bringing machine 'k8s-02' up with 'virtualbox' provider...
|
||||||
|
Bringing machine 'k8s-03' up with 'virtualbox' provider...
|
||||||
|
==> k8s-01: Box 'bento/ubuntu-14.04' could not be found. Attempting to find and install...
|
||||||
|
...
|
||||||
|
...
|
||||||
|
k8s-03: Running ansible-playbook...
|
||||||
|
|
||||||
|
PLAY [k8s-cluster] *************************************************************
|
||||||
|
|
||||||
|
TASK [setup] *******************************************************************
|
||||||
|
ok: [k8s-03]
|
||||||
|
ok: [k8s-01]
|
||||||
|
ok: [k8s-02]
|
||||||
|
...
|
||||||
|
...
|
||||||
|
PLAY RECAP *********************************************************************
|
||||||
|
k8s-01 : ok=157 changed=66 unreachable=0 failed=0
|
||||||
|
k8s-02 : ok=137 changed=59 unreachable=0 failed=0
|
||||||
|
k8s-03 : ok=86 changed=51 unreachable=0 failed=0
|
||||||
|
|
||||||
|
$ vagrant ssh k8s-01
|
||||||
|
vagrant@k8s-01:~$ kubectl get nodes
|
||||||
|
NAME STATUS AGE
|
||||||
|
k8s-01 Ready 45s
|
||||||
|
k8s-02 Ready 45s
|
||||||
|
k8s-03 Ready 45s
|
||||||
|
```
|
||||||
Loading…
Reference in New Issue