用 kubeadm 在 Ubuntu 上快速构建 Kubernetes 测试集群

本文将介绍如何在 Ubuntu server 16.04 版本上安装 kubeadm，并利用 kubeadm 快速的在 Ubuntu server 版本 16.04 上构建一个 kubernetes 的基础的测试集群，用来做学习和测试用途，当前（2018-04-14）最新的版本是 1.10.1。参考文档包括 kubernetes 官方网站的 kubeadm 安装文档以及利用 kubeadm 创建集群这两个文档。

生产用途的环境，需要考虑各个组件的高可用，建议参考 Kubernetes 的官方的相关的安装文档。

概述

本次安装建议至少 4 台服务器或者虚拟机，每台服务器 4G 内存，2 个 CPU 核心以上，基本架构为 1 台 master 节点，3 台 slave 节点。整个安装过程将在 Ubuntu 服务器上安装完 kubeadm，以及安装 kubernetes 的基本集群，包括 canal 网络，另后台存储可参考本书的最佳实践中的存储管理内容。本次安装一共 4 个节点，节点信息如下:

角色	主机名	IP 地址
Master	Ubuntu-master	192.168.5.200
Slave	ubuntu-1	192.168.5.201
Slave	ubuntu-2	192.168.5.202
Slave	ubuntu-3	192.168.5.203

准备工作

默认方式安装 Ubuntu Server 版本 16.04
配置主机名映射，每个节点

# cat /etc/hosts
127.0.0.1    localhost
192.168.0.200   Ubuntu-master
192.168.0.201   Ubuntu-1
192.168.0.202   Ubuntu-2
192.168.0.203   Ubuntu-3

如果连接 gcr 网站不方便，无法下载镜像，会导致安装过程卡住，可以下载我导出的镜像包，

我导出的镜像网盘链接

，解压缩以后是多个个 tar 包，使用

  docker load< xxxx.tar

导入各个文件即可）。

在所有节点上安装 kubeadm

查看 apt 安装源如下配置，使用阿里云的系统和 kubernetes 的源。

$ cat /etc/apt/sources.list
# 系统安装源
deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted
deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb http://mirrors.aliyun.com/ubuntu/ xenial multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates multiverse
deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse
# kubeadm及kubernetes组件安装源
deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main

安装 docker，可以使用系统源的的 docker.io 软件包，版本 1.13.1，我的系统里是已经安装好最新的版本了。

# apt-get install docker.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
docker.io is already the newest version (1.13.1-0ubuntu1~16.04.2).
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

更新源，可以不理会 gpg 的报错信息。

# apt-get update
Hit:1 http://mirrors.aliyun.com/ubuntu xenial InRelease
Hit:2 http://mirrors.aliyun.com/ubuntu xenial-updates InRelease
Hit:3 http://mirrors.aliyun.com/ubuntu xenial-backports InRelease
Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease [8,993 B]
Ign:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease
Fetched 8,993 B in 0s (20.7 kB/s)
Reading package lists... Done
W: GPG error: https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
W: The repository 'https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

强制安装 kubeadm，kubectl，kubelet 软件包。

# apt-get install -y kubelet kubeadm kubectl --allow-unauthenticated
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  kubernetes-cni socat
The following NEW packages will be installed:
  kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 5 newly installed, 0 to remove and 4 not upgraded.
Need to get 56.9 MB of archives.
After this operation, 410 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  kubernetes-cni kubelet kubectl kubeadm
Authentication warning overridden.
Get:1 http://mirrors.aliyun.com/ubuntu xenial/universe amd64 socat amd64 1.7.3.1-1 [321 kB]
Get:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.6.0-00 [5,910 kB]
Get:3 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.10.1-00 [21.1 MB]
Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.10.1-00 [8,906 kB]
Get:5 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.10.1-00 [20.7 MB]
Fetched 56.9 MB in 5s (11.0 MB/s)
Use of uninitialized value $_ in lc at /usr/share/perl5/Debconf/Template.pm line 287.
Selecting previously unselected package kubernetes-cni.
(Reading database ... 191799 files and directories currently installed.)
Preparing to unpack .../kubernetes-cni_0.6.0-00_amd64.deb ...
Unpacking kubernetes-cni (0.6.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../socat_1.7.3.1-1_amd64.deb ...
Unpacking ....
....

kubeadm 安装完以后，就可以使用它来快速安装部署 Kubernetes 集群了。

使用 kubeadm 安装 Kubernetes 集群

在做好了准备工作之后，下面介绍如何使用 kubeadm 安装 Kubernetes 集群，我们将首先安装 master 节点，然后将 slave 节点一个个加入到集群中去。

使用 kubeadmin 初始化 master 节点

因为使用要使用 canal，因此需要在初始化时加上网络配置参数，设置 kubernetes 的子网为 10.244.0.0/16，注意此处不要修改为其他地址，因为这个值与后续的 canal 的 yaml 值要一致，如果修改，请一并修改。

这个下载镜像的过程涉及翻墙，因为会从 gcr 的站点下载容器镜像。。。（如果大家翻墙不方便的话，可以用我在上文准备工作中提到的导出的镜像）。

如果有能够连接 gcr 站点的网络，那么整个安装过程非常简单。

# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.200
[init] Using Kubernetes version: v1.10.1
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
    [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [ubuntu-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.200]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [ubuntu-master] and IPs [192.168.0.200]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 28.003828 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node ubuntu-master as master by adding a label and a taint
[markmaster] Master ubuntu-master tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: rw4enn.mvk547juq7qi2b5f
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.0.200:6443 --token rw4enn.mvk547juq7qi2b5f --discovery-token-ca-cert-hash sha256:ba260d5191213382a806a9a7d92c9e6bb09061847c7914b1ac584d0c69471579

执行如下命令来配置 kubectl。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

这样 master 的节点就配置好了，并且可以使用 kubectl 来进行各种操作了，根据上面的提示接着往下做，将 slave 节点加入到集群。

Slave 节点加入集群

在 slave 节点执行如下的命令，将 slave 节点加入集群，正常的返回信息如下：

#kubeadm join 192.168.0.200:6443 --token rw4enn.mvk547juq7qi2b5f --discovery-token-ca-cert-hash sha256:ba260d5191213382a806a9a7d92c9e6bb09061847c7914b1ac584d0c69471579
[preflight] Running pre-flight checks.
    [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[discovery] Trying to connect to API Server "192.168.0.200:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.200:6443"
[discovery] Requesting info from "https://192.168.0.200:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.0.200:6443"
[discovery] Successfully established connection with API Server "192.168.0.200:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

等待节点加入完毕。加入中状态。

# kubectl get node
NAME            STATUS     ROLES     AGE       VERSION
ubuntu-1        NotReady   <none>    6m        v1.10.1
ubuntu-2        NotReady   <none>    6m        v1.10.1
ubuntu-3        NotReady   <none>    6m        v1.10.1
ubuntu-master   NotReady   master    10m       v1.10.1

在 master 节点查看信息如下状态为节点加入完毕。

root@Ubuntu-master:~# kubectl get pod -n kube-system -o wide
NAME                                    READY     STATUS    RESTARTS   AGE       IP              NODE
etcd-ubuntu-master                      1/1       Running   0          21m       192.168.0.200   ubuntu-master
kube-apiserver-ubuntu-master            1/1       Running   0          21m       192.168.0.200   ubuntu-master
kube-controller-manager-ubuntu-master   1/1       Running   0          22m       192.168.0.200   ubuntu-master
kube-dns-86f4d74b45-wkfk2               0/3       Pending   0          22m       <none>          <none>
kube-proxy-6ddb4                        1/1       Running   0          22m       192.168.0.200   ubuntu-master
kube-proxy-7ngb9                        1/1       Running   0          17m       192.168.0.202   ubuntu-2
kube-proxy-fkhhx                        1/1       Running   0          18m       192.168.0.201   ubuntu-1
kube-proxy-rh4lq                        1/1       Running   0          18m       192.168.0.203   ubuntu-3
kube-scheduler-ubuntu-master            1/1       Running   0          21m       192.168.0.200   ubuntu-master

kubedns 组件需要在网络插件完成安装以后会自动安装完成。

安装网络插件 canal

从 canal 官方文档参考，如下网址下载 2 个文件并且安装，其中一个是配置 canal 的 RBAC 权限，一个是部署 canal 的 DaemonSet。

# kubectl apply -f  https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
clusterrole.rbac.authorization.k8s.io "calico" created
clusterrole.rbac.authorization.k8s.io "flannel" created
clusterrolebinding.rbac.authorization.k8s.io "canal-flannel" created
clusterrolebinding.rbac.authorization.k8s.io "canal-calico" created
# kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/canal/canal.yaml
configmap "canal-config" created
daemonset.extensions "canal" created
customresourcedefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" created
customresourcedefinition.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" created
customresourcedefinition.apiextensions.k8s.io "ippools.crd.projectcalico.org" created
customresourcedefinition.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" created
customresourcedefinition.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org" created
customresourcedefinition.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org" created
serviceaccount "canal" created

查看 canal 的安装状态。

# kubectl get pod -n kube-system -o wide
NAME                                    READY     STATUS    RESTARTS   AGE       IP              NODE
canal-fc94k                             3/3       Running   10         4m        192.168.0.201   ubuntu-1
canal-rs2wp                             3/3       Running   10         4m        192.168.0.200   ubuntu-master
canal-tqd4l                             3/3       Running   10         4m        192.168.0.202   ubuntu-2
canal-vmpnr                             3/3       Running   10         4m        192.168.0.203   ubuntu-3
etcd-ubuntu-master                      1/1       Running   0          28m       192.168.0.200   ubuntu-master
kube-apiserver-ubuntu-master            1/1       Running   0          28m       192.168.0.200   ubuntu-master
kube-controller-manager-ubuntu-master   1/1       Running   0          29m       192.168.0.200   ubuntu-master
kube-dns-86f4d74b45-wkfk2               3/3       Running   0          28m       10.244.2.2      ubuntu-3
kube-proxy-6ddb4                        1/1       Running   0          28m       192.168.0.200   ubuntu-master
kube-proxy-7ngb9                        1/1       Running   0          24m       192.168.0.202   ubuntu-2
kube-proxy-fkhhx                        1/1       Running   0          24m       192.168.0.201   ubuntu-1
kube-proxy-rh4lq                        1/1       Running   0          24m       192.168.0.203   ubuntu-3
kube-scheduler-ubuntu-master            1/1       Running   0          28m       192.168.0.200   ubuntu-master

可以看到 canal 和 kube-dns 都已经运行正常，一个基本功能正常的测试环境就部署完毕了。

此时查看集群的节点状态，版本为最新的版本 v1.10.1。

# kubectl get node
NAME            STATUS    ROLES     AGE       VERSION
ubuntu-1        Ready     <none>    27m       v1.10.1
ubuntu-2        Ready     <none>    27m       v1.10.1
ubuntu-3        Ready     <none>    27m       v1.10.1
ubuntu-master   Ready     master    31m       v1.10.1

让 master 也运行 pod（默认 master 不运行 pod）, 这样在测试环境做是可以的，不建议在生产环境如此操作。

#kubectl taint nodes --all node-role.kubernetes.io/master-
node "ubuntu-master" untainted
taint "node-role.kubernetes.io/master:" not found
taint "node-role.kubernetes.io/master:" not found
taint "node-role.kubernetes.io/master:" not found

参考

Overview of kubeadm - kubernetest.io

使用 kubeadm 在 Ubuntu Server 16.04 上快速构建测试集群