diff --git a/README.md b/README.md index 906e31f..2239aa7 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ ## 使用指南 -基本k8s集群安装完成后，需要安装一些常用插件(`kubedns` `dashboard` `ingress`等)；接着介绍一些集群操作场景和思路；然后介绍一些应用部署实践，请根据这份[目录](docs/guide/index.md)阅读你所感兴趣的内容。 +基本k8s集群安装完成后，需要安装一些常用插件(`kubedns` `dashboard` `ingress`等)；接着介绍一些集群操作场景和思路；然后介绍一些应用部署实践，请根据这份[目录](docs/guide/index.md)阅读你所感兴趣的内容。尚在更新中... ## 参考阅读 diff --git a/docs/05-安装calico网络组件.md b/docs/05-安装calico网络组件.md index a78dba0..736c8f5 100644 --- a/docs/05-安装calico网络组件.md +++ b/docs/05-安装calico网络组件.md @@ -33,10 +33,6 @@ calico-node需要在所有master节点和node节点安装 ``` bash roles/calico/ -├── files -│   ├── ca.pem -│   ├── etcd-key.pem -│   └── etcd.pem ├── tasks │   └── main.yml └── templates @@ -46,9 +42,28 @@ roles/calico/ ``` 请在另外窗口打开[roles/calico/tasks/main.yml](../roles/calico/tasks/main.yml) 文件，对照看以下讲解内容。 -### 准备与etcd集群交互的证书 +### 创建calico 证书申请 -这里为了方便直接复制使用了etcd1节点的证书 +``` bash +{ + "CN": "calico", + "hosts": [], + "key": { + "algo": "rsa", + "size": 2048 + }, + "names": [ + { + "C": "CN", + "ST": "HangZhou", + "L": "XS", + "O": "k8s", + "OU": "System" + } + ] +} +``` +然后生成证书 ### 创建 calico-node 的服务文件 [calico-node.service.j2](../roles/calico/templates/calico-node.service.j2) @@ -64,8 +79,8 @@ PermissionsStartOnly=true ExecStart={{ bin_dir }}/docker run --net=host --privileged --name=calico-node \ -e ETCD_ENDPOINTS={{ ETCD_ENDPOINTS }} \ -e ETCD_CA_CERT_FILE=/etc/calico/ssl/ca.pem \ - -e ETCD_CERT_FILE=/etc/calico/ssl//etcd.pem \ - -e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem \ + -e ETCD_CERT_FILE=/etc/calico/ssl/calico.pem \ + -e ETCD_KEY_FILE=/etc/calico/ssl/calico-key.pem \ -e CALICO_LIBNETWORK_ENABLED=true \ -e CALICO_NETWORKING_BACKEND=bird \ -e CALICO_DISABLE_FILE_LOGGING=true \ @@ -92,11 +107,11 @@ RestartSec=10 WantedBy=multi-user.target ``` + 详细配置参数请参考[calico官方文档](https://docs.projectcalico.org/v2.6/reference/node/configuration) ++ calico-node是以docker容器运行在host上的，因此需要把之前的证书目录 /etc/calico/ssl挂载到容器中 + 配置ETCD_ENDPOINTS 、CA、证书等，所有{{ }}变量与ansible hosts文件中设置对应 + 配置集群POD网络 CALICO_IPV4POOL_CIDR={{ CLUSTER_CIDR }} + 本K8S集群运行在自有kvm虚机上，虚机间没有网络ACL限制，因此可以设置CALICO_IPV4POOL_IPIP=off，如果运行在公有云虚机上可能需要打开这个选项 + 配置FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT 默认允许Pod到Node的网络流量，更多[felix配置选项](https://docs.projectcalico.org/v2.6/reference/felix/configuration) -+ calico-node是以docker容器运行在host上的，因此需要把之前的证书目录 /etc/calico/ssl挂载到容器中 ### 启动calico-node @@ -108,8 +123,8 @@ WantedBy=multi-user.target "cniVersion": "0.1.0", "type": "calico", "etcd_endpoints": "{{ ETCD_ENDPOINTS }}", - "etcd_key_file": "/etc/calico/ssl/etcd-key.pem", - "etcd_cert_file": "/etc/calico/ssl//etcd.pem", + "etcd_key_file": "/etc/calico/ssl/calico-key.pem", + "etcd_cert_file": "/etc/calico/ssl/calico.pem", "etcd_ca_cert_file": "/etc/calico/ssl/ca.pem", "log_level": "info", "mtu": 1500, @@ -136,51 +151,74 @@ metadata: spec: datastoreType: "etcdv2" etcdEndpoints: {{ ETCD_ENDPOINTS }} - etcdKeyFile: /etc/calico/ssl/etcd-key.pem - etcdCertFile: /etc/calico/ssl/etcd.pem + etcdKeyFile: /etc/calico/ssl/calico-key.pem + etcdCertFile: /etc/calico/ssl/calico.pem etcdCACertFile: /etc/calico/ssl/ca.pem ``` ### 验证calico网络 -执行calico安装 `ansible-playbook 05.calico.yml` 成功后 +执行calico安装 `ansible-playbook 05.calico.yml` 成功后可以验证如下： + +**查看网卡和路由信息** ``` bash -# 查看网卡和路由信息 -ip a #...省略其他网卡信息，可以看到包含类似cali1c548f86afd@if3这样的网卡 -4: cali1c548f86afd@if3: mtu 1500 qdisc noqueue state UP group default - link/ether ae:88:58:31:82:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0 - inet6 fe80::ac88:58ff:fe31:8260/64 scope link +ip a #...省略其他网卡信息，可以看到包含类似cali1cxxx的网卡 +3: caliccc295a6d4f@if4: mtu 1500 qdisc noqueue state UP group default + link/ether 12:79:2f:fe:8d:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet6 fe80::1079:2fff:fefe:8d28/64 scope link valid_lft forever preferred_lft forever +5: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1 + link/ipip 0.0.0.0 brd 0.0.0.0 +# tunl0网卡现在不用管，是默认生成的，当开启IPIP 特性时使用的隧道 -route -n # 看到类似如下路由表 +route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface -0.0.0.0 192.168.1.254 0.0.0.0 UG 0 0 0 ens3 -192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens3 +0.0.0.0 10.100.80.1 0.0.0.0 UG 0 0 0 ens3 +10.100.80.0 0.0.0.0 255.255.255.0 U 0 0 0 ens3 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 -172.20.33.128 0.0.0.0 255.255.255.255 UH 0 0 0 cali1c548f86afd +172.20.3.64 10.100.80.65 255.255.255.192 UG 0 0 0 ens3 172.20.33.128 0.0.0.0 255.255.255.192 U 0 0 0 * -172.20.104.0 192.168.1.3 255.255.255.192 UG 0 0 0 ens3 -172.20.166.128 192.168.1.4 255.255.255.192 UG 0 0 0 ens3 +172.20.33.129 0.0.0.0 255.255.255.255 UH 0 0 0 caliccc295a6d4f +172.20.104.0 10.100.80.37 255.255.255.192 UG 0 0 0 ens3 +172.20.166.128 10.100.80.36 255.255.255.192 UG 0 0 0 ens3 +``` -# 查看所有calico节点状态 +**查看所有calico节点状态** + +``` bash calicoctl node status Calico process is running. IPv4 BGP status -+--------------+-------------------+-------+------------+-------------+ -| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | -+--------------+-------------------+-------+------------+-------------+ -| 192.168.1.2 | node-to-node mesh | up | 2017-11-30 | Established | -| 192.168.1.3 | node-to-node mesh | up | 2017-11-30 | Established | -| 192.168.1.4 | node-to-node mesh | up | 2017-11-30 | Established | -+--------------+-------------------+-------+------------+-------------+ ++--------------+-------------------+-------+----------+-------------+ +| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | ++--------------+-------------------+-------+----------+-------------+ +| 10.100.80.34 | node-to-node mesh | up | 12:34:00 | Established | +| 10.100.80.35 | node-to-node mesh | up | 12:34:00 | Established | +| 10.100.80.63 | node-to-node mesh | up | 12:34:01 | Established | +| 10.100.80.36 | node-to-node mesh | up | 12:34:00 | Established | +| 10.100.80.65 | node-to-node mesh | up | 12:34:00 | Established | +| 10.100.80.37 | node-to-node mesh | up | 12:34:15 | Established | ++--------------+-------------------+-------+----------+-------------+ +``` -IPv6 BGP status -No IPv6 peers found. +**BGP 协议是通过TCP 连接来建立邻居的，因此可以用netstat 命令验证 BGP Peer** -# 查看集群ipPool情况 +``` bash +netstat -antlp|grep ESTABLISHED|grep 179 +tcp 0 0 10.100.80.66:179 10.100.80.35:41316 ESTABLISHED 28479/bird +tcp 0 0 10.100.80.66:179 10.100.80.36:52823 ESTABLISHED 28479/bird +tcp 0 0 10.100.80.66:179 10.100.80.65:56311 ESTABLISHED 28479/bird +tcp 0 0 10.100.80.66:42000 10.100.80.37:179 ESTABLISHED 28479/bird +tcp 0 0 10.100.80.66:179 10.100.80.34:40243 ESTABLISHED 28479/bird +tcp 0 0 10.100.80.66:179 10.100.80.63:48979 ESTABLISHED 28479/bird +``` + +**查看集群ipPool情况** + +``` bash calicoctl get ipPool -o yaml - apiVersion: v1 kind: ipPool @@ -189,4 +227,3 @@ calicoctl get ipPool -o yaml spec: nat-outgoing: true ``` - diff --git a/docs/06-安装kube-master节点.md b/docs/06-安装kube-master节点.md index 0514869..e7051e8 100644 --- a/docs/06-安装kube-master节点.md +++ b/docs/06-安装kube-master节点.md @@ -92,7 +92,7 @@ ExecStart={{ bin_dir }}/kube-apiserver \ --kubelet-https=true \ --anonymous-auth=false \ --basic-auth-file={{ ca_dir }}/basic-auth.csv \ - --experimental-bootstrap-token-auth \ + --enable-bootstrap-token-auth \ --token-auth-file={{ ca_dir }}/token.csv \ --service-cluster-ip-range={{ SERVICE_CIDR }} \ --service-node-port-range={{ NODE_PORT_RANGE }} \ diff --git a/roles/ca/tasks/main.yml b/roles/ca/tasks/main.yml index 569b1b6..735eb2e 100644 --- a/roles/ca/tasks/main.yml +++ b/roles/ca/tasks/main.yml @@ -3,7 +3,6 @@ with_items: - "{{ bin_dir }}" - "{{ ca_dir }}" - - "{{ base_dir }}/roles/calico/files/" - "{{ base_dir }}/roles/prepare/files/" - name: 下载证书工具 CFSSL @@ -25,6 +24,3 @@ - name: 准备分发 CA证书 shell: "cd {{ ca_dir }} && cp ca* {{ base_dir }}/roles/prepare/files/" -- name: 复制 CA证书生成calico secret 使用 - shell: "cp -f {{ ca_dir }}/ca.pem {{ base_dir }}/roles/calico/files/" - diff --git a/roles/calico/tasks/main.yml b/roles/calico/tasks/main.yml index e31ef6e..157d83e 100644 --- a/roles/calico/tasks/main.yml +++ b/roles/calico/tasks/main.yml @@ -5,7 +5,7 @@ - /etc/cni/net.d - name: 复制CA 证书到calico 证书目录 - copy: src=ca.pem dest=/etc/calico/ssl/ca.pem + copy: src={{ ca_dir }}/ca.pem dest=/etc/calico/ssl/ca.pem - name: 创建calico 证书请求 template: src=calico-csr.json.j2 dest=/etc/calico/ssl/calico-csr.json @@ -20,14 +20,8 @@ - name: 创建 calico 的 systemd unit 文件 template: src=calico-node.service.j2 dest=/etc/systemd/system/calico-node.service -- name: daemon-reload - shell: systemctl daemon-reload - -- name: enable-calico - shell: systemctl enable calico-node - -- name: 重新启动calico - shell: systemctl restart calico-node +- name: 启动calico 服务 + shell: systemctl daemon-reload && systemctl enable calico-node && systemctl restart calico-node - name: 下载calico cni plugins和calicoctl 客户端 copy: src={{ base_dir }}/bin/{{ item }} dest={{ bin_dir }}/{{ item }} mode=0755 diff --git a/roles/docker/tasks/main.yml b/roles/docker/tasks/main.yml index cfc1185..355b98b 100644 --- a/roles/docker/tasks/main.yml +++ b/roles/docker/tasks/main.yml @@ -17,23 +17,17 @@ - name: docker国内镜像加速 copy: src=daemon.json dest=/etc/docker/daemon.json -- name: 创建docker的systemd unit文件 - template: src=docker.service.j2 dest=/etc/systemd/system/docker.service - -- name: daemon-reload - shell: systemctl daemon-reload - -- name: enable-docker - shell: systemctl enable docker - - name: flush-iptables shell: "iptables -F && iptables -X \ && iptables -F -t nat && iptables -X -t nat \ && iptables -F -t raw && iptables -X -t raw \ && iptables -F -t mangle && iptables -X -t mangle" -- name: start-docker - shell: systemctl restart docker +- name: 创建docker的systemd unit文件 + template: src=docker.service.j2 dest=/etc/systemd/system/docker.service + +- name: 开启docker 服务 + shell: systemctl daemon-reload && systemctl enable docker && systemctl restart docker ## 可选 ------安装docker查询镜像 tag的小工具---- - name: apt更新缓存刷新 diff --git a/roles/etcd/tasks/main.yml b/roles/etcd/tasks/main.yml index b00dcad..42783ef 100644 --- a/roles/etcd/tasks/main.yml +++ b/roles/etcd/tasks/main.yml @@ -23,11 +23,5 @@ - name: 创建etcd的systemd unit文件 template: src=etcd.service.j2 dest=/etc/systemd/system/etcd.service -- name: daemon-reload - shell: systemctl daemon-reload - -- name: enable-etcd - shell: systemctl enable etcd - -- name: 重启etcd服务 - shell: systemctl restart etcd +- name: 开启etcd服务 + shell: systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd