mirror of https://github.com/easzlab/kubeasz.git
259 lines
8.9 KiB
Markdown
259 lines
8.9 KiB
Markdown
|
# calico 配置 BGP Route Reflectors
|
|||
|
|
|||
|
BGP 路由反射器(Route Reflectors)是解决BGP扩展性问题的有效方式;没有 RR 时AS内的BGP路由器之间需要两两建立连接(IBGP全互联),引入 RR 后,其他 BGP 路由器只需要与它建立连接交换路由信息。更多相关知识请查阅思科/华为相关网络文档。
|
|||
|
|
|||
|
calico-node 版本 v3.3 开始支持内建路由反射器,非常方便,因此使用 calico 作为网络插件可以支持大规模的 K8S 集群。
|
|||
|
|
|||
|
本文档主要讲解配置 BGP Route Reflectors,建议预先阅读[基础calico文档](calico.md)。
|
|||
|
|
|||
|
## 前提条件
|
|||
|
|
|||
|
实验环境为按照kubeasz安装的2主2从集群,calico 版本 v3.3.2
|
|||
|
|
|||
|
```
|
|||
|
$ kubectl get node
|
|||
|
NAME STATUS ROLES AGE VERSION
|
|||
|
192.168.1.1 Ready,SchedulingDisabled master 178m v1.13.1
|
|||
|
192.168.1.2 Ready,SchedulingDisabled master 178m v1.13.1
|
|||
|
192.168.1.3 Ready node 178m v1.13.1
|
|||
|
192.168.1.4 Ready node 178m v1.13.1
|
|||
|
$ kubectl get pod -n kube-system -o wide | grep calico
|
|||
|
calico-kube-controllers-77487546bd-jqrlc 1/1 Running 0 179m 192.168.1.3 192.168.1.3 <none> <none>
|
|||
|
calico-node-67t5m 2/2 Running 0 179m 192.168.1.1 192.168.1.1 <none> <none>
|
|||
|
calico-node-drmhq 2/2 Running 0 179m 192.168.1.2 192.168.1.2 <none> <none>
|
|||
|
calico-node-rjtkv 2/2 Running 0 179m 192.168.1.4 192.168.1.4 <none> <none>
|
|||
|
calico-node-xtspl 2/2 Running 0 179m 192.168.1.3 192.168.1.3 <none> <none>
|
|||
|
```
|
|||
|
查看当前集群中BGP连接情况:可以看到集群中4个节点两两建立了 BGP 连接
|
|||
|
|
|||
|
```
|
|||
|
$ ansible all -m shell -a '/opt/kube/bin/calicoctl node status'
|
|||
|
192.168.1.3 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |
|
|||
|
| 192.168.1.2 | node-to-node mesh | up | 03:08:18 | Established |
|
|||
|
| 192.168.1.4 | node-to-node mesh | up | 03:08:19 | Established |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
|
|||
|
192.168.1.2 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| 192.168.1.4 | node-to-node mesh | up | 03:08:17 | Established |
|
|||
|
| 192.168.1.3 | node-to-node mesh | up | 03:08:18 | Established |
|
|||
|
| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
|
|||
|
192.168.1.1 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| 192.168.1.2 | node-to-node mesh | up | 03:08:21 | Established |
|
|||
|
| 192.168.1.3 | node-to-node mesh | up | 03:08:21 | Established |
|
|||
|
| 192.168.1.4 | node-to-node mesh | up | 03:08:21 | Established |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
|
|||
|
192.168.1.4 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
| 192.168.1.2 | node-to-node mesh | up | 03:08:17 | Established |
|
|||
|
| 192.168.1.3 | node-to-node mesh | up | 03:08:19 | Established |
|
|||
|
| 192.168.1.1 | node-to-node mesh | up | 03:08:20 | Established |
|
|||
|
+--------------+-------------------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
```
|
|||
|
## 配置全局禁用全连接(BGP full mesh)
|
|||
|
|
|||
|
```
|
|||
|
$ cat << EOF | calicoctl create -f -
|
|||
|
apiVersion: projectcalico.org/v3
|
|||
|
kind: BGPConfiguration
|
|||
|
metadata:
|
|||
|
name: default
|
|||
|
spec:
|
|||
|
logSeverityScreen: Info
|
|||
|
nodeToNodeMeshEnabled: false
|
|||
|
asNumber: 64512
|
|||
|
EOF
|
|||
|
```
|
|||
|
|
|||
|
上述命令配置完成后,马上可以看到之前所有的bgp连接都消失了,查看命令: `calicoctl node status`
|
|||
|
|
|||
|
## 配置 BGP node 与 Route Reflector 的连接建立规则
|
|||
|
|
|||
|
``` bash
|
|||
|
$ cat << EOF | calicoctl create -f -
|
|||
|
kind: BGPPeer
|
|||
|
apiVersion: projectcalico.org/v3
|
|||
|
metadata:
|
|||
|
name: peer-to-rrs
|
|||
|
spec:
|
|||
|
# 规则1:普通 bgp node 与 rr 建立连接
|
|||
|
nodeSelector: !has(i-am-a-route-reflector)
|
|||
|
peerSelector: has(i-am-a-route-reflector)
|
|||
|
|
|||
|
---
|
|||
|
kind: BGPPeer
|
|||
|
apiVersion: projectcalico.org/v3
|
|||
|
metadata:
|
|||
|
name: rr-mesh
|
|||
|
spec:
|
|||
|
# 规则2:route reflectors 之间也建立连接
|
|||
|
nodeSelector: has(i-am-a-route-reflector)
|
|||
|
peerSelector: has(i-am-a-route-reflector)
|
|||
|
EOF
|
|||
|
```
|
|||
|
|
|||
|
上述命令配置完成后,可以使用查看命令:`calicoctl get bgppeer` `calicoctl get bgppeer rr-mesh -o yaml`
|
|||
|
|
|||
|
## 选择并配置 Route Reflector 节点
|
|||
|
|
|||
|
首先查看当前集群中的节点:
|
|||
|
|
|||
|
```
|
|||
|
$ calicoctl get node -o wide
|
|||
|
NAME ASN IPV4 IPV6
|
|||
|
k8s401 (64512) 192.168.1.1/24
|
|||
|
k8s402 (64512) 192.168.1.2/24
|
|||
|
k8s403 (64512) 192.168.1.3/24
|
|||
|
k8s404 (64512) 192.168.1.4/24
|
|||
|
```
|
|||
|
|
|||
|
可以在集群中选择1个或多个节点作为 rr 节点,这里先选择节点:k8s401
|
|||
|
|
|||
|
``` bash
|
|||
|
# 1.先导出 node k8s401 的配置,准备修改
|
|||
|
$ calicoctl get node k8s401 --export -o yaml |tee rr01.yml
|
|||
|
apiVersion: projectcalico.org/v3
|
|||
|
kind: Node
|
|||
|
metadata:
|
|||
|
creationTimestamp: null
|
|||
|
name: k8s401
|
|||
|
spec:
|
|||
|
bgp:
|
|||
|
ipv4Address: 192.168.1.1/24
|
|||
|
ipv4IPIPTunnelAddr: 172.20.7.128
|
|||
|
orchRefs:
|
|||
|
- nodeName: 192.168.1.1
|
|||
|
orchestrator: k8s
|
|||
|
|
|||
|
# 2.修改上述 rr01.yml 的配置如下
|
|||
|
apiVersion: projectcalico.org/v3
|
|||
|
kind: Node
|
|||
|
metadata:
|
|||
|
creationTimestamp: null
|
|||
|
name: k8s401
|
|||
|
labels:
|
|||
|
# 设置标签
|
|||
|
i-am-a-route-reflector: true
|
|||
|
spec:
|
|||
|
bgp:
|
|||
|
ipv4Address: 192.168.1.1/24
|
|||
|
ipv4IPIPTunnelAddr: 172.20.7.128
|
|||
|
# 设置集群ID
|
|||
|
routeReflectorClusterID: 224.0.0.1
|
|||
|
orchRefs:
|
|||
|
- nodeName: 192.168.1.1
|
|||
|
orchestrator: k8s
|
|||
|
|
|||
|
# 3.应用修改后的 rr node 配置
|
|||
|
$ calicoctl apply -f rr01.yml
|
|||
|
```
|
|||
|
|
|||
|
## 查看增加 rr 之后的bgp 连接情况
|
|||
|
|
|||
|
```
|
|||
|
$ ansible all -m shell -a '/opt/kube/bin/calicoctl node status'
|
|||
|
192.168.1.4 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
| 192.168.1.1 | global | up | 11:02:55 | Established |
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
|
|||
|
192.168.1.3 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
| 192.168.1.1 | global | up | 11:02:55 | Established |
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
|
|||
|
192.168.1.1 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+---------------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+---------------+-------+----------+-------------+
|
|||
|
| 192.168.1.2 | node specific | up | 11:02:55 | Established |
|
|||
|
| 192.168.1.3 | node specific | up | 11:02:55 | Established |
|
|||
|
| 192.168.1.4 | node specific | up | 11:02:55 | Established |
|
|||
|
+--------------+---------------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
|
|||
|
192.168.1.2 | SUCCESS | rc=0 >>
|
|||
|
Calico process is running.
|
|||
|
|
|||
|
IPv4 BGP status
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
| 192.168.1.1 | global | up | 11:02:55 | Established |
|
|||
|
+--------------+-----------+-------+----------+-------------+
|
|||
|
|
|||
|
IPv6 BGP status
|
|||
|
No IPv6 peers found.
|
|||
|
```
|
|||
|
可以看到所有其他节点都与两个rr节点建立bgp连接。
|
|||
|
|
|||
|
## 再增加一个 rr 节点
|
|||
|
|
|||
|
步骤同上述选择第1个 rr 节点,这里省略;添加成功后可以看到所有其他节点都与两个rr节点建立bgp连接,两个rr节点之间也建立bgp连接。
|
|||
|
|
|||
|
## 参考文档
|
|||
|
|
|||
|
[Calico 使用指南:Route Reflectors](https://docs.projectcalico.org/v3.3/usage/routereflector)
|
|||
|
[BGP路由反射器基础](https://www.sohu.com/a/140033025_761420)
|
|||
|
|
|||
|
|