kubernetes-sigs
/
kubespray
mirror of https://github.com/kubernetes-sigs/kubespray.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update MetalLB and switch to CRD notation. (#9120) 

Signed-off-by: Jeroen Rijken <jeroen.rijken@xs4all.nl>
pull/9986/head
Jeroen Rijken 2023-04-14 10:14:41 +02:00 committed by GitHub
parent 73ce6aef97
commit 709ae1d244
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 2187 additions and 343 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -177,7 +177,7 @@ Note: Upstart/SysV init based OS types are not supported.
- [krew](https://github.com/kubernetes-sigs/krew) v0.4.3
- [argocd](https://argoproj.github.io/) v2.6.7
- [helm](https://helm.sh/) v3.11.2
- [metallb](https://metallb.universe.tf/) v0.12.1
- [metallb](https://metallb.universe.tf/) v0.13.9
- [registry](https://github.com/distribution/distribution) v2.8.1
- Storage Plugin
- [cephfs-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.0-k8s1.11

202
docs/metallb.md
View File

@ -14,58 +14,121 @@ kube_proxy_strict_arp: true
## Install
You have to explicitly enable the MetalLB extension and set an IP address range from which to allocate LoadBalancer IPs.
You have to explicitly enable the MetalLB extension.
```yaml
metallb_enabled: true
metallb_speaker_enabled: true
metallb_avoid_buggy_ips: true
metallb_ip_range:
- 10.5.0.0/16
```
By default only the MetalLB BGP speaker is allowed to run on control plane nodes. If you have a single node cluster or a cluster where control plane are also worker nodes you may need to enable tolerations for the MetalLB controller:
```yaml
metallb_controller_tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: ""
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
operator: "Equal"
value: ""
effect: "NoSchedule"
metallb_config:
controller:
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: ""
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
operator: "Equal"
value: ""
effect: "NoSchedule"
```
## Pools
First you need to specify all of the pools you are going to use:
```yaml
metallb_config:
address_pools:
primary:
ip_range:
- 192.0.1.0-192.0.1.254
auto_assign: true
pool1:
ip_range:
- 192.0.2.1-192.0.2.1
auto_assign: false # When set to false, you need to explicitly set the loadBalancerIP in the service!
pool2:
ip_range:
- 192.0.2.2-192.0.2.2
auto_assign: false
```
## Layer2 Mode
Pools that need to be configured in layer2 mode, need to be specified in a list:
```yaml
metallb_config:
layer2:
- primary
```
## BGP Mode
When operating in BGP Mode MetalLB needs to have defined upstream peers:
When operating in BGP Mode MetalLB needs to have defined upstream peers and link the pool(s) specified above to the correct peer:
```yaml
metallb_protocol: bgp
metallb_ip_range:
- 10.5.0.0/16
metallb_peers:
- peer_address: 192.0.2.1
peer_asn: 64512
my_asn: 4200000000
- peer_address: 192.0.2.2
peer_asn: 64513
my_asn: 4200000000
```
metallb_config:
Some upstream BGP peers may require password authentication:
layer3:
defaults:
```yaml
metallb_protocol: bgp
metallb_ip_range:
- 10.5.0.0/16
metallb_peers:
- peer_address: 192.0.2.1
peer_asn: 64512
my_asn: 4200000000
password: "changeme"
peer_port: 179 # The TCP port to talk to. Defaults to 179, you shouldn't need to set this in production.
hold_time: 120s # Requested BGP hold time, per RFC4271.
communities:
vpn-only: "1234:1"
NO_ADVERTISE: "65535:65282"
metallb_peers:
peer1:
peer_address: 192.0.2.1
peer_asn: 64512
my_asn: 4200000000
communities:
- vpn-only
address_pool:
- pool1
# (optional) The source IP address to use when establishing the BGP session. In most cases the source-address field should only be used with per-node peers, i.e. peers with node selectors which select only one node. CURRENTLY NOT SUPPORTED
source_address: 192.0.2.2
# (optional) The router ID to use when connecting to this peer. Defaults to the node IP address.
# Generally only useful when you need to peer with another BGP router running on the same machine as MetalLB.
router_id: 1.2.3.4
# (optional) Password for TCPMD5 authenticated BGP sessions offered by some peers.
password: "changeme"
peer2:
peer_address: 192.0.2.2
peer_asn: 64513
my_asn: 4200000000
communities:
- NO_ADVERTISE
address_pool:
- pool2
# (optional) The source IP address to use when establishing the BGP session. In most cases the source-address field should only be used with per-node peers, i.e. peers with node selectors which select only one node. CURRENTLY NOT SUPPORTED
source_address: 192.0.2.1
# (optional) The router ID to use when connecting to this peer. Defaults to the node IP address.
# Generally only useful when you need to peer with another BGP router running on the same machine as MetalLB.
router_id: 1.2.3.5
# (optional) Password for TCPMD5 authenticated BGP sessions offered by some peers.
password: "changeme"
```
When using calico >= 3.18 you can replace MetalLB speaker by calico Service LoadBalancer IP advertisement.
@ -75,30 +138,61 @@ In this scenario you should disable the MetalLB speaker and configure the `calic
```yaml
metallb_speaker_enabled: false
metallb_avoid_buggy_ips: true
metallb_ip_range:
- 10.5.0.0/16
calico_advertise_service_loadbalancer_ips: "{{ metallb_ip_range }}"
metallb_config:
address_pools:
primary:
ip_range:
- 10.5.0.0/16
auto_assign: true
layer2:
- primary
calico_advertise_service_loadbalancer_ips: "{{ metallb_config.address_pools.primary.ip_range }}"
```
If you have additional loadbalancer IP pool in `metallb_additional_address_pools` , ensure to add them to the list.
If you have additional loadbalancer IP pool in `metallb_config.address_pools` , ensure to add them to the list.
```yaml
metallb_speaker_enabled: false
metallb_ip_range:
- 10.5.0.0/16
metallb_additional_address_pools:
kube_service_pool_1:
ip_range:
- 10.6.0.0/16
protocol: "bgp"
auto_assign: false
avoid_buggy_ips: true
kube_service_pool_2:
ip_range:
- 10.10.0.0/16
protocol: "bgp"
auto_assign: false
avoid_buggy_ips: true
metallb_config:
address_pools:
primary:
ip_range:
- 10.5.0.0/16
auto_assign: true
pool1:
ip_range:
- 10.6.0.0/16
auto_assign: true
pool2:
ip_range:
- 10.10.0.0/16
auto_assign: true
layer2:
- primary
layer3:
defaults:
peer_port: 179
hold_time: 120s
communities:
vpn-only: "1234:1"
NO_ADVERTISE: "65535:65282"
metallb_peers:
peer1:
peer_address: 10.6.0.1
peer_asn: 64512
my_asn: 4200000000
communities:
- vpn-only
address_pool:
- pool1
peer2:
peer_address: 10.10.0.1
peer_asn: 64513
my_asn: 4200000000
communities:
- NO_ADVERTISE
address_pool:
- pool2
calico_advertise_service_loadbalancer_ips:
- 10.5.0.0/16
- 10.6.0.0/16

62
inventory/sample/group_vars/k8s_cluster/addons.yml
View File

@ -170,11 +170,6 @@ cert_manager_enabled: false
# MetalLB deployment
metallb_enabled: false
metallb_speaker_enabled: "{{ metallb_enabled }}"
# metallb_ip_range:
# - "10.5.0.50-10.5.0.99"
# metallb_pool_name: "loadbalanced"
# metallb_auto_assign: true
# metallb_avoid_buggy_ips: false
# metallb_speaker_nodeselector:
# kubernetes.io/os: "linux"
# metallb_controller_nodeselector:
@ -197,25 +192,50 @@ metallb_speaker_enabled: "{{ metallb_enabled }}"
# operator: "Equal"
# value: ""
# effect: "NoSchedule"
# metallb_version: v0.12.1
# metallb_version: v0.13.9
# metallb_protocol: "layer2"
# metallb_port: "7472"
# metallb_memberlist_port: "7946"
# metallb_additional_address_pools:
# kube_service_pool:
# ip_range:
# - "10.5.1.50-10.5.1.99"
# protocol: "layer2"
# auto_assign: false
# avoid_buggy_ips: false
# metallb_protocol: "bgp"
# metallb_peers:
# - peer_address: 192.0.2.1
# peer_asn: 64512
# my_asn: 4200000000
# - peer_address: 192.0.2.2
# peer_asn: 64513
# my_asn: 4200000000
# metallb_config:
# address_pools:
# primary:
# ip_range:
# - 10.5.0.0/16
# auto_assign: true
# pool1:
# ip_range:
# - 10.6.0.0/16
# auto_assign: true
# pool2:
# ip_range:
# - 10.10.0.0/16
# auto_assign: true
# layer2:
# - primary
# layer3:
# defaults:
# peer_port: 179
# hold_time: 120s
# communities:
# vpn-only: "1234:1"
# NO_ADVERTISE: "65535:65282"
# metallb_peers:
# peer1:
# peer_address: 10.6.0.1
# peer_asn: 64512
# my_asn: 4200000000
# communities:
# - vpn-only
# address_pool:
# - pool1
# peer2:
# peer_address: 10.10.0.1
# peer_asn: 64513
# my_asn: 4200000000
# communities:
# - NO_ADVERTISE
# address_pool:
# - pool2
argocd_enabled: false
# argocd_version: v2.6.7

2
roles/download/defaults/main.yml
View File

@ -1166,7 +1166,7 @@ dashboard_metrics_scraper_tag: "v1.0.8"
metallb_speaker_image_repo: "{{ quay_image_repo }}/metallb/speaker"
metallb_controller_image_repo: "{{ quay_image_repo }}/metallb/controller"
metallb_version: v0.12.1
metallb_version: v0.13.9
downloads:
netcheck_server:

31
roles/kubernetes-apps/metallb/tasks/main.yml
View File

@ -15,8 +15,8 @@
fail:
msg: "metallb_peers is mandatory when metallb_protocol is bgp and metallb_speaker_enabled"
when:
- metallb_protocol == 'bgp' and metallb_speaker_enabled
- metallb_peers is not defined or not metallb_peers
- metallb_config.layer3 is defined and metallb_speaker_enabled
- metallb_config.metallb_peers is not defined or not metallb_config.metallb_peers
- name: Kubernetes Apps | Check that the deprecated 'matallb_auto_assign' variable is not used anymore
fail:
@ -45,11 +45,29 @@
src: "{{ item }}.j2"
dest: "{{ kube_config_dir }}/{{ item }}"
mode: 0644
with_items: ["metallb.yml", "metallb-config.yml"]
with_items: ["metallb.yml", "metallb-config.yml", "pools.yaml", "layer2.yaml", "layer3.yaml"]
register: "rendering"
when:
- "inventory_hostname == groups['kube_control_plane'][0]"
- name: Kubernetes Apps | Create MetalLB resources and replace existing
k8s:
definition: "{{ lookup('template', 'metallb.yaml') }}"
- name: Kubernetes Apps | Wait for MetalLB controller to be running
k8s_info:
kind: Deployment
namespace: metallb-system
name: controller
wait: True
wait_sleep: 10
wait_timeout: 360
wait_condition:
status: "True"
type: Available
register: result
until: result is not failed
- name: Kubernetes Apps | Install and configure MetalLB
kube:
name: "MetalLB"
@ -60,3 +78,10 @@
with_items: "{{ rendering.results }}"
when:
- "inventory_hostname == groups['kube_control_plane'][0]"
- name: Kubernetes Apps | Delete MetalLB ConfigMap
k8s:
name: config
kind: ConfigMap
namespace: metallb-system
state: absent

19
roles/kubernetes-apps/metallb/templates/layer2.yaml.j2 100644
View File

@ -0,0 +1,19 @@
#jinja2: trim_blocks: True, lstrip_blocks: True
# yamllint disable-file
---
# Create layer2 configuration
{% for entry in metallb_config.layer2 %}
---
# L2 Configuration
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: "{{ entry }}"
namespace: metallb-system
spec:
ipAddressPools:
- "{{ entry }}"
{% endfor %}

125
roles/kubernetes-apps/metallb/templates/layer3.yaml.j2 100644
View File

@ -0,0 +1,125 @@
#jinja2: trim_blocks: True, lstrip_blocks: True
# yamllint disable-file
---
# Create layer3 configuration
{% if metallb_config.layer3.communities is defined %}
{% for community_name, community in metallb_config.layer3.communities.items() %}
---
apiVersion: metallb.io/v1beta1
kind: Community
metadata:
name: "{{ community_name }}"
namespace: metallb-system
spec:
communities:
- name: "{{ community_name }}"
value: "{{ community }}"
{% endfor %}
{% endif %}
---
apiVersion: metallb.io/v1beta1
kind: Community
metadata:
name: well-known
namespace: metallb-system
spec:
communities:
- name: no-export
value: 65535:65281
- name: no-advertise
value: 65535:65282
- name: local-as
value: 65535:65283
- name: nopeer
value: 65535:65284
# BGPAdvertisement is used to advertise address pools to the BGP peer. Specific pools can be listed to be advertised.
# Local BGP Advertisement specifies that the IP specified in the address pool will be used as remote source address for traffic entering your cluster from the remote peer.
# When using this option, be sure to use a subnet and routable IP for your address pool.
# This is good: 10.0.0.10/24. This is also good: 10.0.0.129/25. This is bad: 10.0.0.0/24. This is also bad: 10.0.0.128/25.
# In this example, 10.0.0.10 will be used as the remote source address.
# This is also bad: 10.0.0.10-10.0.0.25. Remember: you are working with aggregationLength, which specifies a subnet, not an IP range!
# The no-advertise community is set on the local advertisement to prevent this route from being published to the BGP peer.
# Your aggregationLength ideally is the same size as your address pool.
{% for peer_name, peer in metallb_config.layer3.metallb_peers.items() %}
{% if peer.aggregation_length is defined and peer.aggregation_length <= 30 %}
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: "{{ peer_name }}-local"
namespace: metallb-system
spec:
aggregationLength: 32
aggregationLengthV6: 128
communities:
- no-advertise
localpref: "{{ peer.localpref | default ("100") }}"
ipAddressPools:
{% for address_pool in peer.address_pool %}
- "{{ address_pool }}"
{% endfor %}
{% endif %}
# External BGP Advertisement. The IP range specied in the address pool is advertised to the BGP peer.
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: "{{ peer_name }}-external"
namespace: metallb-system
spec:
{% if peer.aggregation_length is defined and peer.aggregation_length <= 30 %}
aggregationLength: {{ peer.aggregation_length }}
{% endif %}
ipAddressPools:
{% for address_pool in peer.address_pool %}
- "{{ address_pool }}"
{% endfor %}
{% if peer.communities is defined %}
{% for community in peer.communities %}
communities:
- "{{ community }}"
{% endfor %}
{% endif %}
# Configuration for the BGP peer.
---
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
name: "{{ peer_name }}"
namespace: metallb-system
spec:
myASN: {{ peer.my_asn }}
peerASN: {{ peer.peer_asn }}
peerAddress: {{ peer.peer_address }}
{% if peer.peer_port is defined %}
peerPort: {{ peer.peer_port }}
{% else %}
peerPort: {{ metallb_config.layer3.defaults.peer_port }}
{% endif -%}
{% if peer.password is defined %}
password: "{{ peer.password }}"
{% endif -%}
{% if peer.router_id is defined %}
routerID: "{{ peer.router_id }}"
{% endif -%}
{% if peer.hold_time is defined %}
holdTime: {{ peer.hold_time }}
{% elif metallb_config.layer3.defaults.hold_time is defined %}
holdTime: {{ metallb_config.layer3.defaults.hold_time }}
{% endif -%}
{% if peer.multihop is defined %}
ebgpMultiHop: {{ peer.multihop }}
{% endif -%}
{% endfor %}

54
roles/kubernetes-apps/metallb/templates/metallb-config.yml.j2
View File

@ -1,54 +0,0 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
{% if metallb_peers | length > 0 %}
peers:
{% for peer in metallb_peers %}
- peer-address: {{ peer.peer_address }}
peer-asn: {{ peer.peer_asn }}
my-asn: {{ peer.my_asn }}
{% if peer.password is defined %}
password: "{{ peer.password }}"
{% endif %}
{% if peer.source_address is defined %}
source-address: {{ peer.source_address }}
{% endif %}
{% if peer.node_selectors is defined %}
node-selectors:
{{ peer.node_selectors | to_yaml(indent=2, width=1337) | indent(8) }}
{% endif %}
{% endfor %}
{% endif %}
address-pools:
- name: {{ metallb_pool_name }}
protocol: {{ metallb_protocol }}
addresses:
{% for ip_range in metallb_ip_range %}
- {{ ip_range }}
{% endfor %}
{% if metallb_auto_assign == false %}
auto-assign: false
{% endif %}
{% if metallb_avoid_buggy_ips == true %}
avoid-buggy-ips: true
{% endif %}
{% if metallb_additional_address_pools is defined %}{% for pool in metallb_additional_address_pools %}
- name: {{ pool }}
protocol: {{ metallb_additional_address_pools[pool].protocol }}
addresses:
{% for ip_range in metallb_additional_address_pools[pool].ip_range %}
- {{ ip_range }}
{% endfor %}
{% if metallb_additional_address_pools[pool].auto_assign is defined %}
auto-assign: {{ metallb_additional_address_pools[pool].auto_assign }}
{% endif %}
{% if metallb_additional_address_pools[pool].avoid_buggy_ips is defined %}
avoid-buggy-ips: {{ metallb_additional_address_pools[pool].avoid_buggy_ips }}
{% endif %}
{% endfor %}
{% endif %}

2011
roles/kubernetes-apps/metallb/templates/metallb.yml.j2
View File

File diff suppressed because it is too large Load Diff

22
roles/kubernetes-apps/metallb/templates/pools.yaml.j2 100644
View File

@ -0,0 +1,22 @@
#jinja2: trim_blocks: True, lstrip_blocks: True
# yamllint disable-file
---
# Create all pools
{% for pool_name, pool in metallb_config.address_pools.items() %}
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
namespace: metallb-system
name: "{{ pool_name }}"
spec:
addresses:
{% for ip_range in pool.ip_range %}
- "{{ ip_range }}"
{% endfor %}
autoAssign: {{ pool.auto_assign }}
avoidBuggyIPs: true
{% endfor %}