kubespray/docs/operations/nodes.md

# Adding/replacing a node

Modified from [comments in #3471](https://github.com/kubernetes-sigs/kubespray/issues/3471#issuecomment-530036084)

## Limitation: Removal of first kube_control_plane and etcd-master

Currently you can't remove the first node in your kube_control_plane and etcd-master list. If you still want to remove this node you have to:

### 1) Change order of current control planes

Modify the order of your control plane list by pushing your first entry to any other position. E.g. if you want to remove `node-1` of the following example:

```yaml
  children:
    kube_control_plane:
      hosts:
        node-1:
        node-2:
        node-3:
    kube_node:
      hosts:
        node-1:
        node-2:
        node-3:
    etcd:
      hosts:
        node-1:
        node-2:
        node-3:
```

change your inventory to:

```yaml
  children:
    kube_control_plane:
      hosts:
        node-2:
        node-3:
        node-1:
    kube_node:
      hosts:
        node-2:
        node-3:
        node-1:
    etcd:
      hosts:
        node-2:
        node-3:
        node-1:
```

## 2) Upgrade the cluster

run `upgrade-cluster.yml` or `cluster.yml`. Now you are good to go on with the removal.

## Adding/replacing a worker node

This should be the easiest.

### 1) Add new node to the inventory

### 2) Run `scale.yml`

You can use `--limit=NODE_NAME` to limit Kubespray to avoid disturbing other nodes in the cluster.

Before using `--limit` run playbook `facts.yml` without the limit to refresh facts cache for all nodes.

### 3) Remove an old node with remove-node.yml

With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.

If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars: `-e node=NODE_NAME -e reset_nodes=false -e allow_ungraceful_removal=true`.
Use this flag even when you remove other types of nodes like a control plane or etcd nodes.

### 4) Remove the node from the inventory

That's it.

## Adding/replacing a control plane node

### 1) Run `cluster.yml`

Append the new host to the inventory and run `cluster.yml`. You can NOT use `scale.yml` for that.

### 2) Restart kube-system/nginx-proxy

In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.

```sh
# run in every host
docker ps | grep k8s_nginx-proxy_nginx-proxy | awk '{print $1}' | xargs docker restart

# or with containerd
crictl ps | grep nginx-proxy | awk '{print $1}' | xargs crictl stop
```

### 3) Remove old control plane nodes

With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.

## Replacing a first control plane node

### 1) Change control plane nodes order in inventory

from

```ini
[kube_control_plane]
 node-1
 node-2
 node-3
```

to

```ini
[kube_control_plane]
 node-2
 node-3
 node-1
```

### 2) Remove old first control plane node from cluster

With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=node-1` to the playbook to limit the execution to the node being removed.
If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.

### 3) Edit cluster-info configmap in kube-public namespace

`kubectl  edit cm -n kube-public cluster-info`

Change ip of old kube_control_plane node with ip of live kube_control_plane node (`server` field). Also, update `certificate-authority-data` field if you changed certs.

### 4) Add new control plane node

Update inventory (if needed)

Run `cluster.yml` with `--limit=kube_control_plane`

## Adding an etcd node

You need to make sure there are always an odd number of etcd nodes in the cluster. In such a way, this is always a replacement or scale up operation. Either add two new nodes or remove an old one.

### 1) Add the new node running cluster.yml

Update the inventory and run `cluster.yml` passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`.
If the node you want to add as an etcd node is already a worker or control plane node in your cluster, you have to remove him first using `remove-node.yml`.

Run `upgrade-cluster.yml` also passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`. This is necessary to update all etcd configuration in the cluster.

At this point, you will have an even number of nodes.
Everything should still be working, and you should only have problems if the cluster decides to elect a new etcd leader before you remove a node.
Even so, running applications should continue to be available.

If you add multiple etcd nodes with one run, you might want to append `-e etcd_retries=10` to increase the amount of retries between each etcd node join.
Otherwise the etcd cluster might still be processing the first join and fail on subsequent nodes. `etcd_retries=10` might work to join 3 new nodes.

### 2) Add the new node to apiserver config

In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure the new etcd nodes are present in the apiserver command line parameter `--etcd-servers=...`.

## Removing an etcd node

### 1) Remove an old etcd node

With the node still in the inventory, run `remove-node.yml` passing `-e node=NODE_NAME` as the name of the node that should be removed.
If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.

### 2) Make sure only remaining nodes are in your inventory

Remove `NODE_NAME` from your inventory file.

### 3) Update kubernetes and network configuration files with the valid list of etcd members

Run `cluster.yml` to regenerate the configuration files on all remaining nodes.

### 4) Remove the old etcd node from apiserver config

In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure only active etcd nodes are still present in the apiserver command line parameter `--etcd-servers=...`.

### 5) Shutdown the old instance

That's it.