mirror of https://github.com/ceph/ceph-ansible.git
README-MULTISITE: Fix syntax issues from markdownlint
This commit makes the following changes:
- Remove trailing whitespace;
- Use consistent header levels;
- Fix code blocks;
- Remove hard tabs;
- Fix ordered lists;
- Fix bare URLs;
- Use markdown list of sections.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 2c244425ec)
pull/5862/head
Benoît Knecht
Dimitri Savineau
parent
0d8ba2e60c
commit
564a33c0c9
|
|
@ -1,5 +1,4 @@
|
|||
RGW Multisite
|
||||
=============
|
||||
# RGW Multisite
|
||||
|
||||
This document contains directions for configuring the RGW Multisite in ceph-ansible.
|
||||
Multisite replication can be configured either over multiple Ceph clusters or in a single Ceph cluster to isolate RGWs from eachother.
|
||||
|
|
@ -7,32 +6,36 @@ Multisite replication can be configured either over multiple Ceph clusters or in
|
|||
The first two sections are refreshers on working with ansible inventory and RGW Multisite.
|
||||
The next 4 sections are instructions on deploying the following multisite scenarios:
|
||||
|
||||
Scenario #1: Single Realm with Multiple Ceph Clusters\
|
||||
Scenario #2: Single Ceph Cluster with Multiple Realms\
|
||||
Scenario #3: Multiple Realms over Multiple Ceph Clusters\
|
||||
Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances on a Host\
|
||||
- Scenario #1: Single Realm with Multiple Ceph Clusters
|
||||
- Scenario #2: Single Ceph Cluster with Multiple Realms
|
||||
- Scenario #3: Multiple Realms over Multiple Ceph Clusters
|
||||
- Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances on a Host
|
||||
|
||||
# Working with Ansible Inventory
|
||||
## Working with Ansible Inventory
|
||||
|
||||
If you are familiar with basic ansible terminology, working with inventory files, and variable precedence feel free to skip this section.
|
||||
|
||||
## The Inventory File
|
||||
### The Inventory File
|
||||
|
||||
Ceph-ansible starts up all the different daemons in a Ceph cluster.
|
||||
Each daemon (osd.0, mon.1, rgw.a) is given a line in the inventory file. Each line is called a **host** in ansible.
|
||||
Each type of daemon (osd, mon, rgw, mgr, etc.) is given a **group** with its respective daemons in the ansible inventory file.
|
||||
|
||||
Here is an example of an inventory file (in .ini format) for a ceph cluster with 1 ceph-mgr, 4 rgws, 3 osds, and 2 mons:
|
||||
```
|
||||
|
||||
```ansible-inventory
|
||||
[mgrs]
|
||||
mgr-001 ansible_ssh_host=192.168.224.48 ansible_ssh_port=22
|
||||
|
||||
[rgws]
|
||||
rgw-001 ansible_ssh_host=192.168.216.145 ansible_ssh_port=22 radosgw_address=192.168.216.145
|
||||
rgw-002 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.178
|
||||
|
||||
[osds]
|
||||
osd-001 ansible_ssh_host=192.168.230.196 ansible_ssh_port=22
|
||||
osd-002 ansible_ssh_host=192.168.226.21 ansible_ssh_port=22
|
||||
osd-003 ansible_ssh_host=192.168.176.118 ansible_ssh_port=22
|
||||
|
||||
[mons]
|
||||
mon-001 ansible_ssh_host=192.168.210.155 ansible_ssh_port=22 monitor_address=192.168.210.155
|
||||
mon-002 ansible_ssh_host=192.168.179.111 ansible_ssh_port=22 monitor_address=192.168.179.111
|
||||
|
|
@ -41,13 +44,13 @@ Here is an example of an inventory file (in .ini format) for a ceph cluster with
|
|||
Notice there are 4 groups defined here: mgrs, rgws, osds, mons.
|
||||
There is one host (mgr-001) in mgrs, 2 hosts (rgw-001, rgw-002) in rgws, 3 hosts (osd-001, osd-002, osd-003) in osds, and 2 hosts (mon-001, mon-002) in mons.
|
||||
|
||||
## group_vars
|
||||
### group_vars
|
||||
|
||||
In the ceph-ansible tree there is a directory called `group_vars`. This directory has a collection of .yml files for variables set for each of the groups.
|
||||
The rgw multisite specific variables are defined in `all.yml`. This file has variables that apply to all groups in the inventory.
|
||||
When a variable, for example if `rgw_realm: usa`, is set in `group_vars/all.yml`, `usa` will be the value for `rgw_realm` for all of the rgws.
|
||||
|
||||
## host_vars
|
||||
### host_vars
|
||||
|
||||
If you want to set any of the variables defined in `group_vars` for a specific host you have two options.
|
||||
One option is to edit the line in the inventory file for the host you want to configure. In the above inventory each mon and rgw has a host specific variable for its address.
|
||||
|
|
@ -58,7 +61,7 @@ The values for the variables set in `host_vars` have a higher precedence than th
|
|||
|
||||
Consider this the file `host_vars/rgw-001`:
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_realm: usa
|
||||
rgw_zonegroup: alaska
|
||||
rgw_zone: juneau
|
||||
|
|
@ -72,11 +75,11 @@ system_secret_key: alaskasecretkey
|
|||
Even if `rgw_realm` is set to `france` in `group_vars/all.yml`, `rgw_realm` will evaluate to `usa` for tasks run on `rgw-001`.
|
||||
This is because Ansible gives higher precedence to the values set in `host_vars` over `group_vars`.
|
||||
|
||||
For more information on working with inventory in Ansible please visit: https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html.
|
||||
For more information on working with inventory in Ansible please visit: <https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html>.
|
||||
|
||||
# Brief Multisite Overview
|
||||
## Brief Multisite Overview
|
||||
|
||||
## RGW Multisite terminology
|
||||
### RGW Multisite terminology
|
||||
|
||||
If you are familiar with RGW multisite in detail, feel free to skip this section.
|
||||
|
||||
|
|
@ -89,13 +92,13 @@ A Ceph **cluster** in multisite has 1 or more rgws that use the same backend OSD
|
|||
|
||||
There can be mutliple clusters in one realm, multiple realms in a single cluster, or multiple realms over multiple clusters.
|
||||
|
||||
## RGW Realms
|
||||
### RGW Realms
|
||||
|
||||
A realm allows the RGWs inside of it to be independent and isolated from RGWs outside of the realm. A realm contains one or more zonegroups.
|
||||
|
||||
Realms can contain 1 or more clusters. There can also be more than 1 realm in a cluster.
|
||||
|
||||
## RGW Zonegroups
|
||||
### RGW Zonegroups
|
||||
|
||||
Similar to zones a zonegroup can be either **master zonegroup** or a **secondary zonegroup**.
|
||||
|
||||
|
|
@ -103,7 +106,7 @@ Similar to zones a zonegroup can be either **master zonegroup** or a **secondary
|
|||
There can only be one master zonegroup per realm. There can be any number of secondary zonegroups in a realm.
|
||||
Zonegroups that are not master must have `rgw_zonegroupmaster` set to false.
|
||||
|
||||
## RGW Zones
|
||||
### RGW Zones
|
||||
|
||||
A zone is a collection of RGW daemons. A zone can be either **master zone** or a **secondary zone**.
|
||||
|
||||
|
|
@ -116,17 +119,17 @@ A secondary zone pulls a realm in order to sync data to it.
|
|||
Finally, The variable `rgw_zone` is set to "default" to enable compression for clusters configured without rgw multi-site.
|
||||
If multisite is configured `rgw_zone` should not be set to "default".
|
||||
|
||||
For more defail information on multisite please visit: https://docs.ceph.com/docs/master/radosgw/multisite/
|
||||
For more defail information on multisite please visit: <https://docs.ceph.com/docs/master/radosgw/multisite/>.
|
||||
|
||||
# Deployment Scenario #1: Single Realm & Zonegroup with Multiple Ceph Clusters
|
||||
## Deployment Scenario #1: Single Realm & Zonegroup with Multiple Ceph Clusters
|
||||
|
||||
## Requirements
|
||||
### Requirements
|
||||
|
||||
* At least 2 Ceph clusters
|
||||
* 1 RGW per cluster
|
||||
* Jewel or newer
|
||||
|
||||
## Configuring the Master Zone in the Primary Cluster
|
||||
### Configuring the Master Zone in the Primary Cluster
|
||||
|
||||
This will setup a realm, master zonegroup and master zone in the Ceph cluster.
|
||||
Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration.
|
||||
|
|
@ -134,13 +137,14 @@ If there is one more that one rgw being deployed in this configuration, the rgw(
|
|||
|
||||
1. Generate System Access and System Secret Keys
|
||||
|
||||
```
|
||||
```bash
|
||||
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys.txt
|
||||
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys.txt
|
||||
```
|
||||
|
||||
2. Edit `group_vars/all.yml` for the 1st cluster
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_multisite: true
|
||||
|
||||
rgw_zone: juneau
|
||||
|
|
@ -159,21 +163,22 @@ rgw_zone_user_display_name: "Edward Lewis"
|
|||
system_access_key: 6kWkikvapSnHyE22P7nO
|
||||
system_secret_key: MGecsMrWtKZgngOHZdrd6d3JxGO5CPWgT2lcnpSt
|
||||
```
|
||||
|
||||
**Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`. Both values always need to be defined when running multisite.
|
||||
|
||||
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated.
|
||||
|
||||
3. Run the ceph-ansible playbook for the 1st cluster
|
||||
|
||||
## Configuring the Secondary Zone in a Separate Cluster
|
||||
### Configuring the Secondary Zone in a Separate Cluster
|
||||
|
||||
This will setup a realm, master zonegroup and master zone in the secondary Ceph cluster.
|
||||
Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration.
|
||||
If there is one more that one rgw being deployed in this configuration, the rgw(s) will be added to the secondary zone.
|
||||
|
||||
4. Edit `group_vars/all.yml` for the 2nd cluster
|
||||
1. Edit `group_vars/all.yml` for the 2nd cluster
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_multisite: true
|
||||
|
||||
rgw_zone: fairbanks
|
||||
|
|
@ -203,35 +208,36 @@ rgw_pullhost: rgw-001-hostname
|
|||
|
||||
**Note:** `system_access_key`, and `system_secret_key` should match what you used in the Primary Cluster
|
||||
|
||||
5. Run the ceph-ansible playbook on your 2nd cluster
|
||||
2. Run the ceph-ansible playbook on your 2nd cluster
|
||||
|
||||
## Conclusion
|
||||
### Conclusion
|
||||
|
||||
You should now have a master zone on cluster0 and a secondary zone on cluster1 in an Active-Active mode.
|
||||
|
||||
# Deployment Scenario #2: Single Ceph Cluster with Multiple Realms
|
||||
## Deployment Scenario #2: Single Ceph Cluster with Multiple Realms
|
||||
|
||||
## Requirements
|
||||
### Requirements
|
||||
|
||||
* Jewel or newer
|
||||
|
||||
## Configuring Multiple Realms in a Single Cluster
|
||||
### Configuring Multiple Realms in a Single Cluster
|
||||
|
||||
This configuration will a single Ceph cluster with multiple realms.
|
||||
Each of the rgws in the inventory should have a file in `host_vars` where the realm, zone, and zonegroup can be set for the rgw along with other variables.
|
||||
|
||||
1. Generate System Access and System Secret Keys for each realm
|
||||
|
||||
```
|
||||
```bash
|
||||
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-1.txt
|
||||
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-1.txt
|
||||
|
||||
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-2.txt
|
||||
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-2.txt
|
||||
```
|
||||
|
||||
2. Edit `group_vars/all.yml` for the cluster
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_multisite: true
|
||||
```
|
||||
|
||||
|
|
@ -241,7 +247,7 @@ As previously learned, all values set here will be set on all rgw hosts. `rgw_mu
|
|||
|
||||
Here is an example of the file `host_vars/rgw-001` for the `rgw-001` entry in the `[rgws]` section of for the example ansible inventory.
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: true
|
||||
rgw_zonesecondary: false
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -259,7 +265,7 @@ system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB
|
|||
|
||||
Here is an example of the file `host_vars/rgw-002` for the `rgw-002` entry in the `[rgws]` section of for the example ansible inventory.
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: true
|
||||
rgw_zonesecondary: false
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -283,14 +289,14 @@ system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
|
|||
|
||||
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated.
|
||||
|
||||
5. Run the ceph-ansible playbook on your cluster
|
||||
4. Run the ceph-ansible playbook on your cluster
|
||||
|
||||
## Conclusion
|
||||
### Conclusion
|
||||
|
||||
The RGWs in the deployed cluster will be split up into 2 realms: `france` and `usa`. France has a zonegroup named `idf` and usa has one called `alaska`.
|
||||
`Idf` has a master zone called `paris`. `Alaska` has a master zone called `juneau`.
|
||||
|
||||
# Deployment Scenario #3: Multiple Realms over Multiple Ceph Clusters
|
||||
## Deployment Scenario #3: Multiple Realms over Multiple Ceph Clusters
|
||||
|
||||
The multisite playbooks in ceph-ansible are flexible enough to create many realms, zonegroups, and zones that span many clusters.
|
||||
|
||||
|
|
@ -300,7 +306,7 @@ The host_vars for the rgws in the second cluster would have `rgw_zonesecondary`
|
|||
|
||||
The inventory for the rgws section of the master cluster for this example looks like:
|
||||
|
||||
```
|
||||
```ansible-inventory
|
||||
[rgws]
|
||||
rgw-001 ansible_ssh_host=192.168.216.145 ansible_ssh_port=22 radosgw_address=192.168.216.145
|
||||
rgw-002 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.178
|
||||
|
|
@ -308,13 +314,13 @@ The inventory for the rgws section of the master cluster for this example looks
|
|||
|
||||
The inventory for the rgws section of the secondary cluster for this example looks like:
|
||||
|
||||
```
|
||||
```ansible-inventory
|
||||
[rgws]
|
||||
rgw-003 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.199
|
||||
rgw-004 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.194.109
|
||||
```
|
||||
|
||||
## Requirements
|
||||
### Requirements
|
||||
|
||||
* At least 2 Ceph clusters
|
||||
* at least 2 RGW in the master cluster and the secondary clusters
|
||||
|
|
@ -322,7 +328,7 @@ The inventory for the rgws section of the secondary cluster for this example loo
|
|||
|
||||
1. Generate System Access and System Secret Keys for each realm
|
||||
|
||||
```
|
||||
```bash
|
||||
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-1.txt
|
||||
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-1.txt
|
||||
|
||||
|
|
@ -331,9 +337,10 @@ echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | h
|
|||
|
||||
...
|
||||
```
|
||||
|
||||
2. Edit `group_vars/all.yml` for the cluster
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_multisite: true
|
||||
```
|
||||
|
||||
|
|
@ -343,7 +350,7 @@ As per the previous example, all values set here will be set on all rgw hosts.
|
|||
|
||||
Here is an example of the file `host_vars/rgw-001` for the the master cluster.
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: true
|
||||
rgw_zonesecondary: false
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -361,7 +368,7 @@ system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB
|
|||
|
||||
Here is an example of the file `host_vars/rgw-002` for the the master cluster.
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: true
|
||||
rgw_zonesecondary: false
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -383,7 +390,7 @@ system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
|
|||
|
||||
Here is an example of the file `host_vars/rgw-003` for the `rgw-003` entry in the `[rgws]` section for a secondary cluster.
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: false
|
||||
rgw_zonesecondary: true
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -404,7 +411,7 @@ rgw_pullhost: rgw-001-hostname
|
|||
|
||||
Here is an example of the file `host_vars/rgw-004` for the `rgw-004` entry in the `[rgws]` section for a secondary cluster.
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: false
|
||||
rgw_zonesecondary: true
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -425,13 +432,13 @@ rgw_pullhost: rgw-002-hostname
|
|||
|
||||
6. Run the ceph-ansible playbook on your secondary cluster.
|
||||
|
||||
## Conclusion
|
||||
### Conclusion
|
||||
|
||||
There will be 2 realms in this configuration, `france` and `usa`, with RGWs and RGW zones in both clusters. Cluster0 will has the master zones and Cluster1 has the secondary zones.
|
||||
|
||||
Data is realm france will be replicated over both clusters and remain isolated from rgws in realm usa and vice versa.
|
||||
|
||||
# Deployment Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances
|
||||
## Deployment Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances
|
||||
|
||||
More than 1 RGW can be running on a single host. To configure multisite for a host with more than one rgw instance running on the host, `rgw_instances` must be configured.
|
||||
|
||||
|
|
@ -439,7 +446,7 @@ Each item in `rgw_instances` (declared in a host_vars file) represents an RGW on
|
|||
|
||||
Here is an example:
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_instances:
|
||||
- instance_name: rgw1
|
||||
rgw_realm: usa
|
||||
|
|
@ -453,11 +460,11 @@ rgw_instances:
|
|||
system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
|
||||
```
|
||||
|
||||
## Setting rgw_instances for a host in the master zone
|
||||
### Setting rgw_instances for a host in the master zone
|
||||
|
||||
Here is an example of a host_vars for a host (ex: rgw-001 in the examples) containing 2 rgw_instances:
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: true
|
||||
rgw_zonesecondary: false
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -489,7 +496,7 @@ This example starts up 2 rgws on host rgw-001. `rgw1` is configured to be in rea
|
|||
|
||||
The variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/.
|
||||
|
||||
## Setting rgw_instances for a host in a secondary zone
|
||||
### Setting rgw_instances for a host in a secondary zone
|
||||
|
||||
To start up multiple rgws on a host that are in a secondary zone, `endpoint` must be added to rgw_instances.
|
||||
|
||||
|
|
@ -497,7 +504,7 @@ The value of `endpoint` should be the endpoint of an RGW in the master zone of t
|
|||
|
||||
Here is an example of a host_vars for a host containing 2 rgw_instances in a secondary zone:
|
||||
|
||||
```
|
||||
```yaml
|
||||
rgw_zonemaster: false
|
||||
rgw_zonesecondary: true
|
||||
rgw_zonegroupmaster: true
|
||||
|
|
@ -531,6 +538,6 @@ This example starts up 2 rgws on the host that will pull the realm from the rgws
|
|||
|
||||
Just like the example on the master zone, the variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/.
|
||||
|
||||
## Conclusion
|
||||
### Conclusion
|
||||
|
||||
`rgw_instances` can be used in host_vars for multisite deployments like scenarios 2 and 3
|
||||
|
|
|
|||
Loading…
Reference in New Issue