ceph
/
ceph-ansible
mirror of https://github.com/ceph/ceph-ansible.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

README-MULTISITE: Fix syntax issues from markdownlint 

This commit makes the following changes:

- Remove trailing whitespace;
- Use consistent header levels;
- Fix code blocks;
- Remove hard tabs;
- Fix ordered lists;
- Fix bare URLs;
- Use markdown list of sections.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
pull/5538/head
Benoît Knecht 2020-09-03 09:01:16 +02:00 committed by Guillaume Abrioux
parent ba3512a8fc
commit 2c244425ec
1 changed files with 243 additions and 236 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

479
README-MULTISITE.md
View File

@ -1,53 +1,56 @@
RGW Multisite # RGW Multisite
=============
This document contains directions for configuring the RGW Multisite in ceph-ansible. This document contains directions for configuring the RGW Multisite in ceph-ansible.
Multisite replication can be configured either over multiple Ceph clusters or in a single Ceph cluster to isolate RGWs from eachother. Multisite replication can be configured either over multiple Ceph clusters or in a single Ceph cluster to isolate RGWs from eachother.
The first two sections are refreshers on working with ansible inventory and RGW Multisite. The first two sections are refreshers on working with ansible inventory and RGW Multisite.
The next 4 sections are instructions on deploying the following multisite scenarios: The next 4 sections are instructions on deploying the following multisite scenarios:
Scenario #1: Single Realm with Multiple Ceph Clusters\ - Scenario #1: Single Realm with Multiple Ceph Clusters
Scenario #2: Single Ceph Cluster with Multiple Realms\ - Scenario #2: Single Ceph Cluster with Multiple Realms
Scenario #3: Multiple Realms over Multiple Ceph Clusters\ - Scenario #3: Multiple Realms over Multiple Ceph Clusters
Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances on a Host\ - Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances on a Host
# Working with Ansible Inventory ## Working with Ansible Inventory
If you are familiar with basic ansible terminology, working with inventory files, and variable precedence feel free to skip this section. If you are familiar with basic ansible terminology, working with inventory files, and variable precedence feel free to skip this section.
## The Inventory File ### The Inventory File
Ceph-ansible starts up all the different daemons in a Ceph cluster. Ceph-ansible starts up all the different daemons in a Ceph cluster.
Each daemon (osd.0, mon.1, rgw.a) is given a line in the inventory file. Each line is called a **host** in ansible. Each daemon (osd.0, mon.1, rgw.a) is given a line in the inventory file. Each line is called a **host** in ansible.
Each type of daemon (osd, mon, rgw, mgr, etc.) is given a **group** with its respective daemons in the ansible inventory file. Each type of daemon (osd, mon, rgw, mgr, etc.) is given a **group** with its respective daemons in the ansible inventory file.
Here is an example of an inventory file (in .ini format) for a ceph cluster with 1 ceph-mgr, 4 rgws, 3 osds, and 2 mons: Here is an example of an inventory file (in .ini format) for a ceph cluster with 1 ceph-mgr, 4 rgws, 3 osds, and 2 mons:
```
```ansible-inventory
[mgrs] [mgrs]
mgr-001 ansible_ssh_host=192.168.224.48 ansible_ssh_port=22 mgr-001 ansible_ssh_host=192.168.224.48 ansible_ssh_port=22
[rgws] [rgws]
rgw-001 ansible_ssh_host=192.168.216.145 ansible_ssh_port=22 radosgw_address=192.168.216.145 rgw-001 ansible_ssh_host=192.168.216.145 ansible_ssh_port=22 radosgw_address=192.168.216.145
rgw-002 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.178 rgw-002 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.178
[osds] [osds]
osd-001 ansible_ssh_host=192.168.230.196 ansible_ssh_port=22 osd-001 ansible_ssh_host=192.168.230.196 ansible_ssh_port=22
osd-002 ansible_ssh_host=192.168.226.21 ansible_ssh_port=22 osd-002 ansible_ssh_host=192.168.226.21 ansible_ssh_port=22
osd-003 ansible_ssh_host=192.168.176.118 ansible_ssh_port=22 osd-003 ansible_ssh_host=192.168.176.118 ansible_ssh_port=22
[mons] [mons]
mon-001 ansible_ssh_host=192.168.210.155 ansible_ssh_port=22 monitor_address=192.168.210.155 mon-001 ansible_ssh_host=192.168.210.155 ansible_ssh_port=22 monitor_address=192.168.210.155
mon-002 ansible_ssh_host=192.168.179.111 ansible_ssh_port=22 monitor_address=192.168.179.111 mon-002 ansible_ssh_host=192.168.179.111 ansible_ssh_port=22 monitor_address=192.168.179.111
``` ```
Notice there are 4 groups defined here: mgrs, rgws, osds, mons. Notice there are 4 groups defined here: mgrs, rgws, osds, mons.
There is one host (mgr-001) in mgrs, 2 hosts (rgw-001, rgw-002) in rgws, 3 hosts (osd-001, osd-002, osd-003) in osds, and 2 hosts (mon-001, mon-002) in mons. There is one host (mgr-001) in mgrs, 2 hosts (rgw-001, rgw-002) in rgws, 3 hosts (osd-001, osd-002, osd-003) in osds, and 2 hosts (mon-001, mon-002) in mons.
## group_vars ### group_vars
In the ceph-ansible tree there is a directory called `group_vars`. This directory has a collection of .yml files for variables set for each of the groups. In the ceph-ansible tree there is a directory called `group_vars`. This directory has a collection of .yml files for variables set for each of the groups.
The rgw multisite specific variables are defined in `all.yml`. This file has variables that apply to all groups in the inventory. The rgw multisite specific variables are defined in `all.yml`. This file has variables that apply to all groups in the inventory.
When a variable, for example if `rgw_realm: usa`, is set in `group_vars/all.yml`, `usa` will be the value for `rgw_realm` for all of the rgws. When a variable, for example if `rgw_realm: usa`, is set in `group_vars/all.yml`, `usa` will be the value for `rgw_realm` for all of the rgws.
## host_vars ### host_vars
If you want to set any of the variables defined in `group_vars` for a specific host you have two options. If you want to set any of the variables defined in `group_vars` for a specific host you have two options.
One option is to edit the line in the inventory file for the host you want to configure. In the above inventory each mon and rgw has a host specific variable for its address. One option is to edit the line in the inventory file for the host you want to configure. In the above inventory each mon and rgw has a host specific variable for its address.
@ -58,7 +61,7 @@ The values for the variables set in `host_vars` have a higher precedence than th
Consider this the file `host_vars/rgw-001`: Consider this the file `host_vars/rgw-001`:
``` ```yaml
rgw_realm: usa rgw_realm: usa
rgw_zonegroup: alaska rgw_zonegroup: alaska
rgw_zone: juneau rgw_zone: juneau
@ -72,38 +75,38 @@ system_secret_key: alaskasecretkey
Even if `rgw_realm` is set to `france` in `group_vars/all.yml`, `rgw_realm` will evaluate to `usa` for tasks run on `rgw-001`. Even if `rgw_realm` is set to `france` in `group_vars/all.yml`, `rgw_realm` will evaluate to `usa` for tasks run on `rgw-001`.
This is because Ansible gives higher precedence to the values set in `host_vars` over `group_vars`. This is because Ansible gives higher precedence to the values set in `host_vars` over `group_vars`.
For more information on working with inventory in Ansible please visit: https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html. For more information on working with inventory in Ansible please visit: <https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html>.
# Brief Multisite Overview ## Brief Multisite Overview
## RGW Multisite terminology ### RGW Multisite terminology
If you are familiar with RGW multisite in detail, feel free to skip this section. If you are familiar with RGW multisite in detail, feel free to skip this section.
Rados gateways (RGWs) in multisite replication are grouped into zones. Rados gateways (RGWs) in multisite replication are grouped into zones.
A group of 1 or more RGWs can be grouped into a **zone**.\ A group of 1 or more RGWs can be grouped into a **zone**.\
A group of 1 or more zones can be grouped into a **zonegroup**.\ A group of 1 or more zones can be grouped into a **zonegroup**.\
A group of 1 or more zonegroups can be grouped into a **realm**.\ A group of 1 or more zonegroups can be grouped into a **realm**.\
A Ceph **cluster** in multisite has 1 or more rgws that use the same backend OSDs.\ A Ceph **cluster** in multisite has 1 or more rgws that use the same backend OSDs.\
There can be mutliple clusters in one realm, multiple realms in a single cluster, or multiple realms over multiple clusters. There can be mutliple clusters in one realm, multiple realms in a single cluster, or multiple realms over multiple clusters.
## RGW Realms ### RGW Realms
A realm allows the RGWs inside of it to be independent and isolated from RGWs outside of the realm. A realm contains one or more zonegroups. A realm allows the RGWs inside of it to be independent and isolated from RGWs outside of the realm. A realm contains one or more zonegroups.
Realms can contain 1 or more clusters. There can also be more than 1 realm in a cluster. Realms can contain 1 or more clusters. There can also be more than 1 realm in a cluster.
## RGW Zonegroups ### RGW Zonegroups
Similar to zones a zonegroup can be either **master zonegroup** or a **secondary zonegroup**. Similar to zones a zonegroup can be either **master zonegroup** or a **secondary zonegroup**.
`rgw_zonegroupmaster` specifies whether the zonegroup will be the master zonegroup in a realm. `rgw_zonegroupmaster` specifies whether the zonegroup will be the master zonegroup in a realm.
There can only be one master zonegroup per realm. There can be any number of secondary zonegroups in a realm. There can only be one master zonegroup per realm. There can be any number of secondary zonegroups in a realm.
Zonegroups that are not master must have `rgw_zonegroupmaster` set to false. Zonegroups that are not master must have `rgw_zonegroupmaster` set to false.
## RGW Zones ### RGW Zones
A zone is a collection of RGW daemons. A zone can be either **master zone** or a **secondary zone**. A zone is a collection of RGW daemons. A zone can be either **master zone** or a **secondary zone**.
@ -116,17 +119,17 @@ A secondary zone pulls a realm in order to sync data to it.
Finally, The variable `rgw_zone` is set to "default" to enable compression for clusters configured without rgw multi-site. Finally, The variable `rgw_zone` is set to "default" to enable compression for clusters configured without rgw multi-site.
If multisite is configured `rgw_zone` should not be set to "default". If multisite is configured `rgw_zone` should not be set to "default".
For more defail information on multisite please visit: https://docs.ceph.com/docs/master/radosgw/multisite/ For more defail information on multisite please visit: <https://docs.ceph.com/docs/master/radosgw/multisite/>.
# Deployment Scenario #1: Single Realm & Zonegroup with Multiple Ceph Clusters ## Deployment Scenario #1: Single Realm & Zonegroup with Multiple Ceph Clusters
## Requirements ### Requirements
* At least 2 Ceph clusters * At least 2 Ceph clusters
* 1 RGW per cluster * 1 RGW per cluster
* Jewel or newer * Jewel or newer
## Configuring the Master Zone in the Primary Cluster ### Configuring the Master Zone in the Primary Cluster
This will setup a realm, master zonegroup and master zone in the Ceph cluster. This will setup a realm, master zonegroup and master zone in the Ceph cluster.
Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration. Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration.
@ -134,163 +137,166 @@ If there is one more that one rgw being deployed in this configuration, the rgw(
1. Generate System Access and System Secret Keys 1. Generate System Access and System Secret Keys
``` ```bash
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys.txt echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys.txt
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys.txt echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys.txt
``` ```
2. Edit `group_vars/all.yml` for the 1st cluster 2. Edit `group_vars/all.yml` for the 1st cluster
``` ```yaml
rgw_multisite: true rgw_multisite: true
rgw_zone: juneau rgw_zone: juneau
rgw_zonegroup: alaska rgw_zonegroup: alaska
rgw_realm: usa rgw_realm: usa
rgw_zonemaster: true rgw_zonemaster: true
rgw_zonesecondary: false rgw_zonesecondary: false
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_zone_user: edward.lewis rgw_zone_user: edward.lewis
rgw_zone_user_display_name: "Edward Lewis" rgw_zone_user_display_name: "Edward Lewis"
system_access_key: 6kWkikvapSnHyE22P7nO system_access_key: 6kWkikvapSnHyE22P7nO
system_secret_key: MGecsMrWtKZgngOHZdrd6d3JxGO5CPWgT2lcnpSt system_secret_key: MGecsMrWtKZgngOHZdrd6d3JxGO5CPWgT2lcnpSt
``` ```
**Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`. Both values always need to be defined when running multisite.
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated. **Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`. Both values always need to be defined when running multisite.
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated.
3. Run the ceph-ansible playbook for the 1st cluster 3. Run the ceph-ansible playbook for the 1st cluster
## Configuring the Secondary Zone in a Separate Cluster ### Configuring the Secondary Zone in a Separate Cluster
This will setup a realm, master zonegroup and master zone in the secondary Ceph cluster. This will setup a realm, master zonegroup and master zone in the secondary Ceph cluster.
Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration. Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration.
If there is one more that one rgw being deployed in this configuration, the rgw(s) will be added to the secondary zone. If there is one more that one rgw being deployed in this configuration, the rgw(s) will be added to the secondary zone.
4. Edit `group_vars/all.yml` for the 2nd cluster 1. Edit `group_vars/all.yml` for the 2nd cluster
``` ```yaml
rgw_multisite: true rgw_multisite: true
rgw_zone: fairbanks rgw_zone: fairbanks
rgw_zonegroup: alaska rgw_zonegroup: alaska
rgw_realm: usa rgw_realm: usa
rgw_zonemaster: false rgw_zonemaster: false
rgw_zonesecondary: true rgw_zonesecondary: true
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_zone_user: edward.lewis rgw_zone_user: edward.lewis
rgw_zone_user_display_name: "Edward Lewis" rgw_zone_user_display_name: "Edward Lewis"
system_access_key: 6kWkikvapSnHyE22P7nO system_access_key: 6kWkikvapSnHyE22P7nO
system_secret_key: MGecsMrWtKZgngOHZdrd6d3JxGO5CPWgT2lcnpSt system_secret_key: MGecsMrWtKZgngOHZdrd6d3JxGO5CPWgT2lcnpSt
rgw_pull_proto: http rgw_pull_proto: http
rgw_pull_port: 8080 rgw_pull_port: 8080
rgw_pullhost: rgw-001-hostname rgw_pullhost: rgw-001-hostname
``` ```
**Note:** `rgw_zonemaster` should have the value of `false` and `rgw_zonesecondary` should be `true` **Note:** `rgw_zonemaster` should have the value of `false` and `rgw_zonesecondary` should be `true`
**Note:** The variables `rgw_pull_port`, `rgw_pull_proto`, `rgw_pullhost`, are joined together to make an endpoint string needed to create secondary zones. This endpoint is of one of the RGW endpoints in a master zone in the zonegroup and realm you want to create secondary zones in. This endpoint **must be resolvable** from the mons and rgws in the cluster the secondary zone(s) are being created in. **Note:** The variables `rgw_pull_port`, `rgw_pull_proto`, `rgw_pullhost`, are joined together to make an endpoint string needed to create secondary zones. This endpoint is of one of the RGW endpoints in a master zone in the zonegroup and realm you want to create secondary zones in. This endpoint **must be resolvable** from the mons and rgws in the cluster the secondary zone(s) are being created in.
**Note:** `system_access_key`, and `system_secret_key` should match what you used in the Primary Cluster **Note:** `system_access_key`, and `system_secret_key` should match what you used in the Primary Cluster
5. Run the ceph-ansible playbook on your 2nd cluster 2. Run the ceph-ansible playbook on your 2nd cluster
## Conclusion ### Conclusion
You should now have a master zone on cluster0 and a secondary zone on cluster1 in an Active-Active mode. You should now have a master zone on cluster0 and a secondary zone on cluster1 in an Active-Active mode.
# Deployment Scenario #2: Single Ceph Cluster with Multiple Realms ## Deployment Scenario #2: Single Ceph Cluster with Multiple Realms
## Requirements ### Requirements
* Jewel or newer * Jewel or newer
## Configuring Multiple Realms in a Single Cluster ### Configuring Multiple Realms in a Single Cluster
This configuration will a single Ceph cluster with multiple realms. This configuration will a single Ceph cluster with multiple realms.
Each of the rgws in the inventory should have a file in `host_vars` where the realm, zone, and zonegroup can be set for the rgw along with other variables. Each of the rgws in the inventory should have a file in `host_vars` where the realm, zone, and zonegroup can be set for the rgw along with other variables.
1. Generate System Access and System Secret Keys for each realm 1. Generate System Access and System Secret Keys for each realm
``` ```bash
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-1.txt echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-1.txt
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-1.txt echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-1.txt
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-2.txt
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-2.txt
```
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-2.txt
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-2.txt
```
2. Edit `group_vars/all.yml` for the cluster 2. Edit `group_vars/all.yml` for the cluster
``` ```yaml
rgw_multisite: true rgw_multisite: true
``` ```
As previously learned, all values set here will be set on all rgw hosts. `rgw_multisite` be set to `true` for all rgw hosts so multisite playbooks can run on all rgws. As previously learned, all values set here will be set on all rgw hosts. `rgw_multisite` be set to `true` for all rgw hosts so multisite playbooks can run on all rgws.
3. Create & edit files in `host_vars/` to create realms, zonegroups, and master zones. 3. Create & edit files in `host_vars/` to create realms, zonegroups, and master zones.
Here is an example of the file `host_vars/rgw-001` for the `rgw-001` entry in the `[rgws]` section of for the example ansible inventory. Here is an example of the file `host_vars/rgw-001` for the `rgw-001` entry in the `[rgws]` section of for the example ansible inventory.
``` ```yaml
rgw_zonemaster: true rgw_zonemaster: true
rgw_zonesecondary: false rgw_zonesecondary: false
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_realm: france rgw_realm: france
rgw_zonegroup: idf rgw_zonegroup: idf
rgw_zone: paris rgw_zone: paris
radosgw_address: "{{ _radosgw_address }}" radosgw_address: "{{ _radosgw_address }}"
radosgw_frontend_port: 8080 radosgw_frontend_port: 8080
rgw_zone_user: jacques.chirac rgw_zone_user: jacques.chirac
rgw_zone_user_display_name: "Jacques Chirac" rgw_zone_user_display_name: "Jacques Chirac"
system_access_key: P9Eb6S8XNyo4dtZZUUMy system_access_key: P9Eb6S8XNyo4dtZZUUMy
system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB
``` ```
Here is an example of the file `host_vars/rgw-002` for the `rgw-002` entry in the `[rgws]` section of for the example ansible inventory. Here is an example of the file `host_vars/rgw-002` for the `rgw-002` entry in the `[rgws]` section of for the example ansible inventory.
``` ```yaml
rgw_zonemaster: true rgw_zonemaster: true
rgw_zonesecondary: false rgw_zonesecondary: false
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_realm: usa rgw_realm: usa
rgw_zonegroup: alaska rgw_zonegroup: alaska
rgw_zone: juneau rgw_zone: juneau
radosgw_address: "{{ _radosgw_address }}" radosgw_address: "{{ _radosgw_address }}"
radosgw_frontend_port: 8080 radosgw_frontend_port: 8080
rgw_zone_user: edward.lewis rgw_zone_user: edward.lewis
rgw_zone_user_display_name: "Edward Lewis" rgw_zone_user_display_name: "Edward Lewis"
system_access_key: yu17wkvAx3B8Wyn08XoF system_access_key: yu17wkvAx3B8Wyn08XoF
system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
``` ```
**Note:** Since `rgw_realm`, `rgw_zonegroup`, and `rgw_zone` differ between files, a new realm, zonegroup, and master zone are created containing rgw-001 and rgw-002 respectively. **Note:** Since `rgw_realm`, `rgw_zonegroup`, and `rgw_zone` differ between files, a new realm, zonegroup, and master zone are created containing rgw-001 and rgw-002 respectively.
**Note:** `rgw_zonegroupmaster` is set to true in each of the files since it will be the only zonegroup in each realm. **Note:** `rgw_zonegroupmaster` is set to true in each of the files since it will be the only zonegroup in each realm.
**Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`. **Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`.
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated. **Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated.
5. Run the ceph-ansible playbook on your cluster 4. Run the ceph-ansible playbook on your cluster
## Conclusion ### Conclusion
The RGWs in the deployed cluster will be split up into 2 realms: `france` and `usa`. France has a zonegroup named `idf` and usa has one called `alaska`. The RGWs in the deployed cluster will be split up into 2 realms: `france` and `usa`. France has a zonegroup named `idf` and usa has one called `alaska`.
`Idf` has a master zone called `paris`. `Alaska` has a master zone called `juneau`. `Idf` has a master zone called `paris`. `Alaska` has a master zone called `juneau`.
# Deployment Scenario #3: Multiple Realms over Multiple Ceph Clusters ## Deployment Scenario #3: Multiple Realms over Multiple Ceph Clusters
The multisite playbooks in ceph-ansible are flexible enough to create many realms, zonegroups, and zones that span many clusters. The multisite playbooks in ceph-ansible are flexible enough to create many realms, zonegroups, and zones that span many clusters.
@ -300,21 +306,21 @@ The host_vars for the rgws in the second cluster would have `rgw_zonesecondary`
The inventory for the rgws section of the master cluster for this example looks like: The inventory for the rgws section of the master cluster for this example looks like:
``` ```ansible-inventory
[rgws] [rgws]
rgw-001 ansible_ssh_host=192.168.216.145 ansible_ssh_port=22 radosgw_address=192.168.216.145 rgw-001 ansible_ssh_host=192.168.216.145 ansible_ssh_port=22 radosgw_address=192.168.216.145
rgw-002 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.178 rgw-002 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.178
``` ```
The inventory for the rgws section of the secondary cluster for this example looks like: The inventory for the rgws section of the secondary cluster for this example looks like:
``` ```ansible-inventory
[rgws] [rgws]
rgw-003 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.199 rgw-003 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.215.199
rgw-004 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.194.109 rgw-004 ansible_ssh_host=192.168.215.178 ansible_ssh_port=22 radosgw_address=192.168.194.109
``` ```
## Requirements ### Requirements
* At least 2 Ceph clusters * At least 2 Ceph clusters
* at least 2 RGW in the master cluster and the secondary clusters * at least 2 RGW in the master cluster and the secondary clusters
@ -322,124 +328,125 @@ The inventory for the rgws section of the secondary cluster for this example loo
1. Generate System Access and System Secret Keys for each realm 1. Generate System Access and System Secret Keys for each realm
``` ```bash
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-1.txt echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-1.txt
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-1.txt echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-1.txt
echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-2.txt echo system_access_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) > multi-site-keys-realm-2.txt
echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-2.txt echo system_secret_key: $(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) >> multi-site-keys-realm-2.txt
...
```
...
```
2. Edit `group_vars/all.yml` for the cluster 2. Edit `group_vars/all.yml` for the cluster
``` ```yaml
rgw_multisite: true rgw_multisite: true
``` ```
As per the previous example, all values set here will be set on all rgw hosts. As per the previous example, all values set here will be set on all rgw hosts.
3. Create & edit files in `host_vars/` to create realms, zonegroups, and master zones on cluster #1. 3. Create & edit files in `host_vars/` to create realms, zonegroups, and master zones on cluster #1.
Here is an example of the file `host_vars/rgw-001` for the the master cluster. Here is an example of the file `host_vars/rgw-001` for the the master cluster.
``` ```yaml
rgw_zonemaster: true rgw_zonemaster: true
rgw_zonesecondary: false rgw_zonesecondary: false
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_realm: france rgw_realm: france
rgw_zonegroup: idf rgw_zonegroup: idf
rgw_zone: paris rgw_zone: paris
radosgw_address: "{{ _radosgw_address }}" radosgw_address: "{{ _radosgw_address }}"
radosgw_frontend_port: 8080 radosgw_frontend_port: 8080
rgw_zone_user: jacques.chirac rgw_zone_user: jacques.chirac
rgw_zone_user_display_name: "Jacques Chirac" rgw_zone_user_display_name: "Jacques Chirac"
system_access_key: P9Eb6S8XNyo4dtZZUUMy system_access_key: P9Eb6S8XNyo4dtZZUUMy
system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB
``` ```
Here is an example of the file `host_vars/rgw-002` for the the master cluster. Here is an example of the file `host_vars/rgw-002` for the the master cluster.
``` ```yaml
rgw_zonemaster: true rgw_zonemaster: true
rgw_zonesecondary: false rgw_zonesecondary: false
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_realm: usa rgw_realm: usa
rgw_zonegroup: alaska rgw_zonegroup: alaska
rgw_zone: juneau rgw_zone: juneau
radosgw_address: "{{ _radosgw_address }}" radosgw_address: "{{ _radosgw_address }}"
radosgw_frontend_port: 8080 radosgw_frontend_port: 8080
rgw_zone_user: edward.lewis rgw_zone_user: edward.lewis
rgw_zone_user_display_name: "Edward Lewis" rgw_zone_user_display_name: "Edward Lewis"
system_access_key: yu17wkvAx3B8Wyn08XoF system_access_key: yu17wkvAx3B8Wyn08XoF
system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
``` ```
4. Run the ceph-ansible playbook on your master cluster. 4. Run the ceph-ansible playbook on your master cluster.
5. Create & edit files in `host_vars/` for the entries in the `[rgws]` section of the inventory on the secondary cluster. 5. Create & edit files in `host_vars/` for the entries in the `[rgws]` section of the inventory on the secondary cluster.
Here is an example of the file `host_vars/rgw-003` for the `rgw-003` entry in the `[rgws]` section for a secondary cluster. Here is an example of the file `host_vars/rgw-003` for the `rgw-003` entry in the `[rgws]` section for a secondary cluster.
``` ```yaml
rgw_zonemaster: false rgw_zonemaster: false
rgw_zonesecondary: true rgw_zonesecondary: true
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_realm: france rgw_realm: france
rgw_zonegroup: idf rgw_zonegroup: idf
rgw_zone: versailles rgw_zone: versailles
radosgw_address: "{{ _radosgw_address }}" radosgw_address: "{{ _radosgw_address }}"
radosgw_frontend_port: 8080 radosgw_frontend_port: 8080
rgw_zone_user: jacques.chirac rgw_zone_user: jacques.chirac
rgw_zone_user_display_name: "Jacques Chirac" rgw_zone_user_display_name: "Jacques Chirac"
system_access_key: P9Eb6S8XNyo4dtZZUUMy system_access_key: P9Eb6S8XNyo4dtZZUUMy
system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB
rgw_pull_proto: http rgw_pull_proto: http
rgw_pull_port: 8080 rgw_pull_port: 8080
rgw_pullhost: rgw-001-hostname rgw_pullhost: rgw-001-hostname
``` ```
Here is an example of the file `host_vars/rgw-004` for the `rgw-004` entry in the `[rgws]` section for a secondary cluster. Here is an example of the file `host_vars/rgw-004` for the `rgw-004` entry in the `[rgws]` section for a secondary cluster.
``` ```yaml
rgw_zonemaster: false rgw_zonemaster: false
rgw_zonesecondary: true rgw_zonesecondary: true
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
rgw_multisite_proto: http rgw_multisite_proto: http
rgw_realm: usa rgw_realm: usa
rgw_zonegroup: alaska rgw_zonegroup: alaska
rgw_zone: juneau rgw_zone: juneau
radosgw_address: "{{ _radosgw_address }}" radosgw_address: "{{ _radosgw_address }}"
radosgw_frontend_port: 8080 radosgw_frontend_port: 8080
rgw_zone_user: edward.lewis rgw_zone_user: edward.lewis
rgw_zone_user_display_name: "Edward Lewis" rgw_zone_user_display_name: "Edward Lewis"
system_access_key: yu17wkvAx3B8Wyn08XoF system_access_key: yu17wkvAx3B8Wyn08XoF
system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
rgw_pull_proto: http rgw_pull_proto: http
rgw_pull_port: 8080 rgw_pull_port: 8080
rgw_pullhost: rgw-002-hostname rgw_pullhost: rgw-002-hostname
``` ```
6. Run the ceph-ansible playbook on your secondary cluster. 6. Run the ceph-ansible playbook on your secondary cluster.
## Conclusion ### Conclusion
There will be 2 realms in this configuration, `france` and `usa`, with RGWs and RGW zones in both clusters. Cluster0 will has the master zones and Cluster1 has the secondary zones. There will be 2 realms in this configuration, `france` and `usa`, with RGWs and RGW zones in both clusters. Cluster0 will has the master zones and Cluster1 has the secondary zones.
Data is realm france will be replicated over both clusters and remain isolated from rgws in realm usa and vice versa. Data is realm france will be replicated over both clusters and remain isolated from rgws in realm usa and vice versa.
# Deployment Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances ## Deployment Scenario #4: Multiple Realms over Multiple Ceph Clusters with Multiple Instances
More than 1 RGW can be running on a single host. To configure multisite for a host with more than one rgw instance running on the host, `rgw_instances` must be configured. More than 1 RGW can be running on a single host. To configure multisite for a host with more than one rgw instance running on the host, `rgw_instances` must be configured.
Each item in `rgw_instances` (declared in a host_vars file) represents an RGW on that host. In each item is the multisite configuration for that RGW. Each item in `rgw_instances` (declared in a host_vars file) represents an RGW on that host. In each item is the multisite configuration for that RGW.
Here is an example: Here is an example:
``` ```yaml
rgw_instances: rgw_instances:
- instance_name: rgw1 - instance_name: rgw1
rgw_realm: usa rgw_realm: usa
@ -453,11 +460,11 @@ rgw_instances:
system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY system_secret_key: 5YZfaSUPqxSNIkZQQA3lBZ495hnIV6k2HAz710BY
``` ```
## Setting rgw_instances for a host in the master zone ### Setting rgw_instances for a host in the master zone
Here is an example of a host_vars for a host (ex: rgw-001 in the examples) containing 2 rgw_instances: Here is an example of a host_vars for a host (ex: rgw-001 in the examples) containing 2 rgw_instances:
``` ```yaml
rgw_zonemaster: true rgw_zonemaster: true
rgw_zonesecondary: false rgw_zonesecondary: false
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
@ -485,19 +492,19 @@ rgw_instances:
system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB system_secret_key: qqHCUtfdNnpHq3PZRHW5un9l0bEBM812Uhow0XfB
``` ```
This example starts up 2 rgws on host rgw-001. `rgw1` is configured to be in realm usa and `rgw2` is configured to be in realm france. This example starts up 2 rgws on host rgw-001. `rgw1` is configured to be in realm usa and `rgw2` is configured to be in realm france.
The variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/. The variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/.
## Setting rgw_instances for a host in a secondary zone ### Setting rgw_instances for a host in a secondary zone
To start up multiple rgws on a host that are in a secondary zone, `endpoint` must be added to rgw_instances. To start up multiple rgws on a host that are in a secondary zone, `endpoint` must be added to rgw_instances.
The value of `endpoint` should be the endpoint of an RGW in the master zone of the realm that is resolvable from the host.`rgw_pull_{proto, host, port}` are not necessary since `endpoint` is a combination of all three. The value of `endpoint` should be the endpoint of an RGW in the master zone of the realm that is resolvable from the host.`rgw_pull_{proto, host, port}` are not necessary since `endpoint` is a combination of all three.
Here is an example of a host_vars for a host containing 2 rgw_instances in a secondary zone: Here is an example of a host_vars for a host containing 2 rgw_instances in a secondary zone:
``` ```yaml
rgw_zonemaster: false rgw_zonemaster: false
rgw_zonesecondary: true rgw_zonesecondary: true
rgw_zonegroupmaster: true rgw_zonegroupmaster: true
@ -527,10 +534,10 @@ rgw_instances:
endpoint: https://rgw-001-hostname:8081 endpoint: https://rgw-001-hostname:8081
``` ```
This example starts up 2 rgws on the host that will pull the realm from the rgws on rgw-001 above. `rgw3` is pulling from the rgw endpoint in realm usa in the master zone example above (instance name rgw1). `rgw4` is pulling from the rgw endpoint in realm france in the master zone example above (instance name rgw2). This example starts up 2 rgws on the host that will pull the realm from the rgws on rgw-001 above. `rgw3` is pulling from the rgw endpoint in realm usa in the master zone example above (instance name rgw1). `rgw4` is pulling from the rgw endpoint in realm france in the master zone example above (instance name rgw2).
Just like the example on the master zone, the variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/. Just like the example on the master zone, the variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/.
## Conclusion ### Conclusion
`rgw_instances` can be used in host_vars for multisite deployments like scenarios 2 and 3 `rgw_instances` can be used in host_vars for multisite deployments like scenarios 2 and 3