Notice there are 4 groups defined here: mgrs, rgws, osds, mons.
There is one host (mgr-000) in mgrs, 2 hosts (rgws-000, rgws-001) in rgws, 3 hosts (osd-000, osd-001, osd-002) in osds, and 2 hosts (mon-000, mon-001) in mons.
## group_vars
In the ceph-ansible tree there is a directory called `group_vars`. This directory has a collection of .yml files for variables set for each of the groups.
The rgw multisite specific variables are defined in `all.yml`. This file has variables that apply to all groups in the inventory.
When a variable, for example if `rgw_realm: usa`, is set in `group_vars/all.yml`, `usa` will be the value for `rgw_realm` for all of the rgws.
## host_vars
If you want to set any of the variables defined in `group_vars` for a specific host you have two options.
One option is to edit the line in the inventory file for the host you want to configure. In the above inventory each mon and rgw has a host specific variable for its address.
The preferred option is to create a directory called `host_vars` at the root of the ceph-ansible tree.
In `host_vars/` there can be files with the same name as the host (ex: osd-000, mgr-000, rgw-001) that set variables for each host.
The values for the variables set in `host_vars` have a higher precedence than the values in `group_var`.
Consider this the file `host_vars/rgw-000`:
```
rgw_realm: usa
rgw_zonegroup: alaska
rgw_zone: juneau
rgw_zonemaster: true
rgw_zonesecondary: false
system_access_key: alaskaaccesskey
system_secret_key: alaskasecretkey
```
Even if `rgw_realm` is set to `france` in `group_vars/all.yml`, `rgw_realm` will evaluate to `usa` for tasks run on `rgw-000`.
This is because Ansible gives higher precedence to the values set in `host_vars` over `group_vars`.
For more information on working with inventory in Ansible please visit: https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html.
# Brief Multisite Overview
## RGW Multisite terminology
If you are familiar with RGW multisite in detail, feel free to skip this section.
**Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`. Both values always need to be defined when running multisite.
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated.
3. Run the ceph-ansible playbook for the 1st cluster
## Configuring the Secondary Zone in a Separate Cluster
This will setup a realm, master zonegroup and master zone in the secondary Ceph cluster.
Since there is only 1 realm, 1 zonegroup, and 1 zone for all the rgw hosts, only `group_vars/all.yml` needs to be edited for mulitsite conifguration.
If there is one more that one rgw being deployed in this configuration, the rgw(s) will be added to the secondary zone.
**Note:** `rgw_zonemaster` should have the value of `false` and `rgw_zonesecondary` should be `true`
**Note:** The variables `rgw_pull_port`, `rgw_pull_proto`, `rgw_pullhost`, are joined together to make an endpoint string needed to create secondary zones. This endpoint is of one of the RGW endpoints in a master zone in the zonegroup and realm you want to create secondary zones in. This endpoint **must be resolvable** from the mons and rgws in the cluster the secondary zone(s) are being created in.
**Note:** `system_access_key`, and `system_secret_key` should match what you used in the Primary Cluster
5. Run the ceph-ansible playbook on your 2nd cluster
## Conclusion
You should now have a master zone on cluster0 and a secondary zone on cluster1 in an Active-Active mode.
# Deployment Scenario #2: Single Ceph Cluster with Multiple Realms
## Requirements
* Jewel or newer
## Configuring Multiple Realms in a Single Cluster
This configuration will a single Ceph cluster with multiple realms.
Each of the rgws in the inventory should have a file in `host_vars` where the realm, zone, and zonegroup can be set for the rgw along with other variables.
1. Generate System Access and System Secret Keys for each realm
As previously learned, all values set here will be set on all rgw hosts. `rgw_multisite` be set to `true` for all rgw hosts so multisite playbooks can run on all rgws.
**Note:** Since `rgw_realm`, `rgw_zonegroup`, and `rgw_zone` differ between files, a new realm, zonegroup, and master zone are created containing rgws-000 and rgws-001 respectively.
**Note:** `rgw_zonegroupmaster` is set to true in each of the files since it will be the only zonegroup in each realm.
**Note:** `rgw_zonemaster` should have the value of `true` and `rgw_zonesecondary` should be `false`.
**Note:** replace the `system_access_key` and `system_secret_key` values with the ones you generated.
5. Run the ceph-ansible playbook on your cluster
## Conclusion
The RGWs in the deployed cluster will be split up into 2 realms: `france` and `usa`. France has a zonegroup named `idf` and usa has one called `alaska`.
`Idf` has a master zone called `paris`. `Alaska` has a master zone called `juneau`.
# Deployment Scenario #3: Multiple Realms over Multiple Ceph Clusters
A multisite configuration consisting of multiple realms across multiple clusters can be configured by having files in `host_vars` for the rgws in each cluster similar to scenario #2.
The host_vars for the rgws in the second cluster would have `rgw_zonesecondary` set to true and the additional `rgw_pull` variables as seen in scenario #2
4. Run the ceph-ansible playbook on your master cluster.
5. Create & edit files in `host_vars/` for the entries in the `[rgws]` section of the inventory on the secondary cluster.
Here is an example of the file `host_vars/rgws-000` for the `rgws-000` entry in the `[rgws]` section of for the example ansible inventory if it was for a secondary cluster.
Here is an example of the file `host_vars/rgws-001` for the `rgws-001` entry in the `[rgws]` section of for the example ansible inventory if it was for a secondary cluster.
There will be 2 realms in this configuration, `france` and `usa`, with RGWs and RGW zones in both clusters. Cluster0 will has the master zones and Cluster1 has the secondary zones.
Data is realm france will be replicated over both clusters and remain isolated from rgws in realm usa and vice versa.
More than 1 RGW can be running on a single host. To configure multisite for a host with more than one rgw instance running on the host, `rgw_instances` must be configured.
Each item in `rgw_instances` (declared in a host_vars file) represents an RGW on that host. In each item is the multisite configuration for that RGW.
The variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/.
## Setting rgw_instances for a host in a secondary zone
To start up multiple rgws on a host that are in a secondary zone, `endpoint` must be added to rgw_instances.
The value of `endpoint` should be the endpoint of an RGW in the master zone of the realm that is resolvable from the host.`rgw_pull_{proto, host, port}` are not necessary since `endpoint` is a combination of all three.
Here is an example of a host_vars for a host containing 2 rgw_instances in a secondary zone:
This example starts up 2 rgws on host that will pull the realm from the rgws on rgw-000 above. `rgw2` is pulling from the rgw endpoint in realm usa in the master zone example above (instance name rgw0). `rgw3` is pulling from the rgw endpoint in realm france in the master zone example above (instance name rgw1).
Just like the example on the master zone, the variables `rgw_zonemaster`, `rgw_zonesecondary`, `rgw_zonegroupmaster`, `rgw_multisite_proto` cannot be set in an item of rgw_instances. All of these variables must be set in group_vars/ or host_vars/.
