Commit Graph

5567 Commits (37db27eefabefb1416dec6b816b7a26240230333)
 

Author SHA1 Message Date
Guillaume Abrioux b8db1166c5 nfs/rgw: support enforcing keys
if one sets `ceph_nfs_rgw_access_key` and/or `ceph_nfs_rgw_secret_key`,
the nfs/rgw user creation won't take those variables into account and it
will generate a user with automatically generated credentials.
It ends up with a mismatch between what will be set in ganesha.conf and
the created user.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2010754

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2021-10-26 16:39:58 +02:00
Per Abildgaard Toft c5e4851a3f shrink-osd: fix regression because of a wrong regex
968891f449 introduced a regression.
The regex is wrong because it doesn't allow to shrink osds with id
greater than 9

Fixes: #6950

Signed-off-by: Per Abildgaard Toft <per@minfejl.dk>
(cherry picked from commit 84118a3063)
2021-10-26 16:39:33 +02:00
Guillaume Abrioux 3f4abb09b4 shrink-osd: check osd id format
This adds a check early in order to ensure the format of osd ids passed
is correct.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2005734

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 968891f449)
2021-10-26 16:39:33 +02:00
Guillaume Abrioux 2c9fc7f517 rolling_update: modify default health_osd_check_*
let's do more retries with a shorter delay.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 50a21d695e)
2021-10-26 16:39:09 +02:00
Guillaume Abrioux a15448e5cc tests: add new scenario subset_update
new scenario in order to test the subset upgrade approach using tags.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit fb8a66149b)
2021-10-25 23:22:35 +02:00
Guillaume Abrioux 3dd96da652 rolling_update: fix pre and post osd upgrade play
when using --limit osds, the play before and after osd upgrade are
skipped because we use `hosts: "{{ mon_group_name | default('mons') }}[0]"`
using `hosts: "{{ osds_group_name | default('osds') }}" with
`delegate_to` to the first monitor addresses this issue.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit fc9f87c45f)
2021-10-25 23:22:35 +02:00
Guillaume Abrioux dc1a4c29ea update: support upgrading a subset of nodes
It can be useful in a large cluster deployment to split the upgrade and
only upgrade a group of nodes at a time.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2014304

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e5cf9db2b0)
2021-10-25 23:22:35 +02:00
Guillaume Abrioux c5a8343417 tests: ensure ca-certificates is up to date
otherwise the `rpm_key` module fails because it can't verify the
certificate.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2021-10-02 15:48:31 +02:00
Guillaume Abrioux a890d6a043 tests: remove all references to ceph_stable_release
this is legacy and not needed anymore.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f277a39dfe)
2021-10-02 15:48:31 +02:00
Seena Fallah 7495de3120 ceph-defaults: set ceph_stable_release default to the stable branch release
ceph_stable_release is a legacy from the time where a single branch of ceph-ansible supported more than one release of ceph

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit fb99626987)
2021-10-02 15:48:31 +02:00
Guillaume Abrioux a8026b3516 tests: set rgw_instances in collect-logs.yml
in order to gather rgw logs, we need rgw_instances to be set.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit c2e46fe5a5)
2021-09-30 17:54:31 +02:00
Guillaume Abrioux dda2368892 tests: update collect-logs.yml playbook
- change `ceph -s` output to json-pretty.
- gather rgw logs
- add `health detail` command

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit b2ccc7234a)
2021-09-30 17:49:29 +02:00
Guillaume Abrioux 6dd85f3344 tests: move collect-logs.yml to ceph-ansible repo
related ceph-build PR: ceph/ceph-build#1914

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 702564518b)
2021-09-29 16:41:40 +02:00
Seena Fallah 0a93de938b purge: add remove_docker tag
This can help to skip docker removal tasks

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit ff39c8d70b)
2021-09-14 20:50:06 +02:00
Seena Fallah 0ede37b2ec purge: add container_binary needed for zap osds
`container_binary` isn't set anymore in the purge osd play because of a
regression introduced by 60aa70a.
The CI didn't catch it because the play purging node-exporter sets this
variable for all nodes before we run the purge osd play.

This commit fixes this regression.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit a51ce767ca)
2021-09-09 14:40:53 +02:00
Dimitri Savineau ce80ed946c ceph-defaults: set quay.io as the default registry
Because the ceph container images are now only pushed to the quay.io
registry then this updates the default registry value.
The docker.io registry can still be used but doesn't receive updated
container images.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit e7b43c1fc6)
2021-09-09 13:48:14 +02:00
Seena Fallah e7b0af31c3 ceph-container-engine: allow override container_package_name and container_service_name
Only include specific variables when they are undefined

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 95bce32270)
2021-09-09 13:25:00 +02:00
Dimitri Savineau 0d670c7942 purge-dashboard: remove cid files
This adds the service cid file cleanup as supported in the classic purge
playbook since b9dd253

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1786691

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit cddc23f511)
2021-09-08 12:05:33 -04:00
Dimitri Savineau 3aba6c8e4e tests/rgw: use json format output for user info
If the radosgw user already exists then we need to have the output in json
format because we are expecting to load the output with json.loads()
Otherwise we have pytest failure like:

```console
self = <json.decoder.JSONDecoder object at 0x7fa2f00a5fd0>, s = '', idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.

        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.

        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit f2bd8ae70f)
2021-08-27 14:40:37 -04:00
Dimitri Savineau e1dd35f6d6 tests/rgw: add timeout 5s to radosgw-admin command
If the radosgw daemons aren't up and running correctly (like not registered
in the servicemap or the OSD are down) then the radosgw-admin will hang
forever.
Jenkins will kill the jobs after 3h but we don't want to wait until this global
timeout.
Adding the timeout 5 command to the radosgw-admin commands (which is already
present on other ceph calls) allows the job to fail earlier.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit f01ae82eec)
2021-08-27 14:40:37 -04:00
Dimitri Savineau 8f0b0d1285 container: explicitly pull monitoring images
We don't pull the monitoring container images (alertmanager, prometheus,
node-exporter and grafana) in a dedicated task like we're doing for the
ceph container image.
This means that the container image pull is done during the start of the
systemd service.
By doing this, pulling the image behind a proxy isn't working with podman.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1995574

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 5bb7240f87)
2021-08-23 16:21:19 -04:00
Guillaume Abrioux c7cd688f2e iscsi: don't set default value for trusted_ip_list
It restricts access to the iSCSI API.
It can be left empty if the API isn't going to be access from outside the
gateway node

Even though this seems to be a limited use case, it's better to leave it
empty by default than having a meaningless default value.

We could make this variable mandatory but that would be a breaking
change. Let's just add a logic in the template in order to set this
variable in the configuration file only if it was specified by users.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1994930

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
Co-authored-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 6802b8dddd)
2021-08-19 12:06:58 -04:00
Guillaume Abrioux 20583e83dd containers: introduce target systemd unit
This adds ceph-*.target systemd unit files support for containerized
deployments.
This also fixes a regression introduced by PR #6719 (rgw and nfs systemd
units not getting purged)

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1962748

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 09ef465f62)
2021-08-18 13:43:01 -04:00
Guillaume Abrioux 6ebbda8cef roles: remove leftover from pr #4319
pr #4319 introduced some uesless `become: true` on systemd tasks.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1db8fa8989)
2021-08-18 11:08:39 -04:00
Guillaume Abrioux 4f76dedd9e Vagrantfile: fallback on 'varant_variables.yml.sample'
When using a vagrant command from the root directory of the repo, it
throws an error if no 'vagrant_variables.yml' file is present.

```
Message: Errno::ENOENT: No such file or directory @ rb_sysopen - /home/guits/workspaces/ceph-ansible/vagrant_variables.yml
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 3d27f9e7dc)
2021-08-18 11:08:13 -04:00
Guillaume Abrioux 2d38d8266b update: gather facts only one time
this play doesn't need to gather facts from localhost

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit c14e9114ba)
2021-08-17 15:47:38 -04:00
Dimitri Savineau ffe01c7ff5 ceph-mon: do not log monitor keyring
We don't want to display the keyring in the ansible log.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit e44075abd6)
2021-08-12 13:31:12 +02:00
Guillaume Abrioux c55c87d3c5 common: do not log keyring secret
let's not display any keyring secret by default in ansible log.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1980744

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7511195738)
2021-08-11 17:01:22 -04:00
Benoît Knecht a8346af4f7 ceph-rgw: Work around Jinja2 < 2.8 missng eq test
EL7 ships with Jinja2 version 2.7, which is missing the `eq` test.

Work around this by using `match` instead.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2021-08-11 13:53:44 +02:00
Benoît Knecht 66426e1316 ceph-rgw: Set pg_num on RGW pool if required
If the `pg_num` value specified in `rgw_create_pools` is different from the
actual value in the cluster, apply it with `ceph osd pool set`.

This corresponds to the behavior of the `ceph_pool` module used in Ceph Ansible
5.0 onward.

Also avoid setting the pool application if it's already done.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2021-08-11 13:53:44 +02:00
Dimitri Savineau 712a9c4403 switch2container: fix mon quorum check
This was reverted by 7ddbe74

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1990733

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2021-08-10 10:03:02 +02:00
Dimitri Savineau 6897153ab7 ceph-dashboard: fix TLS cert openssl generation
With OpenSSL version prior 1.1.1 (like CentOS 7 with 1.0.2k), the -addext
doesn't exist.
As a solution, this uses the default openssl.cnf configuration file as a
template and add the subjectAltName in the v3_ca section. This temp openssl
configuration file is removed after the TLS certificate creation.
This patch also move the run_once statement at the block level.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1978869

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 5e0ace7e54)
2021-08-09 15:14:48 -04:00
Guillaume Abrioux 02750a94cc dashboard: subj_alt_names fact refactor
the current way the variable is built results in:

```
2021-08-03 04:18:23,020 - ceph.ceph - INFO - ok: [ceph-sangadi-4x-indpt6-node1-installer] => changed=false
  ansible_facts:
    subj_alt_names: |-
      subjectAltName=ceph-sangadi-4x-indpt6-node1-installer/subjectAltName=10.0.210.223/subjectAltName=ceph-sangadi-4x-indpt6-node1-installersubjectAltName=ceph-sangadi-4x-indpt6-node2/subjectAltName=10.0.210.252/subjectAltName=ceph-sangadi-4x-indpt6-node2/
```

which is incorrect.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1978869

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 6f1a0634f7)
2021-08-09 15:14:48 -04:00
VasishtaShastry 4ae9f321ac Fixes typo in rgw-add-users-buckets playbook
Signed-off-by: VasishtaShastry <vipin.indiasmg@gmail.com>
(cherry picked from commit 478d9fdcb6)
2021-08-09 14:31:55 -04:00
Dimitri Savineau e1e22933a7 add-osd: use container_exec_cmd fact from mon host
Because we're delegating the task to the first monitor node, we need to be
sure that the container_exec_cmd fact is the one from that node too otherwise
we could have a mismatch on the ceph-mon container name.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1990772

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2021-08-09 15:48:23 +02:00
Teoman ONAY 3d4e15cebf podman pids.max default value is 2048, docker's one is 4096 which are
sufficient for the default value (512) of rgw thread pool size.
But if its value is increased near to the pids-limit value,
it does not leave place for the other processes to spawn and run within
the container and the container crashes.

pids-limit set to unlimited regardless of the container engine.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1987041

Signed-off-by: Teoman ONAY <tonay@redhat.com>
(cherry picked from commit 9b5d97adb9)
2021-08-05 11:04:31 -04:00
Dimitri Savineau 03ed9e111c infra: use dedicated variables for balancer status
The balancer status is registered during the cephadm-adopt, rolling_update
and swith2container playbooks. But it is also used in the ceph-handler role
which is included in those playbooks too.
Even if the ceph-handler tasks are skipped for rolling_update and
switch2container, the balancer_status variable is erased with the skip task
result.

play1:
  register: balancer_status
play2:
  register: balancer_status <-- skipped
play3:
  when: (balancer_status.stdout | from_json)['active'] | bool

This leads to issue like:

The conditional check '(balancer_status.stdout | from_json)['active'] | bool'
failed. The error was: Unexpected templating type error occurred on
({% if (balancer_status.stdout | from_json)['active'] | bool %} True
{% else %} False {% endif %}): expected string or buffer.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1982054

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 386661699b)
2021-08-04 11:48:13 -04:00
Dimitri Savineau 1044940304 osds: use osd pool ls instead of osd dump command
The ceph osd pool ls detail command is a subset of the ceph osd dump
command.

$ ceph osd dump --format json|wc -c
10117
$ ceph osd pool ls detail --format json|wc -c
4740

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 06471a4b82)
2021-08-03 14:03:35 -04:00
Dimitri Savineau 5c6921e553 rolling_update: get ceph version when mons exist
eec3878 introduced a regression for upgrade scenarios where there's no
monitor nodes at all (like ganesha standalone, external clients, etc..)

TASK [get the ceph release being deployed] ************************************
task path: infrastructure-playbooks/rolling_update.yml:121
Thursday 29 July 2021  15:55:29 +0000 (0:00:00.484)       0:00:15.802 *********
fatal: [client0]: FAILED! =>
  msg: '''dict object'' has no attribute ''mons'''

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit e87a47cf0c)
2021-08-03 12:42:28 -04:00
Benoît Knecht bacaa654b1 infrastructure-playbooks: Get Ceph info in check mode
In the `set osd flags` block, run the Ceph commands that gather information
from the cluster (and don't make any changes to it) even when running in check
mode.

This allows the tasks that depend on the variables set by those tasks to
succeed in check mode.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit d7653dca95)
2021-08-02 15:54:34 +02:00
Benoît Knecht 9668137daf ceph-handler: Fix osd handler in check mode
Run the Ceph commands that only gather information (without making any changes
to the cluster) when running Ansible in check mode.

This allows the tasks that depend on the variables set by those tasks to
succeed in check mode.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 498acd7527)
2021-08-02 15:54:34 +02:00
Dimitri Savineau 1270d5964a library: remove unused module import
Move the import at the top of the file and remove unused module import.

- E402 module level import not at top of file
- F401 'xxxx' imported but unused

This also removes the '# noqa E402' statement from the code.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 2138a00a32)
2021-08-02 15:53:00 +02:00
Wong Hoi Sing Edison 5bf73d4766 library: flake8 ceph-ansible modules
This commit ensure all ceph-ansible modules pass flake8 properly.

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>
(cherry picked from commit beda1fe773)
2021-08-02 15:53:00 +02:00
Dimitri Savineau deac21c6bf ceph-defaults: add missing grafana dashboards
The radosgw-sync-overview and rbd-details grafana dashboars were missing
from the list.

Closes: #6758

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit f0ccf3ebf0)
2021-07-27 10:53:54 -04:00
Guillaume Abrioux 77171216fb update: check the ceph release
Check early which Ceph release is going to be deployed and fail if it
doesn't correspond to the ceph-ansible version being used.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1978643

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit eec38784ec)
2021-07-26 14:10:24 -04:00
Dimitri Savineau c39e7cb151 alertmanager: allow disable dashboard tls verify
When using self-signed/untrusted CA certificates, alertmanager displays
an error in logs. With this commit this should make those messages
disappear.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1936299

Co-authored-by: Guillaume Abrioux <gabrioux@redhat.com>

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 9f77b929d1)
2021-07-26 13:19:13 -04:00
Guillaume Abrioux 72bbc8285e dashboard: support dedicated network for the dashboard
This introduces a new variable `dashboard_network` in order to support
deploying the dashboard on a different subnet.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1927574

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f4f73b6197)
2021-07-26 13:19:03 -04:00
Dimitri Savineau 00e0ebc911 multisite: use node fqdn for endpoints when https
When the rgw_multisite_proto variable is set to https then we shoudn't use
the IP address in the zone endpoints list but the node FQDN to match the
TLS certificate CN.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1965504

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit ad05a08160)
2021-07-26 17:54:13 +02:00
Guillaume Abrioux 907fb08956 purge: support osd_auto_discovery
This adds a task that zaps by osd id so we can support the scenario
where osds were deployed with `osd_auto_discovery` is true.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1876860

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 4144074a50)
2021-07-26 17:53:06 +02:00
Guillaume Abrioux 3dcfbc2edf purge: merge playbooks
This refactor merges the two playbooks so we only have to maintain 1
playbook.
(Symlink the old purge-container-cluster.yml playbook for backward
 compatibility).

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 17cd83bf3a)
2021-07-26 17:53:06 +02:00