Commit Graph

5552 Commits (ae452a86dc3ee8f79a548c08cc2a9121d307403e)
 

Author SHA1 Message Date
Guillaume Abrioux 968dd3830a ceph_key: support using different keyring
Currently the `ceph_key` module doesn't support using a different
keyring than `client.admin`.
This commit adds the possibility to use a different keyring.

Usage:
```
      ceph_key:
        name: "client.rgw.myrgw-node.rgw123"
        cluster: "ceph"
        user: "client.bootstrap-rgw"
        user_key: /var/lib/ceph/bootstrap-rgw/ceph.keyring
        dest: "/var/lib/ceph/radosgw/ceph-rgw.myrgw-node.rgw123/keyring"
        caps:
          osd: 'allow rwx'
          mon: 'allow rw'
          import_key: False
        owner: "ceph"
        group: "ceph"
        mode: "0400"
```

Where:
`user` corresponds to `-n (--name)`
`user_key` corresponds to `-k (--keyring)`

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 12e6260266)
2020-10-06 10:31:34 +02:00
Guillaume Abrioux 2a3b563c7e rgw: fix multi instances scaleout in baremetal
When rgw and osd are collocated, the current workflow prevents from
scaling out the radosgw_num_instances parameter when rerunning the
playbook in baremetal deployments.

When ceph-osd notifies handlers, it means rgw handlers are triggered
too. The issue with this is that they are triggered before the role
ceph-rgw is run.
In the case a scaleout operation is expected on `radosgw_num_instances`
it causes an issue because keyrings haven't been created yet so the new
instances won't start.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1881313

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit a802fa2810)
2020-10-06 10:31:34 +02:00
Guillaume Abrioux 5db74194b2 tests: reboot and test idempotency on collocation
test reboot and idempotency on collocation scenario.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f83f798206)
2020-10-06 10:31:34 +02:00
Dimitri Savineau a5f19b7864 ceph_key: remove backward compatibility
It's time to remove this backward compatibility. Users had enough time
to convert their openstack_keys and key values.
We now fail in ceph-validate if the caps key isn't set.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit c960362639)
2020-10-06 10:09:16 +02:00
Guillaume Abrioux a40ea7e712 infrastructure-playbooks: drop add-osd playbook
This playbook isn't needed anymore, we can achieve this operation by
running main playbook with `--limit` option.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 20718582da)
2020-10-06 10:03:40 +02:00
Guillaume Abrioux 32be163360 ceph-osd: refact `docker_exec_start_osd`
This commit drops nested jinja construction in this set_fact task.
It also rename it to `container_exec_start_osd`

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit ff95fa9c32)
2020-10-06 09:54:50 +02:00
Guillaume Abrioux 88a4d39978 flake8: fix pep8 syntax on tests/functional/tests/
tests/conftest.py and tests present in tests/functional/tests/ has been
missed from previous commit

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 8596f1d52c)
2020-10-06 08:54:43 +02:00
Guillaume Abrioux df54883fdf flake8: fix all tests/library/*.py files
This commit modifies all *.py files in ./tests/library/ so flake8
passes.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e49a5241f0)
(cherry picked from commit fb98f436848189e26480697b23f45b28f51a6ccd)
2020-10-02 09:00:56 -04:00
Guillaume Abrioux e5df63f34f tests: refact flake8 workflow
drop ricardochaves/python-lint action and use `run` steps instead.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f2d3432cad)
(cherry picked from commit 7378909c7b8d6a6285f14ea6c7c8987fae73939d)
2020-10-02 09:00:56 -04:00
Guillaume Abrioux 80879df44d defaults: change defaults value
this commit changes defaults value in default pool definitions.

there's no need to define `pg_num`, `pgp_num`, `size` and `min_size`,
`ceph_pool` module will use the current default if needed.

This also drops the 3 following `set_fact` in `ceph-facts`:

- osd_pool_default_pg_num,
- osd_pool_default_pgp_num,
- osd_pool_default_size_num

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit c101cb3931)
2020-10-02 09:32:53 +02:00
Guillaume Abrioux b01f1dc5c9 ceph_pool: update tests
update test_ceph_pool.py due to recent refact

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 8f5db079ae)
2020-10-02 09:32:53 +02:00
Guillaume Abrioux 28b475cd72 ceph_pool: improve pg_autoscaler support
This commit modifies how the `pg_autoscaler` feature is handled by the
ceph_pool module.

1/ If a pool has the pg_autoscaler feature enabled, we shouldn't try to
update pg/pgp.
2/ Make it more readable

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 740df379b7)
2020-10-02 09:32:53 +02:00
Guillaume Abrioux 9fe7ab6bc6 ceph_pool: pep8
Adopt pep8 syntax in ceph_pool module

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 787878f0c3)
2020-10-02 09:32:53 +02:00
Guillaume Abrioux cb44f655fc ceph_pool: refact module
remove complexity about current defaults in running cluster

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 29fc115f4a)
2020-10-02 09:32:53 +02:00
Guillaume Abrioux 7fc78939b1 library: remove legacy file
This file is a leftover and should have been removed when we dropped the
validate module.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 8603cba9ab)
2020-10-01 23:31:11 +02:00
Guillaume Abrioux ed6ae6815d tests: add github workflows
Add github workflow. Especially for flake8 for now.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 1ee626a1b3)
2020-10-01 09:26:13 -04:00
Wong Hoi Sing Edison 32a2f04cbc library: flake8 ceph-ansible modules
This commit ensure all ceph-ansible modules pass flake8 properly.

Signed-off-by: Wong Hoi Sing Edison <hswong3i@gmail.com>
Co-authored-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 268a39ca0e)
2020-10-01 09:26:13 -04:00
Seena Fallah 10fc2d1d92 ceph-facts: add get default crush rule from running monitor
In case of deploying new monitor node to an existing cluster,
osd_pool_default_crush_rule should be taken from running monitor because
ceph-osd role won't be run and the new monitor will have different
osd_pool_default_crush_role from other monitors.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit ff9f4d138f)
2020-09-29 12:15:09 -04:00
Guillaume Abrioux 72c73ac2bc fs2bs: support `osd_auto_discovery` scenario
This commit adds the `osd_auto_discovery` scenario support in the
filestore-to-bluestore playbook.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1881523

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
Co-authored-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 8b1eeef18a)
2020-09-29 16:28:43 +02:00
Ali Maredia 9f58d4a3d1 rgw multisite: check connection for realm endpoint
This commit adds connection checks before realm pulls
Curls are performed on the endpoint being pulled from
the mons and the rgws

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1731158

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 902575369c)
2020-09-29 09:24:59 -04:00
Tyler Bishop e3284b20ac facts: support device aliases for (dedicated|bluestore_wal)_devices
Just likve `devices`, this commit adds the support for linux device aliases for
`dedicated_devices` and `bluestore_wal_devices`.

Signed-off-by: Tyler Bishop <tbishop@liquidweb.com>
(cherry picked from commit ee4b8804ae)
2020-09-29 09:24:17 -04:00
Dimitri Savineau 6294244c4f ceph-handler: set handler on xxx_stat result
In non containerized deployment we check if the service is running
via the socket file presence.
This is done via the xxx_socket_stat variable that check the file
socket in the /var/run/ceph/ directory.
In some scenarios, we could have the socket file still present in
that directory but not used by any process.
That's why we have the xxx_stat variable which clean those leftovers.

The problem here is that we're set the variable for the handlers status
(like handler_mon_status) based on xxx_socket_stat instead of xxx_stat.
That means we will trigger the handlers if there's an old socket file
present on the system without any process associated.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1866834

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 733596582d)
2020-09-29 09:23:45 -04:00
Dimitri Savineau 77c5115ad2 ceph-iscsi: create pool once from monitor
af9f6684 introduced a regression on the ceph iscsi pool creation
because it was delegated to the first monitor node before that change.
This patch restores the initial worflow.
When the iscsi node doesn't have the admin keyring then the pool
creation fails.
This commit also ensures that the pool creation is only executed once
when having multiple iscsi nodes.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 501b8e0fd3)
2020-09-29 09:23:14 -04:00
Benoît Knecht ab458a9592 library: Fix new-style modules check mode
Running the `ceph_crush.py`, `ceph_key.py` or `ceph_volume.py` modules in check
mode resulted in the following error:

```
New-style module did not handle its own exit
```

This was due to the fact that they simply returned a `dict` in that case,
instead of calling `module.exit_json()`.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 85dd405814)
2020-09-28 20:41:51 -04:00
Dimitri Savineau babc8a05fd ceph-mds: remove unused block condition
Since af9f6684 the cephfs pool(s) creation don't use the fs_pools_created
variable anymore because the ceph_pool module is idempotent.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 1db4dc807c)
2020-09-28 20:39:25 -04:00
Seena Fallah 9b0f45431d ceph-facts: check for mon socket in its own host
delegate to its own host after checking mon socket to findout if mon socket is in-use or not.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 69f7e35382)
2020-09-28 20:38:52 -04:00
Guillaume Abrioux 5538dd8b3b Revert "ceph-rgw: remove ceph_pool state and default value"
This reverts commit ba3512a8fc.

(cherry picked from commit bf7b044c9a)
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-09-28 16:47:45 -04:00
Raffael a8f6324b48 doc: Update methods.rst
Based on the discussion in issue #5392 I added now this paragraph to this page.

Signed-off-by: Raffael Luthiger <r.luthiger@huanga.com>
(cherry picked from commit 9af86d250b)
2020-09-25 15:19:47 -04:00
Benoît Knecht cfeb6b8403 README-MULTISITE: Fix syntax issues from markdownlint
This commit makes the following changes:

- Remove trailing whitespace;
- Use consistent header levels;
- Fix code blocks;
- Remove hard tabs;
- Fix ordered lists;
- Fix bare URLs;
- Use markdown list of sections.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 2c244425ec)
2020-09-25 14:38:58 -04:00
Dimitri Savineau 7e2e11320d ceph-rgw: remove ceph_pool state and default value
Since the state is now optional and default values are handled in the
ceph_pool module itself.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit ba3512a8fc)
2020-09-25 14:05:34 -04:00
Dimitri Savineau 1fa1b5b751 ceph_pool: add idempotency to absent state
When using the "absent" state on a non existing pool then the ceph_pool
module will fail and return a python traceback.

Instead we should check if the pool exit or not and execute the pool
deletion according to the result.
The state changed is now set when the pool is actually deleted.

This also disable add_file_common_args because we don't manipulate
files with this module.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 047a3e2653)
2020-09-25 13:55:20 -04:00
Dimitri Savineau 8d49d97582 ceph-config: remove ceph_release from ceph.conf.j2
We don't use ceph_release variable in the ceph.conf jinja template.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 62bd41f0d4)
2020-09-25 13:37:36 -04:00
Guillaume Abrioux 9d04b8ca8b ansible.cfg: remove cfg file in infrastructure-playbooks
There's no need ot have a copy of this file in infrastructure-playbooks
directory.
playbooks in that directory can be run from the root dir of
ceph-ansible.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f906caa6da)
2020-09-25 11:12:39 -04:00
Guillaume Abrioux 113eadad72 ansible.cfg: set force_valid_group_names param
As of 2.10, group names containing a dash are invalid.
However, setting this option makes it still possible to use a dash in
group names and prevent this warning to show up.
It might need to be definitely addressed in a future ansible release.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1880476

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 6938ed1302)
2020-09-25 11:12:39 -04:00
Kefu Chai 354abf903a docs: update URLs to point to the RTD links
Fixes #5798
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit f3a78371d9)
2020-09-25 10:47:31 -04:00
Dmitriy Rabotyagov f996a232d7 Remove libjemalloc1 installation task
libjemalloc1 package is not required neither for ganesha dependency nor
for the package build process. So this task can be simply dropped.

Signed-off-by: Dmitriy Rabotyagov <noonedeadpunk@ya.ru>
(cherry picked from commit 297532ca41)
2020-09-24 14:02:27 -04:00
Dimitri Savineau 17f2dd8f86 library/ceph_key: set no_log on secret
We don't need to show this information during the module execution.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit a3f4e2b4d1)
2020-09-24 09:39:47 -04:00
Guillaume Abrioux b714e04d91 facts: refact `ceph_uid` fact
There's no need to set this fact with a `set_fact`
We can achieve this in `ceph-defaults`

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit bcc673f66c)
2020-09-21 12:26:19 -04:00
Dimitri Savineau c0654fa37a container: quote registry password
When using a quote in the registry password then we have the following
error:

The error was: ValueError: No closing quotation

To fix this we need to use the quote filter.

Close: https://bugzilla.redhat.com/show_bug.cgi?id=1880252

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 6dcfdf17d4)
2020-09-18 14:25:44 -04:00
Guillaume Abrioux 7e29b22c76 facts: fix 'set_fact rgw_instances with rgw multisite'
the current condition doesn't work, as soon as the first iteration is
done the condition makes next iterations skip since `rgw_instances` got
set with the first iteration.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1859872

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit ff19c1d851)
2020-09-18 10:35:20 -04:00
Dimitri Savineau fb43400903 ceph-infra: include iscsi nodes for logrotate
The iscsi nodes aren't included in the logrotate condition.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 85643edfe3)
2020-09-17 14:49:35 -04:00
Guillaume Abrioux 3e53d2a811 infra: support log rotation for tcmu-runner
This commit adds the log rotation support for tcmu-runner.

ceph-container related PR: ceph/ceph-container#1726

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1873915

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f576c02ff7)
2020-09-16 21:35:48 -04:00
Dimitri Savineau 8f175eb60c container: add optional http(s) proxy option
When using a http(s) proxy with either docker or podman we can rely on
the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables.
But with ansible, even if those variables are defined in a source file
then they aren't loaded during the container pull/login tasks.
This implements the http(s) proxy support with docker/podman.
Both implementations are different:
  1/ docker doesn't rely en the environment variables with the CLI.
Thos are needed by the docker daemon via systemd.
  2/ podman uses the environment variables so we need to add them to
the login/pull tasks.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1876692

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit bda3581294)
2020-09-16 11:32:14 -04:00
Dimitri Savineau 95a073cb3b ceph-prometheus: update pool stat counter
Since [1] The bytes_used pool counter in prometheus has been renamed
to stored.

Closes: #5781

[1] https://github.com/ceph/ceph/commit/71fe9149

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit e54b924eaf)
2020-09-16 10:08:46 -04:00
Dimitri Savineau aaf1139242 switch2container: chown symlink for devices
If the OSD directory is using symlinks for referencing devices (like
block, db, wal for bluestore and journal for filestore) then the chown
command could fail to change the owner:group on some system.

$ ls -hl /var/lib/ceph/osd/ceph-0/
total 28K
lrwxrwxrwx 1 ceph ceph 92 Sep 15 01:53 block -> /dev/ceph-45113532-95ca-471b-bd75-51de46f1339c/osd-data-570a1aee-60c0-44c9-8036-ffed7d67a4e6
-rw------- 1 ceph ceph 37 Sep 15 01:53 ceph_fsid
-rw------- 1 ceph ceph 37 Sep 15 01:53 fsid
-rw------- 1 ceph ceph 55 Sep 15 01:53 keyring
-rw------- 1 ceph ceph  6 Sep 15 01:53 ready
-rw------- 1 ceph ceph  3 Sep 15 02:00 require_osd_release
-rw------- 1 ceph ceph 10 Sep 15 01:53 type
-rw------- 1 ceph ceph  2 Sep 15 01:53 whoami
$ find /var/lib/ceph/osd/ceph-0 -not -user 167 -execdir chown 167:167 {} +
chown: cannot dereference './block': Permission denied
$ find /var/lib/ceph/osd/ceph-0 -not -user 167
/var/lib/ceph/osd/ceph-0/block

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit da4280e243)
2020-09-15 15:30:12 -04:00
Dimitri Savineau 8757fdfb4a switch2container: remove deb systemd units
When running the switch2container playbook on a Debian based system
then the systemd unit path isn't the same than Red Hat based system.
Because the systemd unit files aren't removed then the new container
systemd unit isn't take in count.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit c1af69a7e7)
2020-09-15 15:30:12 -04:00
Guillaume Abrioux edcdbe5601 purge: remove potential socket leftover
This commit ensure we remove any socket left by ceph and the
`ceph-osd-run.sh` script.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1861755

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5e91e0f3e2)
2020-09-14 16:50:49 -04:00
Guillaume Abrioux 4b55dcdc0d tests: do not run node_exporter test on clients
We need to skip these tests on client nodes since we don't deploy
node_exporter on them anymore

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5650a6d7d0)
2020-09-14 16:13:11 -04:00
Dimitri Savineau 25ba7f5314 node-exporter: exclude client nodes
We don't need to install node-exporter on client node because there's
no ceph services running on them.
This also makes sure we use the group name variables in the prometheus
service template instead of hardcoding the values.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit b105549ed8)
2020-09-14 16:13:11 -04:00
Dimitri Savineau 24698e7f4b dashboard: use run_once at block level
Instead of using run_once: true on each tasks in a block section, we
can use the run_once statement at the block level.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
(cherry picked from commit 2c4af70abd)
2020-09-14 15:54:22 -04:00