Commit Graph

5724 Commits (ac0342b72e045b96a31bed23c6baeceeecca7f23)
 

Author SHA1 Message Date
Guillaume Abrioux 5c4ae5356d osd: add tag on 'wait for all osd to be up' task
This allows skipping this task if really desired.
Use it carefully. Use it at your own risk.

Fixes: #6073

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-12-01 11:00:25 +01:00
Dimitri Savineau 1831b4955f ceph-client: use group_by instead of add_host
Instead of iterate over all client nodes with a loop sequentially, we
can use the group_by ansible buildin.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-12-01 10:58:48 +01:00
Dimitri Savineau c3ed124d31 library: add cephadm_bootstrap module
This adds cephadm_bootstrap ansible module for replacing the command module
usage with the cephadm bootstrap command.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-12-01 10:30:05 +01:00
Dimitri Savineau 5da593604a library: add ceph_osd_flag module
This adds ceph_osd_flag ansible module for replacing the command module
usage with the ceph osd set/unset commands.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-12-01 10:29:11 +01:00
Guillaume Abrioux d40dd764e0 iscsigw: remove `--cap-add=all` from `podman run` cmd
As of podman `2.0.5`, `--cap-add` and `--privileged` are exclusive
options.

```
Nov 30 13:56:30 magna089 podman[171677]: Error: invalid config provided: CapAdd and privileged are mutually exclusive options
```

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1902149

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-30 12:24:11 -05:00
Guillaume Abrioux c68b124ba8 container: remove `--ignore` from `podman rm` command
As of podman 2.0.5, `--ignore` param conflicts with `--storage`.
```
Nov 30 13:53:10 magna089 podman[164443]: Error: --storage conflicts with --volumes, --all, --latest, --ignore and --cidfile
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-30 12:24:11 -05:00
Dimitri Savineau 0b5b1de963 library: add ceph_osd module
This adds ceph_osd ansible module for replacing the command module
usage with the ceph osd destroy/down/in/out/purge/rm commands.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-30 16:53:45 +01:00
Dimitri Savineau eaf0ebfc85 library: add ceph_mgr_module module
This adds ceph_mgr_module ansible module for replacing the command module
usage with the ceph mgr module enable/disable commands.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-30 16:52:02 +01:00
Dimitri Savineau c610a1b411 tests: use ca_test_common with ceph_volume_simple
To avoid duplicate code.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-30 10:05:45 +01:00
Dimitri Savineau d76fbb38d5 improve plugins/filter testing
- The plugins/filter directory wasn't present in the flake8 workflow
configuration.
- Fix the flake8 syntax.
- Add the directory to PYTHONPATH environment variable for pytest
to avoid importing the plugin filter via sys.
- Add unittest on missing netaddr module import.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-30 10:05:01 +01:00
Guillaume Abrioux 0b05620597 switch2containers: do not stop ceph.target in osd play
`ceph.target` should be disabled only. Otherwise, in collocation
scenario you stop other collocated services in the OSD play which isn't
what we want to do. Each daemon has its corresponding play for managing
the transition to container.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1901865

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-27 09:34:01 -05:00
Dimitri Savineau eb452d35bc alertmanager/prometheus: fix owner/group
Set the owner/group on alertmanager and prometheus directories and
files to nobody and nogroup (uid and gid 65534) to avoid permission
issues.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1901543

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-27 14:02:42 +01:00
Dimitri Savineau 239a14279d tests: add module_utils directory to flake8/pytest
This adds the module_utils and associated test directory into the flake8
and pytest workflow configuration.
It also moves the ca_common module_utils test file from tests/library to
it's own directory tests/module_utils.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-27 06:35:56 +01:00
Guillaume Abrioux 67467fd87c module_utils: refactor
- update `generate_ceph_cmd()` so `user_key` is automatically built from
`cluster` and `user` params.

- update and add testing.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-26 11:21:09 +01:00
Guillaume Abrioux 926a1b2f7b library: use exit_module from ca_common
pr #6054 got merged but could have been updated to use `ca_common` for
`exit_module()`

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-26 09:13:31 +01:00
Dimitri Savineau 0078230f1d rhcs: default to containerized deployment
Starting RHCS 5, only containerized deployment is available.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-25 10:46:16 +01:00
Dimitri Savineau 0cc34fa54e library: remove useless condition in ceph_volume
Since the action values are already defined as a list of choices in
ansible then we will never enter into this condition.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-25 10:44:48 +01:00
Dimitri Savineau 3baac5ad5b library: add ceph_volume_simple_{activate,scan}
This adds ceph_volume_simple_{activate,scan} ansible modules for replacing
the command module usage with the ceph-volume simple activate/scan commands.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-25 10:09:42 +01:00
Guillaume Abrioux 970c6a4ee6 mon: refact initial keyring generation
adding monitor is no longer possible because we generate a new mon
keyring each time the playbook is run.

Fixes: #5864

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-25 09:34:44 +01:00
Guillaume Abrioux 5ff2ca270f mon: replace `command` task by `copy`
We can achieve this task using `copy` module.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-25 09:34:44 +01:00
Dimitri Savineau 40a87c4b92 ceph-iscsi: set the pool name in the config file
When using a custom pool for iSCSI gateway then we need to set the pool
name in the configuration otherwise the default rbd pool name will be
used.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-24 20:41:54 +01:00
Guillaume Abrioux 6ba4c8c70b tests: update pytest workflow
This sets the `PYTHONPATH` environment variable so we can run pytest
with the recent module refactor (module_utils introduction)

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-24 10:38:28 +01:00
Guillaume Abrioux 7c72632418 library: import ca_test_common in test_ceph_key
since we added `ca_test_common.py` let's use it in `test_ceph_key.py`

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-24 10:38:28 +01:00
Guillaume Abrioux 81233dd963 rgw: call `ceph_ec_profile` when needed
Let's replace `command` tasks with `ceph_ec_profile` calls

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-24 10:38:28 +01:00
Guillaume Abrioux 497d27dcf1 library: add ec_profile module
This commit adds a new module `ceph_ec_profile` to manage erasure code
profiles.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-24 10:38:28 +01:00
Guillaume Abrioux d7fd46842d library: add module_utils namespace
This adds a new `module_utils` namespace in order to avoid defining same
functions in each module.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-24 10:38:28 +01:00
Guillaume Abrioux 94c37b9de8 tests: use github workflow for nbsp char check
Let's use a github workflow instead of travis for this.

With this commit we can get rid of Travis.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 195d88fcda lint: ignore 302,303,505 errors
ignore 302,303 and 505 errors

[302] Using command rather than an argument to e.g. file
[303] Using command rather than module
[505] referenced files must exist

they aren't relevant on these tasks.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux c948b668eb lint: do not use 'local_action'
Fix ansible-lint 504 error:

[504] Do not use 'local_action', use 'delegate_to: localhost'

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux dfc7e6e4bd lint: trailing whitespace
Fix ansible-lint 201 error:

[201] Trailing whitespace

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 97dd9218dd lint: all tasks should be named
Fix ansible-lint 502 error:

[502] All tasks should be named

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 11b4bf5083 lint: use shell only when shell functionality is required
Fix ansible-lint 305 error:

[305] Use shell only when shell functionality is required

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 2011e4dbc8 lint: don't compare to literal true/false
Fix ansible lint 601 error:

[601] Don't compare to literal True/False

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 9fba6eecfa lint: variables should have spaces before and after
Fix ansible lint 206 error:

[206] Variables should have spaces before and after: {{ var_name }}

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 5450de58b3 lint: commands should not change things
Fix ansible lint 301 error:

[301] Commands should not change things if nothing needs doing

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 1879c26eb9 lint: set pipefail on shell tasks
Fix ansible lint 306 error:

[306] Shells that use pipes should set the pipefail option

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux d4400f911a tests: use github workflow for ansible-lint
let's use github workflow instead of travis.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-23 08:33:47 +01:00
Guillaume Abrioux 873fc8ec0f osd: ensure /var/lib/ceph/osd/{cluster}-{id} is present
This commit ensures that the `/var/lib/ceph/osd/{{ cluster }}-{{ osd_id }}` is
present before starting OSDs.

This is needed specificly when redeploying an OSD in case of OS upgrade
failure.
Since ceph data are still present on its devices then the node can be
redeployed, however those directories aren't present since they are
initially created by ceph-volume. We could recreate them manually but
for better user experience we can ask ceph-ansible to recreate them.

NOTE:
this only works for OSDs that were deployed with ceph-volume.
ceph-disk deployed OSDs would have to get those directories recreated
manually.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1898486

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-19 09:20:28 +01:00
Dimitri Savineau e150df789e ceph-facts: fix read osd pool default crush fact
We don't need to use run_once on that task when having running monitors
otherwise the read task could be skip and the set task will fail.

The conditional check 'crush_rule_variable.rc == 0' failed. The error
was: error while evaluating conditional (crush_rule_variable.rc == 0):
'dict object' has no attribute 'rc'

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1898856

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-18 12:55:43 -05:00
Dimitri Savineau 3e79f0322a tests: use github workflow for pytest
Move the pytest testing from TravisCI to Github workflow.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-17 17:52:03 +01:00
Guillaume Abrioux f5ba6d9b01 containers: modify bindmount option
This commit changes the bind mount option for the mount point
`/var/lib/ceph` in the systemd template for mon and mgr containers. This
is needed in case of collocating mon/mgr with osds using dmcrypt
scenario.
Once mon/mgr got converted to containers, the dmcrypt layer sub mount is
still seen in `/var/lib/ceph`. For some reason it makes the
corresponding devices busy so any other container can't open/close it.
As a result, it prevents osds from starting properly.

Since it only happens on the nodes converted before the OSD play, the idea is
to bind mount `/var/lib/ceph` on mon and mgr with the `rshared` option
so once the sub mount is unmounted, it is propagated inside the
container so it doesn't see that mount point.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1896392

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-17 09:19:23 -05:00
Dimitri Savineau 35ed9977aa switch2container: chown symlink in mon/mgr plays
fa2bb3a only fix the symlink owner/group issue in the OSD play. If the
OSDs are collocated with other services like MONs and MGRs then the
chown command will fail.

$ find /var/lib/ceph/osd/ceph-0 -not -user 167 -execdir chown 167:167 {} +
chown: cannot dereference './block': Permission denied

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1896448

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-16 13:40:57 -05:00
Guillaume Abrioux 5ba7824c55 container: force rm --storage on ExecStartPre
This is a workaround to avoid error like following:
```
Error: error creating container storage: the container name "ceph-mgr-magna022" is already in use by "4a5f674e113f837a0cc561dea5d2cd55d16ca159a647b7794ab06c4c276ef701"
```

that doesn't seem to be 100% reproducible but it shows up after a
reboot. The only workaround we came up with at the moment is to run
`podman rm --storage <container>` before starting it.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1887716

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-16 10:38:40 -05:00
Benoît Knecht c5f7343a2f ceph-facts: Fix osd_pool_default_crush_rule fact
The `osd_pool_default_crush_rule` is set based on `crush_rule_variable`, which
is the output of a `grep` command.

However, two consecutive tasks can set that variable, and if the second task is
skipped, it still overwrites the `crush_rule_variable`, leading the
`osd_pool_default_crush_rule` to be set to `ceph_osd_pool_default_crush_rule`
instead of the output of the first task.

This commit ensures that the fact is set right after the `crush_rule_variable`
is assigned, before it can be overwritten.

Closes #5912

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2020-11-13 09:36:49 +01:00
Gaudenz Steinlin 4d1fdd2b05 config: Always use osd_memory_target if set
The osd_memory_target variable was only used if it was higher than the
calculated value based on the number of OSDs. This is changed to always
use the value if it is set in the configuration. This allows this value
to be intentionally set lower so that it does not have to be changed
when more OSDs are added later.

Signed-off-by: Gaudenz Steinlin <gaudenz.steinlin@cloudscale.ch>
2020-11-13 09:13:58 +01:00
Guillaume Abrioux 2fa17520c4 main: followup on pr 6012
This tag can be set at the play level.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-12 15:31:31 -05:00
Dimitri Savineau fa2bb3af86 switch2container: disable ceph-osd enabled-runtime
When deploying the ceph OSD via the packages then the ceph-osd@.service
unit is configured as enabled-runtime.
This means that each ceph-osd service will inherit from that state.
The enabled-runtime systemd state doesn't survive after a reboot.
For non containerized deployment the OSD are still starting after a
reboot because there's the ceph-volume@.service and/or ceph-osd.target
units that are doing the job.

$ systemctl list-unit-files|egrep '^ceph-(volume|osd)'|column -t
ceph-osd@.service     enabled-runtime
ceph-volume@.service  enabled
ceph-osd.target       enabled

When switching to containerized deployment we are stopping/disabling
ceph-osd@XX.servive, ceph-volume and ceph.target and then removing the
systemd unit files.
But the new systemd units for containerized ceph-osd service will still
inherit from ceph-osd@.service unit file.

As a consequence, if an OSD host is rebooting after the playbook execution
then the ceph-osd service won't come back because they aren't enabled at
boot.

This patch also adds a reboot and testinfra run after running the switch
to container playbook.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1881288

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-12 20:05:39 +01:00
Francesco Pantano fafd5f871a Add ceph_client tag to execute or skip the playbook
There are some use cases where there's a need to skip the execution
of the ceph-ansible client role even though the client section of the
inventory isn't empty.
This can happen in contexts where the services are colocated or when
a all-in-one deployment is performed.
The purpose of this change is adding a 'ceph_client' tag to avoid
altering the ceph-ansible execution flow but at the same time be able
to include or exclude a set of tasks using this tag.

Signed-off-by: Francesco Pantano <fpantano@redhat.com>
2020-11-12 13:44:49 +01:00
Dimitri Savineau 3e49258377 rolling_update: always run cv simple scan/activate
There's no need to use a condition on the ceph release for the
ceph-volume simple commands.

Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>
2020-11-10 14:01:10 +01:00
Guillaume Abrioux 5cadfea42e dashboard: change dashboard_grafana_api_no_ssl_verify default value
This sets the `dashboard_grafana_api_no_ssl_verify` default value
according to the length of `dashboard_crt` and `dashboard_key`.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
2020-11-04 10:00:48 +01:00