containers: modify bindmount option

This commit changes the bind mount option for the mount point
`/var/lib/ceph` in the systemd template for mon and mgr containers. This
is needed in case of collocating mon/mgr with osds using dmcrypt
scenario.
Once mon/mgr got converted to containers, the dmcrypt layer sub mount is
still seen in `/var/lib/ceph`. For some reason it makes the
corresponding devices busy so any other container can't open/close it.
As a result, it prevents osds from starting properly.

Since it only happens on the nodes converted before the OSD play, the idea is
to bind mount `/var/lib/ceph` on mon and mgr with the `rshared` option
so once the sub mount is unmounted, it is propagated inside the
container so it doesn't see that mount point.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1896392

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit f5ba6d9b01)

roles/ceph-mgr/templates/ceph-mgr.service.j2

@@ -22,7 +22,7 @@ ExecStart=/usr/bin/{{ container_binary }} run --rm --net=host \
 {% endif %}
   --memory={{ ceph_mgr_docker_memory_limit }} \
   --cpus={{ ceph_mgr_docker_cpu_limit }} \
-  -v /var/lib/ceph:/var/lib/ceph:z \
+  -v /var/lib/ceph:/var/lib/ceph:z,rshared \
   -v /etc/ceph:/etc/ceph:z \
   -v /var/run/ceph:/var/run/ceph:z \
   -v /etc/localtime:/etc/localtime:ro \

roles/ceph-mon/templates/ceph-mon.service.j2

@@ -21,7 +21,7 @@ ExecStart=/usr/bin/{{ container_binary }} run --rm --name ceph-mon-%i \
 {% endif %}
   --memory={{ ceph_mon_docker_memory_limit }} \
   --cpus={{ ceph_mon_docker_cpu_limit }} \
-  -v /var/lib/ceph:/var/lib/ceph:z \
+  -v /var/lib/ceph:/var/lib/ceph:z,rshared \
   -v /etc/ceph:/etc/ceph:z \
   -v /var/run/ceph:/var/run/ceph:z \
   -v /etc/localtime:/etc/localtime:ro \