2014-03-04 02:08:51 +08:00
---
# This playbook does a rolling update for all the Ceph services
# Change the value of serial: to adjust the number of server to be updated.
#
2014-07-08 21:39:42 +08:00
# The four roles that apply to the ceph hosts will be applied: ceph-common,
# ceph-mon, ceph-osd and ceph-mds. So any changes to configuration, package updates, etc,
2014-03-04 02:08:51 +08:00
# will be applied as part of the rolling update process.
#
# /!\ DO NOT FORGET TO CHANGE THE RELEASE VERSION FIRST! /!\
2015-04-15 03:50:39 +08:00
- hosts : all
tasks :
- debug : msg="gather facts on all hosts for following reference"
2014-03-04 02:08:51 +08:00
- hosts : mons
serial : 1
sudo : True
2016-01-29 23:54:59 +08:00
vars :
upgrade_ceph_packages : True
2015-03-27 01:51:11 +08:00
pre_tasks :
- name : Compress the store as much as possible
command : ceph tell mon.{{ ansible_hostname }} compact
2014-03-04 02:08:51 +08:00
roles :
2015-03-27 01:51:11 +08:00
- ceph-common
- ceph-mon
2014-03-04 02:08:51 +08:00
post_tasks :
2015-03-27 01:51:11 +08:00
- name : Check if sysvinit
stat : >
This update has the following changes for the rolling update playbook.
1. Change how sysvinit ceph is determined to be enabled
For mons, the playbook checks if sysvinit is enabled by trying to
stat /var/lib/ceph/mon/ceph-{{ ansible_hostname }}/sysvinit [1].
However, that file is not created when a monitor is configured to
use sysvinit, instead, Ansible's service module is used [2]. Ansible
2.0 can verify if a service is enabled and does this by checking for
a glob, in this context it would be '/etc/rc?.d/S??ceph' [3]. Because
Ansible 1.9 does not support this feature, this change updates
rolling_update.yml by checking if sysvinit is enabled by having stat
glob for the same pattern and following the symlink. This is done only
for the mons.
2. Change how sysvinit ceph is restarted
The playbook passes the argument "mon" to the sysv init script
but, the init script does not necessarily take that argument and
it failed when tested on a RHEL7 system. However, dropping the
argument and just using Ansible's service module for state=restarted
worked so this change does not have this line for when
monsysvinit.stat.exists. A similar change is in this pull request
for the OSD restart (removing args=osd).
A second ceph mon restart command is run regardless of any conditions
being met. I am not sure why the service is restarted in the case of
upstart or sysvinit and then restarted again. I am going to assume
there is a subtle reason for this and not touch this second run but I
added a condition so that when ansible_os_family is Red Hat, then mon
is not passed as an argument, otherwise it is restarted as was already
in place.
[1] https://github.com/ceph/ceph-ansible/blob/v1.0.3/rolling_update.yml#L32-L33
[2] https://github.com/ceph/ceph-ansible/blob/v1.0.3/roles/ceph-mon/tasks/start_monitor.yml#L42-L45
[3] https://github.com/ansible/ansible-modules-core/blob/stable-2.0/system/service.py#L492-L493
2016-04-08 11:55:15 +08:00
path=/etc/rc?.d/S??ceph
follow=yes
2015-03-27 01:51:11 +08:00
register : monsysvinit
- name : Check if upstart
stat : >
path=/var/lib/ceph/mon/ceph-{{ ansible_hostname }}/upstart
register : monupstart
- name : Restart the monitor after compaction (Upstart)
service : >
name=ceph-mon
state=restarted
args=id={{ ansible_hostname }}
when : monupstart.stat.exists == True
- name : Restart the monitor after compaction (Sysvinit)
service : >
name=ceph
state=restarted
when : monsysvinit.stat.exists == True
- name : restart monitor(s)
service : >
name=ceph
state=restarted
args=mon
This update has the following changes for the rolling update playbook.
1. Change how sysvinit ceph is determined to be enabled
For mons, the playbook checks if sysvinit is enabled by trying to
stat /var/lib/ceph/mon/ceph-{{ ansible_hostname }}/sysvinit [1].
However, that file is not created when a monitor is configured to
use sysvinit, instead, Ansible's service module is used [2]. Ansible
2.0 can verify if a service is enabled and does this by checking for
a glob, in this context it would be '/etc/rc?.d/S??ceph' [3]. Because
Ansible 1.9 does not support this feature, this change updates
rolling_update.yml by checking if sysvinit is enabled by having stat
glob for the same pattern and following the symlink. This is done only
for the mons.
2. Change how sysvinit ceph is restarted
The playbook passes the argument "mon" to the sysv init script
but, the init script does not necessarily take that argument and
it failed when tested on a RHEL7 system. However, dropping the
argument and just using Ansible's service module for state=restarted
worked so this change does not have this line for when
monsysvinit.stat.exists. A similar change is in this pull request
for the OSD restart (removing args=osd).
A second ceph mon restart command is run regardless of any conditions
being met. I am not sure why the service is restarted in the case of
upstart or sysvinit and then restarted again. I am going to assume
there is a subtle reason for this and not touch this second run but I
added a condition so that when ansible_os_family is Red Hat, then mon
is not passed as an argument, otherwise it is restarted as was already
in place.
[1] https://github.com/ceph/ceph-ansible/blob/v1.0.3/rolling_update.yml#L32-L33
[2] https://github.com/ceph/ceph-ansible/blob/v1.0.3/roles/ceph-mon/tasks/start_monitor.yml#L42-L45
[3] https://github.com/ansible/ansible-modules-core/blob/stable-2.0/system/service.py#L492-L493
2016-04-08 11:55:15 +08:00
when : not ansible_os_family == "RedHat"
- name : restart monitor(s)
service : >
name=ceph
state=restarted
when : ansible_os_family == "RedHat"
2015-03-27 01:51:11 +08:00
2015-04-17 10:04:52 +08:00
- name : select a running monitor
set_fact : mon_host={{ item }}
with_items : groups.mons
when : item != inventory_hostname
2015-03-27 01:51:11 +08:00
- name : Waiting for the monitor to join the quorum...
shell : >
2015-03-31 04:13:29 +08:00
ceph -s | grep monmap | sed 's/.*quorum//' | egrep -sq {{ ansible_hostname }}
2015-03-27 01:51:11 +08:00
register : result
until : result.rc == 0
retries : 5
delay : 10
2015-04-17 10:04:52 +08:00
delegate_to : "{{ mon_host }}"
2015-03-27 01:51:11 +08:00
2014-03-04 02:08:51 +08:00
- hosts : osds
serial : 1
sudo : True
2016-01-29 23:54:59 +08:00
vars :
upgrade_ceph_packages : True
2015-03-27 01:51:11 +08:00
pre_tasks :
2015-04-28 01:08:05 +08:00
- name : Set OSD flags
command : ceph osd set {{ item }}
with_items :
- noout
- noscrub
- nodeep-scrub
2015-04-14 21:49:53 +08:00
delegate_to : "{{ groups.mons[0] }}"
2015-03-27 01:51:11 +08:00
2014-03-04 02:08:51 +08:00
roles :
2015-03-27 01:51:11 +08:00
- ceph-common
- ceph-osd
2014-03-04 02:08:51 +08:00
post_tasks :
2015-03-27 01:51:11 +08:00
- name : Check if sysvinit
2015-08-07 13:07:51 +08:00
shell : stat /var/lib/ceph/osd/ceph-*/sysvinit
2015-03-27 01:51:11 +08:00
register : osdsysvinit
2015-07-29 00:21:15 +08:00
failed_when : false
2015-03-27 01:51:11 +08:00
- name : Check if upstart
2015-08-07 13:07:51 +08:00
shell : stat /var/lib/ceph/osd/ceph-*/upstart
2015-03-27 01:51:11 +08:00
register : osdupstart
2015-07-29 00:21:15 +08:00
failed_when : false
2015-03-27 01:51:11 +08:00
- name : Gracefully stop the OSDs (Upstart)
service : >
name=ceph-osd-all
state=restarted
2015-08-07 13:07:51 +08:00
when : osdupstart.rc == 0
2015-03-27 01:51:11 +08:00
- name : Gracefully stop the OSDs (Sysvinit)
service : >
name=ceph
state=restarted
2015-08-07 13:07:51 +08:00
when : osdsysvinit.rc == 0
2015-03-27 01:51:11 +08:00
- name : Waiting for clean PGs...
shell : >
2015-03-31 04:13:29 +08:00
test "$(ceph pg stat | sed 's/^.*pgs://;s/active+clean.*//;s/ //')" -eq "$(ceph pg stat | sed 's/pgs.*//;s/^.*://;s/ //')" && ceph health | egrep -sq "HEALTH_OK|HEALTH_WARN"
2015-03-27 01:51:11 +08:00
register : result
until : result.rc == 0
retries : 10
delay : 10
2015-04-14 21:49:53 +08:00
delegate_to : "{{ groups.mons[0] }}"
2015-03-27 01:51:11 +08:00
2015-04-28 01:08:05 +08:00
- name : Unset OSD flags
command : ceph osd unset {{ item }}
with_items :
- noout
- noscrub
- nodeep-scrub
2015-04-14 21:49:53 +08:00
delegate_to : "{{ groups.mons[0] }}"
2015-03-27 01:51:11 +08:00
2014-03-04 02:08:51 +08:00
- hosts : mdss
serial : 1
sudo : True
2016-01-29 23:54:59 +08:00
vars :
upgrade_ceph_packages : True
2015-03-27 01:51:11 +08:00
2014-03-04 02:08:51 +08:00
roles :
2015-03-27 01:51:11 +08:00
- ceph-common
- ceph-mds
2014-03-04 02:08:51 +08:00
post_tasks :
2015-03-27 01:51:11 +08:00
- name : Check if sysvinit
stat : >
path=/var/lib/ceph/mon/ceph-{{ ansible_hostname }}/sysvinit
register : mdssysvinit
- name : Check if upstart
stat : >
path=/var/lib/ceph/mon/ceph-{{ ansible_hostname }}/upstart
register : mdsupstart
- name : Restart the metadata server (Upstart)
service : >
name=ceph-mds
state=restarted
args=id={{ ansible_hostname }}
when : mdsupstart.stat.exists == True
- name : Restart the metadata server (Sysvinit)
service : >
name=ceph
state=restarted
args=mds
when : mdssysvinit.stat.exists == True
- hosts : rgws
serial : 1
sudo : True
2016-01-29 23:54:59 +08:00
vars :
upgrade_ceph_packages : True
2015-03-27 01:51:11 +08:00
roles :
- ceph-common
2015-07-25 00:14:59 +08:00
- ceph-rgw
2015-03-27 01:51:11 +08:00
post_tasks :
- name : restart rados gateway server(s)
service : >
name={{ item }}
state=restarted
with_items :
- radosgw
when : radosgw_frontend == 'civetweb'
- name : restart rados gateway server(s)
service : >
name={{ item }}
state=restarted
with_items :
- apache2
- radosgw
when : radosgw_frontend == 'apache'