ceph-ansible/docs/source/osds/scenarios.rst

OSD Scenarios
=============

There are a few *scenarios* that are supported and the differences are mainly
based on the Ceph tooling required to provision OSDs, but can also affect how
devices are being configured to create an OSD.

Supported values for the required ``osd_scenario`` variable are:

* :ref:`collocated <osd_scenario_collocated>`
* :ref:`non-collocated <osd_scenario_non_collocated>`
* :ref:`lvm <osd_scenario_lvm>`

Since the Ceph mimic release, it is preferred to use the :ref:`lvm scenario
<osd_scenario_lvm>` that uses the ``ceph-volume`` provisioning tool. Any other
scenario will cause deprecation warnings.

All the scenarios mentionned above support both containerized and non-containerized cluster.
As a reminder, deploying a containerized cluster can be done by setting ``containerized_deployment``
to ``True``.

.. _osd_scenario_lvm:

lvm
---

This OSD scenario uses ``ceph-volume`` to create OSDs, primarily using LVM, and
is only available when the Ceph release is luminous or newer.

**It is the preferred method of provisioning OSDs.**

It is enabled with the following setting::


    osd_scenario: lvm

Other (optional) supported settings:

- ``osd_objectstore``: Set the Ceph *objectstore* for the OSD. Available options
  are ``filestore`` or ``bluestore``.  You can only select ``bluestore`` with
  the Ceph release is luminous or greater. Defaults to ``filestore`` if unset.

- ``dmcrypt``: Enable Ceph's encryption on OSDs using ``dmcrypt``.
    Defaults to ``false`` if unset.

- ``osds_per_device``: Provision more than 1 OSD (the default if unset) per device.


Simple configuration
^^^^^^^^^^^^^^^^^^^^

With this approach, most of the decisions on how devices are configured to
provision an OSD are made by the Ceph tooling (``ceph-volume lvm batch`` in
this case).  There is almost no room to modify how the OSD is composed given an
input of devices.

To use this configuration, the ``devices`` option must be populated with the
raw device paths that will be used to provision the OSDs.


.. note:: Raw devices must be "clean", without a gpt partition table, or
          logical volumes present.


For example, for a node that has ``/dev/sda`` and ``/dev/sdb`` intended for
Ceph usage, the configuration would be:


.. code-block:: yaml

   osd_scenario: lvm
   devices:
     - /dev/sda
     - /dev/sdb

In the above case, if both devices are spinning drives, 2 OSDs would be
created, each with its own collocated journal.

Other provisioning strategies are possible, by mixing spinning and solid state
devices, for example:

.. code-block:: yaml

   osd_scenario: lvm
   devices:
     - /dev/sda
     - /dev/sdb
     - /dev/nvme0n1

Similar to the initial example, this would end up producing 2 OSDs, but data
would be placed on the slower spinning drives (``/dev/sda``, and ``/dev/sdb``)
and journals would be placed on the faster solid state device ``/dev/nvme0n1``.
The ``ceph-volume`` tool describes this in detail in
`the "batch" subcommand section <http://docs.ceph.com/docs/master/ceph-volume/lvm/batch/>`_


Other (optional) supported settings:

- ``crush_device_class``: Sets the CRUSH device class for all OSDs created with this
  method (it is not possible to have a per-OSD CRUSH device class using the *simple*
  configuration approach). Values *must be* a string, like
  ``crush_device_class: "ssd"``


Advanced configuration
^^^^^^^^^^^^^^^^^^^^^^

This configuration is useful when more granular control is wanted when setting
up devices and how they should be arranged to provision an OSD. It requires an
existing setup of volume groups and logical volumes (``ceph-volume`` will **not**
create these).

To use this configuration, the ``lvm_volumes`` option must be populated with
logical volumes and volume groups. Additionally, absolute paths to partitions
*can* be used for ``journal``, ``block.db``, and ``block.wal``.

.. note:: This configuration uses ``ceph-volume lvm create`` to provision OSDs

Supported ``lvm_volumes`` configuration settings:

- ``data``: The logical volume name or full path to a raw device (an LV will be
  created using 100% of the raw device)

- ``data_vg``: The volume group name, **required** if ``data`` is a logical volume.

- ``crush_device_class``: CRUSH device class name for the resulting OSD, allows
  setting set the device class for each OSD, unlike the global ``crush_device_class``
  that sets them for all OSDs.

.. note:: If you wish to set the ``crush_device_class`` for the OSDs
          when using ``devices`` you must set it using the global ``crush_device_class``
          option as shown above. There is no way to define a specific CRUSH device class
          per OSD when using ``devices`` like there is for ``lvm_volumes``.


``filestore`` objectstore variables:

- ``journal``: The logical volume name or full path to a partition.

- ``journal_vg``: The volume group name, **required** if ``journal`` is a logical volume.

.. warning:: Each entry must be unique, duplicate values are not allowed


``bluestore`` objectstore variables:

- ``db``: The logical volume name or full path to a partition.

- ``db_vg``: The volume group name, **required** if ``db`` is a logical volume.

- ``wal``: The logical volume name or full path to a partition.

- ``wal_vg``: The volume group name, **required** if ``wal`` is a logical volume.


.. note:: These ``bluestore`` variables are optional optimizations. Bluestore's
          ``db`` and ``wal`` will only benefit from faster devices. It is possible to
          create a bluestore OSD with a single raw device.

.. warning:: Each entry must be unique, duplicate values are not allowed


``bluestore`` example using raw devices:

.. code-block:: yaml

   osd_objectstore: bluestore
   osd_scenario: lvm
   lvm_volumes:
     - data: /dev/sda
     - data: /dev/sdb

.. note:: Volume groups and logical volumes will be created in this case,
          utilizing 100% of the devices.

``bluestore`` example with logical volumes:

.. code-block:: yaml

   osd_objectstore: bluestore
   osd_scenario: lvm
   lvm_volumes:
     - data: data-lv1
       data_vg: data-vg1
     - data: data-lv2
       data_vg: data-vg2

.. note:: Volume groups and logical volumes must exist.


``bluestore`` example defining ``wal`` and ``db`` logical volumes:

.. code-block:: yaml

   osd_objectstore: bluestore
   osd_scenario: lvm
   lvm_volumes:
     - data: data-lv1
       data_vg: data-vg1
       db: db-lv1
       db_vg: db-vg1
       wal: wal-lv1
       wal_vg: wal-vg1
     - data: data-lv2
       data_vg: data-vg2
       db: db-lv2
       db_vg: db-vg2
       wal: wal-lv2
       wal_vg: wal-vg2

.. note:: Volume groups and logical volumes must exist.


``filestore`` example with logical volumes:

.. code-block:: yaml

   osd_objectstore: filestore
   osd_scenario: lvm
   lvm_volumes:
     - data: data-lv1
       data_vg: data-vg1
       journal: journal-lv1
       journal_vg: journal-vg1
     - data: data-lv2
       data_vg: data-vg2
       journal: journal-lv2
       journal_vg: journal-vg2

.. note:: Volume groups and logical volumes must exist.


.. _osd_scenario_collocated:

collocated
----------

.. warning:: This scenario is deprecated in the Ceph mimic release, and fully
             removed in newer releases. It is recommended to used the
             :ref:`lvm scenario <osd_scenario_lvm>` instead

This OSD scenario uses ``ceph-disk`` to create OSDs with collocated journals
from raw devices.

Use ``osd_scenario: collocated`` to enable this scenario. This scenario also
has the following required configuration options:

- ``devices``

This scenario has the following optional configuration options:

- ``osd_objectstore``: defaults to ``filestore`` if not set. Available options are ``filestore`` or ``bluestore``.
  You can only select ``bluestore`` if the Ceph release is luminous or greater.

- ``dmcrypt``: defaults to ``false`` if not set.

This scenario supports encrypting your OSDs by setting ``dmcrypt: True``.

If ``osd_objectstore: filestore`` is enabled both 'ceph data' and 'ceph journal' partitions
will be stored on the same device.

If ``osd_objectstore: bluestore`` is enabled 'ceph data', 'ceph block', 'ceph block.db', 'ceph block.wal' will be stored
on the same device. The device will get 2 partitions:

- One for 'data', called 'ceph data'

- One for 'ceph block', 'ceph block.db', 'ceph block.wal' called 'ceph block'

Example of what you will get:

.. code-block:: console

   [root@ceph-osd0 ~]# blkid /dev/sda*
   /dev/sda: PTTYPE="gpt"
   /dev/sda1: UUID="9c43e346-dd6e-431f-92d8-cbed4ccb25f6" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="749c71c9-ed8f-4930-82a7-a48a3bcdb1c7"
   /dev/sda2: PARTLABEL="ceph block" PARTUUID="e6ca3e1d-4702-4569-abfa-e285de328e9d"

An example of using the ``collocated`` OSD scenario with encryption would look like:

.. code-block:: yaml

   osd_scenario: collocated
   dmcrypt: true
   devices:
     - /dev/sda
     - /dev/sdb


.. _osd_scenario_non_collocated:

non-collocated
--------------

.. warning:: This scenario is deprecated in the Ceph mimic release, and fully
             removed in newer releases. It is recommended to used the
             :ref:`lvm scenario <osd_scenario_lvm>` instead

This OSD scenario uses ``ceph-disk`` to create OSDs from raw devices with journals that
exist on a dedicated device.

Use ``osd_scenario: non-collocated`` to enable this scenario. This scenario also
has the following required configuration options:

- ``devices``

This scenario has the following optional configuration options:

- ``dedicated_devices``: defaults to ``devices`` if not set

- ``osd_objectstore``: defaults to ``filestore`` if not set. Available options are ``filestore`` or ``bluestore``.
  You can only select ``bluestore`` with the Ceph release is luminous or greater.

- ``dmcrypt``: defaults to ``false`` if not set.

This scenario supports encrypting your OSDs by setting ``dmcrypt: True``.

If ``osd_objectstore: filestore`` is enabled 'ceph data' and 'ceph journal' partitions
will be stored on different devices:
- 'ceph data' will be stored on the device listed in ``devices``
- 'ceph journal' will be stored on the device listed in ``dedicated_devices``

Let's take an example, imagine ``devices`` was declared like this:

.. code-block:: yaml

   devices:
     - /dev/sda
     - /dev/sdb
     - /dev/sdc
     - /dev/sdd

And ``dedicated_devices`` was declared like this:

.. code-block:: yaml

   dedicated_devices:
     - /dev/sdf
     - /dev/sdf
     - /dev/sdg
     - /dev/sdg

This will result in the following mapping:

- ``/dev/sda`` will have ``/dev/sdf1`` as journal

- ``/dev/sdb`` will have ``/dev/sdf2`` as a journal

- ``/dev/sdc`` will have ``/dev/sdg1`` as a journal

- ``/dev/sdd`` will have ``/dev/sdg2`` as a journal


If ``osd_objectstore: bluestore`` is enabled, both 'ceph block.db' and 'ceph block.wal' partitions will be stored
on a dedicated device.

So the following will happen:

- The devices listed in ``devices`` will get 2 partitions, one for 'block' and one for 'data'. 'data' is only 100MB
  big and do not store any of your data, it's just a bunch of Ceph metadata. 'block' will store all your actual data.

- The devices in ``dedicated_devices`` will get 1 partition for RocksDB DB, called 'block.db' and one for RocksDB WAL, called 'block.wal'

By default ``dedicated_devices`` will represent block.db

Example of what you will get:

.. code-block:: console

   [root@ceph-osd0 ~]# blkid /dev/sd*
   /dev/sda: PTTYPE="gpt"
   /dev/sda1: UUID="c6821801-2f21-4980-add0-b7fc8bd424d5" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="f2cc6fa8-5b41-4428-8d3f-6187453464d0"
   /dev/sda2: PARTLABEL="ceph block" PARTUUID="ea454807-983a-4cf2-899e-b2680643bc1c"
   /dev/sdb: PTTYPE="gpt"
   /dev/sdb1: PARTLABEL="ceph block.db" PARTUUID="af5b2d74-4c08-42cf-be57-7248c739e217"
   /dev/sdb2: PARTLABEL="ceph block.wal" PARTUUID="af3f8327-9aa9-4c2b-a497-cf0fe96d126a"

There is more device granularity for Bluestore ONLY if ``osd_objectstore: bluestore`` is enabled by setting the
``bluestore_wal_devices`` config option.

By default, if ``bluestore_wal_devices`` is empty, it will get the content of ``dedicated_devices``.
If set, then you will have a dedicated partition on a specific device for block.wal.

Example of what you will get:

.. code-block:: console

   [root@ceph-osd0 ~]# blkid /dev/sd*
   /dev/sda: PTTYPE="gpt"
   /dev/sda1: UUID="39241ae9-d119-4335-96b3-0898da8f45ce" TYPE="xfs" PARTLABEL="ceph data" PARTUUID="961e7313-bdb7-49e7-9ae7-077d65c4c669"
   /dev/sda2: PARTLABEL="ceph block" PARTUUID="bff8e54e-b780-4ece-aa16-3b2f2b8eb699"
   /dev/sdb: PTTYPE="gpt"
   /dev/sdb1: PARTLABEL="ceph block.db" PARTUUID="0734f6b6-cc94-49e9-93de-ba7e1d5b79e3"
   /dev/sdc: PTTYPE="gpt"
   /dev/sdc1: PARTLABEL="ceph block.wal" PARTUUID="824b84ba-6777-4272-bbbd-bfe2a25cecf3"

An example of using the ``non-collocated`` OSD scenario with encryption, bluestore and dedicated wal devices would look like:

.. code-block:: yaml

   osd_scenario: non-collocated
   osd_objectstore: bluestore
   dmcrypt: true
   devices:
     - /dev/sda
     - /dev/sdb
   dedicated_devices:
     - /dev/sdc
     - /dev/sdc
   bluestore_wal_devices:
     - /dev/sdd
     - /dev/sdd