ceph-ansible/docs/source/osds/scenarios.rst

OSD Scenario
============

As of stable-4.0, the following scenarios are not supported anymore since they are associated to ``ceph-disk``:

* `collocated`
* `non-collocated`

Since the Ceph luminous release, it is preferred to use the :ref:`lvm scenario
<osd_scenario_lvm>` that uses the ``ceph-volume`` provisioning tool. Any other
scenario will cause deprecation warnings.

``ceph-disk`` was deprecated during the ceph-ansible 3.2 cycle and has been removed entirely from Ceph itself in the Nautilus version.
At present (starting from stable-4.0), there is only one scenario, which defaults to ``lvm``, see:

* :ref:`lvm <osd_scenario_lvm>`

So there is no need to configure ``osd_scenario`` anymore, it defaults to ``lvm``.

The ``lvm`` scenario mentioned above support both containerized and non-containerized cluster.
As a reminder, deploying a containerized cluster can be done by setting ``containerized_deployment``
to ``True``.

If you want to skip OSD creation during a ``ceph-ansible run``
(e.g. because you have already provisioned your OSDs but disk IDs have
changed), you can skip the ``prepare_osd`` tag i.e. by specifying
``--skip-tags prepare_osd`` on the ``ansible-playbook`` command line.

.. _osd_scenario_lvm:

lvm
---

This OSD scenario uses ``ceph-volume`` to create OSDs, primarily using LVM, and
is only available when the Ceph release is luminous or newer.
It is automatically enabled.

Other (optional) supported settings:

- ``osd_objectstore``: Set the Ceph *objectstore* for the OSD. Available options
  are ``filestore`` or ``bluestore``.  You can only select ``bluestore`` with
  the Ceph release is luminous or greater. Defaults to ``bluestore`` if unset.

- ``dmcrypt``: Enable Ceph's encryption on OSDs using ``dmcrypt``.
    Defaults to ``false`` if unset.

- ``osds_per_device``: Provision more than 1 OSD (the default if unset) per device.


Simple configuration
^^^^^^^^^^^^^^^^^^^^

With this approach, most of the decisions on how devices are configured to
provision an OSD are made by the Ceph tooling (``ceph-volume lvm batch`` in
this case).  There is almost no room to modify how the OSD is composed given an
input of devices.

To use this configuration, the ``devices`` option must be populated with the
raw device paths that will be used to provision the OSDs.


.. note:: Raw devices must be "clean", without a gpt partition table, or
          logical volumes present.


For example, for a node that has ``/dev/sda`` and ``/dev/sdb`` intended for
Ceph usage, the configuration would be:


.. code-block:: yaml

   devices:
     - /dev/sda
     - /dev/sdb

In the above case, if both devices are spinning drives, 2 OSDs would be
created, each with its own collocated journal.

Other provisioning strategies are possible, by mixing spinning and solid state
devices, for example:

.. code-block:: yaml

   devices:
     - /dev/sda
     - /dev/sdb
     - /dev/nvme0n1

Similar to the initial example, this would end up producing 2 OSDs, but data
would be placed on the slower spinning drives (``/dev/sda``, and ``/dev/sdb``)
and journals would be placed on the faster solid state device ``/dev/nvme0n1``.
The ``ceph-volume`` tool describes this in detail in
`the "batch" subcommand section <https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/>`_

This option can also be used with ``osd_auto_discovery``, meaning that you do not need to populate
``devices`` directly and any appropriate devices found by ansible will be used instead.

.. code-block:: yaml

   osd_auto_discovery: true

Other (optional) supported settings:

- ``crush_device_class``: Sets the CRUSH device class for all OSDs created with this
  method (it is not possible to have a per-OSD CRUSH device class using the *simple*
  configuration approach). Values *must be* a string, like
  ``crush_device_class: "ssd"``


Advanced configuration
^^^^^^^^^^^^^^^^^^^^^^

This configuration is useful when more granular control is wanted when setting
up devices and how they should be arranged to provision an OSD. It requires an
existing setup of volume groups and logical volumes (``ceph-volume`` will **not**
create these).

To use this configuration, the ``lvm_volumes`` option must be populated with
logical volumes and volume groups. Additionally, absolute paths to partitions
*can* be used for ``journal``, ``block.db``, and ``block.wal``.

.. note:: This configuration uses ``ceph-volume lvm create`` to provision OSDs

Supported ``lvm_volumes`` configuration settings:

- ``data``: The logical volume name or full path to a raw device (an LV will be
  created using 100% of the raw device)

- ``data_vg``: The volume group name, **required** if ``data`` is a logical volume.

- ``crush_device_class``: CRUSH device class name for the resulting OSD, allows
  setting set the device class for each OSD, unlike the global ``crush_device_class``
  that sets them for all OSDs.

.. note:: If you wish to set the ``crush_device_class`` for the OSDs
          when using ``devices`` you must set it using the global ``crush_device_class``
          option as shown above. There is no way to define a specific CRUSH device class
          per OSD when using ``devices`` like there is for ``lvm_volumes``.


``filestore`` objectstore variables:

- ``journal``: The logical volume name or full path to a partition.

- ``journal_vg``: The volume group name, **required** if ``journal`` is a logical volume.

.. warning:: Each entry must be unique, duplicate values are not allowed


``bluestore`` objectstore variables:

- ``db``: The logical volume name or full path to a partition.

- ``db_vg``: The volume group name, **required** if ``db`` is a logical volume.

- ``wal``: The logical volume name or full path to a partition.

- ``wal_vg``: The volume group name, **required** if ``wal`` is a logical volume.


.. note:: These ``bluestore`` variables are optional optimizations. Bluestore's
          ``db`` and ``wal`` will only benefit from faster devices. It is possible to
          create a bluestore OSD with a single raw device.

.. warning:: Each entry must be unique, duplicate values are not allowed


``bluestore`` example using raw devices:

.. code-block:: yaml

   osd_objectstore: bluestore
   lvm_volumes:
     - data: /dev/sda
     - data: /dev/sdb

.. note:: Volume groups and logical volumes will be created in this case,
          utilizing 100% of the devices.

``bluestore`` example with logical volumes:

.. code-block:: yaml

   osd_objectstore: bluestore
   lvm_volumes:
     - data: data-lv1
       data_vg: data-vg1
     - data: data-lv2
       data_vg: data-vg2

.. note:: Volume groups and logical volumes must exist.


``bluestore`` example defining ``wal`` and ``db`` logical volumes:

.. code-block:: yaml

   osd_objectstore: bluestore
   lvm_volumes:
     - data: data-lv1
       data_vg: data-vg1
       db: db-lv1
       db_vg: db-vg1
       wal: wal-lv1
       wal_vg: wal-vg1
     - data: data-lv2
       data_vg: data-vg2
       db: db-lv2
       db_vg: db-vg2
       wal: wal-lv2
       wal_vg: wal-vg2

.. note:: Volume groups and logical volumes must exist.


``filestore`` example with logical volumes:

.. code-block:: yaml

   osd_objectstore: filestore
   lvm_volumes:
     - data: data-lv1
       data_vg: data-vg1
       journal: journal-lv1
       journal_vg: journal-vg1
     - data: data-lv2
       data_vg: data-vg2
       journal: journal-lv2
       journal_vg: journal-vg2

.. note:: Volume groups and logical volumes must exist.