ceph-ansible/docs/source/testing/running.rst

.. _running_tests:

Running Tests
=============

Although tests run continuously in CI, a lot of effort was put into making it
easy to run in any environment, as long as a couple of requirements are met.


.. _dependencies:

Dependencies
------------

There are some Python dependencies, which are listed in a ``requirements.txt``
file within the ``tests/`` directory. These are meant to be installed using
Python install tools (pip in this case):

.. code-block:: console

   pip install -r tests/requirements.txt

For virtualization, either libvirt or VirtualBox is needed (there is native
support from the harness for both). This makes the test harness even more
flexible as most platforms will be covered by either VirtualBox or libvirt.


.. _running_a_scenario:

Running a scenario
------------------

Tests are driven by ``tox``, a command line tool to run a matrix of tests defined in
a configuration file (``tox.ini`` in this case at the root of the project).

For a thorough description of a scenario see :ref:`test_scenarios`.

To run a single scenario, make sure it is available (should be defined from
``tox.ini``) by listing them:

.. code-block:: console

   tox -l

In this example, we will use the ``luminous-ansible2.4-xenial_cluster`` one.  The
harness defaults to ``VirtualBox`` as the backend, so if you have that
installed in your system then this command should just work:

.. code-block:: console

   tox -e luminous-ansible2.4-xenial_cluster

And for libvirt it would be:

.. code-block:: console

   tox -e luminous-ansible2.4-xenial_cluster -- --provider=libvirt

.. warning::

   Depending on the type of scenario and resources available, running
   these tests locally in a personal computer can be very resource intensive.

.. note::

   Most test runs take between 20 and 40 minutes depending on system
   resources

The command should bring up the machines needed for the test, provision them
with ``ceph-ansible``, run the tests, and tear the whole environment down at the
end.


The output would look something similar to this trimmed version:

.. code-block:: console

   luminous-ansible2.4-xenial_cluster create: /Users/alfredo/python/upstream/ceph-ansible/.tox/luminous-ansible2.4-xenial_cluster
   luminous-ansible2.4-xenial_cluster installdeps: ansible==2.4.2, -r/Users/alfredo/python/upstream/ceph-ansible/tests/requirements.txt
   luminous-ansible2.4-xenial_cluster runtests: commands[0] | vagrant up --no-provision --provider=virtualbox
   Bringing machine 'client0' up with 'virtualbox' provider...
   Bringing machine 'rgw0' up with 'virtualbox' provider...
   Bringing machine 'mds0' up with 'virtualbox' provider...
   Bringing machine 'mon0' up with 'virtualbox' provider...
   Bringing machine 'mon1' up with 'virtualbox' provider...
   Bringing machine 'mon2' up with 'virtualbox' provider...
   Bringing machine 'osd0' up with 'virtualbox' provider...
   ...


After all the nodes are up, ``ceph-ansible`` will provision them, and run the
playbook(s):

.. code-block:: console

   ...
   PLAY RECAP *********************************************************************
   client0                    : ok=4    changed=0    unreachable=0    failed=0
   mds0                       : ok=4    changed=0    unreachable=0    failed=0
  mon0                       : ok=4    changed=0    unreachable=0    failed=0
   mon1                       : ok=4    changed=0    unreachable=0    failed=0
   mon2                       : ok=4    changed=0    unreachable=0    failed=0
   osd0                       : ok=4    changed=0    unreachable=0    failed=0
   rgw0                       : ok=4    changed=0    unreachable=0    failed=0
   ...


Once the whole environment is all running the tests will be sent out to the
hosts, with output similar to this:

.. code-block:: console

   luminous-ansible2.4-xenial_cluster runtests: commands[4] | testinfra -n 4 --sudo -v --connection=ansible --ansible-inventory=/Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster/hosts /Users/alfredo/python/upstream/ceph-ansible/tests/functional/tests
   ============================ test session starts ===========================
   platform darwin -- Python 2.7.8, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /Users/alfredo/python/upstream/ceph-ansible/.tox/luminous-ansible2.4-xenial_cluster/bin/python
   cachedir: ../../../../.cache
   rootdir: /Users/alfredo/python/upstream/ceph-ansible/tests, inifile: pytest.ini
   plugins: testinfra-1.5.4, xdist-1.15.0
   [gw0] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
   [gw1] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
   [gw2] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
   [gw3] darwin Python 2.7.8 cwd: /Users/alfredo/python/upstream/ceph-ansible/tests/functional/ubuntu/16.04/cluster
   [gw0] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
   [gw1] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
   [gw2] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
   [gw3] Python 2.7.8 (v2.7.8:ee879c0ffa11, Jun 29 2014, 21:07:35)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
   gw0 [154] / gw1 [154] / gw2 [154] / gw3 [154]
   scheduling tests via LoadScheduling

   ../../../tests/test_install.py::TestInstall::test_ceph_dir_exists[ansible:/mon0]
   ../../../tests/test_install.py::TestInstall::test_ceph_dir_is_a_directory[ansible:/mon0]
   ../../../tests/test_install.py::TestInstall::test_ceph_conf_is_a_file[ansible:/mon0]
   ../../../tests/test_install.py::TestInstall::test_ceph_dir_is_a_directory[ansible:/mon1]
   [gw2] PASSED ../../../tests/test_install.py::TestCephConf::test_ceph_config_has_mon_host_line[ansible:/mon0]
   ../../../tests/test_install.py::TestInstall::test_ceph_conf_exists[ansible:/mon1]
   [gw3] PASSED ../../../tests/test_install.py::TestCephConf::test_mon_host_line_has_correct_value[ansible:/mon0]
   ../../../tests/test_install.py::TestInstall::test_ceph_conf_is_a_file[ansible:/mon1]
   [gw1] PASSED ../../../tests/test_install.py::TestInstall::test_ceph_command_exists[ansible:/mon1]
   ../../../tests/test_install.py::TestCephConf::test_mon_host_line_has_correct_value[ansible:/mon1]
   ...

Finally the whole environment gets torn down:

.. code-block:: console

   luminous-ansible2.4-xenial_cluster runtests: commands[5] | vagrant destroy --force
   ==> osd0: Forcing shutdown of VM...
   ==> osd0: Destroying VM and associated drives...
   ==> mon2: Forcing shutdown of VM...
   ==> mon2: Destroying VM and associated drives...
   ==> mon1: Forcing shutdown of VM...
   ==> mon1: Destroying VM and associated drives...
   ==> mon0: Forcing shutdown of VM...
   ==> mon0: Destroying VM and associated drives...
   ==> mds0: Forcing shutdown of VM...
   ==> mds0: Destroying VM and associated drives...
   ==> rgw0: Forcing shutdown of VM...
   ==> rgw0: Destroying VM and associated drives...
   ==> client0: Forcing shutdown of VM...
   ==> client0: Destroying VM and associated drives...


And a brief summary of the scenario(s) that ran is displayed:

.. code-block:: console

   ________________________________________________ summary _________________________________________________
     luminous-ansible2.4-xenial_cluster: commands succeeded
     congratulations :)