Packaging (using pyproject.toml)

The pyproject.toml allows for easy packaging and installation:

$ pip install pyyc

or through the complete cloning of the gitlab repository:

$ git clone https://gitlab.in2p3.fr/ycopin/pyyc  # cloning repo
$ cd pyyc
$ pip install .                                  # local installation

Package initialization (__init__.py)

The __init__.py file in each package directory describes the initialization of the package, i.e. the actions (usually imports, but also definitions, etc.) to be performed at package import.

>>> import pyyc  # Will run pyyc/__init__.py and all subsequent initializations
Initialization top-level module
Initialization sub-package A module A1
Initialization sub-package A module A2
Initialization sub-package B module + sub-package A module A1
>>> pyyc.subpkgA.modA1.version
'sub-package A module A1'
>>> pyyc.subpkgB.version
'sub-package B module + sub-package A module A1'

Note

The print-outs in the __init__.py are just for educational purpose!

Main entries (__main__.py)

The main program can be called in different ways:

  • as the __main__.py entry of a module, e.g.:

    $ python -m pyyc arg1 arg2     # Execute top-level pyyc/__main__.py
Initialization top-level module
Initialization sub-package A module A1
Initialization sub-package A module A2
Initialization sub-package B module + sub-package A module A1
---------------------- MAIN ----------------------
Command line arguments: ['arg1', 'arg2']

    There can be only one package main, corresponding to the if __name == "__main__" part of the __main__.py file.

  • as console scripts defined as entry points in pyproject.toml:

    [project.scripts]
# Entry points and automatic script creation
# script 'pyyc' corresponds to function 'main()' in file '__main__.py'
pyyc = "pyyc.__main__:main"
# 'pyyc_addition' corresponds to function 'main_addition()' in '__main__.py'
pyyc_addition = "pyyc.__main__:main_addition"

    These entry points are converted to plain scripts at installation:

    $ pyyc arg1 arg2               # Execute pyyc/__main__.py:main()
Initialization top-level module
Initialization sub-package A module A1
Initialization sub-package A module A2
Initialization sub-package B module + sub-package A module A1
---------------------- MAIN ----------------------
Command line arguments: ['arg1', 'arg2']

$ pyyc_addition 1 2            # Execute pyyc/__main__.py:main_addition()
Initialization top-level module
Initialization sub-package A module A1
Initialization sub-package A module A2
Initialization sub-package B module + sub-package A module A1
1 + 2 = 3

Reference: https://setuptools.pypa.io/en/latest/userguide/entry_point.html

How-to generate documentation

The sample code is documented using the documentation generator sphinx within the dedicated directory docs/, typically for a first use:

[docs/]$ sphinx-quickstart  # initiate the documentation tool
[docs/]$ # edit 'conf.py' to your needs (see below)
[docs/]$ sphinx-apidoc -o . ../pyyc  # automatic generation of documentation files
Creating file ./pyyc.rst.
Creating file ./pyyc.subpkgA.rst.
Creating file ./pyyc.subpkgB.rst.
Creating file ./modules.rst.
[docs/]$ # include these documentation files in 'index.rst' (see below)
[docs/]$ make html          # build the documentation as a website
[docs/]$ firefox _build/html/index.html

As an illustration, the online version of this documentation, using the sphinx configuration file conf.py, is available at https://ycopin.pages.in2p3.fr/pyyc/. The automatically generated code documentation part is under Code documentation.

Documentation configuration (conf.py)

In particular, the use of the auto-documentation extension sphinx.ext.autodoc requires few non-trivial lines in conf.py:

  • set-up the path to code sources for extraction of docstrings:

    import os, sys, re

sys.path.insert(0, os.path.abspath(".."))

  • add sphinx.ext.autodoc in the list of sphinx extensions extensions = [...] (or initially add option --ext-autodoc to sphinx-quickstart)

  • configure the autodoc extension:

    # Autodoc configuration
autodoc_default_options = {
    "members": True,  # Document all members
    "undoc-members": True,  # ... including undocumented ones
    "ignore-module-all": True,  # do not stick to __all__
}
autoclass_content = "both"  # Insert class and __init__ docstrings
autodoc_member_order = "bysource"  # Keep source order

Documentation content (index.rst)

The index.rst file is the top-level documentation file, which will include links to all the other documentation .rst files under the specific .. toctree:: command, e.g.:

.. toctree::
   :caption: Code documentation
   :titlesonly:

   pyyc
   pyyc.subpkgA
   pyyc.subpkgB
   modules

How-to include notebooks

If any, you can add notebooks directly to your documentation with the nbsphinx extension (which needs to be installed).

  • add nbsphinx in the list of sphinx extensions extensions = [...]

  • add notebooks stored in docs/notebooks/ directory in a dedicated documentation section, e.g.:

    Notebooks
=========

.. toctree::
   :titlesonly:

   notebooks/pyyc.ipynb

How-to run code tests

Note

see Test definition for definition of tests.

Doctests

Doctests (i.e. small examples and tests directly included in the docstrings) can be performed on documented source code with either standard package doctest or with external package pytest:

[pyyc/]$ python -m doctest -v mod.py
[pyyc/]$ pytest --doctest-modules -v mod.py

Dedicated tests

Tests gathered in the tests/ directory shall be performed using pytest, e.g.:

[tests/]$ pytest -v test_mod.py

pytest will actually auto-discover all the tests from top-level directory, and read configuration (if any) from pyproject.toml:

$ pytest

Test coverage

coverage will run the test suite and look for parts of the code which have been (and more importantly not been) tested.

$ coverage run -m pytest
$ coverage report
Name                       Stmts   Miss  Cover
----------------------------------------------
pyyc/__init__.py               5      0   100%
pyyc/mod.py                   44      2    95%
pyyc/subpkgA/__init__.py       5      0   100%
pyyc/subpkgA/modA1.py          2      0   100%
pyyc/subpkgA/modA2.py          2      0   100%
pyyc/subpkgB/__init__.py       3      0   100%
pyyc/subpkgB/modB.py           3      0   100%
tests/__init__.py              0      0   100%
tests/test_mod.py             46      0   100%
----------------------------------------------
TOTAL                        110      2    98%

To visualize which parts of the code is documented or not:

$ coverage html
Wrote HTML report to htmlcov/index.html
$ firefox htmlcov/index.html

Coverage can also be estimated directly from pytest using plugin pytest-cov:

$ pytest --cov                    # summary report
$ pytest --cov --cov-report=html  # generate an html interactive report
$ firefox htmlcov/index.html

How-to include data files (e.g. config/)

pyyc.mod.read_config() is an example of a function accessing data file at run-time, using importlib.resources.files (python 3.10+):

>>> from pyyc.mod import read_config
>>> cfg = read_config()  # will look for config file distributed with pyyc package
Reading configuration from .../pyYC/pyyc/config/default.cfg...
>>> cfg['DEFAULT']['version']
'cfg-1.0'

This is controlled by section [tool.setuptools.package-data] in pyproject.toml:

[tool.setuptools.package-data]        # Package data
pyyc = ["*.cfg"]

Reference: https://setuptools.pypa.io/en/latest/userguide/datafiles.html#accessing-data-files-at-runtime

How-to set-up Gitlab continuous integration

If the code is hosted in a GitLab repository, one can use Continuous methods with a dedicated configuration file .gitlab-ci.yml in the top-level directory. The provided configuration will build and deploy the documentation as a static website hosted on GitLab pages, namely https://ycopin.pages.in2p3.fr/pyyc/ for pyyc.

Attention

the documentation pages might not be visible to everyone after deployment; check page visibility setting in Settings / General / Visibility, project features, permissions.

More elements

For French-speaking users, you can have a look at the online course Analyse scientifique avec Python, and in particular to the packaging section.

To do