Sphinx Documentation

How to generate documentation websites for your Python package using Sphinx, including generating API documentation automatically, build multiple versions across releases and automatically build and host them on GitHub Pages.

*This is going on my blog* Picture by Me.

Pre-requisites

The instructions here assume that you have your Python Packaging well structured, under version control and backed up on GitHub.

Initial Setup

Sphinx comes with the sphinx-quickstart interactive tool which will help setup your repository with a basic conf.py and Makefile. There are a number of command line options but it is also interactive so you can answer questions to configure your setup. I like to keep the source and build directories separate and so use the --sep flag as well as the --makefile flag to generate a Makefile for building documentation on GNU/Linux or OSX (if you use M$-Win the use the --batchfile flag instead).

I keep documentation under docs/ directory within the root of the package directory.

cd ~/path/to/package
mkdir docs
cd docs
sphinx-quickstart --makefile

`conf.py`

Configuration is via a conf.py the automatically generated conf.py produced by sphinx-quickstart. It is well commented and instructive on how to use it to configure Sphinx and contains details on adding/modifying various sections of the this file.

Key sections are the list of extensions that your documentation uses.

`index.rst`

The front-page of your website, typically index.html for static sites, is derived from index.rst. You can write welcome details about your project and link to other pages you have written. Typically I write all but the front matter in Markdown.

Welcome to my packages documentation
====================================

This is my package, there are many packages like it but this one is mine.

Including Markdown

I already know Markdown fairly well and would rather use that to write documents (as I do with thisblog). Fortunately Sphinx can incorporate documentation written in Markdown using the myst_parser package. Simply include it in the extensions.

extensions = [
    "myst_parser",
]

By default it works with extensions of .md but if there are other flavours you wish to include (e.g. .Rmd for RMarkdown or .qmd for Quarto) you add them to the source_suffix in docs/conf.py

source_suffix = {".rst": "restructuredtext", ".md": "markdown"}

In your index.rst you can then list the Markdown filenames, without extensions. For example if you have an installation.md and configuration.md place them in the same directory as index.rst (the root docs/) directory and have in your index.rst have…

Welcome to my packages documentation
====================================

This is my package, there are many packages like it but this one is mine.

.. toctree::
   :maxdepth: 1
   :caption: Getting Started
   introduction
   configuration

Markdown Tables

If you have tables in Markdown (and its likely that you will) then you will need the sphinx-markdown-tables package which ensures they are rendered correctly.

Mermaid Diagrams

Further Sphinx has support for Mermaid diagrams that have been written in Markdown documents via the sphinxcontrib-mermaid package. This means that you can include all sorts of neat diagrams such as Git Graph.

%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true,'showCommitLabel': true, 'rotateCommitLabel': true}} }%%
gitGraph
    commit
    commit
    branch bug1
    checkout main
    commit
    checkout bug1
    commit
    commit
    checkout main
    branch bug2
    checkout bug2
    commit
    commit
    checkout bug1
    commit
    checkout main
    merge bug1 tag: "v0.1.1"
    checkout bug2
    commit
    commit
    checkout main
    merge bug2 tag: "v0.1.2"
    commit

Including API Documentation

As you write your package it is good practice include docstrings for each module/class/method/function that you write. For Python there are several different styles for writing these, my personal preference is for numpydoc style but regardless of your preference you should write them. They are invaluable to users (including your future self) to understand how the code works and as many modern Integrated Development Environments (IDEs) supporting functionality to show the documentation for functions as you type they are an invaluable reference. If you’re an Emacs user then you can leverage the numpydoc package to automatically insert NumPy docstrings in Python function definitions based on the function definition, it automatically detects names, type hints, exceptions and return types to generate the docstring (yet another reason to use Emacs!).

Whilst it is useful to have this API available in an IDE as you work it is also useful to include the reference on a packages website and this is relatively straight-forward with Sphinx which provides several tools and extensions.

`sphinx-apidoc`

The first is the the sphinx-apidoc command to generate documentation from the embedded docstrings. This is a command line tool that could be added to the Makefile.

`sphinx-autoapi`

However, rather than learning the intricacies of using this command the package Sphinx extensions sphinx-autoapi can be leveraged to automatically build the API documentation for you. This is particularly useful when you come to build multiple versions of your documentation as it means you do not have to include the .rst files that sphinx-apidoc generates in your repository they are generated on the fly when Sphinx builds each version of the documentation.

Configuration is via docs/conf.py and the package needs referencing in the extensions section then configuring at a bare minimum which directories to generate documentation from. i

extensions = [
    "autoapi.extension",
]

# -- autoapi configuration ---------------------------------------------------
autotype_api = "python"
autoapi_dirs = ["../mypackage"]

There are a lot of subtle configuration options and I would recommend reading the documentation and working through the Tutorials and How To Guides.

This has the added advantage that it works with ReadTheDocs.

Sphinx Autosummary

In addition the sphinx_ext_autosummary automates summarising the API docstrings.

Add the package as a dependency to the extensions…

extensions = [
    "sphinx.ext.autosummary"
]

Under the index.rst you should include a section header for the api that references an api.rst page for inclusion.

.. toctree::
   :maxdepth: 2
   :caption: API

   api

And then create the api.rst page which need only have the following. By including :recursive: the sub-modules will be included automatically.

API
===

.. autosummary::
   :recursive:
   :toctree: generated

   mypackage

Multiple Versions

Over time code and in turn documentation changes, not just the API but the documents written to demonstrate installation and usage of software. Not everyone always uses the latest version of your software and so it can be useful to provision documentation for each version that is available. Fortunately the Sphinx extension sphinx-multiversion makes this relatively painless.

You need to include it in the list of extensions of docs/conf.py

extensions = [
    "sphinx_multiversion",
]

Configuring Versions

Sidebar

For versions to not just be built but available you need to include a section on your site that allows selecting which version of the documentation to view. Sidebars are included via HTML templates and you need to configure the path to this directory and the name of the HTML file within it. The following options in the conf.py configure the _templates directory and within it the versioning.html file.

templates_path = [
    "_templates",
]

html_sidebars =  {"**":   "versioning.html",}

The versioning.html file can take a number of formats, refer to the documentation for all options, but the following is an example.

{% if versions %}
<h3>{{ _('Versions') }}</h3>
<ul>
  {%- for item in versions %}
  <li><a href="{{ item.url }}">{{ item.name }}</a></li>
  {%- endfor %}
</ul>
{% endif %}

Ensure this file is under Git version control, it is needed to build your pages on GitHub.

Tags/Branches

If no options are set then sphinx-multiversion will build documentation for all branhces, which is probably undesirable. Typically you want to restrict this to the released versions which are identified by git tags and perhaps your main/master branch. If you prefix your tags with v and you want to build the documentation for the HEAD of your main (or master) branch then you should set options as shown below for sphinx-multiversion. I like to be able to test documentation builds and so I have a section that allows me to include a given branch.

smv_tag_whitelist = r"^v\d+.*$"  # Tags beginning with v#
smv_branch_whitelist = r"^main$"  # main branch
# If testing changes locally comment out the above and the smv_branch_whitelist below instead. Replace the branch name
# you are working on ("ns-rse/testing-branch" in the example below) with the branch you are working on and run...
#
# cd docs
# sphinx-multiversion . _build/html
#
# smv_branch_whitelist = r"^(main|ns-rse/testing-branch)$"  # main branch
smv_released_pattern = r"^tags/.*$"  # Tags only
# smv_released_pattern = r"^(/.*)|(main).*$"  # Tags and HEAD of main
smv_outputdir_format = "{ref.name}"

If you are testing locally be sure to revert the commented sections so that the branch is not built on GitHub Pages.

Themes

There are a number of different themes available for including in your package. Which is used is defined by the html_theme variable in docs/conf.py. I like the pydata-sphinx-theme that is used by Pandas/Matplotlib.

html_theme = "pydata_sphinx_theme"

Package Dependencies

Since the documentation is part of your package it is important to include all of the dependencies that are required for building the documentation dependencies of your package so they can be easily installed and are available to Sphinx (since Sphinx will try loading anything listed in your docs/conf.py). These days you should really be using pyproject.toml to configure and manage your package, if you are unfamiliar with the packaging process see my post on Python Packaging.

[project.optional-dependencies]

docs = [
  "Sphinx",
  "myst_parser",
  "numpydoc",
  "pydata_sphinx_theme",
  "sphinx-autoapi",
  "sphinx-autodoc-typehints",
  "sphinx-multiversion",
  "sphinx_markdown_tables",
  "sphinx_rtd_theme",
  "sphinxcontrib-mermaid",
]

Ensure all of these dependencies are installed in your Virtual Environment.

cd ~/path/to/package
pip install .[docs]

Building Documentation

You are now ready to build your documentation locally.

cd ~/path/to/package/docs
mkdir -p _build/html
sphinx-multiversion . _build/html

Output should reside under the ~/path/to/package/docs/_build/html/ directory and there should be a directory for every tag as well as main (or master).

Deploying on GitHub Pages

The final stage is to leverage GitHub Pages to host your documentation. This can be achieved using a GitHub Action. These are a way of running certain tasks automatically on GitHub in response to certain actions. You can configure your actions to use those defined by others. I found the action-sphinx-docs-to-gh-pages action for generating Sphinx documentation but it didn’t support generating API documentation nor multiple versions of documentation so I have forked it and added this functionality (I intend to work with the authors and push the changes upstream).

To use this action you need to create a file in ~/path/to/package/.github/workflows/ called sphinx_docs_to_gh_pages.yaml and copy and paste the following YAML.

name: Sphinx docs to gh-pages

on:
  push:
  workflow_dispatch:

jobs:
  sphinx_docs_to_gh-pages:
    runs-on: ubuntu-latest
    name: Sphinx docs to gh-pages
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Setup Python
        uses: actions/setup-python@v4.3.0
        with:
          python-version: 3.9
      - name: Installing the Documentation requirements
        run: |
          pip3 install .[docs]
      - name: Running Sphinx to gh-pages Action
        uses: ns-rse/action-sphinx-docs-to-gh-pages@main
        with:
          # When testing set this branch to your branch, when working switch to main. It WILL fail if not
          # defined as it defaults to 'main'.
          branch: main
          dir_docs: docs
          sphinxapiexclude: '../*setup* ../*tests* ../*.ipynb ../demo.py ../make_baseline.py ../jupyter_notebook_config.py ../demo_ftrs.py'
          sphinxapiopts: '--separate -o . ../'
          sphinxopts: ''
          multiversion: true
          multiversionopts: ''

Save, add and commit to your Git repository and push the changes to GitHub.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2023,
  author = {Shephard, Neil},
  title = {Sphinx {Documentation}},
  date = {2023-05-07},
  url = {https://blog.nshephard.dev/posts/sphinx-docs/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2023. “Sphinx Documentation.” May 7, 2023. https://blog.nshephard.dev/posts/sphinx-docs/.