blog.nshephard.dev

Pytest testmon

Neil Shephard — Wed, 24 Sep 2025 00:00:00 GMT

Pytest is an excellent testing framework for Python with a wealth of plugins which extend the functionality. This blog post covers using the pytest-testmon to speed up the feedback loop whilst working on and developing your code.

Pretty Stones by Me!

Test, test, test

It is good practice to have a comprehensive suite of both unit, integration and regression tests when writing and developing software. The former ensure at a fine-grained level that each of the functions or methods that you write work as expected whilst the later ensure that they all function together as expected.

Pytest is a popular alternative framework for writing and running your test suite to the standard libraries unittests and is easy to incorporate into your project. Including and configuring it via pyproject.toml might look something like the following which includes pytest-cov to summarise coverage and pytest-mpl for testing Matplotlib output. Configuration of coverage is included that omits certain files.

[project.optional-dependencies]
tests = [
  "pytest",
  "pytest-cov",
  "pytest-mpl"
]

[tool.pytest.ini_options]
minversion = "8.0"
addopts = ["--cov", "--mpl", "-ra", "--strict-config", "--strict-markers"]
log_level = "INFO"
log_cli = true
log_cli_level = "INFO"
testpaths = [
    "tests",
]
filterwarnings = [
    "ignore::DeprecationWarning",
    "ignore::PendingDeprecationWarning",
    "ignore::UserWarning"
]
xfail_strict = true

[tool.coverage.run]
source = ["src"]
omit = [
  "**/_version.py",
  "*tests*",
  "**/__init__*",
]

Tests are typically organised with a single file per Python sub-module with at least one test for each function. If you’re not familiar with pytest check out its documentation for more on how to organise your test suite.

Development cycle

A typical development cycle to add a new feature might involve the following cycle if you are following the Test Drive Development paradigm.

Write a unit test.
Write code to add functionality.
Run test suite to see if it passes.

If it fails return to 2.
If it passes all is good.

Alternatively if fixing bugs you might…

Add new test to cover the bug.
Run test to check new test fails.
Attempt to fix the bug
Run test suite to see if it passes.

If it fails return to 3.
If it passes all is good.

In either scenario you will typically iterate over the steps a few times until everything is working using pytest to run your test suite root of your packages directory.

pytest

But if you’ve a large test suite you probably want to avoid running all tests each time and you can do so by telling pytest which file to run tests on by providing it as an argument. Here we run only the test_submodule1.py tests.

pytest tests/test_submodule1.py

You can have even finer grained control over your tests. For example if you want to only run an individual test within tests/test_submodules.py you can do that too by specifying the name of the test you want to run after the test file name, so if you had test_function1() defined in tests/test_submodules.py you would run the following.

pytest tests/test_submodule1.py::test_function1

Similarly if you were developing a new feature and writing function2 in submodule1 you could have a test test_function2() and explicitly test it with.

pytest tests/test_submodule1.py::test_function2

As you iterate through the development cycle you could manually run your unit tests to find out if they pass or fail, but that doesn’t account for any integration tests where your functions are used as part of a pipeline and knowing what tests are affected by changes can be challenging, particularly in a large code base where you would want to avoid running the whole test suite each time.

Commits as Units of Work

In software development small units of related work are typically aggregated into commits within Git (or other version control systems) which means we have a natural point at which to undertake testing, i.e. when we are ready to add our work to the Git history via a commit. Thus before making a commit we should make sure that our test suite completes. Naturally we could run the whole suite with pytest (and even parallelise it so it runs faster with pytest-xdist) but as a software package grows so does the number of tests and it might take a while to run the test suite. We’ve seen already that it is possible to run individual tests as desired but this might take a little work if there are several tests that need running when fixing bugs and regression tests need to be run as well as unit tests.

Wouldn’t it be great if we could automatically run only the tests that are affected by the changes that we have made when making a commit?

Enter `pytest-testmon`

This is where pytest-testmon comes in useful, it “selects and executes only tests you need to run” and achieves this by “collecting dependencies between tests and all executed code and comparing the dependencies against changes.”. Basically, given the dummy examples above it would identify the tests that involve either function1() or function2() and only run those tests.

Installation

You can install pytest-testmon in your virtual environment and run it manually. Before you can use it though you have to allow it to analyse your code base and build a database of how the tests relate to the packages modules, functions, classes and methods, so after installing it you should run pytest --testmon to create becomes useful it has to establish how functions, classes and methods relate to tests and you can use it thought pytest-testmon needs to analyse your code base and build a database of how the test relates to the packages modules and functions/methods.

pip install pytest-testmon
pytest --testmon

This creates, by default, the files .testmondata, .testmondata-shm and .testmondata-wal which comprise is an SQLite database of the functions, classes and methods and how they relate to your test suite. Once you have installed and built your database subsequent calls to pytest --testmon will work out what tests are affected by changes and only run those.

You will probably want to have git ignore these files and can do so by adding them to either the local .gitignore or your global .gitignore

.testmondata*

This is an improvement as you now can run pytest --testmon to ensure that only tests affected by the changes you are making are run and it helps shorten the feedback loop because you only have to wait for a subset of the tests to run. But it’s still running the tests manually before you make your git commit. We want to automate this step, how can we ensure pytest --testmon is run prior to commits to make sure tests pass on only the changed files?

Hello pre-commit my old friend

I’m a big fan of pre-commit and have written about it many times. We can define a local pre-commit hook to run pytest --testmon automatically before each commit is made which not only removes the task of having to remember to run pytest --testmon manually but ensures that our test suite passes before we can make commits.

Configure pre-commit

We define the local hook in the repositories .pre-commit-config.yaml file which will run pytest --testmon on each commit. The repo: local is defined so that it only runs on local systems and not in Continuous Integration (CI), typically the whole test suite will be run in CI anywaym and give it appropriately named ìd and name values. The entry is the command we wish to run, i.e. pytest --testmon and we set the language: system. Finally we restrict it to files that end in \.py$ so that it won’t run if you change say your documentation files that are written in Markdown (.md).

repos:
  - repo: local
    hooks:
      - id: pytest
        name: Pytest (testmon)
        entry: pytest --testmon
        language: system
        files: \.py$

Package Upgrades

You may find periodically that pytest-testmon reports that the installed packages have changed and the database needs updating. If this is the case simply run pytest --testmon again manually and it should update the database.

Add `pytest-testmon` to package dependencies

You will want to make sure that anyone else who wishes to contribute to your code also uses pytest-testmon to help ensure that pull requests that are made against your repository continue to pass all tests.Here we add it under the dev subset of optional dependencies in the projects pyproject.toml as only those developing the software will want to install it, and we want to avoid installing it unnecessarily in say Continuous Integration.

[project.optional-dependencies]
dev = [
  "pytest-testmon",
]

If you don’t already have a tool.pytest.ini_options section in your pyproject.toml now would be a good time to add one (see the example at the start of this article). You don’t need any specific configuration for pytest --testmon adding, just make sure its an optional dependency as above.

Update your documentation

It is recommended to add a section to your documentations Contributing section that advises how to install the developer dependencies and use pytest-testmon including which files to ignore.

pip install -e .[dev]

Test It Out

Add a new test, fix a bug or add a new feature (with a test of course!) and git add -u . && git commit -m "bug: Fixing bug 5" and you should find that only the tests that affect the changed code are run as part of the pre-commit hook. 🪄

Summary

Speeding up the feedback loop for software development by pushing the identification of errors in your code further “left”, i.e. earlier in the development cycle is really useful and leveraging pytest-testmon as a pre-commit hook is a simple way of achieving this. It helps you capture and fix changes that break your test suite before making commits to your Git history which helps remove the fixing tests commits that litter many projects.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2025,
  author = {Shephard, Neil},
  title = {Pytest Testmon},
  date = {2025-09-24},
  url = {https://blog.nshephard.dev/posts/pytest-testmon/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2025. “Pytest Testmon.” September 24, 2025. https://blog.nshephard.dev/posts/pytest-testmon/.

Emacs Tips & Tweaks (2025-09-05)- vundo, regex builder and more

Neil Shephard — Fri, 05 Sep 2025 00:00:00 GMT

Improving my Emacs configuration is an open-ended hobby. Recently I discovered a couple of things that I’ve incorporated into my daily usage that I’ve found really useful and thought I’d document and share them here.

Taking Time to Reflect by Me

`vundo`

Emacs has a complicated undo history and I often find myself going round in circles. Enter vundo which is a really neat little package that makes navigating the undo history much easier by providing a graphical overview of the sometimes branching undo history. By default it comes with the key-binding C-c v which will popup a buffer at the bottom showing a graph of your undo history. Switch to this buffer and you can use keys to navigate to older points in the undo history. If you want to know what differs just hit d. If you enable vundo-popup-mode then the buffer will appear whenever you hit the traditional C-x u to undo some changes. I have this set as a hook for a couple of modes.

(use-package vundo
  :ensure t
  :hook
  (prog-mode . vundo-popup-mode)
  (text-mode . vundo-popup-mode))

Regex Builder

I’m not great at writing Regular Expressions (aka regex) and regularly resort to online builders (e.g. regexr or regex101) but these a) lack Emacs’ flavour of regex’s; b) don’t contain what is in the buffer I’m editing and want to search/replace.

Enter re-builder which when invoked brings up a buffer into which you can start typing your regular expression and matches in the buffer you are editing are highlighted. Pretty handy for finding things at least, but what about replacing regex matches, which is my typical recourse for using a regex?

Well as with most things, someone else has already thought about this and I found this excellent post on Bridging Islands in Emacs by Karthink. I won’t repeat the contents here and recommend you read the article and absorb it into your configuration. You can see mine here.

Now when I use C-M-% which is the default for query-replace-rexp it instead calls re-builder and I can start writing my regular expression and see matches. Once I’ve got the regex correct simply hit Ret and I’m prompted to enter what I want to replace matches with.

One thing I’ve not quite sussed out yet is the different input methods of regex’s this takes, it seems more backslashes are required in one method than “standard” but re-builder translates things back to the standard notation on saving.

Tony Aldon has made a good video that shows how to use re-builder.

Yank from kill ring

Copying (yanking) and pasting (insert) is a common task in Emacs and a history of what you have copied is kept in the “kill-ring”. You can insert older items if you know their position in this history by preceding C-y with C-# (where #) represents the Nth most recent yank. But what if you can’t remember how many yanks back the item you want to insert is?

Obviously Emacs has your back as you can use M-y which pops up a buffer containing the kill-ring history which you can easily move through and hit Ret to insert the item you want. Simple but a game changer that I now use on a daily basis.

`use-package`

use-package is a commonly used package that helps simplify installing and configuring packages. I have the following snippet early in my init.el to ensure it is installed and can be used subsequently to load and configure all packages.

;;   https://ianyepan.github.io/posts/setting-up-use-package/
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

I noticed recently that the GitHub repo was archived in 2025-08-23 and the notice in the README.md indicates that it is now a core Emacs package meaning I can simply configure the package with the following instead (configuration details are already in the official documentation).

(use-package use-package
  :config
  (setq use-package-always-ensure t)
  (setq use-package-expand-minimally t))

Emacs pre-commit

I’m a big fan of pre-commit (see previous posts) and already use the lisp-format hook on my configuration files. Recently I clocked there was a new linter in the form of pre-commit-elisp which I’ve added to the hooks I use on my configuration repository. Configuration is simple by adding the following to .pre-commit-config.yaml.

  - repo: https://github.com/jamescherti/pre-commit-elisp
    rev: v1.0.0
    hooks:
      # Validate that all parentheses in .el files are correctly balanced
      - id: elisp-check-parens
      # Byte-compile .el files to identify compilation errors early
      # - id: elisp-byte-compile
      # Optional:
      # NOTE: This change the indentation.
      - id: elisp-indent

As you can see I opted not to byte-compile all files by commenting that line out.

Summary

Emacs is a 🐰 🕳️ but that is part of the fun and I always enjoy discovering new features and packages that make my life that little bit easier. Now all I need to do is go through and re-write my Emacs configuration as a literate org-mode and make sure I’m Using use-package the right way.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2025,
  author = {Shephard, Neil},
  title = {Emacs {Tips} \& {Tweaks} (2025-09-05)- Vundo, Regex Builder
    and More},
  date = {2025-09-05},
  url = {https://blog.nshephard.dev/posts/emacs_202509/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2025. “Emacs Tips & Tweaks (2025-09-05)- Vundo, Regex Builder and More.” September 5, 2025. https://blog.nshephard.dev/posts/emacs_202509/.

Debugging GitHub Actions with tmate

Neil Shephard — Tue, 10 Jun 2025 00:00:00 GMT

If you work with remote computer systems regularly you are probably familiar with SSH for making your connections. You might also have come across tmux, its forktmate or the GNU programme screen all of which allow you to maintain and/or share a session on a remote system.

Returning from Bowfell Buttress by Me

The ability to connect remotely to a machine is invaluable, but what does that have to do with debugging GitHub Actions? You have your development environment on your local computer where you have made sure all tests pass and the documentation builds. Well often tests are run on a multitiude of operating systems and you may not always have access to these locally, even with the availability of emulators, for example licensing issues may prevent you from emulating some systems.

This is where tmate comes in handy because there is a GitHub Actions which sets up tmate on the runner that was spun on the GitHub servers and if you’ve setup your SSH keys correctly then you can connect to this machine and set about debugging errors.

The official documentation is comprehensive and well written and I would recommend you read through it if you want to know more. Here I give a brief overview of how I use tmate in my work. Its pretty basic but has been invaluable.

Setup and Configuration

SSH Keys

It is essential that you have your public SSH key stored in your GitHub account. A while back GitHub started enforcing this for development so chances are you have already got this setup, if not follow the instructions . GitHub have an excellent series of posts on connecting to GitHub with SSH.

Work through those if you need to setup SSH. Once you have setup your SSH keys you are ready to add the tmate step to your GitHub Action.

Testing on multiple OS’s and versions

You can configure your GitHub Action to run on a range of operating systems by setting up single or multi-dimension matrix under the strategy. The example below¹ tests multiple operating systems and multiple versions of Python.

jobs:
  build:
    name: Test (${{ matrix.python-version }}, ${{ matrix.os }})
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: ["ubuntu-22.04", "ubuntu-20.04", "ubuntu-latest", "macos-latest", "windows-latest"]
        python-version: ["3.10", "3.11", "3.12", "3.13"]
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
    ...

`tmate` GitHub Action

To use tmate to debug your workflow you need to add the action-tmate to the .github/workflows/.yaml you wish to debug. There are a lot of options available but a very basic configuration is shown below and you should add this as the last step in your workflow.

steps:
  ...
  - name: Setup tmate session
    if: ${{ failure() }}
    uses: mxschmitt/action-tmate@v3
    timeout-minutes: 5

This will use the mxschmitt/action-tmate step only if there is a failure and will make the SSH session on the runner available for just 5 minutes. Its quite likely that it will take me than 5 minutes to work out what is causing the error and so after have encountered an error I typically increase this amount to allow me longer to work on debugging the problem.

Usage

Once you have staged and committed the changes to you have made to your workflow to include the action-tmate step using it is pretty straight-forward. If everything runs ok you will see this in your workflows log because the step only runs on failure.

No errors so the tmate action did not run

If however there is a failure in one of the preceding steps of your workflow then the tmate job kicks in and you will see the following in your jobs logs.

tmate kicks in on failure of a preceding job

The key bit of information here are the lines at the bottom which are repeated…

2025-06-03T15:29:21.4815179Z openssh-client is already the newest version (1:9.6p1-3ubuntu13.11).
2025-06-03T15:29:21.4816354Z xz-utils is already the newest version (5.6.1+really5.4.5-1ubuntu0.2).
2025-06-03T15:29:21.4817296Z 0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.
2025-06-03T15:29:22.4080716Z ssh 7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io
2025-06-03T15:29:22.4113791Z
2025-06-03T15:29:22.4125076Z SSH: ssh 7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io
2025-06-03T15:29:22.4125623Z or: ssh -i  7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io
2025-06-03T15:29:27.4141023Z SSH: ssh 7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io
2025-06-03T15:29:27.4141669Z or: ssh -i  7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io

These messages are shown after the mxschmitt/action-tmate job has installed tmate on the GitHub runner (/remote server) and it is now available for you to connect to using SSH. If you have your SSH key loaded in a Keychain² or other SSH key manager then you can use the first form.

ssh 7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io

If you get a Permission Denied error then try the second form, substituting for the private key that corresponds to the one you have on your GitHub account.

ssh -i  7Nus3JTkxyGCmscUBKKV8fp3z@nyc1.tmate.io

Once connected you have all the tools in the current version of the host operating system along with those your job installed, including the package and its dependencies you are testing or using in the workflow that has failed (remember this is the machine on which the job failed and your workflow sets up the environment for testing/building 😉).

You can now set about debugging why the job failed in more detail. You may want to run pytest --lf -x to re-run the last failed jobs and start poking around. If you use pdb to debug then you can add breakpoint() at the point you want to start debugging.

Astute readers will have noticed that you are time-limited as the configuration for this step in the workflow specified timeout-minutes: 5 which means the runner and your SSH connection will terminate after five minutes. Typically it’s going to take you longer to solve your problem so you should tweak your /.github/workflow/.yaml to increase this value on your local copy, commit and push the changes to your remote branch on GitHub where they will trigger another run of your jobs which should fail and you will have longer to spend connected debugging. Once you’ve solved the problem reduce it back to a few minutes.

Conclusion

Being able to remotely connect to GitHub runners allows you to investigate what went wrong quickly on operating systems you may not have at your disposal locally. Its a quick and efficient way to investigate in finer detail why tests and builds might be failing on a specific operating system or version of the software you are testing.

Footnotes

This is based on the TopoStats tests.yaml package that I work on.↩︎
Keychain’s are a convenience programmes for managing ssh-agent that are long-running. They mean you can enter your SSH password once, typically on login, and your private keys are held in memory and used whenever you attempt to connect via SSH to a remote machine.↩︎

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2025,
  author = {Shephard, Neil},
  title = {Debugging {GitHub} {Actions} with Tmate},
  date = {2025-06-10},
  url = {https://blog.nshephard.dev/posts/tmate/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2025. “Debugging GitHub Actions with Tmate.” June 10, 2025. https://blog.nshephard.dev/posts/tmate/.

Emacs Yasnippet

Neil Shephard — Wed, 07 May 2025 00:00:00 GMT

I’ve used Emacs for around 20 years but it wasn’t until about 8 years ago that it all started to click and I realised the benefit of having a highly customisable environment for doing all your development work, note taking, agenda organising (and much more) because its much more fun tinkering with your configuration than it is getting work done!

Recently I revisited Yasnippet and it finally clicked with me so I thought I’d write a short post as much to help me solidify my understanding as to perhaps help others.

Autumn Colours by Me

Yasnippet

What is YASnippet? Its a templating system for Emacs which saves you time on writing boiler-plate code/text. Whilst all the cool-kids these days might be “vibe-coding” with Large Language Models and paying Microsoft for the privilege of accessing Co-Pilot there is a lot to be said for understanding and crafting your own tools.

Installation

YASnippet is a frame work to make it useful you will want to also install yasnippet-templates.

Installation is pretty straight-forward and the same for all Emacs packages. Ideally you should have MELPA configured as a repository and you can then M-: package-install yasnippet and M-: package-install yasnippet-templates. I use use-package for installing and managing my packages so somewhere deep in my Emacs config I have the following.

(use-package yasnippet
  :ensure t
  :config
  (yas-reload-all)
  (add-hook 'prog-mode-hook #'yas-minor-mode)
  (add-hook 'markdown-mode-hook #'yas-minor-mode))

(use-package yasnippet-snippets
  :ensure t)

This installs both packages and adds a hook so that whenever the prog-mode-hook is called yas-minor-mode is enabled which is also enabled.

Inserting templates

Until you learn the muscle memory for the key-bindings for the various templates available in each mode your easiest option is to use the YASnippet menu that is available when yas-minor-mode is enabled in a buffer as there are too many snippets for to learn instantly (for me anyway).

As an example we will use Python, but most languages have an abundance of templates. You must of course have a buffer(/file) open in python-mode which is simple, just open anything.py and it should be in the correct mode. Lets open C-x C-f ~/tmp/example.py, if you don’t have YASnippet in your menu bar you can M-x yas-snippet-mode and it should appear.

Start by adding a module description to the top of the file.

"""This is an example Python file that we have populated with YASnippet templates."""

from dataclasses import dataclass

We now want to add a class definition. We could write it but lets use a template instead. As we don’t know the key-bindings (yet) we go use the menu and select YASnippet > python-mode > object orientated > dataclass and it will insert a class definition with the @dataclass decorator.

"""This is an example Python file that we have populated with YASnippet templates."""

from dataclasses import dataclass

@dataclass
class class:

The class name (class:) is highlighted but probably isn’t the name you want for your class, but start typing and it will be replaced. Here we enter person

"""This is an example Python file that we have populated with YASnippet templates."""

from dataclasses import dataclass

@dataclass
class person:

Using Key-bindings to insert

You might have noticed that the menu you opened and navigated through had a bunch of characters on the right-hand-side next to each command, these are the key-bindings you can use to insert the template automatically so it’s worth paying attention to these (for inserting this dataclass template the key-binding was dc but more on those below)

Once you are familiar with some of the snippets that you want to use you can learn the key-bindings to insert them. These are all documented in the snippets themselves so you can browse the YASnippet Templates repository and looking at each to find out the key-bindings.

We’ll add a dunder __eq__ method to the class but rather than go through the menus I’ll tell you the key-binding to use, type _eq and then hit and the template will be inserted. This is how you can type key-sequences and not always have them replaced by the template, you have to hit afterwards.

"""This is an example Python file that we have populated with YASnippet templates."""

from dataclasses import dataclass

@dataclass
class class:

    def __eq__(self, other):
        return

You now have a template with a skeleton for the __eq__ dunder method! Obviously the templating can’t know how you want to test for equality so you have to define that yourself which will be dependent on what your class is. For simplicity in this example we’ll just test that self == other which should return True or False.

"""This is an example Python file that we have populated with YASnippet templates."""

from dataclasses import dataclass

@dataclass
class class:

    def __eq__(self, other):
        return self == other

Creating your own templates

My main motivation for looking at this system was that I made a dumb mistake when writing some Python code. I was using the getter/setter design pattern in Python to make attributes using the @property fixture and stupidly didn’t include a return self._some_attribute in the getter and wasted too much time looking for my error before a colleague kindly pointed out my error (thanks Sylvia 🙏).

I therefore set about writing my own template to insert a decorated getter and associated setter method for Python classes and it was incredibly easy. I place the following code in ~/.config/emacs/snippets/python-mode/setter which binds to the sg key sequence and it “Just Worked(TM)”!

# -*- mode: snippet -*-
# name: setter-getter
# key: sg
# expand-env: ((yas-indent-line 'fixed))
# --
    @property
    def ${1:attribute}(self) -> ${2:type}:
        """
        Greeter for the ''${1}'' attribute.

        Returns
        -------
        ${2}
            Returns the value of ''${1}''.
        """
        return self._${1}

    @${1}.setter
    def ${1}(self, value: ${2}) -> None:
        """
        Setter for the ''${1}'' attribute.

        Parameters
        ----------
        value : ${2}
            Value to set for ${1}.
        """
self._${1} = value

There is a lot going on here, there is a header (lines starting with #) and then the snippet itself.

The header has some key/value pairs, the name is setter-getter and this is bound to the key-sequence sg so we know that we can type sg to insert the template. I’ve not yet sussed out the expand-env as I copied it from an existing template but I think it preserves the indenting.

The template then follows and it’s a bit more complex than the examples so far. What does ${1:attribute}, ${2:type} and the related ${1} and ${2} mean/do? Well these are ways of inserting names of your own choosing to and having them filled in automatically so you don’t have to find/replace them after inserting the template.

Lets give it a go. Insert the template by hitting sg, you should have the following inserted under your Class definition.

        @property
        def attribute(self) -> type:
            """
            Greeter for the ''attribute'' attribute.

            Returns
            -------
            type
                Returns the value of ''attribute''.
            """
            return self._attribute

        @attribute.setter
        def attribute(self, value: type) -> None:
            """
            Setter for the ''attribute'' attribute.

            Parameters
            ----------
            value : type
                Value to set for attribute.
            """
            self._attribute = value

This template is as it appears in the definition, but two words are highlighted, the attribute and type which if we look back at our template are the values associated with ${1} and ${2} respectively. As with inserting the class earlier we can start typing where the cursor is, adjacent to the first occurrence of attribute and Emacs/YASnippet will auto-magically replace all occurrences with what you type, here I’ve typed age…

    @property
    def age(self) -> type:
        """
        Greeter for the ''age'' attribute.

        Returns
        -------
        type
            Returns the value of ''age''.
        """
        return self._age

    @age.setter
    def age(self, value: type) -> None:
        """
        Setter for the ''age'' attribute.

        Parameters
        ----------
        value : type
            Value to set for age.
        """
        self._age = value

Hit again and the cursor moves to the next “parameter” in our template, in this example type and you can again start typing and Emacs/YASnippet will replace it with whatever you type, I’ve opted for int | float.

    @property
    def age(self) -> int | float:
        """
        Greeter for the ''age'' attribute.

        Returns
        -------
        int | float
            Returns the value of ''age''.
        """
        return self._age

    @age.setter
    def age(self, value: int | float) -> None:
        """
        Setter for the ''age'' attribute.

        Parameters
        ----------
        value : int | float
            Value to set for age.
        """
        self._age = value

Hit again and the cursor moves to the end of the template and you are done. You can go back and edit the docstrings if you want to make them more informative if you want.

More Templates

YASnippet templates are not the only resource, there are others listed on the YASnippet GitHub Page page under Where are the snippets? which I haven’t investigated yet as I’ve only so much time to spend in front of a computer.

Summary

YASnippet are a really powerful tool for templating common boiler plate code and can save you a considerable number of key-strokes by using a templates. Whilst there may be an overhead in learning and memorising the keystrokes for doing so the menu system is a saviour (as the author of Mastering Emacs advocates for learning Emacs). Best of all you are free to create your own templates that are customised to your own requirements and I would advocate that learning this is a better way of spending your time than asking a statistical model ever more refined questions until you get the answer you want (aka “prompt engineering” or “vibe coding”).

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2025,
  author = {Shephard, Neil},
  title = {Emacs {Yasnippet}},
  date = {2025-05-07},
  url = {https://blog.nshephard.dev/posts/emacs-yasnippet/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2025. “Emacs Yasnippet.” May 7, 2025. https://blog.nshephard.dev/posts/emacs-yasnippet/.

Version Control with Jujutsu

Neil Shephard — Sat, 08 Feb 2025 00:00:00 GMT

I’d bread about and heard a good things about Jujutsu, a new Version Control System developed by Martin von Zweigbergk at Google, although it is not an official Google product and is Open Source Software licensed under the Apache 2.0 license. It has its own internal/native backend but also supports using Git in the background to undertake all tasks. As such it can be used as a drop-in replacement for Git. A big difference from the familiar Git model is that the working copy is automatically committed. If you check out a commit this introduces a new working-copy commit where changes are recorded (no more detached HEAD state!). I recently purchased a refurbished ThinkPad T490s and decided to give NixOS a whirl so what better opportunity to try Jujutsu than to use it to version control my /etc/nixos directory.

This post covers my experience of getting started with Jujutsu.

Installation

Whilst notionally installing and using it on NixOS I use both Gentoo and Arch Linux so have installed it on those systems too.

Gentoo

jj is available in the Gentoo Overlays guru.

eselect repository enable guru
eix-sync
emerge -av dev-vcs/jj

NixOS

jj is also available for NixOS. You can use flake or HomeManager, I opted for the former and added the following to /etc/nixos/flake.nix. Just add the following to /etc/nixos/configuration.nix to install the most recent release system wide.

environment.systemPackages = with pkgs{
...
jujutsu
...
};

Then either nixos-rebuild test to install and test the installation (you should now have jj in your command line) and if happy nixos-rebuild switch to make the change permanent and available on the next boot.

Arch Linux

Stable version is included in packages, although you can keep track of development using the AUR _jj-git package.

pacman -Syu jj

Shell Integration

There is integration with both Bash and ZSH as well as other shells.

Bash

source <(jj util completion bash)

ZSH

autoload -U compinit
compinit
source <(jj util completion zsh)

Initial Configuration

As with Git you need to add your user.name and user.email to the configuration.

jj config set --user user.name "Your Name"
jj config set --user user.email "your@email.address"

We can check the configuration…

```sh
jj config list
ui.editor = "/usr/bin/nano"
ui.pager = "less"
user.email = "nshephard@protonmail.com"
user.name = "Neil Shephard"

This looks ok but I found that /usr/bin/nano is wrong for a NixOS system….

which nano
/run/current-system-sw/bin/nano

..so updated that along with a switch from less to most for paging.

jj config set --user ui.editor "/run/current-system/sw/bin/nano"
config set --user ui.pager "/run/current-system/sw/bin/most"

Usage

I wanted to keep /etc/nixos under version control using jj and as advised in Steve’s Jujutsu Tutorial opted to use the Git backend.

cd /etc/nixos
jj git init
jj st
    Working copy changes:
    A agenix.nix
    A audio.nix
    A bluetooth.nix
    A configuration.nix
    A configuration.nix~
    A flake.lock
    A flake.nix
    A flake.nix~
    A hardware-configuration.nix
    A home.nix
    A home.nix~
    A luks.nix
    A secrets/secrets.nix
    A secrets/slack-vpn.age
    A xfce.nix
    A xfce.nix~
    Working copy : vpqtltus 35cc03d2 (no description set)
    Parent commit: zzzzzzzz 00000000 (empty) (no description set)

First things first, I’d already edited some files using Emacs and didn’t want the temporary files it leaves behind included. According to the documentation there is no .jjignore yet so we use .gitignore instead listing our files and patterns there.

*~
\#*

…but the files are already being tracked because they were present when the repository was initialised. We therefore need to untrack them with…

jj file untrack

This failed because ~ is used in the syntax for jj filesets which is a method of defining patterns of files. I tried a few things but in the end couldn’t suss it out in the five minutes so took the brute force option of rm -rf .jj and initialising the repository anew. We can look at the status with jj st and it will use our configured pager (which I’ve set to most) to show the changes.

jj st
Working copy changes:
A .gitignore
A agenix.nix
A audio.nix
A bluetooth.nix
A configuration.nix
A flake.lock
A flake.nix
A hardware-configuration.nix
A home.nix
A luks.nix
A secrets/secrets.nix
A secrets/slack-vpn.age
A xfce.nix
Working copy : nxzzlvzo 43e6338e (no description set)
Parent commit: zzzzzzzz 00000000 (empty) (no description set)

We can see the changes (i.e. all new files) are already noted as being under the working copy. We can also use jj describe to look at the changes and add a description. If we use the -m "A message" flag and value we can add a message and it will replace the no description set shown by jj st. With jj describe the information is opened up in an editor, and if a message has already been set it will be shown at the top. Note that lines beginning with JJ will be removed (i.e. they are comment lines). On adding a message or changing it the commit ID changes, the change ID remains the same but the commit ID changes over time allowing us to refer to individual commits rather than a whole change set.

jj describe -m "Initial commit with jj :)"
Working copy now at: nxzzlvzo d2d192ec Initial commit with jj :)
Parent commit      : zzzzzzzz 00000000 (empty) (no description set)

New commits

We’re ready to make some changes, but unlike Git we can make our commit first rather than after having made the changes. We do this with jj new

jj new
Working copy now at: mkrknnyv d20e2368 (empty) (no description set)
Parent commit      : nxzzlvzo d2d192ec Initial commit with jj :)

We can now modify a file, in this case I tidied up /etc/nixos/configuration.nix and put all network.* options within a network = {...}; block and similar aggregated all nix.* options into a nix = {...}; block.

jj st
Working copy changes:
M configuration.nix
Working copy : mkrknnyv 31f1c759 (no description set)
Parent commit: nxzzlvzo d2d192ec Initial commit with jj :)

We can see the full commit history with jj log (no surprisese there!)

@  mkrknnyv nshephard@protonmail.com 2024-12-21 16:20:53 31f1c759
│  (no description set)
○  nxzzlvzo nshephard@protonmail.com 2024-12-21 15:48:47 d2d192ec
│  Initial commit with jj :)
◆  zzzzzzzz root() 00000000

It’s interesting to note that the bold/highlighting of the start of commit hashes gives you an indication of the unique component of that hash (but you’ll have to take my word for that as I’ve not bothered to copy that over to the blog!).

It is important to note that there is no need to explicitly make a commit, the work done/changes are already part of the current commit. When you are ready to start the next piece of work you jj new (optionally with -m "") to start a new piece of work.

This naturally leads to the question of how to undo work that you have done With jj you can move back to commits using jj edit @- or referring to the commit directly with jj edit and then use

jj edit @-
jj abandon <hash_of_latest_commit>

Diffing

You can view differences with jj diff and it will show the differences between the current “HEAD” and the previous commit. I use difftastic (see Configuration section below) so have colourized output which isn’t shown below.

    home.nix --- Nix
    152 152       urldecode = "python3 -c 'import sys, urllib.parse as ul; print(ul.unquote_plus(sys.stdin.read()))'";
    153 153       urlencode = "python3 -c 'import sys, urllib.parse as ul; print(ul.quote_plus(sys.stdin.read()))'";
    154 154     };
    ... 155     # initExtra = ''
    ... 156     #   if command -v keychain > /dev/null 2>&1; then eval $(keychain --eval --nogui ${keyFilename} --quiet); fi
    ... 157     # '';
    155 158   };
    156 159
    157 160   programs.emacs = {

    configuration.nix --- 1/2 --- Nix
    177   # Some programs need SUID wrappers, can be configured further or are      177   # Some programs need SUID wrappers, can be configured further or are
    178   # started in user sessions.                                               178   # started in user sessions.
    179   # programs.mtr.enable = true;                                             179   # programs.mtr.enable = true;
    180   programs.gnupg.agent = {                                                  180   programs = {
    ...                                                                             181     gnupg.agent = {
    181     enable = true;                                                          182       enable = true;
    ...                                                                             183       # enableSSHSupport = true;
    ...                                                                             184     };
    182     enableSSHSupport = true;                                                185     ssh.startAgent = true;
    183   };                                                                        186   };
    184                                                                             187
    185   # List services that you want to enable:                                  188   # List services that you want to enable:
    186   services = {                                                              189   services = {

    configuration.nix --- 2/2 --- Nix
    210 213     fprintd = {
    211 214       enable = true;
    212 215     };
    ... 216     # yubikey
    ... 217     yubikey-agent = {
    ... 218       enable = true;
    ... 219     };
    213 220   };
    214 221   # Open ports in the firewall.
    215 222   # networking.firewall.allowedTCPPorts = [ ... ];

If you want to look at differences between two specific commits you can use the --from and --to options (the former likely being more useful than the later).

Remotes

I wanted to back my work up remotely and have a few options the ubiquitous GitHub, GitLab, or my self-hosted Forgejo. I opted for the later which is hosted on the VPS I pay for with OVH.

jj git remote add origin <git@forgejo.nshephard.dev>:nshephard/crow.git

However trying to push failed with a rather cryptic and unhelpful message.

jj git push
Changes to push to origin:
  Add bookmark trunk to bfce9c9ab2aa
Error: failed to connect to forgejo.nshephard.dev: Invalid argument; class=Os (2)

I use a non-standard port for SSH on my server (i.e. not 22 ). The “trick” here was to use the scp like syntax to specifying the url under the remote in the git configuration which resides in .jj/repo/store/git/config

[remote "origin"]
    url = ssh://git@forgejo.nshephard.dev:2222/~/nshephard/crow.git

Success, I can reach the remote, but it fails to authenticate.

jj git push
Changes to push to origin:
  Add bookmark trunk to bfce9c9ab2aa
Error: failed to authenticate SSH session: Unable to extract public key from private key file: Wrong passphrase or invalid/unrecognized private key file format; class=Ssh (23)
Hint: Jujutsu uses libssh2, which doesn't respect ~/.ssh/config. Does `ssh -F /dev/null` to the host work?

Checking my ~/.ssh/config and my Forgejo configuration and I realised that I have it configured to run as user forgejo.

[remote "origin"]
    url = ssh://forgejo@forgejo.nshephard.dev:2222/nshephard/crow.git

Bookmarks (aka branches)

These are mainly for compatibility with Git, jj actually prefers to use anonymous rather than named branches (sometimes called a “branchless” workflow). You create a bookmark at a given point and it stays there until you move it. This is kind of weird compared to Git where commits are stacked on top of each other to make branches and you are always checked out on the HEAD commit at the top or otherwise in a “detached” status.

Create a bookmark with…

jj bookmark create <name>

If you want to move a bookmark after its creation you can do so…

jj bookmark move <bookmark_name> --to <revision>

Note that the default --to is @ so jj bookrmark move will move it to your current location, whether that is the tip or not.

Merging Branches

I found when it came to pushing to my Forgejo instance where I had created the repository I had to first jj git pull to get the initial commit there and then setup remote tracking.

jj git push
    Warning: Non-tracking remote bookmark trunk@origin exists
    Hint: Run `jj bookmark track trunk@origin` to import the remote bookmark.
    Nothing changed.

jj git fetch
    bookmark: trunk@origin [new] untracked

jj bookmark track trunk@origin

    jj log
    ◆  myqxkksp nshephard@noreply.forgejo.nshephard.dev 2024-12-21 15:47:13 trunk@origin 12c4747e
    │  Initial commit
    │ @  tossulss nshephard@protonmail.com 2024-12-23 07:28:16 trunk bfce9c9a
    │ │  Add pcscd to services for GnuPG pinentry
    │ ○  olrmoynt nshephard@protonmail.com 2024-12-22 23:03:58 dcd199d9
    │ │  Adding tree to systemPackages
    │ ○  wpttsonz nshephard@protonmail.com 2024-12-22 22:39:32 ca5be2ed
    │ │  ZSH home.nix configuration
    │ ○  twrvtqty nshephard@protonmail.com 2024-12-22 12:27:39 bb4c6bca
    │ │  system: emacs daemon for user
    │ ○  xuzumvqs nshephard@protonmail.com 2024-12-21 22:56:16 ee5dd297
    │ │  Add btop and htop to system.Packages
    │ ○  zrxyptxn nshephard@protonmail.com 2024-12-21 22:47:29 75fbf987
    │ │  Minor tweaks to mark ends of blocks in xfce.nix
    │ ○  zkowzsvy nshephard@protonmail.com 2024-12-21 20:41:10 c20e2a66
    │ │  Adding difftastic
    │ ○  mkrknnyv nshephard@protonmail.com 2024-12-21 16:31:47 18b4e7b1
    │ │  Tidying up nix and network sections
    │ ○  nxzzlvzo nshephard@protonmail.com 2024-12-21 15:48:47 d2d192ec
    ├─╯  Initial commit with jj :)
    ◆  zzzzzzzz root() 00000000
    #+end_

At this point the two “branches” (trunk@origin and the local trunk) have diverged and are in conflict, preventing me from pushing

jj git push Warning: Bookmark trunk is conflicted Hint: Run jj bookmark list to inspect, and use jj bookmark set to fix it up. Nothing changed.

There is an old command in Jujutsu to jj merge but, as the help informs you, it has been deprecated in favour of jj new. This isn’t too dissimilar to Git though since “merges” are just commits that bring two branches together. The syntax for this is jj new [OPTIONS] [REVISIONS], by default the REVISIONS is simply @ the current “HEAD”, but specifying more than one will merge the two together. You can of course include -m "Message about merging". Taking the above output from jj log I can make a merge with the following (the minimal hashes are highlighted in the terminal but not above and here the to my refers to merging to the latest commit).

jj new -m "merge: local work with remote init" to my
Working copy now at: oupkqwzo da0ebd37 (conflict) (empty) merge: local work with remote init
Parent commit      : tossulss bfce9c9a trunk?? | Add pcscd to services for GnuPG pinentry
Parent commit      : myqxkksp 12c4747e trunk?? trunk@origin | Initial commit
Added 2 files, modified 1 files, removed 0 files
There are unresolved conflicts at these paths:
.gitignore    2-sided conflict

Conflicts

The manual covers conflict resolution and its worth reading that. That I encountered merge conflicts isn’t entirely unexpected I had created .gitignore both locally and on the remote so bringing them together the is natural. Lets look at this…

cat .gitignore

<<<<<<< Conflict 1 of 1
%%%%%%% Changes from base to side #1
+## Emacs temporary files
+*~
+\#*
+++++++ Contents of side #2
# ---> Nix
# Ignore build outputs from performing a nix-build or `nix build` command
result
result-*

>>>>>>> Conflict 1 of 1 ends

This is fairly similar to Git merge conflicts, but I like the side 1~/~side 2 notation (you get the same in Git by default but its occluded and you have to read up to understand that first bit delimited by ‘<<<<<<<<’ is from the current branch and the other bit is from the branch that is being merged).

I know I want both of these included in .gitignore so I make the changes, removing all the conflict markup and save the file. This tidies up the current commit, there is no need to make another commit to take a snapshot of those changes as there is in Git. However the bookmarks are still in conflict so we need to set that to the correct commit.

jj st

Working copy changes:
M .gitignore
Working copy : oupkqwzo f10df751 merge: local work with remote init
Parent commit: tossulss bfce9c9a trunk?? | Add pcscd to services for GnuPG pinentry
Parent commit: myqxkksp 12c4747e trunk?? trunk@origin | Initial commit
These bookmarks have conflicts:
trunk
Use `jj bookmark list` to see details. Use `jj bookmark set <name> -r <rev>` to resolve.

jj bookmark set trunk -r ou

jj st

Working copy changes:
M .gitignore
Working copy : oupkqwzo f10df751 trunk* | merge: local work with remo
Parent commit: tossulss bfce9c9a Add pcscd to services for GnuPG pine
Parent commit: myqxkksp 12c4747e trunk@origin | Initial commit

jj log

    @    oupkqwzo nshephard@protonmail.com 2024-12-23 11:51:49 trunk* f10df751
    ├─╮  merge: local work with remote init
    │ ◆  myqxkksp nshephard@noreply.forgejo.nshephard.dev 2024-12-21 15:47:13 trunk@origin 12c4747e
    │ │  Initial commit
    ○ │  tossulss nshephard@protonmail.com 2024-12-23 07:28:16 bfce9c9a
    │ │  Add pcscd to services for GnuPG pinentry
    ○ │  olrmoynt nshephard@protonmail.com 2024-12-22 23:03:58 dcd199d9
    │ │  Adding tree to systemPackages
    ○ │  wpttsonz nshephard@protonmail.com 2024-12-22 22:39:32 ca5be2ed
    │ │  ZSH home.nix configuration
    ○ │  twrvtqty nshephard@protonmail.com 2024-12-22 12:27:39 bb4c6bca
    │ │  system: emacs daemon for user
    ○ │  xuzumvqs nshephard@protonmail.com 2024-12-21 22:56:16 ee5dd297
    │ │  Add btop and htop to system.Packages
    ○ │  zrxyptxn nshephard@protonmail.com 2024-12-21 22:47:29 75fbf987
    │ │  Minor tweaks to mark ends of blocks in xfce.nix
    ○ │  zkowzsvy nshephard@protonmail.com 2024-12-21 20:41:10 c20e2a66
    │ │  Adding difftastic
    ○ │  mkrknnyv nshephard@protonmail.com 2024-12-21 16:31:47 18b4e7b1
    │ │  Tidying up nix and network sections
    ○ │  nxzzlvzo nshephard@protonmail.com 2024-12-21 15:48:47 d2d192ec
    ├─╯  Initial commit with jj :)
    ◆  zzzzzzzz root() 00000000

We’ve merged out branches but trunk@origin is behind that merge we can bring that up-to-date by pushing

jj git push

Changes to push to origin:
  Move forward bookmark trunk from 12c4747edb21 to f10df751ab04
Warning: The working-copy commit in workspace 'default' became immutable, so a new commit has been created on top of it.
Working copy now at: tuwxwnqw 4e3890c1 (empty) (no description set)
Parent commit      : oupkqwzo f10df751 trunk | merge: local work with remote init

jj log

    @  tuwxwnqw nshephard@protonmail.com 2024-12-23 11:58:36 4e3890c1
    │  (empty) (no description set)
    ◆  oupkqwzo nshephard@protonmail.com 2024-12-23 11:51:49 trunk f10df751
    │  merge: local work with remote init
    ~

Not sure where the rest of the commit history is but it is showing up on the ForgeJo repository commit history. I’ll return to that later.

Revisions and Revsets

A revision set or “revset” is a range of commits and jj has its own language for describing refsets.

Symbols

We’ve already encountered @ which points to our current working copy that we have checked out (sometimes “HEAD” but could be elsewhere in history).

Operators

The tutorial notes the following common operators.

x & y: changes that are in both x and y.
x | y: changes that are in either x or y.
::x Ancestors of x.
x:: Descendants of x.

We found that we couldn’t review the history of the current checked out commit (@) any more but lets see if we can use this new knowledge to find view the log history. We want to look at all ancestors so we can use ::t to view the ancestors of the most recent, empty, commit.

****NB**** It might be worth adding a description with jj describe before undertaking work, remember that the changes in the working directory are always part of the current commit.

jj log -r ::t
@  tuwxwnqw nshephard@protonmail.com 2024-12-23 11:58:36 4e3890c1
│  (empty) (no description set)
◆    oupkqwzo nshephard@protonmail.com 2024-12-23 11:51:49 trunk f10df751
├─╮  merge: local work with remote init
│ ◆  myqxkksp nshephard@noreply.forgejo.nshephard.dev 2024-12-21 15:47:13 12c4747e
│ │  Initial commit
◆ │  tossulss nshephard@protonmail.com 2024-12-23 07:28:16 bfce9c9a
│ │  Add pcscd to services for GnuPG pinentry
◆ │  olrmoynt nshephard@protonmail.com 2024-12-22 23:03:58 dcd199d9
│ │  Adding tree to systemPackages
◆ │  wpttsonz nshephard@protonmail.com 2024-12-22 22:39:32 ca5be2ed
│ │  ZSH home.nix configuration
◆ │  twrvtqty nshephard@protonmail.com 2024-12-22 12:27:39 bb4c6bca
│ │  system: emacs daemon for user
◆ │  xuzumvqs nshephard@protonmail.com 2024-12-21 22:56:16 ee5dd297
│ │  Add btop and htop to system.Packages
◆ │  zrxyptxn nshephard@protonmail.com 2024-12-21 22:47:29 75fbf987
│ │  Minor tweaks to mark ends of blocks in xfce.nix
◆ │  zkowzsvy nshephard@protonmail.com 2024-12-21 20:41:10 c20e2a66
│ │  Adding difftastic
◆ │  mkrknnyv nshephard@protonmail.com 2024-12-21 16:31:47 18b4e7b1
│ │  Tidying up nix and network sections
◆ │  nxzzlvzo nshephard@protonmail.com 2024-12-21 15:48:47 d2d192ec
├─╯  Initial commit with jj :)
◆  zzzzzzzz root() 00000000

Functions

The revset language also includes a number of functions that help filter log messages such as author(), description(), ancestors(x, depth) (an extended version of ::x) and parents().

I’m not going to dig too deep into these at the moment as I have limited use for them right now but see the Figuring out where our changes are with revsets - Steve’s Jujutsu Tutorial and the Revset language of the official documentation.

Configuration

You can edit the configuration either at the --user or --repo level with jj config edit --[user|repo] (to find the path of the users configuration file use jj config path --user, repository configuration is in .jj/repo/config.toml). These are TOML files.

I enabled color using the brilliant difftastic

[user]
    name = "Neil Shephard"
    email = "nshephard@protonmail.com"

[ui]
    editor = "/run/current-system/sw/bin/nano"
    pager = "/run/current-system/sw/bin/most"
    color = "always"
    ## Use Difftastic by default
    diff.tool = ["difft", "--color=always", "$left", "$right"]

…there are a lot more configuration options available (see configuration documentation for full details).

Workflow

Two popular workflows are described in the tutorial, the Squash Workflow and the Edit Workflow.

Squash Workflow

This is kind of link git commit --amend where changes are added to the existing HEAD commit of the branch. The jj workflow has at it’s head (denoted by @ in the jj log output) the “unstaged” changes and jj squash adds them to the previous commit, which is typically created before making any changes with a description of the intended work (you could do this with git commit -a --allow-empty -m "bug: I'm going to squash a bug!" ) and then repeatedly git commit --amend as we complete the work. With jj squash workflow though it encourages making smaller more atomic commits and reduces the amount of “/fixing an error/typo” commits by those averse to using --amend. By default all files are included but you can specify just those files you want to include by listing them.

Jujutsu also allows interactive selection of lines to edit via the -i flag. A terminal interface opens and it is possible to select which lines to include prior to making the commit. After having selected all the changes simply hit `c` to confirm them.

If you decide you don’t want to keep the work you can jj abandon the work in progress and it reverts all changes. In fact jj squash offers much of the functionality of git rebase -i.

Edit Workflow

Continuing from the previous example we make some more changes, but rather than using new, because there is already an empty change there as we squashed the existing changes into the previous commit leaving @ empty, we use jj describe -m "message" to add a message to the empty commit that we are not going to squash. Now make the changes and when ready to start a new piece of work you can use jj new -m "".

Editing older commits

In Git this can be done either by adding a git commit --fixup or using git rebase -i tp interactively squash commits. In jj though we can use jj new -B @ -m "a new message" and what this does is add a new commit before the ~@ commit (other references can be used if you want to modify a commit further back in the commit history). You get for free a rebase of descendant commits, of course conflicts can arise but this command will always complete without resolving the conflicts (yet!).

The “HEAD” of the “branch” has been moved to this commit and changes can be made and saved (they’re already included as there is no staging in Jujutsu). When done you can return to the “HEAD” using jj edit or the convenience shortcut jj next --edit which moves @ to the “child” commit and allows editing.

You can edit earlier commits with jj edit @- for the previous commit or jj edit

IDE/Interfaces

Being an Emacs user I naturally wanted to use Jujutsu via Emacs and was hoping for a Magit equivalent. Being considerably newer there isn’t anything quite as powerful as Magit just yet but there is work in progress in the form of jujutsushi - A emacs interface to jujutsu and jujutsu.el: An Emacs interface for jujutsu (although the former’s author has stated they have deprioritized development in light of the later, see here).

The wiki is a useful resource on IDE integration

Conclusion

After a few weeks or so tinkering with Jujutsu/_jj I’ve found there are a number of features that differ from my Git experience to date. Having a mental model of Version Control is important for these to make sense. Obviously I need to spend longer working with the system to have a deeper understanding and appreciation of how it works and a better comparison to Git, but first impressions are good, although switching full scale would mean abandoning the amazing Magit which is one of the best Emacs packages going, but there are some Emacs packages for working with Jujutsu in the pipeline.

All changes are “staged”.
Branch names are redundant but are available (as “bookmarks”) for compatibility with Git and these need updating to the most recent commit.
Moving around commits seems more intuitive and there is no warning about the dangers of being in a “detached HEAD” state.
As a consequence it’s easy to update changes that should have been in older commits.
When this happens rebasing descendent commits is free, even if conflicts arise, they are still committed. They will need resolving eventually but you can do this once on the commit you wish to rather than repeatedly and having to rely on git rerere

I’d highly recommend reading some other people’s blogs on Jujutsu for a more technical understanding and broader view. In particular I liked the post Jujutsu VCS Introduction and Patterns | Kuba Martin.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2025,
  author = {Shephard, Neil},
  title = {Version {Control} with {Jujutsu}},
  date = {2025-02-08},
  url = {https://blog.nshephard.dev/posts/jujutsu/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2025. “Version Control with Jujutsu.” February 8, 2025. https://blog.nshephard.dev/posts/jujutsu/.

Quarto Custom domain on GitHub Pages

Neil Shephard — Thu, 14 Nov 2024 00:00:00 GMT

This blog is published using the excellent Quarto framework to GitHub Pages which offers free website hosting. However, astute readers familiar with this sort of setup will have noticed that the domain is no the usual [github-user].github.io and instead it resolves to blog.nshephard.dev. This post explains how to achieve that as it took me a little bit of work to sort out properly.

The Bernia Ridge by Me

Why?

I found that despite setting a custom domain in the GitHub pages Settings (Settings > Pages > Custom domain) each time I published the site (e.g. on a new blog post) the CNAME file disappeared from the gh-pages branch and the domain didn’t resolve. I searched but couldn’t find a solution to this so raised an issue to include it in the quarto-actions.

In the meantime I added a custom step to the publish.yaml to add the CNAME file to the gh-pages branch after Render and Publish which worked, but I was subsequently pointed to a cleaner existing way of achieving the same result.

What isn’t covered

This post doesn’t cover…

How to setup a blog/website using Quarto.
How to write Quarto Markdown.
How to publish a blog or website using Quarto, they have excellent documentation
Setting a CNAME with your DNS registrar.

On this last point, how you set your CNAME to redirect your custom domain to that of the GitHub Pages is dependent on the domain registrar service you use. I use OVH.

What is covered

How to add a CNAME file to the gh-pages branch that is produced by the quarto-dev/quarto-actions/publish GitHub Action so that the pages resolve to a custom domain.

The Problem

If you use plain GitHub Pages then you can set a custom CNAME by going to Settings > Pages > Custom Domain and entering the address there. However, as documented (see item 4) this doesn’t work if you use a custom GitHub Action to publish your website, which is the case when you use the quarto-dev/quarto-actions/publish GitHub Action to publish using Quarto.

Why? Because each time the action runs it regenerates the content of the gh-pages branch on the runner and pushes it to your repository and doesn’t include a custom CNAME file.

Solution 1

The solution I initially hit upon was to add an extra step to my .github/workflows/publish.yaml that…

Checks out the gh-pages branch.
Creates the CNAME file with the domain I use.
Adds this to the gh-pages branch.
Pushes back up-stream.

You should already have added a publish action to your repository which runs the quarto-dev/quarto-actions/publish GitHub Action. This runs quarto publish $TARGET where $TARGET is set by the chosen output configured in the target argument. In this example gh-pages

      - name: Render and Publish
        uses: quarto-dev/quarto-actions/publish@v2
        with:
          target: gh-pages
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

After this you should add the following, substituting blog.nshephard.dev for your own domain.

      - name: Add CNAME file to gh-pages branch
        run: |
          rm renv/activate.R
          git switch gh-pages
          git pull
          echo 'blog.nshephard.dev > CNAME
          git add CNAME
          git commit -m "Adding CNAME"
          git push

This…

Removes the lingering renv/activate.R which prevents switching branches.
Switches branches to gh-pages.
git pull to get the just published version of the branch.
Creates the CNAME file with the domain you specify.
Adds (stages) the CNAME file to the gh-pages branch.
Commits the change.
Pushes the commit to origin (i.e. the gh-pages branch on GitHub)

This worked as the CNAME file is added each time the workflow runs but, unsurprisingly, there is a simpler solution.

Solution 2 - The correct way of doing this

Turns out the answer provided by @cscheid is, unsurprisingly, much simpler.

You need to add CNAME in the _quarto.yaml so that it is included in the project.

project:
  type: website
  resources:
    ...
    - "CNAME"

You also have to create and add the CNAME file with your domain to the repository.

echo 'blog.nshephard.dev` > CNAME
git add CNAME _quarto.yaml
git commit -m "Adding CNAME to repository"
git push

Then, because it’s included in the resources, the file will carry through/persist when the publishing action runs to create the gh-pages branch with each new update and run on GitHub Actions :magic:.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Quarto {Custom} Domain on {GitHub} {Pages}},
  date = {2024-11-14},
  url = {https://blog.nshephard.dev/posts/quarto-cname/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Quarto Custom Domain on GitHub Pages.” November 14, 2024. https://blog.nshephard.dev/posts/quarto-cname/.

Pytest Matplotlib

Neil Shephard — Tue, 24 Sep 2024 00:00:00 GMT

Pytest is an excellent framework for writing tests in Python. Sometimes the code you want to test will generate images using Matplotlib or related libraries based around it such as Seaborn or plotnine and ideally your pytest suite should check that your functions generate the images you expect. There is an excellent pytest plugin to help with this pytest-mpl and this post covers how to use it and a very useful feature to help you investigate differences in images when your tests of images fail.

Histograms (not plotted using Matplotlib!) by Me.

Installation

Setup a new virtual environment, here I use virtualenvwrapper to create a temporary virtual environment (which will be deleted on deactivation) and install the pytest-mpl package (it pulls in matplotlib, pytest and a bunch of other dependencies).

mktmpenv
pip install pytest-mpl

Function and Test

We need a simple example to write tests for. As with previous posts I’ve made the code available in the pytest-examples repository. You can clone this repository to follow along and play around with this worked example.

git clone git@github.com:ns-rse/pytest-examples.git
cd pytest-examples

`scatter()`

This repository contains the module pytestexample/mpl_example.py and the function scatter() to plot a scatter-plot of two random variables along with a few common options.

#| label: python-test
#| code-fold: true
#| code-link: true
#| code-tools: true
#| eval: false
#| echo: true
"""Example code for pytest-mpl exposition."""

import numpy as np
import numpy.typing as npt
import matplotlib.pyplot as plt


def scatter(
    n_obs: int,
    figsize: tuple[int, int],
    title: str | None = None,
    seed: int = 3513387,
) -> tuple:
    """
    Generate a scatter plot of two random variables.


    Parameters
    ----------
    n_obs : int
        Number of random observations to generate.
    figsize : tuple[int, int]
        Shape to plot.
    seed : int
        Seed for pseudo-random number generation.

    Returns
    -------
    tuple(fig, ax)

    """
    # Generate two random sets of numbers
    rng = np.random.default_rng(seed)
    x = np.random.randn(n_obs)
    y = np.random.randn(n_obs)
    # Create the figure
    fig = plt.figure(figsize=figsize)
    ax = fig.add_subplot()
    ax.scatter(x, y)
    plt.title(title)

    return (fig, ax)

Defining a test

We now need a test to check that, for a given set of inputs, the same output is always returned. Following the naming conventions used by pytest we create this at tests/test_mpl_example.py. Matplotlib provides its own image_comparison decorator from Matplotlib itself which requires at a bare minimum the baseline_image. I’m going to skip over its usage and instead introduce the pytest-mpl extension as it offers a few extra features which are really neat. Instead of using the @image_comparsion() decorator we can add the pytest-mpl package to the optional test dependencies in pyproject.toml.

[project.optional-dependencies]
tests = [
  "py",
  "pytest",
  "pytest-cov",
  "pytest-mpl",
  "pytest-tmp-files",
]

We use the @pytest.mark.mpl_image_compare() decorator to mark a test as having Matplotlib output to indicate that we want to compare images. We need to set the baseline_dir which is where the images against which tests will be compared are stored here it is set to baseline which is relative to the position of the file. We then call our function, in this case mpl_example.scatter() with different sets of parameters (courtesy of pytest parameterisation).

The test itself must return fig of the desired plot so that it can be compared to the reference image each time the test is run.

Note

We also use the pytest.mark.parametrize() to setup two test scenarios.

"""Tests of the mpl_example module."""

import pytest

from pytest_examples import mpl_example


@pytest.mark.mpl_image_compare(baseline_dir="baseline")
@pytest.mark.parametrize(
    ("n_obs", "figsize", "title", "seed"),
    [
        pytest.param(300, (6, 6), "", 3513387, id="300 points; 6x6; no title"),
        pytest.param(3000, (6, 6), "Lots of points!", 3513387, id="3000 points; 6x6; Lots of points"),
    ],
)
def test_scatter(n_obs: int, figsize: tuple[int, int], title: str, seed: int) -> None:
    """Test of the scatter() function."""
    fig, _ = mpl_example.scatter(n_obs, figsize, title, seed)
    return fig

Once you have decorated your test you need to generate the images against which subsequent tests are to be compared to. This is done using the --mpl-generate-path= flag and providing an appropriate argument. The path is relative to where pytest is running from, in this example we use tests/baseline. We can restrict the test to the specific one we are working by specifying the path to the file and optionally the test within the file as in the below example where the path to the test file (tests/test_mpl_example.py) and the test name (test_scatter_hist) are separated by double colons (::).

mkdir -p tests/resources/img
pytest --mpl-generate-path=tests/baseline `tests/test_mpl_example.py::test_scatter`

Note

There is subtle difference between the baseline_dir parameter specified in the fixture itself (baseline) and the argument given to --mpl-generate-path= (tests/baseline).

This is because pytest searches for all files beginning with test_ in the directory tests and when running pytest you do so from the directory level above where tests resides which is typically the root of your package. Because the test files reside within the tests/ directory the relative path to the directory the parameter baseline_dir argument must omit this leading directory.

├── ./tests
    ├── ./tests/baseline
    └── ./tests/test_mpl_example.py

The tests are skipped “since generating image.”

❱ pytest --mpl --mpl-generate-path=baseline tests/test_mpl_example.py
======================== test session starts ==========================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
Matplotlib: 3.9.2
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: regtest-2.1.1, anyio-4.6.0, icdiff-0.9, pylint-0.21.0, pytest_tmp_files-0.0.2, syrupy-4.7.1, mpl-0.17.0, cov-5.0.0, mock-3.14.0, xdist-3.6.1, durations-1.3.1
collected 2 items

tests/test_mpl_example.py ss                                                                                                                            [100%]

---------- coverage: platform linux, python 3.12.7-final-0 -----------
Name                            Stmts   Miss  Cover
---------------------------------------------------
pytestexamples/divide.py           16     16     0%
pytestexamples/mpl_example.py      13      0   100%
pytestexamples/shapes.py            5      5     0%
---------------------------------------------------
TOTAL                              34     21    38%

======================== short test summary info =====================
SKIPPED [2] ../../../../../../../home/neil/.virtualenvs/pytest-examples/lib/python3.12/site-packages/pytest_mpl/plugin.py:925: Skipped test, since generating image.
======================== 2 skipped in 2.46s ===========================

We can look at the generated images, nothing fancy just some random dots.

The generated images reside within the tests/baseline/ directory and as the baseline_dir=baseline argument to the @pytest.mark.mpl_image_compare() is relative to the location of the test file itself which is in test we are good to go. We can re-run the tests with the --mpl flag to check they pass.

pytest --mpl tests/test_mpl_example::test_scatter

======================== test session starts ==========================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
Matplotlib: 3.9.2
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: regtest-2.1.1, anyio-4.6.0, icdiff-0.9, pylint-0.21.0, pytest_tmp_files-0.0.2, syrupy-4.7.1, mpl-0.17.0, cov-5.0.0, mock-3.14.0, xdist-3.6.1, durations-1.3.1
collected 2 items

tests/test_mpl_example.py ..                                                                                                                            [100%]

---------- coverage: platform linux, python 3.12.7-final-0 -----------
Name                            Stmts   Miss  Cover
---------------------------------------------------
pytestexamples/divide.py           16     16     0%
pytestexamples/mpl_example.py      13      0   100%
pytestexamples/shapes.py            5      5     0%
---------------------------------------------------
TOTAL                              34     21    38%

=========================== 2 passed in 2.56s ===========================

Update `pytest` options

We don’t want to have to remember to use the --mpl flag each time we run the test and we also want to make sure its used in Continuous Integration. This can be achieved by adding the --mpl flag to the pytest options that are defined in pyproject.toml.

[tool.pytest.ini_options]
  ...
  addopts = ["--cov", "--mpl", "-ra", "--showlocals", "--strict-config", "--strict-markers"]
  ...

Note

Don’t forget to stage and commit these changes to your repository.

Now each time you run your test suite the output of calling the test is compared to the reference images that reside under tests/baseline/ (or whatever directory you specified).

Failing tests

Sometimes your tests might fail. To demonstrate this we change one of the title parameters in the tests to Another title in tests/test_mpl_example.py and re-run the tests, sure enough the test fails. We are told the RMS Value (the Root Mean Square difference value) along with the location of the test files of which there are three images, a copy of the baseline, the result and a difference between the two.

❱ pytest --mpl  tests/test_mpl_example.py
========================== test session starts =======================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
Matplotlib: 3.9.2
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: regtest-2.1.1, anyio-4.6.0, icdiff-0.9, pylint-0.21.0, pytest_tmp_files-0.0.2, syrupy-4.7.1, mpl-0.17.0, cov-5.0.0, mock-3.14.0, xdist-3.6.1, durations-1.3.1
collected 2 items

tests/test_mpl_example.py F.                                                                                                                            [100%]

========================== FAILURES ==================================
________________ test_scatter[300 points; 6x6; no title] ______
Error: Image files did not match.
  RMS Value: 8.551853600243634
  Expected:
    /tmp/tmpn9ucgc47/tests.test_mpl_example.test_scatter_300 points; 6x6; no title/baseline.png
  Actual:
    /tmp/tmpn9ucgc47/tests.test_mpl_example.test_scatter_300 points; 6x6; no title/result.png
  Difference:
    /tmp/tmpn9ucgc47/tests.test_mpl_example.test_scatter_300 points; 6x6; no title/result-failed-diff.png
  Tolerance:
    2

---------- coverage: platform linux, python 3.12.7-final-0 -----------
Name                            Stmts   Miss  Cover
---------------------------------------------------
pytestexamples/divide.py           16     16     0%
pytestexamples/mpl_example.py      13      0   100%
pytestexamples/shapes.py            5      5     0%
---------------------------------------------------
TOTAL                              34     21    38%

========================== short test summary info ======================
FAILED tests/test_mpl_example.py::test_scatter[300 points; 6x6; no title] - Failed: Error: Image files did not match.
========================== 1 failed, 1 passed in 2.45s ==================

Navigating to and viewing these files is pretty easy when there is only one test that has failed but when more than one test fails they are all in their own directory and navigating and viewing them takes a bit longer.

`--mpl-generate-summary`

This is where the --mpl-generate-summary option comes in really handy as it will generate a report of the differences in either html, json or basic-html. Here I’ll show the use of the html option and what it produces.

By default the report is created under /tmp/ but with the --mpl-results-path you can specify a location relative to where the tests are being run from. You do not need to create the directory/path it will be created for you. That said I’m not that bothered about keeping the test image comparisons though so I typically omit the --mpl-generate-path option and instead the output and use the /tmp/ directory, that way my project directory doesn’t get cluttered with files as this is wiped when the computer is rebooted.

Note

If you do use --mpl-results-path to specify a nested directory within your repository you would probably want to exclude it from being included in version control though so add the path to .gitignore.

Note

Remember we have added the --mpl flag to the general tool.pytest.ini_options in pyproject.toml to ensure pytest-mpl extension is enabled and run.

Lets run the tests, which we know will fail, with the --mpl-generate-summary=html option enabled (and not using --mpl-results-path) .

pytest --mpl-generate-summary=html tests/test_mpl_example.py

The output is virtually identical but there is an additional line at the end…

❱ pytest --mpl  --mpl-generate-summary=html tests/test_mpl_example.py
========================== test session starts =======================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
Matplotlib: 3.9.2
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: regtest-2.1.1, anyio-4.6.0, icdiff-0.9, pylint-0.21.0, pytest_tmp_files-0.0.2, syrupy-4.7.1, mpl-0.17.0, cov-5.0.0, mock-3.14.0, xdist-3.6.1, durations-1.3.1
collected 2 items

tests/test_mpl_example.py F.                                                                                                                            [100%]

========================== FAILURES ==================================
_________________ test_scatter[300 points; 6x6; no title; green] _____
Error: Image files did not match.
  RMS Value: 8.551853600243634
  Expected:
    /tmp/tmp1bzvguuq/tests.test_mpl_example.test_scatter_300 points; 6x6; no title; green/baseline.png
  Actual:
    /tmp/tmp1bzvguuq/tests.test_mpl_example.test_scatter_300 points; 6x6; no title; green/result.png
  Difference:
    /tmp/tmp1bzvguuq/tests.test_mpl_example.test_scatter_300 points; 6x6; no title; green/result-failed-diff.png
  Tolerance:
    2

---------- coverage: platform linux, python 3.12.7-final-0 -----------
Name                            Stmts   Miss  Cover
---------------------------------------------------
pytestexamples/divide.py           16     16     0%
pytestexamples/mpl_example.py      13      0   100%
pytestexamples/shapes.py            5      5     0%
---------------------------------------------------
TOTAL                              34     21    38%

========================== short test summary info ======================
FAILED tests/test_mpl_example.py::test_scatter[300 points; 6x6; no title; green] - Failed: Error: Image files did not match.
========================== 1 failed, 1 passed in 2.50s ==================
A summary of test results can be found at: /tmp/tmp1bzvguuq/fig_comparison.html

If we open the test results in our browser we have a nice summary of the tests that have passed and failed.

Overview generated by pytest --mpl-generate-summary=html

If we want to look at a specific failed test we can, on the left we see the baseline image, on the right the test image and in-between a plot showing the differences between the two. We also have the “Root Mean Square Error” reported for the test.

Detailed view of a failed test generated by pytest --mpl-generate-summary=html

Detailed view of a pass test generated by pytest --mpl-generate-summary=html

Conclusion

If your package produces plots its easy to write tests that check they are correctly generated using the pytest-mpl extension and there is a neat convenience option to generate summaries of tests to make viewing the failures and differences in your browser convenient.

Key Points

You have to generate reference images against which the tests are run.
You should add the --mpl option to the tools.pytest.ini_options section of your pyproject.toml under addopts to ensure the extension is used whenever tests are run, be that locally or in Continuous Integration.
If you find tests are failing you can easily generate a web-page summarising the passes and failures using the --mpl-generate-summary=html flag when invoking pytest to get an HTML summary that is easy to navigate.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Pytest {Matplotlib}},
  date = {2024-09-24},
  url = {https://blog.nshephard.dev/posts/pytest-mpl/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Pytest Matplotlib .” September 24, 2024. https://blog.nshephard.dev/posts/pytest-mpl/.

Pennine Way 2024

Neil Shephard — Fri, 30 Aug 2024 00:00:00 GMT

This summer I decided to hike the iconic Pennine Way a 431 kilometre (261 miles) national trail stretching between Kirk Yetholm in the North and Edale in the South. Traditionally the route is hiked from South to North but as I live in Sheffield and have hiked around Kinder, Bleaklow and Blackhill I decided to hike from North to South as even if I didn’t complete the route I would get to see places I wasn’t familiar with and it was also cheaper to buy an advance rail ticket than have to purchase one on the day of return as I had no idea how long it would take.

NB - In order to provide some relevance to the computing theme of this blog I include examples of how to plot GPX points in both R using the leaflet R package and python using the folium package make using the leaflet JavaScript framework very simple.

Not Pennine Way by Me

Planning

A number of years ago I attempted to hike the Pennine Way from Edale and having set off from Edale and reached Crowden in good time decided I could combine the first two days and continued to Standedge. This was a mistake as I wasn’t used to hiking with such weight and its a rough section up over Laddow Rocks. Half way there my knee didn’t like this and I hobbled the rest of the way and called my wife to come and collect me, heading home with my tail between my legs.

I didn’t want to repeat this but I had no idea how long this would take me so booked two and a half weeks off of work allowing me 18 days to complete. I use OpenStreetMap and the excellent BRouter WebUI to plan days of upto 35km hiking although one day came in at 44km which I was wary of as I typically don’t enjoy hiking more than 30km without carrying significant weight so didn’t want to push myself too hard. But I figured I could be flexible and go slower if needed, and it wouldn’t be a problem if I was faster I could just drink more beer!

A kit list of what I took is at the end along with details of what I used and didn’t, but I took everything I needed for camping along with clothes, a stove for boiling water, six dehydrated meals, maps, battery pack and a guide which I would have to read backwards as it covered the route from South to North over 20 days.

Tracking

I use OpenTracks to privately record GPX tracks of my hiking, running and cycling as it stores them only on the device on which they are recorded, although you can of course copy them elsewhere and use them as you like. I did this each day I hiked recording almost complete traces for the whole trip. I had one short drop out and missed restarting after having a meal but after that often left it running whilst stopped for lunch or dinner sto avoid forgetting to restart.

I uploaded each track to an instance of FitTrackee that I host myself which provided a neat way of reviewing the statistics for each days hiking which I have exported and saved to CSV and used in the tables and figures of this write-up.

Details and more information about these and other Open Source tools for self-research that protect the users privacy from the pernicious intrusions made by Big Tech can be found on the Personal Science Wiki.

Pictures

I carried my phone, a Motorola GT100 and a Sony RX100 Mk IV with me along the route. I take a lot of pictures, most are crap but some are half decent and I’ve included in this post the better ones or those that accompany the story well. You can find the full set of pictures on my Pennine Way Flickr album.

Using GPX Tracks

I’ve opted to provide solutions to plotting GPX tracks in both R and Python since I use both in my day to day work. Each code chunk outputs a map and the code can be viewed if you want to learn how to create maps using either language.

This requires…

Finding files with the .gpx extension.
Loading each file GPX file.
Extracting the GPS points (latitude and longitude), speed, elevation and date/time.
Plotting the points on a Leaflet map.

To simplify this process I define functions to carry out the task (rather than copying and pasting the code for each day). Both R and Python have libraries for working with the Leaflet library. Currently I’ve got slightly further with the Python library than the R but I intend to update that and will write a separate blog post on mapping with each of the libraries. Things to do on the maps…

Consistent zoom level on both.
Correctly colouring points by speed or elevation in R.
Start and End markers.
Add pop-ups on hovering over any point to show the metrics (lat/lon, date/time, speed, elevation, accuracy).
Plot the whole journey as one.

R
Python

The R library is named after the Javascript library, i.e. leaflet

## https://rstudio.github.io/leaflet/
## https://www.r-bloggers.com/2022/10/r-and-gpx-how-to-read-and-visualize-gpx-files-in-r/
## https://mastodon.social/@yabellini@fosstodon.org/113181640784237866
library(htmlwidgets)
library(leaflet)
library(gpx)
## List GPX files
gpx_files <- list.files(pattern = "\\.gpx$", recursive = TRUE)
## Remove the 'pennine_way' gpx as we will concatenate the others
gpx_files <- head(gpx_files, -1)
## Strip the common path and extract the date
dates <- stringr::str_split(gpx_files, "/", simplify = TRUE)[,2]
dates <- stringr::str_split(dates, "_", simplify = TRUE)[,1]

gps <- lapply(gpx_files, gpx::read_gpx)
names(gps) <- dates
#gps <- lapply(gps, function(df) df[[1]]$tracks)

#' Convert GPS points to data frame
#'
#' @params
#'
#' gps list GPS track read by gpx::read_gpx().
#' columns list Column names.
#'
#' @returns Data Frame of GPS points.
gps_to_df <- function(gps, columns = c("elevation", "time", "lat", "lng", "speed", "accuracy", "segment_id")) {
    df <- as.data.frame(gps$tracks[[1]])
    colnames(df) <- columns
    df <- df |>
          dplyr::mutate(speed = as.numeric(speed),
                        elevation = as.numeric(elevation),
                        accuracy = as.numeric(accuracy),
                        lat = as.numeric(lat),
                        lng = as.numeric(lng),
                        accuracy = as.numeric(accuracy),
                        time = lubridate::as_datetime(time),
          )
    df
}
gps_df <- lapply(gps, gps_to_df)
names(gps_df) <- dates

#' Plot a gps dataframe
#'
#' @params
#'
#' gps_df list List of GPS Dataframes.
#' date str Date of track to plot
#' color str Variable to colour points as, default 'speed' but 'elevation' also possible
#'
#' @returns Leaflet map
plot_day <- function(gps_df, date, color = "speed") {
    ## Palette not working...yet!
    ## pal <- leaflet::colorNumeric(palette = c("green", "red"),
    ##                             domain = gps_df[date][[color]])
    leaflet::leaflet() |>
        leaflet::addTiles() |>
        leaflet::addPolylines(data = gps_df[date][[1]],
                              lat = ~lat,
                              lng = ~lng,
                              ## color = ~pal(color)
                              )
}

The gpxplotter package provides wrappers around folium for plotting GPX traces, but I opted to do this manually so I could learn more about Folium (I used to use it in a previous job, just need to remember what I used to do!).

# https://towardsdatascience.com/build-interactive-gps-activity-maps-from-gpx-files-using-folium-cf9eebba1fe7
from pathlib import Path
import pandas as pd
import folium
import gpxpy
gpx_files = sorted(Path(".").glob("**/*.gpx"))

def gpx_to_df(filename: Path | str) -> pd.DataFrame:
    """
    Load a GPX track and extract data.

    Parameters
    ----------
    filename: Path | str
        Path to a .gpx file.

    Returns
    -------
    pd.DataFrame
    """
    gpx = gpxpy.parse(filename.open())
    gpx_segments = list(gpx.tracks[0].segments)
    data = []
    for segment in gpx_segments:
        for point_id, point in enumerate(segment.points):
            data.append([point.latitude,
                         point.longitude,
                         point.elevation,
                         point.time,
                         segment.get_speed(point_id)
                       ])
    columns = ["latitude", "longitude", "elevation", "time", "speed"]
    return pd.DataFrame(data, columns = columns)

def tidy_speed(df: pd.DataFrame, speed: str = "speed", threshold: float = 3.5) -> pd.DataFrame:
    """
    Remove extreme speed values from a GPS dataframe.

    Parameters
    ----------
    df: pd.DataFrame
        Pandas DataFrame of GPS points.
    speed: str
        Variable holding speed, default is 'speed' and shouldn't need changing.
    threshold: float
        Threshold for defining extreme speed

    Returns
    -------
    pd.DataFrame
        Pandas Dataframe with values of 'speed' > 'threshold' set to 'threshold'.
    """
    df.loc[df[speed] > threshold, speed] = threshold
    return df

def extract_lat_lon(gps_df: pd.DataFrame, lat: str = "latitude", lon: str = "longitude") -> list[tuple]:
    """
    Extract latitude and longitude from GPX data frame.

    Parameters
    ----------
    gps_df: pd.DataFrame
        Pandas Dataframe with latitude and longitude columns.
    lat: str
        Name of latitude column.
    lon: str
        Name of longitude column.

    Returns
    -------
    list[tuple]
        Returns a list of tuples each of which is the latitude and longitude.
    """
    _array = gps_df[[lat, lon]].to_numpy()
    return [tuple(point) for point in _array]

def plot_gpx_df(gps_df: list[tuple],
                lat: str = "latitude",
                lon: str = "longitude",
                colors: str = "speed",
                colormap: str = "viridis",
                speed_threshold: float = 3.5,
                default_tile: str = "openstreetmap") -> folium.Map:
    """
    Plot gps points on a map.

    Parameters
    ----------
    gps_df: list[tuple]
        A list of tuples of GPS points
    lat: str
        Name of latitude column.
    lon: str
        Name of longitude column.
    colors: str
        What to plot the colour of the line as, options are 'elevation' and 'speed' (default).
    colormap: str
        Colormap to use when plotting points.
    speed_threshold: float
        Speed threshold for defining and resetting extreme values, default is '3.5'.
    default_tile: str
        Default map tile to use
    """
    if speed_threshold > 0.0:
        gps_df = tidy_speed(gps_df, threshold=speed_threshold)
    map = folium.Map(location=[gps_df[lat].mean(),
                               gps_df[lon].mean()],
                     tiles = None)

    folium.TileLayer("openstreetmap", name = "OpenStreetMap").add_to(map)
    folium.TileLayer("http://tile.stamen.com/terrain/{z}/{x}/{y}.jpg",
                     attr="terrain-bcg",
                     name="Terrain Map").add_to(map)
    # Add layer control (not working)
    # folium.LayerControl(collapsed = True).add_to(map)
    points = extract_lat_lon(gps_df, lat, lon)
    folium.ColorLine(points, weight=4, colors=gps_df[colors]).add_to(map)
    # Calculate bounding box (not working)
    # south_west = gps_df[[lat, lon]].min().values.tolist()
    # north_east = gps_df[[lat, lon]].min().values.tolist()
    # map.fit_bounds([south_west, north_east])
    return map

## Get a list of all GPX points
gpx = {Path(_file.stem).stem.split("_")[0]: gpx_to_df(_file) for _file in gpx_files }

Day by Day

Day 1 - Sheffield to Kirk Yetholm

Having packed over the previous few days I weighed the rucksack with 2 litres of water attached, 17.4kg.

This was a walking holiday so I waved goodbye to my wife and daughter around 07:30 feeling fresh and walked to the train station to catch the 08:22 train to Berwick-Upon-Tweed. Being Sheffield this involved a hill but it wasn’t too bad carrying the extra weight of the rucksack. I had poles but hadn’t put them to good use just yet.

The train journey was uneventful, but gave me time to sit and read a book on the Kobo I bought with me. I had bought the Kobo for hiking in the Julian Alps of Slovenia the previous year as I didn’t want to carry multiple books. Having recently started reading Johann Harri’s Stolen Focus : Why You Can’t Pay Attention I expected to get through it and had some fiction to read as well (Haruki Murakami’s IQ84). The countryside passed by as did the first couple of chapters and before long I was getting off at Berwick-Upon-Tweed.

I’d checked the schedule and had a short wait before the bus to Kelso arrived. There was a gentleman trying to straighten one of two buckled wheels on his bike outside the station. Its quite hard to buckle wheels and I figured he’d therefore been in an accident recently so went and asked if he was ok. He said he was but that he’d been knocked off by a car turning across him. He was lucky and only had a few bruises but his bike was not ride-able and he was supposed to be cycling to Wales. He asked me to stand on one side of the wheel whilst he stood on the other to try and straighten it. Unsurprisingly this didn’t work. He then noticed some fence railings and tried levering it straight…unsuccessfully. Not only were both his wheels buckled but the built-in pannier rack that was part of the frame had sheered off so he would have trouble carrying his bags. He wasn’t hurt so I left him to his own devices and explored Castle Vale Park which led down to the River Tweed by the Royal Border Bridge and the remains of Berwick Castle.

Royal Border Bridge, Berwick Upon Tweed | Flickr

A nice way to kill some time but I didn’t want to miss my bus so headed back through Castle Vale Park to wait at the bus stop. It arrived and set-off on time and I bounced through some nice farmland towards Kelso, spying The Cheviot hills in the distance which I would be hiking up in a few hours.

Didn’t have masses of time to explore Kelso but grabbed a baguette and had a pint around the town square and purchased a bottle of Traquair Jacobite Ale (8% ABV) to drink before bed (with limited weight I wanted something strong!).

Another bus journey and I arrived in Kirk Yetholm. I’d clocked someone who looked like they were hiking and as we got of asked where he was heading. He said he was supposed to have been doing an Ultrar-marathon through the area but had injured himself a few days before the start so was taking active recovery by walking and bussing his way along the route. He kindly offered to snap a picture of me next to the Pennine Way map on the bus-stop, bid me farewell and headed off whilst I went for another pint in The Border, the pub that marks the “end” (or start) of the Pennine Way.

The Border Inn | Flickr

Had a nice chat with a local who had lived in Sheffield in the past and was joined by what seemed the proprietor of the pub who said me I should sign the log-book behind the bar even though I was leaving. I duly did so to log the start date somewhere and considered purchasing a t-shirt but a) it was in the wrong direction and b) I hadn’t even started, let alone finished hiking at that point. One of them kindly offered to snap a picture of me in front of the pub’s display of the end of The Way.

Beer at The Border | Having a pint at The Border Inn in Kirk Yetholm | Flickr

Day 1 - 2024-08-14 Kirk Yetholm to Clennell Street

Beer drunk I said farewell, shouldered my pack and started the GPS tracker, setting off up the lane in the heat of the afternoon. Not ideal conditions for hiking but I had a schedule to follow.

Above Kirk Yetholm

Follow The Path

There are two options to this section of The Way, a high option going over various hills and low option for when weather is foul. I didn’t have that excuse and had been planning on stopping for the night at the Auchope Refuge Hut which was on the high route so up the hills I went. These undulated but the path was good and I made good time arriving at the hut in around three hours.

College Burn

Auchope Refuge

With a few more hours of daylight left I decided to push on and find somewhere else to wild camp as my legs weren’t tired and I had plenty of time, even though this involved a steep climb from Auchope Rigg up most of Cairn Hill. I decided in doing so not to do the dog-leg extension of The Way across the flat-ish moor to the summit of The Cheviot, the hill after which the area is named as it added an extra 4km to the journey and would have taken about an hour and I was now on the lookout for somewhere to stop.

Heading downhill there weren’t many options forthcoming, lots of heather with a path through it but as things levelled out I came to a cross-roads with a path marked Clennell Street with a sign indicating that motor vehicles were not allowed between 1st April and 1st May. Quite how anything but motorbikes could get up there I’ve no idea, but there was a flat patch of grass so I donned a long-sleeve top to keep the midges away and setup the tent and bed for the night. That done I had my dinner of some ciabatta and pate along with the bottle of Traquair Jacobite I’d carried along and as the sunset I snuggled down for the first night under canvas feeling pleased to have gone a little further than planned and had no problems hiking up the hills with the weight of my pack.

Clennell Street | Flickr

R
Python

Code

virtualenvwrapper hooks

Neil Shephard — Wed, 31 Jul 2024 00:00:00 GMT

I’ve written previously about virtualenvwrapper which I use to manage my Python Virtual Environments and mentioned the possibility of using hooks but didn’t go into detail.

South Coast of Portugal by Me

Introduction

Just like the various hooks available in Git, virtualenvwrapper also supports hooks that allow scripts to be run in response to various events. These reside under your $VIRTUALENVWRAPPER_HOOK_DIR which by default is the same as your $WORKON_HOME directory and in a typical standard installation will be ~/.virtualenvs.

The available scripts that are recognised are…

get_env_details
initialize
premkvirtualenv
postmkvirtualenv
precpvirtualenv
postcpvirtualenv
preactivate
postactivate

Each of these is a simple shell script and will start with the scripting language to use e.g. #!/usr/bin/bash or #!/usr/bin/zsh depending on your shell. You can then script the actions you wish to take when the script is executed.

Install minimal requirements

I’m a big fan of dotfiles¹, mine are hosted on GitLab, it’s a repository of my configuration files and scripts that I use regularly across multiple computers. Because I’m lazy I wrote a couple of requirements.txt files for installing packages in my virtual environments.

requirements.txt : holds everything I might ever want to use in Python.
python-lsp-requirements.txt

Install packages for setting up a Python Language Server (which I use from Emacs).
venv_minimal_requirements.txt

a minimal set of the most common Python packages I am likely to want when creating a new virtual environment.

Because I have my dotfiles cloned to the same location on every computer (~/dotfiles) I added the following to the ~/.virtualenvs/postmkvirtualenv ² which will install all of the packages listed in ~/dotfiles/python/venv_minimal_requirements.txt whenever a create a new virtual environment, whether that is with mkvritualenv or mktmpenv.

pip install --no-cache-dir -r ~/dotfiles/python/venv_minimal_requirements.txt

This ensured the latest versions of each packages listed in ~/dotfiles/python/venv_minimal_requirements.txt were downloaded and installed as the --no-cache-dir prevents using cached versions of packages.

A smarter script

This served me well for a time, but occasionally I found I didn’t want to install any packages in a new virtual environment (most often when testing new branches using mktmpenv) and I’d have to remember to comment out the line in the hook file (~/.virtualenvs/postmkvirtualenv) before creating the environment. Typically though I’d forget to do this and would have to halt installation of required packages, deactivate the environment, then comment it out and create a new environment.

This quickly became irksome.

But ~/.virtualenvs/postmkvirtualenv is just a script and so we can use a bit of scripting knowledge to make it interactive and ask the user if they want to install the packages listed in venv_minimal_requirements.txt. I found a really useful answer on StackOverflow in the How do I prompt for yes/no/cancel input in a Linux shell script that showed several different ways to prompt the user for a response as to whether they want to do something.

I therefore updated my ~/.virtualenvs/postmkvirtualenv to the following which prompts for a numeric response, 1 for Yes and 2 for No and takes the appropriate action, installing using my original invocation of pip if I want to install packages and enter 1 or installing nothing if I enter 2.

#!/usr/bin/zsh
# This hook is sourced after a new virtualenv is activated.


echo "Do you wish to install minimal requirements (from venv_minimal_requirements.txt)? "
select yn in "Yes" "No"; do
  case $yn in
    Yes ) pip install --no-cache-dir -r ~/dotfiles/python/venv_minimal_requirements.txt; break;;
    No ) echo "No packages installed. install packages with 'pip'.\n"; break;;
  esac
done

NB You may want to tweak the opening shebang if you use the Bash shell.

Conclusion

A little bit of shell scripting knowledge can be really powerful when used in conjunction with “hooks”. This is true of virtualenvwrapper as well as of Git and the pre-commit framework.

Footnotes

There is a wealth of information on what you can do with your dotfiles but that is an article in itself and I’m yet to write it. A useful set of different aliases you could use can be found here↩︎
Actually I create the script in ~/dotfiles/python/postmkvirtualenv and made a symbolic link at ~/.virtualenv/postmkvirtualenv that points to it so that whenever I update or improve this script it is updated across my computers.↩︎

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Virtualenvwrapper Hooks},
  date = {2024-07-31},
  url = {https://blog.nshephard.dev/posts/virtualenv-hooks/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Virtualenvwrapper Hooks.” July 31, 2024. https://blog.nshephard.dev/posts/virtualenv-hooks/.

Pytest Fail and Skip

Neil Shephard — Thu, 25 Apr 2024 00:00:00 GMT

Pytest is an excellent framework for writing tests in Python. Sometimes tests don’t pass though and you might want to mark them as failing or skip them.

Against the Clouds by Me

Pytest has a few decorators available to help with skipping tests using @pytest.mark.skip or @pytest.mark.skipif or allowing tests to fail with @pytest.mark.xfail

We’ll use the pytest-examples repository for looking at how these work.

git clone git@github.com:ns-rse/pytest-examples.git
cd pytest-examples

Why?

There are a number of reasons why you may wish to deliberately and Eric covers them nicely. In brief…

Incompatible Python or Package Version - some tests don’t pass under a certain version.
Platform specific issues - some tests fail on a specific platform.
External dependencies - if you haven’t got round to mocking a service.
Local dependencies - excluding tests running under Continuous Integration that rely on local dependencies.

Choose your Partner - Skip (to my lou)

If you want to unconditionally skip a test prefix it with @pytest.mark.skip(), adding a reason can be useful and there is the argument reason=""to do so, it helps others, including your future self. If we use the tests/test_divide.py from the pytest-examples repository we can skip the redundant test_divide_unparameterised() function as its already covered by the parameterised test that follows.

@pytest.mark.skip(reason="redundant - covered by test_divide()")
def test_divide_unparameterised() -> None:
    """Test the divide function."""
    assert divide(10, 5) == 2

When we run the test we are told it is skipped. To keep things fast we run just that test using the command line version pytest :: but your IDE may support running individual tests (in Emacs you can use pytest.el to the same effect).

❱ pytest tests/test_divide::test_divide_unparameterised
======================================= test session starts ============================================
platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0
Matplotlib: 3.8.4
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: durations-1.2.0, xdist-3.5.0, pytest_tmp_files-0.0.2, mpl-0.17.0, lazy-fixture-0.6.3, cov-5.0.0
collected 1 item

tests/test_divide.py s                                                                            [100%]

---------- coverage: platform linux, python 3.11.9-final-0 -----------
Name                        Stmts   Miss  Cover
-----------------------------------------------
pytest_examples/divide.py      16      8    50%
pytest_examples/shapes.py       5      5     0%
-----------------------------------------------
TOTAL                          21     13    38%

====================================== short test summary info =========================================
SKIPPED [1] tests/test_divide.py:9: redundant - covered by test_divide()
====================================== 1 skipped in 0.59s ==============================================

Choose your Partner - Failing (…)

Nothing in the the old dance about failing but you can selectively allow tests to fail using the pytest.mark.xfail() fixture. If you know a test is going to fail you can, rather than commenting it out, mark it as such. If we update the test condition so we know it will fail we mark that it will fail as follows.

@pytest.mark.xfail(reason="demonstrate expected failure")
def test_divide_unparameterised() -> None:
    """Test the divide function."""
    assert divide(10, 5) == 3

And running pytest on this shows the failure

❱ pytest tests/test_divide.py::test_divide_unparameterised
====================================== test session starts =============================================
platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0
Matplotlib: 3.8.4
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: durations-1.2.0, xdist-3.5.0, pytest_tmp_files-0.0.2, mpl-0.17.0, lazy-fixture-0.6.3, cov-5.0.0
collected 1 item

tests/test_divide.py x                                                                            [100%]


---------- coverage: platform linux, python 3.11.9-final-0 -----------
Name                        Stmts   Miss  Cover
-----------------------------------------------
pytest_examples/divide.py      16      6    62%
pytest_examples/shapes.py       5      5     0%
-----------------------------------------------
TOTAL                          21     11    48%

====================================== short test summary info =========================================
XFAIL tests/test_divide.py::test_divide_unparameterised - demonstrate expected failure
====================================== 1 skipped in 0.59s ==============================================

Conditional Skipping/Failing

The pytest.mark.skipif() and pytest.mark.xfail() fixtures both have the argument condition which allows you to use a Boolean (i.e. a statement that evaluates to True or False) to determine whether they are used. Any Python expression that can be evaluated to True or False can be used and for backwards compatibility strings can still be used. If condition argument is used in pytest.mark.xfail() then the reason argument must also be given indicating why the test is being skipped/is expected to failed.

Here we fail the test only if the Python version is 3.10.*. Note the need to import sys and the use of sys.version_info[:2] to extract a tuple of the major and minor Python version).

import sys

@pytest.mark.xfail(sys.version_info[:2] == (3, 10), reason="Skip under Python 3.10"))
def test_divide_unparameterised() -> None:
    """Test the divide function."""
    assert divide(10, 5) == 3

Skipping/Failing Parameterised Tests

In many instances you can parameterise tests, and you can use the fixtures we’ve covered against the whole test. But what if you want to skip not all of the parameterised tests but only specific ones? This is possible because as covered previously you can use pytest.param() function to define your parameters and give them id="some text" to help identify them. pytest.param() also has a marks= option which allows you to add pytest.mark.* to just that set of parameters and so we can add pytest.mark.xfail() or pytest.mark.skip[if]() to specific sets of parameters.

Instead of placing the fixture before the test so that it applies to all functions, you use the pytest.param() for each set of parameters and add pytest.mark.xfails() (or other variants) as arguments to the marks option.

Here we mark the test with id of zero division error with marks=pytest.mark.xfail as we know that a division by zero test will fail and so that set of parameters should be skipped.

@pytest.mark.parametrize(
    ("a", "b", "expected"),
    [
        pytest.param(10, 5, 2, id="ten divided by five"),
        pytest.param(9, 3, 3, id="nine divided by three"),
        pytest.param(5, 2, 2.5, id="five divided by two"),
        pytest.param(0, 100, 0, id="zero divided by one hundred"),
        pytest.param(
            10, 0, ZeroDivisionError, id="zero division error", marks=pytest.mark.xfail(reason="Expected to fail")),
    ],
)
def test_divide(a: float | int, b: float | int, expected: float) -> None:
    """Test the divide function."""
    assert divide(a, b) == expected

❱ pytest tests/test_divide.py::test_divide
====================================== test session starts =============================================

platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0
Matplotlib: 3.8.4
Freetype: 2.6.1
rootdir: /mnt/work/git/hub/ns-rse/pytest-examples/main
configfile: pyproject.toml
plugins: durations-1.2.0, xdist-3.5.0, pytest_tmp_files-0.0.2, mpl-0.17.0, lazy-fixture-0.6.3, cov-5.0.0
collected 5 items

tests/test_divide.py ....x                                                                        [100%]

---------- coverage: platform linux, python 3.11.9-final-0 -----------
Name                        Stmts   Miss  Cover
-----------------------------------------------
pytest_examples/divide.py      16      3    81%
pytest_examples/shapes.py       5      5     0%
-----------------------------------------------
TOTAL                          21      8    62%

====================================== short test summary info =========================================
XFAIL tests/test_divide.py::test_divide[zero division error] - Expected to fail
====================================== 4 passed, 1 xfailed in 0.37s ====================================

The condition/reason arguments to both pytest.mark.skipif() and pytest.mark.xfail() functions are still valid and can be used to conditionally mark specific sets of parameters to be skipped or indicate if they will fail under certain conditions.

To exclude the test with id of five divided by two under Python 3.10 we would do the following (again note the need to import sys and its use in the cond positional argument).

import sys

...

@pytest.mark.parametrize(
    ("a", "b", "expected"),
    [
        pytest.param(10, 5, 2, id="ten divided by five"),
        pytest.param(9, 3, 3, id="nine divided by three"),
        pytest.param(5, 2, 2.5, id="five divided by two", marks=pytest.mark.xfail(sys.version_info[:2] == (3, 10),
                                                                                  reason="Skip under Python 3.10")),
        pytest.param(0, 100, 0, id="zero divided by one hundred"),
        pytest.param(10, 0, ZeroDivisionError, id="zero division error", marks=pytest.mark.xfail),
    ],
)
def test_divide(a: float | int, b: float | int, expected: float) -> None:
    """Test the divide function."""
    assert divide(a, b) == expected

Summary

Pytest has features which help support test development and allow specific tests to fail or be skipped completely which helps with both test development and with Continuous Integration where test results can vary depending on platform and package versions.

This post stems from a suggestion made by @jni@jni@fosstodon.org during some work I have been contributing to the skan package. Thanks to Juan for the prompt/pointer.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Pytest {Fail} and {Skip}},
  date = {2024-04-25},
  url = {https://blog.nshephard.dev/posts/pytest-xfail/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Pytest Fail and Skip.” April 25, 2024. https://blog.nshephard.dev/posts/pytest-xfail/.

Git - Worktrees

Neil Shephard — Wed, 13 Mar 2024 00:00:00 GMT

A common task in Git workflow is the need to switch/checkout branches and check work that has been done on another branch, running tests, perhaps contributing to code that is out for Pull/Merge Review. If you’ve work in progress on your own branch this means either making a commit or stashing the work to come back to at a later date. Neither of these are particularly problematic as you can git pop stashed work to restore it or git commit --amend, or git commit --fixup and squash commits to maintain small atomic commits and avoid cluttering up the commit history with commits such as “Saving work to review another branch”. But, perhaps unsurprisingly, Git has another way of helping your workflow in this situation. Rather than having branches you can use “worktrees”.

Natures Arches by Me

Normally when you’ve git clone’d a repository all configuration files for working with the repository are saved to the repository directory under .git and all files in their current state on the main branch are also copied to the repository directory. If we clone the pytest-examples directory we can look at its contents using tree -afHD -L 2 (this limits the depth as we don’t need to look deep inside the .git or mypy directories which contain lots of files).

git clone git@github.com:ns-rse/pytest-examples.git
cd pytest-examples
tree -afhD -L 2
[4.0K Mar 11 07:26]  .
├── [ 52K Jan  5 11:26]  ./.coverage
├── [4.0K Mar 11 07:26]  ./.git
│   ├── [ 749 Jan  5 11:30]  ./.git/COMMIT_EDITMSG
│   ├── [ 394 Jan  5 11:28]  ./.git/COMMIT_EDITMSG~
│   ├── [ 479 Feb 17 14:08]  ./.git/config
│   ├── [ 556 Feb 17 14:06]  ./.git/config~
│   ├── [  73 Jan  1 13:24]  ./.git/description
│   ├── [ 222 Mar 11 07:26]  ./.git/FETCH_HEAD
│   ├── [  21 Mar 11 07:26]  ./.git/HEAD
│   ├── [4.0K Jan  1 13:27]  ./.git/hooks
│   ├── [1.3K Mar 11 07:26]  ./.git/index
│   ├── [4.0K Jan  1 13:24]  ./.git/info
│   ├── [4.0K Jan  1 13:24]  ./.git/logs
│   ├── [4.0K Mar 11 07:26]  ./.git/objects
│   ├── [  41 Mar 11 07:26]  ./.git/ORIG_HEAD
│   ├── [ 112 Jan  3 15:57]  ./.git/packed-refs
│   ├── [4.0K Jan  1 13:24]  ./.git/refs
│   └── [4.0K Jan  1 13:31]  ./.git/rr-cache
├── [4.0K Jan  2 11:52]  ./.github
│   └── [4.0K Jan  3 15:57]  ./.github/workflows
├── [3.0K Jan  2 12:06]  ./.gitignore
├── [1.0K Jan  1 13:24]  ./LICENSE
├── [ 293 Jan  2 12:06]  ./.markdownlint-cli2.yaml
├── [4.0K Jan  5 11:27]  ./.mypy_cache
│   ├── [ 12K Jan  5 11:28]  ./.mypy_cache/3.11
│   ├── [ 190 Jan  2 10:39]  ./.mypy_cache/CACHEDIR.TAG
│   └── [  34 Jan  2 10:39]  ./.mypy_cache/.gitignore
├── [1.7K Mar 11 07:26]  ./.pre-commit-config.yaml
├── [ 763 Jan  1 13:25]  ./.pre-commit-config.yaml~
├── [ 18K Jan  2 12:06]  ./.pylintrc
├── [4.8K Mar 11 07:26]  ./pyproject.toml
├── [4.7K Jan  1 17:36]  ./pyproject.toml~
├── [4.0K Jan  1 19:04]  ./.pytest_cache
│   ├── [ 191 Jan  1 19:04]  ./.pytest_cache/CACHEDIR.TAG
│   ├── [  37 Jan  1 19:04]  ./.pytest_cache/.gitignore
│   ├── [ 302 Jan  1 19:04]  ./.pytest_cache/README.md
│   └── [4.0K Jan  1 19:04]  ./.pytest_cache/v
├── [4.0K Mar 11 07:26]  ./pytest_examples
│   ├── [1.3K Mar 11 07:26]  ./pytest_examples/divide.py
│   ├── [ 179 Mar 11 07:26]  ./pytest_examples/__init__.py
│   ├── [4.0K Jan  5 11:18]  ./pytest_examples/__pycache__
│   ├── [ 491 Mar 11 07:26]  ./pytest_examples/shapes.py
│   └── [ 390 Jan  2 13:34]  ./pytest_examples/shapes.py~
├── [4.0K Jan  2 16:09]  ./pytest_examples.egg-info
│   ├── [   1 Jan  2 16:09]  ./pytest_examples.egg-info/dependency_links.txt
│   ├── [3.1K Jan  2 16:09]  ./pytest_examples.egg-info/PKG-INFO
│   ├── [ 481 Jan  2 16:09]  ./pytest_examples.egg-info/requires.txt
│   ├── [ 446 Jan  2 16:09]  ./pytest_examples.egg-info/SOURCES.txt
│   └── [  16 Jan  2 16:09]  ./pytest_examples.egg-info/top_level.txt
├── [ 602 Jan  3 15:57]  ./README.md
├── [   0 Jan  1 13:31]  ./README.md~
├── [4.0K Jan  1 13:30]  ./.ruff_cache
│   ├── [4.0K Jan  2 11:57]  ./.ruff_cache/0.1.8
│   ├── [  43 Jan  1 13:30]  ./.ruff_cache/CACHEDIR.TAG
│   └── [   1 Jan  1 13:30]  ./.ruff_cache/.gitignore
├── [4.0K Mar 11 07:26]  ./tests
│   ├── [ 681 Mar 11 07:26]  ./tests/conftest.py
│   ├── [  26 Jan  2 12:11]  ./tests/conftest.py~
│   ├── [4.0K Jan  5 11:26]  ./tests/__pycache__
│   ├── [1.7K Mar 11 07:26]  ./tests/test_divide.py
│   ├── [1.6K Mar 11 07:26]  ./tests/test_shapes.py
│   └── [   0 Jan  2 13:36]  ./tests/test_shapes.py~
└── [ 460 Jan  2 16:09]  ./_version.py

21 directories, 43 files

Lets create the contributing branch

git switch -c contributing
echo "# Contributing\n\nContributions to this repository are welcome via Pull Requests." > CONTRIBUTING.md

If we want to switch branches without making a commit but save our work in progress as we want to add more to the CONTRIBUTING.md file later we can stash the changes with a message. We then switch to main and create a new branch (citation) for and add a CITATION.cff file.

git stash -m "An example stash"
git switch main
git switch -c citation
echo "cff-version: 1.2.0\ntitle: Pytest Examples\ntype: software" > CITATION.cff
git add CITATION.cff
git commit -m "Adding CITATION.cff"

When we are ready to return to our contributing branch we can switch and git pop the work we stashed. By default the last stash is popped, but its possible to view all the stashes and select which you wish to pop and restore to the current branch.

git switch contributing
git pop

Worktrees rather than branches

Worktrees take a different approach to organising branches. They start with a --bare clone of the repository which implies the --no-checkout flag and means that the files that would normally be found under the /.git directory are copied but are instead placed in the top level of the directory rather than under .git/. No tracked files are copied as they may conflict with these files. You have all the information Git has about the history of the repository and the different commits and branches but none of the actual files.

NB If you don’t explicitly state a target directory to clone to it will be the repository name suffixed with .git, i.e. in this example pytest-examples.git. I recommend sticking with the convention of using the same repository name so will explicitly state it.

cd ..
mv pytest-examples pytest-examples-orig-clone
git clone --bare git@github.com:ns-rse/pytest-examples.git pytest-examples
cd pytest-examples
tree -afhD -L 2
[4.0K Mar 13 07:45]  .
├── [ 129 Mar 13 07:45]  ./config
├── [  73 Mar 13 07:45]  ./description
├── [  21 Mar 13 07:45]  ./HEAD
├── [4.0K Mar 13 07:45]  ./hooks
│   ├── [ 478 Mar 13 07:45]  ./hooks/applypatch-msg.sample
│   ├── [ 896 Mar 13 07:45]  ./hooks/commit-msg.sample
│   ├── [4.6K Mar 13 07:45]  ./hooks/fsmonitor-watchman.sample
│   ├── [ 189 Mar 13 07:45]  ./hooks/post-update.sample
│   ├── [ 424 Mar 13 07:45]  ./hooks/pre-applypatch.sample
│   ├── [1.6K Mar 13 07:45]  ./hooks/pre-commit.sample
│   ├── [ 416 Mar 13 07:45]  ./hooks/pre-merge-commit.sample
│   ├── [1.5K Mar 13 07:45]  ./hooks/prepare-commit-msg.sample
│   ├── [1.3K Mar 13 07:45]  ./hooks/pre-push.sample
│   ├── [4.8K Mar 13 07:45]  ./hooks/pre-rebase.sample
│   ├── [ 544 Mar 13 07:45]  ./hooks/pre-receive.sample
│   ├── [2.7K Mar 13 07:45]  ./hooks/push-to-checkout.sample
│   ├── [2.3K Mar 13 07:45]  ./hooks/sendemail-validate.sample
│   └── [3.6K Mar 13 07:45]  ./hooks/update.sample
├── [4.0K Mar 13 07:45]  ./info
│   └── [ 240 Mar 13 07:45]  ./info/exclude
├── [4.0K Mar 13 07:45]  ./objects
│   ├── [4.0K Mar 13 07:45]  ./objects/info
│   └── [4.0K Mar 13 07:45]  ./objects/pack
├── [ 249 Mar 13 07:45]  ./packed-refs
└── [4.0K Mar 13 07:45]  ./refs
    ├── [4.0K Mar 13 07:45]  ./refs/heads
    └── [4.0K Mar 13 07:45]  ./refs/tags

9 directories, 19 files

What use is that? Well from this point you can instead of using git branch use git worktree add and it will create a directory with the name of the branch which holds all the files in their current state on that branch.

git worktree add main
Preparing worktree (checking out 'main')
HEAD is now at 2f7c382 Merge pull request #6 from ns-rse/ns-rse/tidy-print
tree -afhD -L 2 main/
[4.0K Mar 13 08:13]  main
├── [  64 Mar 13 08:13]  main/.git
├── [4.0K Mar 13 08:13]  main/.github
│   └── [4.0K Mar 13 08:13]  main/.github/workflows
├── [3.0K Mar 13 08:13]  main/.gitignore
├── [1.0K Mar 13 08:13]  main/LICENSE
├── [ 293 Mar 13 08:13]  main/.markdownlint-cli2.yaml
├── [1.7K Mar 13 08:13]  main/.pre-commit-config.yaml
├── [ 18K Mar 13 08:13]  main/.pylintrc
├── [4.8K Mar 13 08:13]  main/pyproject.toml
├── [4.0K Mar 13 08:13]  main/pytest_examples
│   ├── [1.3K Mar 13 08:13]  main/pytest_examples/divide.py
│   ├── [ 179 Mar 13 08:13]  main/pytest_examples/__init__.py
│   └── [ 491 Mar 13 08:13]  main/pytest_examples/shapes.py
├── [ 602 Mar 13 08:13]  main/README.md
└── [4.0K Mar 13 08:13]  main/tests
    ├── [ 681 Mar 13 08:13]  main/tests/conftest.py
    ├── [1.7K Mar 13 08:13]  main/tests/test_divide.py
    └── [1.6K Mar 13 08:13]  main/tests/test_shapes.py

5 directories, 14 files

Each branch can have a worktree added for it and then when you want to switch between them its is simply a case of cding into the worktree (/branch) you wish to work on. You use Git commands within the directory to apply them to that branch and Git keeps track of everything in the usual manner.

Lets create two worktree’s, the contributing and citation we created above when working with branches.

git worktree add contributing
git worktree add citation

You are now free to move between worktrees (/branches) and undertake work on each without having to git stash or git commit work in progress. We can add the CONTRIBUTING.md to the contributing worktree then jump to the citation worktree and add the CITATION.cff

cd contributing
echo "# Contributing\n\nContributions to this repository are welcome via Pull Requests." > CONTRIBUTING.md
cd ../citation
echo "cff-version: 1.2.0\ntitle: Pytest Examples\ntype: software" > CITATION.cff

Neither branches have had the changes committed so Git will not show any differences between them, but we can use diff -qr to compare the directories.

 diff -qr contributing citation
Only in citation: CITATION.cff
Only in contributing: CONTRIBUTING.md
Files contributing/.git and citation/.git differ

If we commit the changes to each we can git diff them.

cd contributing
git add CONTRIBUTING.md
git commit -m "Adding basic CONTRIBUTING.md"
cd ../citation
git add CITATION.cff
git commit -m "Adding basic CITATION.cff"
git diff citation contributing
CITATION.cff --- Text
1 cff-version: 1.2.0
2 title: Pytest Examples
3 type: software

CONTRIBUTING.md --- Text
1 # Contributing
2
3 Contributions to this repository are welcome via Pull Requests

NB The output of git diff may depend on the difftool that you have configured, I use and recommend the brilliant difftastic which has easy integration with Git.

Listing Worktrees

Just as you can git branch --list you can git worktree list

git worktree list
/mnt/work/git/hub/ns-rse/pytest-examples               (bare)
/mnt/work/git/hub/ns-rse/pytest-examples/citation      19ff076 [citation]
/mnt/work/git/hub/ns-rse/pytest-examples/contributing  ad56b91 [contributing]
/mnt/work/git/hub/ns-rse/pytest-examples/main          2f7c382 [main]

Moving Worktrees

You can move worktrees to different directories, these do not even have to be within the bare repository that you cloned as Git keeps track of these in the worktrees/ directory which has a folder for each of the worktrees you create and the file gitdir points to the location of that particular worktree.

cd pytest-examples   # Move to the bare repository
tree -afhD -L 2 worktrees
[4.0K Mar 13 09:27]  worktrees
├── [4.0K Mar 13 09:31]  worktrees/citation
│   ├── [  26 Mar 13 09:31]  worktrees/citation/COMMIT_EDITMSG
│   ├── [   6 Mar 13 09:27]  worktrees/citation/commondir
│   ├── [  55 Mar 13 09:27]  worktrees/citation/gitdir
│   ├── [  25 Mar 13 09:27]  worktrees/citation/HEAD
│   ├── [1.4K Mar 13 09:31]  worktrees/citation/index
│   ├── [4.0K Mar 13 09:27]  worktrees/citation/logs
│   ├── [   0 Mar 13 09:31]  worktrees/citation/MERGE_RR
│   ├── [  41 Mar 13 09:27]  worktrees/citation/ORIG_HEAD
│   └── [4.0K Mar 13 09:27]  worktrees/citation/refs
├── [4.0K Mar 13 09:30]  worktrees/contributing
│   ├── [  29 Mar 13 09:30]  worktrees/contributing/COMMIT_EDITMSG
│   ├── [   6 Mar 13 09:27]  worktrees/contributing/commondir
│   ├── [  59 Mar 13 09:27]  worktrees/contributing/gitdir
│   ├── [  29 Mar 13 09:27]  worktrees/contributing/HEAD
│   ├── [1.4K Mar 13 09:30]  worktrees/contributing/index
│   ├── [4.0K Mar 13 09:27]  worktrees/contributing/logs
│   ├── [   0 Mar 13 09:30]  worktrees/contributing/MERGE_RR
│   ├── [  41 Mar 13 09:27]  worktrees/contributing/ORIG_HEAD
│   └── [4.0K Mar 13 09:27]  worktrees/contributing/refs
└── [4.0K Mar 13 08:13]  worktrees/main
    ├── [   6 Mar 13 08:13]  worktrees/main/commondir
    ├── [  51 Mar 13 08:13]  worktrees/main/gitdir
    ├── [  21 Mar 13 08:13]  worktrees/main/HEAD
    ├── [1.3K Mar 13 08:13]  worktrees/main/index
    ├── [4.0K Mar 13 08:13]  worktrees/main/logs
    ├── [  41 Mar 13 08:13]  worktrees/main/ORIG_HEAD
    └── [4.0K Mar 13 08:13]  worktrees/main/refs

10 directories, 19 files

If we look at the gitdir file in each worktree sub-directory we see where they point to.

cat worktrees/*/gitdir
/mnt/work/git/hub/ns-rse/pytest-examples/citation/.git
/mnt/work/git/hub/ns-rse/pytest-examples/contributing/.git
/mnt/work/git/hub/ns-rse/pytest-examples/main/.git

These mirror the locations reported by git worktree list, albeit with .git appended.

If you want to move a worktree you can do so, here we move citation to ~/tmp.

git worktree move citation ~/tmp

Removing worktrees

It’s simple to remove a worktree after the changes have been merged or it is no longer needed, make sure to “prune” the tree after having done so.

git worktree remove citation
git worktree prune
git worktree list
/mnt/work/git/hub/ns-rse/pytest-examples               (bare)
/mnt/work/git/hub/ns-rse/pytest-examples/contributing  ad56b91 [contributing]
/mnt/work/git/hub/ns-rse/pytest-examples/main          2f7c382 [main]

Conclusion

Git Worktrees are a useful way of structuring your Git workflows if you have to switch branches regularly. They avoid the need to stash work in progress or make commits. If you do choose to use worktrees as an alternative to branches be mindful that you should remove and prune them after you have finished with them, particularly if you have a large codebase.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Git - {Worktrees}},
  date = {2024-03-13},
  url = {https://blog.nshephard.dev/posts/git-worktrees/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Git - Worktrees.” March 13, 2024. https://blog.nshephard.dev/posts/git-worktrees/.

Git - Making Amends and Fixing things up

Neil Shephard — Fri, 08 Mar 2024 00:00:00 GMT

Git is the world’s most popular version control software out there at the moment and if you write code and version control it the chances are you are using it. It’s a complex tool though and there are a bewildering array of options. In this short post we will look at some options for changing commits that have been made.

Crown Shy Trees by Me

Python Examples

We’ll use my pytest-examples as an example for this work but you can do this in any repository you have going. We’ll clone the repository and make a new branch called amend-fixup-tutorial.

git clone git@github.com:ns-rse/pytest-examples.git
cd pytest-examples
git switch -c amend-fixup-tutorial
  Switched to a new branch 'amend-fixup-tutorial'

The Git Dance

A typical Git work flow involves making some changes to one or more files (or adding a new one into the repository). These are staged with git add , or you can use a shortcut git add -u/--update to add all currently tracked files that have been modified if you are not adding anything new, before committing them with a message using git commit -m "".

Lets add a simple CONTRIBUTING.md file to the repository.

echo "# Contributing\n\nContributions via pull requests are welcome." > CONTRIBUTING.md
git add CONTRIBUTING.md
git commit -m "Adding CONTRIBUTING.md"

git log --oneline
  01191a2 (HEAD -> amend-fixup-tutorial) Adding CONTRIBUTING.md

This should be familiar to users of Git whether you use the command line interface (CLI), Emacs’ amazing Git porcelain magit or any other tool such as GitKraken or the support in you IDE such as RStudio or VSCode.

Making Amends

Sometimes you will have made a commit and you realise that you want to add more to it or perhaps you forgot to run your test suite and find that on running it your tests fail so you need to make a correction. In this example we want to be more explicit about how to make contributions and let people know they should fork the branch.

echo "\n Please make a fork of this repository, make your changes and open a Pull Request." >> CONTRIBUTING.md

Now you could make a second commit…

git add -u
git commit -m "Ask for PRs via fork in CONTRIBUTING.md"

git log --oneline
9f0655b (HEAD -> amend-fixup-tutorial) Ask for PRs via fork in CONTRIBUTING.md
01191a2 Adding CONTRIBUTING.md

…and there is nothing wrong with that. However, Git history can get long and complicated when there are lots of small commits, because these two changes to CONTRIBUTING.md are essentially the same piece of work and if we’d been thinking clearly we would have written about making forks in the first place and made a single commit.

Fortunately Git can help here as there is the git commit --amend option which adds the staged changes to the last commit and allows you to edit the last commit message (if nothing is currently staged then you will be prompted to edit the last commit message). We can undo the last commit using git reset HEAD~1 and instead amend the first commit that added the CONTRIBUTING.md

git add -u
git commit --amend

git log --oneline
  4fda15f (HEAD -> amend-fixup-tutorial) Adding CONTRIBUTING.md
cat CONTRIBUTING.md
# Contributing

Contributions via pull requests are welcome.

Please make a fork of this repository, make your changes and open a Pull Request.

We now have one commit which contains the new CONTRIBUTING.md file that contains all the changes we wished to have in the file in the first place and our Git history is slightly more compact.

Fixing things up

Amending commits is great providing the commit you want to change is the last commit you made (i.e. HEAD). But sometimes you might wish to correct a commit further back in your history and git commit --amend is of no use here. Git can however help here with the git commit --fixup command which allows you to mark a commit as being a “fix up” of an older commit. These can then be autosquashed via an interactive Git rebase.

Let’s add a few empty commits to our amend-fixup-tutorial branch to so we can do this.

git commit --allow-empty -m "Empty commit for demonstration purposes"
git commit --allow-empty -m "Another empty commit for demonstration purposes"

git log --oneline
  8061221 (HEAD -> amend-fixup-tutorial) Another empty commit for demonstration purposes
  65587ce Empty commit for demonstration purposes
  4fda15f Adding CONTRIBUTING.md

And let’s expand our CONTRIBUTING.md file further.

echo "\nPlease note this repository uses [pre-commit](https://pre-commit.com) to lint the Python code and Markdown files." >> CONTRIBUTING.md

We want to merge this commit with the first one we made in this tutorial using git commit --fixup. To do this we need to know the hash (4fda15f see output from above git log) or the relative reference of the commit we want which in this case is HEAD~2 as it is three commits back from the current HEAD (which is commit 0, most indexing in computing starts at 0 rather than 1). Use one for the following git commit --fixup commands (adjusting the hash to yours if you are using that option, you can find this using git log --oneline).

git add -u
git commit --fixup 4fda15f
git commit --fixup HEAD~2

We see the commit we have just made starts with fixup! and is then followed by the commit message that it is fixing.

git log --oneline
  97711a4 (HEAD -> amend-fixup-tutorial) fixup! Adding CONTRIBUTING.md
  8061221 Another empty commit for demonstration purposes
  65587ce Empty commit for demonstration purposes
  4fda15f Adding CONTRIBUTING.md

The final step is to perform the automatic squashing via an interactive rebase, again you can either use the hash or the relative reference.

git rebase -i --autosquash 4fda15f
git rebase -i --autosquash HEAD~2

This will open the default editor and because the --autosquash option has been used it will already have marked the commits that need combining with fixup. All you have to do is save the file and exit and we can check the history and look at the contents of the file.

NB If you find that the necessary commit isn’t already marked navigate to that line and delete pick. The lines below the file you have open give instructions on how you can mark commits for different actions, in this case you can replace pick with either f or fixup. Save and exit and the commits are squashed.

git log --oneline
  0fda21e (HEAD -> amend-fixup-tutorial) Another empty commit for demonstration purposes
  65587ce Empty commit for demonstration purposes
  4fda15f Adding CONTRIBUTING.md
cat CONTRIBUTING.md
  # Contributing

  Contributions via pull requests are welcome.

  Please make a fork of this repository, make your changes and open a Pull Request.

  Please note this repository uses [pre-commit](https://pre-commit.com) to lint the Python code and Markdown files.

And you’re all done! If you were doing this for real on a repository you could now git push or continue your work. As this was just an example we can switch branches back to main and force deletion of the branch we created.

git switch main
git branch -D amend-fixup-tutorial

Conclusion

Git has lots of commands to help you maintain a clean history by using --amend and --fixup flags to git commit and in the later case then performing an interactive git rebase -i. This takes a little discipline to get into the practice of but once in the habit of doing so it greatly improves the readability of the Git history and avoids including commit messages such as Fixing typo / Linting code / Fixing tests / I've gone mad!.

If all of this sounds completely unfamiliar to you but you would like to learn more about Git I can highly recommend the introductory course developed by Dr Anna Krystalli Git and GitHub through GitKraken : From Zero to Hero. This course is run regularly by myself and colleagues in Research Software Engineering for post-graduate researchers and staff at the University of Sheffield.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Git - {Making} {Amends} and {Fixing} Things Up},
  date = {2024-03-08},
  url = {https://blog.nshephard.dev/posts/git-amend-fixup/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Git - Making Amends and Fixing Things Up.” March 8, 2024. https://blog.nshephard.dev/posts/git-amend-fixup/.

Git Remotes Revisited

Neil Shephard — Sat, 17 Feb 2024 00:00:00 GMT

I’ve written before about Git Remotes but in my on-going effort to improve my understanding of the tools I use daily I’ve discovered that it is possible to have a single remote push to two different URLs so am revisiting the topic and perhaps writing a little more clearly on it.

Microcosmos by Me

Remotes are where other copies of your repository exist and typically where collaboration occurs (i.e. issue tracking, merge requests, bug reports etc.).

The main remote that a repository is configured to use by default is called origin but it is possible to have multiple remotes tracked by your local copy.

Listing Remotes

List remotes with git remote [-v] the -v flag will show the URLs that are stored for the short-cut.

❱ git remote -v
forgejo forgejo@forgejo.hopto.org:nshephard/mvdate.git (fetch)
forgejo forgejo@forgejo.hopto.org:nshephard/mvdate.git (push)
origin git@gitlab.com:nshephard/mvdate.git (fetch)
origin git@gitlab.com:nshephard/mvdate.git (push)

You can get more information about a remote using git remote show origin

❱ git remote show origin
 * remote origin
   Fetch URL: git@gitlab.com:nshephard/mvdate.git
   Push  URL: git@gitlab.com:nshephard/mvdate.git
   HEAD branch: main
   Remote branches:
     main                                                tracked
     refs/merge-requests/18/head                         new (next fetch will store in remotes/origin
     refs/pullreqs/15                                    stale (use 'git remote prune' to remove)
     refs/remotes/origin/nshephard/update-pre-commit     stale (use 'git remote prune' to remove)
     refs/remotes/origin/nshephard/update-readme         stale (use 'git remote prune' to remove)
   Local branches configured for 'git pull':
     main                            merges with remote main
     nshephard/fix-mtime             merges with remote nshephard/fix-mtime
     nshephard/update-pre-commit     merges with remote nshephard/update-pre-commit
   Local ref configured for 'git push':
     main pushes to main (local out of date)

This can be useful to show you what you need to tidy up if there are lots of stale branches around. In this example I can run git remote prune origin as advised to remove these.

Default Remote

The default remote to push to can be set with the following command, it will likely already be set to origin so this would not change anything.

git config --local remote.pushDefault origin

This adds the following to your .git/config if it wasn’t already there.

[remote]
    pushDefault = origin

Adding Remotes

It is straight forward to add a remote with git remote add where the URL is either the https or the git URL.

❱ git remote add forgejo forgejo@forgejo.hopto.org:nshephard/mvdate.git

This adds details to your .git/config so that it has the following

[remote "origin"]
    url = git@gitlab.com:nshephard/mvdate.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    fetch = +refs/merge-requests/*/head:refs/pullreqs/*
[remote "forgejo"]
    url = ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git
    fetch = +refs/heads/*:refs/remotes/forgejo/*
[remote]
    pushDefault = origin
[branch "main"]
    remote = origin
    merge = refs/heads/main

Two remotes are defined, origin and forgejo, the default to push to is set to origin and the main branch is setup to track the remote origin.

Pushing to specific remote

With two remotes setup you can choose, at the branch level, where to push your changes by specifying the remote you wish to use. If you wanted to push a newly created branch, change-just-for-forgejo, to the newly added forgejo remote you would configure it with.

❱ git switch -c change-just-for-forgejo
❱ git commit --allow-empty -m "Test push just to forgejo"
❱ git push --set-upstream forgejo change-just-for-forgejo
Enumerating objects: 2, done.
Counting objects: 100% (2/2), done.
Writing objects: 100% (2/2), 376 bytes | 376.00 KiB/s, done.
Total 2 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a new pull request for 'change-just-for-forgejo':
remote:   <https://forgejo.hopto.org/nshephard/mvdate/compare/main...change-just-for-forgejo>
remote:
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://forgejo.hopto.org:1234/nshephard/mvdate.git

-   [new branch]      change-just-for-forgejo -> change-just-for-forgejo

branch 'change-just-for-forgejo' set up to track 'forgejo/change-just-for-forgejo'.

Git reports that the local change-just-for-forgejo has been setup to track forgejo/change-just-for-forgejo and the following entry has been added to .git/config

[branch "change-just-for-forgejo"]
    remote "forgejo"
    merge = refs/heads/change-just-for-forgejo

A Note on SSH Ports

If a remote is using a non-standard port for SSH connections (i.e. anything other than 22) then you have to use a different format for specifying the remote URL. Instead of forgejo@forgejo.hopto.org:nshephard/mvdate.git you must explicitly state the protocol (ssh://) and include the port so that it reads ssh://forgejo@forgejo.hopto.org:1234 and so to add it you would be added with the following

❱ git remote add forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git
❱ git remote -v
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (fetch)
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)
origin  git@gitlab.com:nshephard/mvdate.git (fetch)
origin  git@gitlab.com:nshephard/mvdate.git (push)

I use a non-standard port and so use that convention for the remainder of this article. If you do not use a non-standard port you can either change the port (1234) to the default (22) or use the conventional syntax for referring to the remote.

Mirroring Remotes

The really neat thing is that it is possible to have a local repository track multiple remotes, which means when you push your changes it will go to both. You could configure an alias to push to both of the remotes we currently have defined, but there is an excellent post on StackOverflow that shows how to do this with Git itself because each remote can have multiple pushurls.

As we have added a second remote to our configuration our .git/config for mvdate our configuration currently looks like this.

[remote "origin"]
    url = git@gitlab.com:nshephard/mvdate.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    fetch = +refs/merge-requests/*/head:refs/pullreqs/*
[remote "forgejo"]
    url = ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git
    fetch = +refs/heads/*:refs/remotes/forgejo/*
[remote]
    pushDefault = origin
[branch "main"]
    remote = origin
    merge = refs/heads/main
[branch "change-just-for-forgejo"]
    remote = forgejo
    merge = refs/heads/change-just-for-forgejo

As above, there are two remotes are defined, origin and forgejo, the default to push to is set to origin and the main branch is setup to track origin whilst the change-just-for-forgejo branch is setup to track forgejo.

❱ git remote -v
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (fetch)
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)
origin  git@gitlab.com:nshephard/mvdate.git (fetch)
origin  git@gitlab.com:nshephard/mvdate.git (push)

How do we get the origin remote setup and configured to push to both gitlab and forgejo? This can be done using the set-url --add --push options to git remote, below we add forgejo as a push target to origin.

NB Note I use a non-standard SSH port in the following, see above note.

❱ git remote set-url --add --push origin ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git
❱ git remote -v
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (fetch)
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)
origin  git@gitlab.com:nshephard/mvdate.git (fetch)
origin  ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)

But this has removed the original push target under origin which pointed to gitlab so we need to add that back in.

❱ git remote set-url --add --push origin git@gitlab.com:nshephard/mvdate.git
❱ git remote -v
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (fetch)
forgejo ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)
origin  git@gitlab.com:nshephard/mvdate.git (fetch)
origin  ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)
origin  git@gitlab.com:nshephard/mvdate.git (push)

We now have two push targets on origin, one pointing to gitlab.com (using the default port 22) and one pointing to forgejo.hopto.org (on port 1234) and as the default target is origin when we git push it will send the changes to both. We still have the forgejo remote defined and it only tracks the forgejo URL.

We can test this with an empty commit on a new branch, test-both, which we first create.

❱ git switch -c test-both
❱ git commit --allow-empty -m "Testing pushing to GitLab and Forgejo"
[test-both c07caf6] Testing pushing to GitLab and Forgejo
❱ git push
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 210 bytes | 210.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a new pull request for 'test-both':
remote:   <https://forgejo.hopto.org/nshephard/mvdate/compare/main...test-both>
remote:
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://forgejo.hopto.org:1234/nshephard/mvdate.git

-   [new branch]      test-both -> test-both

branch 'test-both' set up to track 'origin/test-both'.
Enumerating objects: 26, done.
Counting objects: 100% (26/26), done.
Writing objects: 100% (26/26), 16.75 KiB | 8.37 MiB/s, done.
Total 26 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for test-both, visit:
remote:   <https://gitlab.com/nshephard/mvdate/-/merge_requests/new?merge_request%5Bsource_branch%5D=test-both>
remote:
To gitlab.com:nshephard/mvdate.git

-   [new branch]      test-both -> test-both

branch 'test-both' set up to track 'origin/test-both'.

The output above shows that the branch test-both was pushed to both the URLs we have configured as push targets to origin and if you visit the repositories you will find the branches now exist there.

Deleting Remotes

In my use case I simply want to push both remotes so that they mirror each other so I can delete the forgejo remote. This will leave the push URL for that remote under the configuration for origin and allows us to set any branch to use the origin as a the remote and any changes will be pushed to both.

There may be instances where you want to leave the additional remote in place if you wanted to push some changes just to that remote so its not essential that you remove it, but if you want to you can delete reference to a remote from your local configuration.

❱ git remote remove forgejo
❱ git remote -v
origin  git@gitlab.com:nshephard/mvdate.git (fetch)
origin  ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git (push)
origin  git@gitlab.com:nshephard/mvdate.git (push)

Because the change-just-for-forgejo was setup to track the forejo remote we would need to change that target, we can do so with the following

❱ git switch change-just-for-forgejo
❱ git branch --set-upstream-to=origin/change-just-for-forgejo change-just-for-forgejo

This changes the remote target for the branches definition and our configuration now looks like the following.

[remote "origin"]
    url = git@gitlab.com:nshephard/mvdate.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    fetch = +refs/merge-requests/*/head:refs/pullreqs/*
    pushurl = ssh://forgejo@forgejo.hopto.org:1234/nshephard/mvdate.git
    pushurl = git@gitlab.com:nshephard/mvdate.git
[remote]
    pushDefault = origin
[branch "main"]
    remote = origin
    merge = refs/heads/main
[branch "change-just-for-forgejo"]
    remote = origin
    merge = refs/heads/change-just-for-forgejo

Conclusion

Having a local repository push to two remotes is a simple way of a mirroring. Whether you have a use case for it depends on what you are doing. I could easily imagine this could get very complicated if changes were pushed by others to each remote, but I wouldn’t be surprised if Git is regularly used in this way by others.

Of course if you want to push a branch that you wish to keep private to one remote only then you would have to be very careful in how you use this setup. The original StackOverflow solution inspired that this post suggests creating an independent remote (e.g. all) so that you can push changes to origin or the second remote (in this example forgejo) and use all only when you wish to push changes to both.

Its been good for me to return to a topic I’ve delved into in the past, this second time round I feel I’ve got a slightly better grasp of what I’m doing and have a neater solution to achieve what is required.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Git {Remotes} {Revisited}},
  date = {2024-02-17},
  url = {https://blog.nshephard.dev/posts/git-remotes-revisited/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Git Remotes Revisited.” February 17, 2024. https://blog.nshephard.dev/posts/git-remotes-revisited/.

Pytest Parameterisation

Neil Shephard — Mon, 01 Jan 2024 00:00:00 GMT

Pytest is an excellent framework for writing tests in Python. One of the neat features it includes is the ability to parameterise your tests which means you can write one test and pass different sets of parameters into it to test the range of actions that the function/method are meant to handle.

Minerals by Me

Example

A simple example to work through is provided in my ns-res/pytest_examples repository. We want to have a state where the function can fail so we’ll use a very simple function that carries out division.

def divide(a: float | int, b: float | int) -> float:
    """Divide a by b.

    Parameters
    ----------
    a: float | int
        Number to be divided.
    b: float | int
        Number to divide by.

    Returns
    -------
    float
        a divided by b.
    """
    try:
        return a / b
    except TypeError as e:
        if not isinstance(a, (int | float)):
            raise TypeError(f"Error 'a' should be int or float, not {type(a)}") from e
        raise TypeError(f"Error 'b' should be int or float, not {type(b)}") from e
    except ZeroDivisionError as e:
        raise ZeroDivisionError(f"Can not divide by {b}, choose another number.") from e

Structuring Tests

Pytest is well written and will automatically find your tests in a few places. Personally I use a flat rather than src/ based package layout and keep my tests in the tests/ directory of the package root. Pytest looks in this directory automatically for files that begin with test_ and within each file for functions/methods that begin with test_.

With the above function we could write the following basic test to make sure it works because we know that if we divide 10 by 5 we should get 2 as the answer.

from pytest_examples.divide import divide


def test_divide_unparameterised() -> None:
    """Test the divide function."""
    assert divide(10, 5) == 2

You can find this test along with others in the tests/test_divide.py file of the accompanying repository.

Parameterising Tests

In order to make our test suite robust we should test more scenarios and edge cases, in particular making sure we capture the exceptions that can be raised. This is where the pytest.mark.parameterize() fixture comes into play. It takes as a first argument a tuple of variables that you are going to define values for and pass into your test. Following it is a list of tuples with the values that you want to include, one for each of the variables you have first defined. Here we define a, b and the expected value of dividing a by b which is the value the divide() function should return.

If we expand the number of scenarios we wish to test using @pytest.mark.parametrize() we can write our test as follows.

import pytest

from divide import divide
@pytest.mark.parametrize(
    ("a", "b", "expected"),
    [
        (10, 5, 2),
        (9, 3, 3),
        (5, 2, 2.5),

    ]
)
def test_divide(a: float | int, b: float | int, expected: float) -> None:
    """Test the divide function."""
    assert divide(a, b) == expected

Parameter set IDs

For some time I simply wrote my tests and if the structure was complicated I used comments to mark the code to indicate what the test was doing. When they (inevitably!) failed there was a cryptically long indication of what had failed based on the filename, test name and the values of the various parameters that were in use at the point of failure. These helped narrow down which test failed but took a bit of mental over-head to decipher.

For the above test without ID’s we can force them to fail by adding 1 to the expected value (i.e. == expected + 1) and the resulting output shows how the parameters are concatenated to indicate which test failed.

======================= short test summary info ====================================
FAILED tests/test_divide.py::test_divide_fail[10-5-2] - assert 2.0 == (2 + 1)
FAILED tests/test_divide.py::test_divide_fail[9-3-3] - assert 3.0 == (3 + 1)
FAILED tests/test_divide.py::test_divide_fail[5-2-2.5] - assert 2.5 == (2.5 + 1)
======================= 3 failed in 0.79s ==========================================

Whilst it is possible to work out which failed test is which if you have many sets of parameters with multiple values and only one or two are failing it can take a while to work out which set has failed.

Recently though I was put onto the pytest.param() function by a toot from @danjac@masto.ai and instantly saw the benefit of using this as it allows us to give each set of parameters a unique id which is then used by Pytest when reporting failures.

@pytest.mark.parameterize(
    ("a", "b", "expected"),
    [
        pytest.param(10, 5, 2, id="ten divided by five"),
        pytest.param(9, 3, 3, id="nine divided by three"),
        pytest.param(5, 2, 2.5, id="five divided by two"),

    ]
)
def test_divide(a: float | int, b: float | int, expected: float) -> None:
    """Test the divide function."""
    assert divide(a, b) == expected

Then if/when a test fails the id parameter is reported for the failed test, making it much easier to narrow down where the failure occurred.

Not only does it allow each set of parameters to be given a unique id = "" to aid with identifying tests that fail it also allows each set of parameters to be marked with marks = <> to indicate the expected behaviour for example pytest.mark.xfail or pytest.mark.skipif.

We could therefore add another set of parameters that should fail because one of the exceptions is raised.

import pytest

from pytest_examples.divide import divide


@pytest.mark.parameterize(
    ("a", "b", "expected"),
    [
        pytest.param(10, 5, 2, id="ten divided by five"),
        pytest.param(9, 3, 3, id="nine divided by three"),
        pytest.param(5, 2, 2.5, id="five divided by two"),
        pytest.param(
            10, 0, ZeroDivisionError, id="zero division error", marks=pytest.mark.xfail
        ),
    ],
)
def test_divide(a: float | int, b: float | int, expected: float) -> None:
    """Test the divide function."""
    assert divide(a, b) == expected

Testing Exceptions

The above example shows that Pytest allows us to combine tests that pass and fail (in the above example a ZeroDivisionError) via parmeterisation. However, whilst tests can and should be parameterised, some consider that it is better to keep tests focused and on-topic and write a separate test for different outcomes such as raising exceptions.

This is slightly different from the way the Pytest documentation suggests to undertake Parameterising conditional raising but there is a school of thought, which I like, which states that testing different states/behaviours should be separate (see the following thread for some discussion Why should unit tests test only one thing?).

With this in mind we can separate out the tests that raise exceptions under different scenarios to their own tests (NB obviously its excessive to parameterise test-divide_zero_division_error()).

@pytest.mark.parametrize(
    ("a", "b", "exception"),
    [
        pytest.param("a", 5, TypeError, id="a is string"),
        pytest.param(9, "b", TypeError, id="b is string"),
        pytest.param([1], 2, TypeError, id="a is list"),
        pytest.param(10, [2], TypeError, id="b is list"),
    ],
)
def test_divide_type_errors(a: float | int, b: float | int, exception: float) -> None:
    """Test that TypeError is raised when objects other than int or float are passed as a and b."""
    with pytest.raises(exception):
        divide(a, b)

@pytest.mark.parametrize(
    ("a", "b", "exception"),
    [
        pytest.param(10, 0, ZeroDivisionError, id="b is zero"),
    ],
)
def test_divide_zero_division_error(a: float | int, b: float | int, exception: float) -> None:
    """Test that ZeroDivsionError is raised when attempting to divide by zero."""
    with pytest.raises(exception):
        divide(a, b)

Parameterising with Fixtures

Fixtures are a common and useful feature of the Pytest framework that allow you to define “defined, reliable and consistent context for the tests”. What this means is that if you always need a particular object, whether that is an instantiated class (a new instance of a class) or something else, you can mark a function with @pytest.fixture() and use it in subsequent tests (often fixtures are defined in tests/conftest.py to keep things tidy, at least that is what I do!)¹.

It can be useful to parameterise fixtures themselves so that they too test a number of different states and this saves writing more sets of parameters under the @pytest.mark.parameterize() decorator of each test.

For this example we use a simple function summarise_shapes() which returns the results of summarising a 2-D Numpy array using scikit-image and its skimage.measure.regionprops() function (see pytest_examples/shapes.py).

"""Summarise Shapes."""
import numpy.typing as npt
from skimage import measure


def summarise_shape(shape: npt.NDArray) -> list:
    """
    Summarise the region properties of a 2D numpy array using Scikit-Image.

    Parameters
    ----------
    shape : npt.NDArray
        2D binary array of a shape.

    Returns
    -------
    list
        List of Region Properties each item describing one labelled region.
    """
    return measure.regionprops(shape)

We want to write some tests for these using fixtures which we define in tests/conftest.py. These define two Numpy 2-D binary arrays of 0’s and 1’s in particular shapes (the names should give an indication of the shapes!)

import numpy as np
import numpy.typing as npt
import pytest

from skimage import draw


@pytest.fixture
def square() -> npt.NDArray:
    """Return a 2D numpy array of a square."""
    square = np.zeros((6, 6), dtype=np.uint8)
    start = (1, 1)
    end = (5, 5)
    rr, cc = draw.rectangle_perimeter(start, end, shape=square.shape)
    square[rr, cc] = 1
    return square


@pytest.fixture
def circle() -> npt.NDArray:
    """Return a 2D numpy array of a circle."""
    circle = np.zeros((7, 7), dtype=np.uint8)
    rr, cc = draw.circle_perimeter(r=4, c=4, radius=2, shape=circle.shape)
    circle[rr, cc] = 1
    return circle

There are two different methods to using these fixtures in parameterised tests.

request.getfixturevalue()

The first uses request.getfixturevalue() which “is a special fixture providing information of the requesting test function.”, in this case the “named fixture function”.

You define the fixture name (in quotes) in the @pytest.mark.parametrize() and then when the parameter, in this case shape, is referred to in the test itself, you wrap it in request.getfixturevalue() and the named fixture is then returned and used.

"""Test the shapes module."""
import pytest

from pytest_examples.shapes import summarise_shape


@pytest.mark.parametrize(
    ("shape", "area", "feret_diameter_max", "centroid"),
    [
        pytest.param("square", 11, 7.810249675906654, (1.3636363636363635, 1.3636363636363635), id="summary of square"),
        pytest.param("circle", 12, 5.385164807134504, (4, 4), id="summary of circle"),
    ],
)
def test_summarise_shape_get_fixture_value(
    shape: str, area: float, feret_diameter_max: float, centroid: tuple, request
) -> None:
    """Test the summarisation of shapes."""
    shape_summary = summarise_shape(request.getfixturevalue(shape))
    assert shape_summary[0]["area"] == area
    assert shape_summary[0]["feret_diameter_max"] == feret_diameter_max
    assert shape_summary[0]["centroid"] == centroid

pytest-lazy-fixture

An alternative is to use the Pytest plugin pytest-lazy-fixture and instead of marking the value to be obtained in the test itself you do so when setting up the parameters by referring to the fixture name as an argument to pytest.lazy_fixture() within @pytest.mark.parametrize().

"""Test the shapes module."""
import pytest

from pytest_examples.shapes import summarise_shape


@pytest.mark.parametrize(
    ("shape", "area", "feret_diameter_max", "centroid"),
    [
        pytest.param(
            pytest.lazy_fixture("square"),
            11,
            7.810249675906654,
            (1.3636363636363635, 1.3636363636363635),
            id="summary of square",
        ),
        pytest.param(pytest.lazy_fixture("circle"), 12, 5.385164807134504, (4, 4), id="summary of circle"),
    ],
)
def test_summarise_shape_lazy_fixture(
    shape: str, area: float, feret_diameter_max: float, centroid: tuple, request
) -> None:
    """Test the summarisation of shapes."""
    shape_summary = summarise_shape(shape)
    print(f"{shape_summary[0]['centroid']=}")
    assert shape_summary[0]["area"] == area
    assert shape_summary[0]["feret_diameter_max"] == feret_diameter_max
    assert shape_summary[0]["centroid"] == centroid

Parameterise Fixtures

The pytest-lazy-fixture also allows fixtures themselves to be parameterised using the pytest_lazyfixture.lazy_fixture() function and demonstrated in the packages README which I’ve reproduced below.

The fixture called some() uses lazy_fixture() to include both the one() and the two() fixtures which return their respective integers. test_func() then checks that the value returned by the some() fixture is in the list [1, 2]. Obviously this example is contrived but it serves to demonstrate how fixtures themselves can be parameterised.

import pytest
from pytest_lazyfixture import lazy_fixture

@pytest.fixture(params=[
    lazy_fixture('one'),
    lazy_fixture('two')
])
def some(request):
    return request.param

@pytest.fixture
def one():
    return 1

@pytest.fixture
def two():
    return 2

def test_func(some):
    assert some in [1, 2]

Conclusion

Pytest is a powerful and flexible suite for writing tests in Python. One of the strengths is the ability to parameterise the tests to test multiple scenarios. This can include both successes and failures, however a common approach is to separate tests based on the expected behaviour, although Pytest allows you the flexibility to choose.

Ultimately though parameterising tests is a simple and effective way of reducing the amount of code you have to write to unit-tests for different aspects of your code.

Footnotes

A caveat to this is the use of Random Number Generators as once seeded these can produce different numbers depending on the order in which the fixture is used but that is beyond the scope of this post.↩︎

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2024,
  author = {Shephard, Neil},
  title = {Pytest {Parameterisation}},
  date = {2024-01-01},
  url = {https://blog.nshephard.dev/posts/pytest-param/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2024. “Pytest Parameterisation.” January 1, 2024. https://blog.nshephard.dev/posts/pytest-param/.

virtualenvwrapper

Neil Shephard — Sat, 23 Dec 2023 00:00:00 GMT

If you use Python heavily you will likely be familiar with Virtual Environments. These provide isolated installs of specific packages that take precedence over any packages installed at the system level. There are lots of tools and frameworks for working with virtual environments such as venv, virtualenv and Conda. This post introduces and shows some of the features of virtualenvwrapper.

Clouds Rising at Sunset by Me

virtualenvwrapper

virtualenvwrapper is…

a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

It has the following main features…

Organizes all of your virtual environments in one place.

Wrappers for managing your virtual environments (create, delete, copy).

Use a single command to switch between environments.

Tab completion for commands that take a virtual environment as argument.

User-configurable hooks for all operations (see Per-User Customization).

Plugin system for creating more sharable extensions (see Extending Virtualenvwrapper).

Installation

Many systems have virtualenvwrapper available in their package manager.

emerge -av virtualenvwrapper          # Gentoo
pacman -Syu python-virtualenvwrapper  # Arch
apt-get install virtualenvwrapper     # Debian

Once installed you need to set two key variables $WORKON_HOME and $PROJECT_HOME and ensure the virtualenvwrapper.sh is sourced on starting a shell. $WORKON_HOME is where your virtual environments will be created and stored, whilst $PROJECT_HOME is where projects will be created if you choose to use the helper functions for making projects. Set these to what you want, my options are below. To find out where virtualenvwrapper.sh is installed on your system use which virtualenvwrapper.sh. Once you’ve decided substitute the values in the following of your .bashrc (Bash)or .zshrc (ZSH) depending on which shell you use.

export WORKON_HOME=${HOME}/.virtualenvs
export PROJECT_HOME=${HOME}/work/git/
source /usr/bin/virtualenvwrapper.sh

Creating a Virtual Environment

This is straight-forward.

mkvirtualenv <env_name>

`postmkvirtualenv`

But what if there are some tools that you want to install each and every time you create a virtual environment, regardless of the project you are working on? For example I use the jedi-language-server and want to have various packages such as ipython, pytest and various extensions, linters such as ruff and pylint available by default in every environment you create. Fortunately there is a simple hook that can be run after the creation of a new environment. The file ~/.virtualenvs/postmkvirtualenv is sourced and run after having run mkvirtualenv and so any commands in there are executed as it is essentially a Bash script.

If you maintain a dotfiles directory and have a file that lists the packages you want installed under ~/dotfiles/python/venv_minimal_requirements.txt then you can have the files listed here installed when creating a new virtual environment by appending the command pip install -r ~/dotfiles/python/venv_minimal_requirements.txt to the ~/.virtualenvs/postmkvirtualenv file.

echo "pip install -r ~/dotfiles/python/venv_minimal_requirements.txt" >> ~/.virtualenvs/postmkvirtualenv

Project Directories

Typically code for a project resides in its own directory and this can be automatically bound to the virtual environment using the mkproject command instead of mkvirtualenv. The project directory is stored in the $PROJECT_HOME path you will have configured during installation. You can then create a project and a virtual environment with…

mkproject new_project

Switching to Project Directories

You can switch to a projects directory automatically on activating a particular virtual environment using setvirtualenvproject from the project directory when a specific environment is activated. Make sure you are in the project directory for the corresponding project!. It adds an entry to the ~/.virtualenv//.project file that reflects the directory associated with the environment. Then when you activate the directory via workon it will automatically change to the project directory.

Deactivating and Removing Virtual Environments

Its straight-forward to deactivate the current virtual environment just type deactivate. Similarly you can remove a virtual environment with rmvirtualenv .

One neat option if you want to keep a virtual environment but install all packages anew is the ability to remove all third-party packages in the current virtual environment using wipeenv.

Temporary Virtual Environments

Sometimes you just want to try something out quickly in a clean Virtual Environment, if for example you are reviewing a Pull Request. virtualenvwrapper can help here as it has the mktmpenv. There are two options here -c|--cd or -n|--no-cd which changes directory post-activation or doesn’t respectively. The environment gets a unique name and will be deleted automatically when it is deactivated.

Drawbacks

There are a couple of drawbacks I’ve found to using using virtualenvwrapper.

The first is that mkproject doesn’t allow nesting of project directories, you have to specify a single directory and it will be created under the $PROJECT_HOME directory with the associated environment name. This doesn’t work for me as I use the structure ~/work/git as the base but then have sub-directories based on the Git Forge (GitHub/GitLab/Codeberg) the repository is associated with and further nesting to reflect the user/organisation within as I have both a personal and work accounts. E.g. ~/work/git/hub/ns-rse/ns-rse.github.io which is the source for this site and associated with my work account (ns-rse) or ~/work/git/lab/nshephard/tcx2gpx which is a project of mine (tcx2gpx) hosted on GitLab. This means that if I wanted to create a project with mkproject based on $PROJECT_HOME being /work/git following this structure I would specify mkproject git/lab/new_project and whilst the directory is created, the virtual environment is created as git/lab/new_project which is truncated to git and you can’t workon git because the activation scripts are nested deeper under git/lab/new_project. Further each environment I created would then conflict. I could probably work around this by creating symbolic links but in practice I just use mkvirtualenv and setvirtualenvproject after I git clone work.

This is a problem specifically of my own creation though, something other users might find causes greater friction is that virtualenvwrapper doesn’t support creating and keeping the virtual environments within the project directory itself. This is never something that I’ve wanted to do myself though as I find it tidier to keep them all in one place and easier to find and remove obsolete environments.

Conclusion

I’ve used virtualenvwrapper for years and its a nice light-weight alternative to other solutions of using Pythons Virtual Environments such as venv or Conda. It has some limitations but its worth giving it a whirl as there are lots of useful helper functions and hooks that smooth the process of creating, using and switching between virtual environments.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2023,
  author = {Shephard, Neil},
  title = {Virtualenvwrapper},
  date = {2023-12-23},
  url = {https://blog.nshephard.dev/posts/virtualenvwrapper/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2023. “Virtualenvwrapper.” December 23, 2023. https://blog.nshephard.dev/posts/virtualenvwrapper/.

Repository Review with Scientific Python

Neil Shephard — Fri, 17 Nov 2023 00:00:00 GMT

I’ve written before about Python Packaging and pre-commit which I’m a big fan of. Today I discovered a really useful tool for checking your packaging configuration and pre-commit configuration from the Scientific Python Development Guide.

Elektro

This development guide is really well written and has Tutorials and Topical Guides that cover package development, style guides, type checking and tests amongst other things.

Repo Review

Of particular use is the Repo-Review - Scientific Python Development Guide which will automatically check a GitHub hosted repository against the Scientific Python development guidelines and make recommendations where improvements can be made to the package configuration and pre-commit configuration.

Having tested it against AFM-SPM/TopoStats main branch there are a lot of useful and simple recommendations made.

More Pre-Commit hooks

More pre-commit hooks is only a good thing in my view, they don’t take that long to run against most reasonably sized repositories, particularly if ruff is used in favour of more traditional tools such as flake8 and isort.

There are a few pre-commit hooks that I wasn’t aware of and will be adding to my projects such as..,

Check-manifest that verifies you have working SDist (source) packages.
Codespell for checking the spelling used in code.
validate-pyproject for validating pyproject.toml.

Conclusion

If you are involved in Python package development of any sort give your repository a review and see what recommendations are suggested. You may not agree with all of the recommendations but the vast majority of tools are highly configurable so you can disable the checks you don’t want applied. It won’t do any harm.

Now I just need to find time to apply MyPy.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2023,
  author = {Shephard, Neil},
  title = {Repository {Review} with {Scientific} {Python}},
  date = {2023-11-17},
  url = {https://blog.nshephard.dev/posts/repository-review/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2023. “Repository Review with Scientific Python.” November 17, 2023. https://blog.nshephard.dev/posts/repository-review/.

Git Remotes

Neil Shephard — Fri, 06 Oct 2023 00:00:00 GMT

Git and various forges such as GitHub GitLab are useful collaborative tools for version controlling, sharing and working collaboratively. Normally a repository resides on your local computer and it tracks a remote (often referred to as origin)

Looking up by me

`.git/config`

When you initialise or clone a repository a hidden directory .git is created. Within this resides the configuration file for the repository .git/config that defines how Git is to behave when performing actions on the repository.

`core`

`remote`

The remote field defines the location of the repository to which code is synced via pushing and pulling. Typically and by default this is called the origin and you have a section that defines the url and the fetch for this remote.

The url field can take two forms, either one based on https in which case the prefix of the value will be https://github.com/ or one based on ssh in which case the prefix will be git@github.com:. What follows is then the user account or organisation (e.g. ns-rse for my personal repositories) followed by / (e.g. for this repository that is ns-rse.github.io).

The fetch field is what in Git parlance is called a Refspec (Reference Specification). This takes the form : and is a method of mapping references on the remote side () to those locally () and it is this that maps the local branches (refs/heads/*) to their counterparts on the remote (refs/remotes/origin/*). If there is a + prefix it tells git to up-date the reference even if there is no fast-forward.

[remote "origin"]
    url = git@github.com:ns-rse/ns-rse.github.io
    fetch = +refs/heads/*:refs/remotes/origin/*

`branch`

What follows is then a series of entries for branch which defines further how each local branch maps to a remote and where it should merge to. Each branch has a name define in the section header and within two parameters are set the remote which by default points to the origin (defined in the above remote) section and a merge field which defines the local reference to the branch under refs/head/.

Having a Private branch

Sometimes the need might arise to develop a feature in private. This is fine if you have no intention of sharing the work with others, you create a branch on your local computer and never push it to a forge. However, there will arise times where you do want to share you work with others whilst keeping things private. This too is possible and can be achieved by creating a private repository on your own account rather than using the original and configuring a branch to track that instead.

Creating a second `remote`

Start by creating a new repository on GitHub/GitLab and making it private from the outset. Do not include any additional information such as .gitignore, README.md or LICENSE.md it should be completely empty.

You can then add it as a new remote to your existing repository in one of two ways.

Git at the Command Line

You should have some instructions shown on GitHub one of which is …or push an existing repository from the command line. The first line of this is telling you how to add a new remote to the repository. called origin and to point towards the repository you have just created. You will likely already have an origin remote defined so you need to choose a different name and point it to the correct location.

git remote add private-work git@github.com:ns-rse/private-work

Check your `.git/config`

If you now look at your .git/config file there should be two entries for [remote ""] one for origin (i.e. [remote "origin"]) and one for the new remote that has just been added [remote "private-work"]. Of course, because .git/config is just a text configuration file you can edit it and enter these values manually yourself if you want to.

NB Either of the above methods will append these options to the bottom of the file so if you can’t see it immediately scroll down.

Track your new remote

Now that you have a secondary remote defined you can create a branch and set it to track the private remote you have created and configured. Create a new branch by using git checkout -b

Command Line

git checkout -b private-feature

You then set the upstream using --set-upstream-to or its shorthand -u, at this stage you don’t have anything to include so make an empty commit

git push --set-upstream-to private-work private-work

Magit

Check `.git/config`

Returning to your .git/config file you should now have an additional entry for a branch underneath the new entry for the remote and the value for remote under branch is the name of the remote.

[remote "private-work"]
 url = git@github.com:ns-rse/private-work.git
 fetch = +refs/heads/*:refs/remotes/private-work/*
[branch "private-feature"]
 remote = private-work
 merge = refs/heads/private-feature

Keeping up to Date

The private repository you setup on GitHub/GitLab should now have a copy of the private-work branch you created locally.

If you are collaborating with others the main/master branch may move ahead of yours as others work is merged in. You therefore need to regularly update your private branch by merging or rebasing from the origin rather than the private-work remote your branch is configured to track. The difference between merging and rebasing is beyond the scope of this article, there are pros and cons to each

git checkout main
git pull
git checkout private-work
git merge main

Or you can merge directly from origin with

git checkout private-feature
git merge origin/main

Checkout someone else’s private branch

Inevitably the need might arise for a collaborator to test your private branch. In which case they need to be granted permission to the private repository by the developer who created it. This is done via Settings > Collaborators and inviting them to work on your repository.

Once they have accepted the invitation they will also have to add a secondary remote and the branch they are working on. They can follow the instructions above to set up a remote, however they should not follow the instructions to branch because that branch already exists on your private repository.

but because .git/config is just a text file you can copy the lines from your configuration and share it with your collaborator and they can add them to their .git/config file. Once these options have been added they

Bonus - Using Magit

If you use Emacs and the amazing Magit you can of course do this via the Magit interface. In the Magit buffer for the repository (e.g. magit: ns-rse.github.io) press M to bring up the Remote transient buffer. You then press a to add a remote and are prompted for a name for the remote (in this example we use the same as above private-work), this can be anything you want other than origin which is already defined, and then the URL this will be git@github.com:/.git.

To create a branch in Magit press b then l to select a local branch and enter main (or master), then make sure its upto date with the origin by Pulling with P then u. Now create a new branch by pressing b and since it will be a new press c to create it. You will be prompted for the name of a branch from which to branch from and then for the name of your branch.

In the Magit buffer for the repository you are working on pushes are made with P. The Transient buffer then offers you a choice of places to push to, one of which is elsewhere which is accessed by the e key. You can then type in the name of the remote tracking branch as defined under the remote entry you configured above which in this case is private-work.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2023,
  author = {Shephard, Neil},
  title = {Git {Remotes}},
  date = {2023-10-06},
  url = {https://blog.nshephard.dev/posts/git-remotes/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2023. “Git Remotes.” October 6, 2023. https://blog.nshephard.dev/posts/git-remotes/.

R Resources

Neil Shephard — Fri, 06 Oct 2023 00:00:00 GMT

R is a statistical programming languages and one of the most popular languages for data analysis, statistics and plotting in academia and industry. Learning a new language can be daunting, particularly if you have no experience of scripting and are used to Graphical User Interfaces (GUIs) where you point and click to perform your statistical analysis.

Fear not though, there are lot of resources and very friendly, enthusiastic and helpful R users out there who can help you on your journey learning R. This post details some of them, and I’d welcome additions.

R, by lizjones Me

Most of these resources are links websites that are free and openly available. Where books are linked they are very often freely available on-line, but there will also often be the possibility of purchasing a hard copy, which you may want to consider doing if you find the resource useful to help support the authors.

R has a number of bodies, organisations and companies associated with it.

The R Foundation a not for profit organisation working in the public interest.
The R Consortium a group organized under an open source governance and foundation model to support the worldwide community of users.
rOpenSci open data, software and reproducibility.

Software

R is software and will need installing on your computer. Because it is Free Open Source Softrware (FOSS) you can download and install it on your computer for free. You will have to install it to use it and the isntr

Git Version Control

It is good practice to version control the code you write, it provides an electronic paper trail of how your code has evolved over time and allows you to keep track not just of the code itself but why it has changed or been written.

These days the most popular version control system is Git and projects are often hosted/backed up on popular “forges” such as GitHub or GitLab. Sign up with an academic email address (@.ac.uk or @.edu) and you will have a few extra benefits.

Learning Git is a whole, vast, topic in and of itself, but to get started with R and Git see the recommendation below. If you are a student or researcher at The University of Sheffield you may want to consider taking the Research Software Engineering (RSE) Teams popular Git, GitHub and GitKraken : Zero to Hero course which runs regularly throughout the year. Sign up to their mailing list and you’ll be notified of when the course runs. Alternatively email them to find out when the next course is scheduled to run.

IDEs

Integrated Development Environments (IDE) are software that help you write code faster and more consistently courtesy of various features such as syntax highlighting, automatic bracket and quote pairing, automatic indentation and a suite of functions for performing common tasks such as version controlling files or rendering documents.

The most popular IDE for R is RStudio Desktop which has excellent support for R, RMarkdown/Quarto and basic Git support. If you are new to version control with Git you may want to consider using GitKraken which provides an intuitive point and click interface for version controlling your files and working with GitHub/GitLab.

My personal preference is to use Emacs and the package Emacs Speaks Statistics (ESS). This is a robust solution (ESS) has been around for decades and you get the convenience of using Emacs and its many packages such as the amazing Magit for carrying out all Git related tasks. It has a steeper learning curve than RStudio but in my opinion is completely worth the effort.

Books

If you’re using R the chances are you want to perform some sort of Statistical Analysis on your data. This often involves cleaning data that has been received, writing code to summarise, tabulate and plot your data, often in a literate manner (which means reports are open and can be reproduced easily). If you read nothing else to get you started using R for this work then you should read R for Data Science by Hadley Wickham and Garrett Grolemund. This is an excellent book that is available for free online.

teacheR - Teach Yourself or Others R by Adam Rawles
Cookbook for R by Winston Chang is a useful reference for many common tasks.

Quarto/RMarkdown

R has its own Markdown language for writing literate documents and a comprehensive resources covering all aspects is R Markdown: The Definitive Guide by Yihui Xie, J.J. Allaire and Garret Grolemund. By writing your work in R Markdown you are performing literate programming and it means your report can updated automatically if the underlying data changes. Output to HTML, PDF, LibreOffice, Microsoft Office and many other formats. The underlying source can be version controlled using Git so that it is documented, backed up (e.g. on GitHub or GitLab) and it is easy to collaborate with colleagues.

More recently Posit (nee RStudio) have developed Quarto the next iteration of RMarkdown. It supports more document types (e.g. blogs and RevealJS slides) and has excellent documentation and a growing number of extensions. If you are just starting out I would recommend using Quarto over RMarkdown.

Git Book

It is good practice to version control your code and literate documents as you develop them. This can be achieved using the version control system Git. Get yourself an account on GitHub and/or GitLab and settle down to read Jenny Bryans excellent Happy Git and GitHub for the useR.

Tidyverse

You will hear a lot about the Tidyverse which is an opinionated collection of R packages designed for data science. They are well worth learning as they make writing code considerably easier than with the base R packages. You won’t need all of the packages immediately but key ones to learn are

dplyr (or
tidyr for tidying your data.
forcats for working with categorical variables.
lubridate for working with date variables.
stringr for working with string variables.

If you’ve large datasets the dtplyr which uses the data.table package in the background but with dplyr code. data.table is considerably faster than dplyr for many operations. This is particularly noticeable when you have large datasets.

Statistics

There is a wealth of resources out there for learning and using R for different topics. The following is that which I’m aware of, if there is an omission please open an issue on my blog

Regression Modeling Strategies by Frank E. Harrell, Jr.
An Introduction to Statistical Learning with Applications in R/Python an excellent book on modern “machine learning” techniques.
Tidy Modeling with R by Max Kuhn and Julia Silge
Applied Predictive Modeling by Max Kuhn and Kjell Johnson (site to accompany physical book)
Hands-On Machine Learning with R by Bradley Boehmke and Brandon Greenwell
Interpretable Machine Learning by Christoph Molnar
The Hitchikers Guide to Responsible Machine Learning by Przemsylaw Biecek, Anna Kozak and Aleksander Zawada
Introduction to Data Science and Advanced Data Science by Rafael A. Irizarry
Telling Stories with Data With Applications in R by Rohan Alexander
Introduction to Modern Statistics (1st Ed)
The Epidemiologist R Handbook R for applied epidemiology and public health
R for Health Data Science by Ewen Harrison and Riinu Pius
Forecasting: Principles and Practice (3rd ed)

Bayesian Statistics

There are some excellent resources for learning Bayesian Analyses with R. Perhaps the most comprehensive and in-depth is Statistical Rethinking by Richard McElreath. He runs regular free courses teaching the material in the book (Statistical Rethinking 2023) and the book content has been translated to other R frameworks and Python. Another very good book is Bayes Rules! An Introduction to Applied Bayesian Modeling. These are both covered in the Bayesian Statistics - Syllabus course by Andrew Heiss.

Statistical Rethinking by Richard McElreath
- 2023 lectures and course material
- Statistical rethinking with brms, ggplot2, and the tidyverse: Second edition
Bayes Rules! An Introduction to Applied Bayesian Modeling by Alicia A. Johnson, Miles Q. Ott and Mine Dogucu
Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. William

Plotting

R has excellent support for producing graphs, figures and data visualisations. There is the base graphics that have been around since the beginning, but more recently the ggplot2 framework introduced by Hadley Wickham which implements Leland Wilkinson’s Grammar of Graphics has been very popular.

Advanced Topics

There is a lot to R than just Statistical analysis and one day you may want to investigate these in greater detail. The links below are to more advanced topics such as writing and maintaining packages or specific tasks such as text mining.

R Packages by Hadley Wickham and Jenny Bryan
Advanced R by Hadley Wickham
Advanced Statistical Computing by Roger D. Peng
Mastering Shiny by Hadley Wickham
Outstanding User Interfaces with Shiny by Kenton Russel
Reproducible Analytical Pipelines by Bruno Rodrigues
Text Mining with R by Julia Silge and David Robinson

CRAN

The Comprehensive R Archive Network (CRAN) is the primary place to look for R packages. It also contains a number of subject specific Task Views which are pages that summarise the packages and resources associated with a particular topic. There are also links to the official manuals, FAQs and user contributed documentation.

The R Journal

The R Journal is the peer-reviewed, open-access scientific journal published by the R Foundation. It includes articles on packages, reviews and proposals, comparisons and benchmarking, applications of existing techniques and special issue articles to accompany conferences or particular topics.

Cheatsheats

Cheatsheets come in handy as a reference to packages and commands. A central repository of cheatsheets is maintained by Postit.

Community

The R community is incredibly supportive, welcoming and helpful. There are over 600 User Groups around the world where R users meet up and share their experience and knowledge and support each other. Sheffield has its own SheffieldR User Group.

R Ladies

R-Ladies is a worldwide organisation whose mission is to promote gender diversity in the R Community. Groups around the world have their own meetups and activities.

R4DS

There is also the R4DS Online Learning Commuity which helps you work through the R for Data Science book. They have an active Slack channel for coordinating the courses and run Tidy Tuesday, a weekly podcast and community activity which is a great way of learning now tasks in R.

NHS R Community

The NHS R Community is focused on applications of R in the NHS Research community. They have blogs, a Slack channel and conferences.

Blogs

The R Bloggers site aggregates blogs from people who write about R and is a brilliant resource. A few highlights are noted as well but R Bloggers is probably the best resource. If you want to subscribe to these most have RSS feeds

Tidyverse blog
Notes from a data witch a blog by Danielle Navarro

Mastodon

Find posts and resources on Mastodon by searching for the #rstats hashtag. Here are some people I follow and find useful information from.

Bots

@rpodcast@podcastindex.social
@rweekly@fosstodon.org toots weekly updates from the R Community
@CRANberriesFeed@mas.to a bot that toots about packages released or updated on CRAN.

People

@e3mma@mastodon.social Emma is a bioinformatician and lecturer at York University who teaches R and reproducibility.
@f2harrel@mastodon.online Frank is the author of the Regression Modelling Strategies book.
@Cmastication@mastodon.social JD Long an avid R user.
@hadleywickham@fosstodon.org doyen of R packages and the Tidyverse.
@HeathrTurnr Heather Turner is a Research Software Engineering Fewllow in Statistics at University of Warrick and an R Foundation board member.
@eddelbuettel@mastodon.social ESS and R developer.
@sje@fosstodon.org ESS developer.
@robjhyndman@aus.social co-author of Forecasting Principles and Practice.
@topepo@fosstodon.org author of Applied Predictive Modelling and co-author of other R books; R package developer (tidymodels).
@annakrystalli@fosstodon.org Research Software Engineer, [ReproHack founder] and editor at rOpenSci

Organisations

@ropensci@fosstodon.org
@Posit@fosstodon.org the company that develops RStudio, Quarto and Shiny

Podcasts

Miscellaneous

Generative Art

Many people enjoy playing with R and ggplot2 to create Generative Art. One example is aRtsy, an R package that implements algorithms for making generative art in a straightforward and standardized manner using ‘ggplot2’ (see archive of posts on @aRtsy_package@mastodon). Another package is aRt by Nicola Rennie.

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2023,
  author = {Shephard, Neil},
  title = {R {Resources}},
  date = {2023-10-06},
  url = {https://blog.nshephard.dev/posts/r-resources/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2023. “R Resources.” October 6, 2023. https://blog.nshephard.dev/posts/r-resources/.

GitLab CI - Automatic Publishing to PyPI

Neil Shephard — Tue, 03 Oct 2023 00:00:00 GMT

I’ve written previously on Python Packaging and in that article included details of how to automate publishing to PyPI from GitHub. This article details how to automatically publish your package to PyPI from GitLab.

Packages with Legs, by Me

Repository Configuration

CI Variables

The environment variables $TWINE_USERNAME (__token__) and $TWINE_PASSWORD which will be the token you generate for publishing on PyPI or Test PyPI. These are saved under the repository _Settings > CI/CD > Varialbes_ section and how to create and save these is described below.

Protecting Tags

This really stumped me I could build and push automatically from the master branch but could not use the - if $CI_COMMIT_TAG condition to publish commits that were tagged. I wrote a post on the GitLab Forums asking how to do this and posted it to Mastodon asking if anyone had any ideas. I got two replies (one from @manu_faktur@mastodon.social and one from @diazona@techhub.social) both asking if I’d protected the tags on my repository.

I had no idea that you could protect tags on GitLab (or GitHub for that matter) so looked up the documentation on Protected tags and sure enough this was possible. Go to settings > Repository > Protected tags and set a wildcard to protect my tags, e.g. v* and the pypi CI job defined below will work as expected, building and uploading to PyPI on tagged commits.

CI Configuration

CI

GitLabs CI/CD is configured via a YAML file .gitlab-ci.yaml in the root of your project folder, a useful reference for writing these files is the .gitlab-ci.yml reference.

An example file from the tcx2gpx package is shown below (see here).

This defines the following…

image - the use of a Docker Python 3.11 image for running the pipeline.
variables - Configures pre-commit to run and automatically fix issues found on pull requests.
stages - the subsequent stages to run (NB the debug stage which prints the environment variables is commented out).
pylint - runs linting on Python 3.10 and 3.11.
pytest - Runs tests on Python 3.10 and 3.11.
pages - Builds the documentation pages.
pypi - Builds and uploads the package to PyPI if the commit has a tag associated.

image: python:3.11

variables: # since we're not using merge request pipelines in this example, # we will configure the pre-commit job to
  run on branch pipelines only.  # If you ARE using merge request pipelines, you can omit this section
  PRE_COMMIT_AUTO_FIX: '1' PRE_COMMIT_DEDUPLICATE_MR_AND_BRANCH: 'false' PRE_COMMIT_AUTO_FIX_BRANCH_ONLY: 'false'


before_script:
    - python --version
    - pip install .

# pre-commit autofix (https://gitlab.com/yesolutions/gitlab-ci-templates /
#                     https://stackoverflow.com/collectives/gitlab/articles/71270196/)
include: remote: https://gitlab.com/yesolutions/gitlab-ci-templates/raw/main/templates/pre-commit-autofix.yaml

stages: # - debug - pylint - pytest - pages - pypi

# print-all-env-vars-job:
#     stage: debug
#     script:
#         - echo "GitLab CI/CD | Print all environment variables"
#         - env

.pylint: script:
        - pip install pylint pytest
        - pylint --rcfile .pylintrc tcx2gpx/
        - pylint --rcfile .pylintrc tests/

pylint-3-10: extends: .pylint stage: pylint image: python:3.10 allow_failure: true

pylint-3-11: extends: .pylint stage: pylint image: python:3.11 allow_failure: true

.pytest: script:
        - pip install pytest pytest-cov
        - python -m "pytest"

pytest-3-10: extends: .pytest stage: pytest image: python:3.10 allow_failure: true

pytest-3-11: extends: .pytest stage: pytest image: python:3.11 coverage: /(?i)total.*?
    (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/

pages: stage: pages rules:
        - if: $CI_COMMIT_BRANCH == "master" script:
        - pip install .[docs]
        - cd docs
        - git fetch --tags
        - git tag -l
        - make html
        - mkdir ../public
        - mv _build/html/* ../public/ artifacts: paths:
            - public


pypi: stage: pypi rules:
        - if: $CI_COMMIT_TAG script:
        - pip install .[pypi]
        - pip install build
        - python -m build
        - twine upload --non-interactive --repository pypi dist/*

The pypi stage is named and a rule is defined that says to only run this stage if the value of the environment variable $CI_COMMIT_TAG is True. This only happens when a commit has a (protected 😉) tag.

The script section then installs the package along with the project.optional-dependencies defined in the pypi section of the pyproject.toml.

The package is then built using build and twine is used to push the to push the built package to PyPI.

PyPI Tokens

You should first test building and deploying to the Test PyPI and when this is working simply switch to using the main PyPI. To do so you will need to create an account on both¹. Once you have set yourself up with an account you can generate an API token to authenticate with PyPI. After verifying your email got to Account Settings and select Add API token. These are generated once so copy and paste it into the .pypirc of your project (add this file to your .gitignore so it doesn’t accidentally get added). Remember to do this twice, once for PyPI and once for Test PyPI and once for PyPI for reference.

[testpypi]
username = __token__
password = pypi-

[pypi]
username = __token__
password = pypi-

In GitLab go to your repositories Settings > CI/CD > Variables and add two new variables TWINE_USERNAME with the value __token__ and TWINE_PASSWORD with the token for your account on Test PyPI (remember it should include the prefix pypi- as shown in the above example .pypirc). You have options on how these variables are used and should ensure that all three check boxes are selected, this enables…

Protect variable Export variable to pipelines running on protected branches and tags only.
Mask variable Mask this variable in job logs if it meets regular expression requirements.
Expand variable reference $ will be treated as the start of a reference to another variable.

Testing

Now that you are setup you can test your configuration. To do so you need to first use the API key from the Test PyPI server that you created as the value for $TWINE_PASSWORD (see above) and set the repository twine --repository option to testpypi. Your pypi stage should look like the following…

pypi:
    stage: pypi
    rules:
        - if: $CI_COMMIT_TAG
    script:
        - pip install .[pypi]
        - pip install build
        - python -m build
        - twine upload --non-interactive --repository testpypi dist/*

Once this is set create a tag for the current commit using the Code > Tags settings from the left menu of your repository and then the New tag button on the top right. The tag you create should match the wild card pattern you have set for protecting tags and it should comply to the Public version identifiers specified in PEP440. On creation it triggers the Pipeline, you can check progress and status by navigating to CI/CD > Pipelines and then viewing it. The pypi job should complete and you should be able to navigate to your package on Test PyPI. You can find it under your account settings.

If you find there is a problem you will have to correct it and either delete the tag you created and try again or increment the version. PyPI, and in turn Test PyPI which is a mirror with the same functionality, does not permit uploading packages with a version number that already exists.

Publishing to PyPI

Once you have successfully published to the Test PyPI you are ready to publish to PyPI. There three things you need to do.

Delete the existing tag, if you want to apply the same tag to publish to PyPI you can do so.
Modify the repository option to point to PyPI --repository pypi (or remove it, the default is PyPI).
Change the key stored in the $TWINE_PASSWORD to that which you generated for PyPI instead of the one used for testing with Test PyPI.

Once you have done so you can create a new tag and the upload will be made to PyPI.

Releases

An alternative way to apply tags to commits is to make a Releases. In creating a release you apply a tag to the current commit. In addition GitLab will build and compress snapshot of the files and you can add Release Notes detailing what has changed. GitLab will automatically build release artifacts of your repository and make them available for download directly from GitLab.

Footnotes

PyPI now enforces Two Factor Authentication (2FA) for new accounts, see 2FA Enforcement for New User Registrations↩︎

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{shephard2023,
  author = {Shephard, Neil},
  title = {GitLab {CI} - {Automatic} {Publishing} to {PyPI}},
  date = {2023-10-03},
  url = {https://blog.nshephard.dev/posts/gitlab-ci-pypi/},
  langid = {en}
}

For attribution, please cite this work as:

Shephard, Neil. 2023. “GitLab CI - Automatic Publishing to PyPI.” October 3, 2023. https://blog.nshephard.dev/posts/gitlab-ci-pypi/.

Pre-commit and R Packaging

Neil Shephard — Sat, 29 Jul 2023 00:00:00 GMT

This post is aimed at getting you up and running with the R precommit Package. This shouldn’t be confused with the Python pre-commit Package although as you might suspect they are closely related.

The R package (precommit) consists of a number of R specific hooks that are run by pre-commit before commits are made and check various aspects of your code for compliance with certain style and coding standards (mostly aspects of R packages and I’ll be posting more on R packaging in due course).

A major part of this post is about getting things setup on Windows, I’ve only given a light overview of some of the hooks and common problems encountered as I’ve gone about using and learning R packaging because not everyone uses Windows.

Most of the work on the R package is by Lorenz Walthert, if you find it useful consider sponsoring his work, these things take a lot of time and effort and whilst they can be used for free are worth supporting.

Erwin Wurm sculpture at Yorkshire Sculpture Park by me

pre-commit

I love using pre-commit in my development pipelines and have blogged about it a few times already. It saves so much hassle (once you are used to it) not just for yourself but also your collaborators who are reviewing your Pull Requests. The R precommit package comes with a set of hooks that can be enabled and configured individually. I’ve recently and reason to start making and R package and as I’ve not used R much for a few years and this was my first time attempting to develop a package, I decided to use the hooks to impose the various style standards and checks that are expected.

I opted to enable all of the hooks. I’ve not covered them all here in detail (yet) but describe some of them below and show how to use some additional hooks from the usethis package too.

codemetar

There is a hook for checking the Codemeta, which is in JSON-LD format is created correctly. The R package codemetar facilitates creating this and pulls metadata from the DESCRIPTION, README.Rmd and other aspects of your package to format them in JSON Codemeta. It comes with a handy function to write the file for you, so after installing you can just run codemetar::write_codemeta() which will create the codemeta.json for you. Remember to run this each and every time you update and of the files from which the metadata is created (although keep an eye on #491 which suggests updating automatically)

roxygenize

Roxygen2 is a package for making the documentation to go with your package, it does this by parsing the documentation strings (“docstrings” for short) that you adorn your functions with that describe the arguments and show example usages. This hook requires additional configuration in .pre-commit-config.yaml as you have to install your package dependencies. Fortunately there is a helper function in the precommit package so you can just run precommit::snippet_generate("additional-deps-roxygenize") and it will output the YAML that you need to add to your .pre-commit-config.yaml. It might look something like the following.

    hooks:
    - id: no-debug-statement
    - id: roxygenize
      additional_dependencies:
        -    data.table
        -    dplyr
        -    dtplyr
        -    duckdb
        -    IMD
        -    lubridate
        -    stringr

style-files

The style-files hook runs the styler package against your code to ensure it follows the tidyverse style guide by default, although it can be configured to use a custom style guide of your own creation.

lintr

The lintr package lints your code automatically. It can be configured by adding a .lintr configuration file to your repository, a simple example is shown below. Note the indented closing parenthesis is important you get a complaint about that and any other formatting issues.

linters: linters_with_defaults(
         line_length_linter(120),
         object_name_linter = NULL,
         object_usage_linter = NULL
  )

`spell-check`

This is a useful hook that checks your spelling and adds unusual words to a custom dictionary inst/WORDLIST.

`deps-in-desc`

This hook ensures that all dependencies that are loaded by your package are listed in the DESCRIPTION file so that when the package is installed the necessary dependencies are also pulled in, fairly essential..

usethis package

The usethis package is a compliment to the devtools package that has a lot of very useful helper functions. Some of these enable additional pre-commit hooks whilst others enable GitHub actions, which are part of Continuous Integration pipelines and I would highly recommend enabling them.

`README.Rmd`

The user_readme_rmd() function automatically generates a README.Rmd template and will also create a pre-commit hook that keeps it synchronised with README.md whenever you update it. This is useful because the later, plain-markdown, file is automatically rendered by GitHub/GitLab/Codeberg as your repositories front-page.

`use_github_action()`

Invoking use_github_action() within your package repository will prompt you for the type of action you wish to add to it. There are, as of writing, three options.

    > use_github_action()
    Which action do you want to add? (0 to exit)
    (See <https://github.com/r-lib/actions/tree/v2/examples> for other options)

    1: check-standard: Run `R CMD check` on Linux, macOS, and Windows
    2: test-coverage: Compute test coverage and report to https://about.codecov.io
    3: pr-commands: Add /document and /style commands for pull requests

Selecting one will write a file to /.github/workflows/.yaml and then print out code to add a badge to your repository.

Selection: 1
    ✔ Adding '*.html' to '.github/.gitignore'
    ✔ Creating '.github/workflows/'
    ✔ Saving 'r-lib/actions/examples/check-standard.yaml@v2' to '.github/workflows/R-CMD-check.yaml'
    • Learn more at <https://github.com/r-lib/actions/blob/v2/examples/README.md>.
    • Copy and paste the following lines into 'README.Rmd':
      <!-- badges: start -->
      [![R-CMD-check](https://github.com/CUREd-Plus/cuRed/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/CUREd-Plus/cuRed/actions/workflows/R-CMD-check.yaml)

blog.nshephard.dev

Pytest testmon

Test, test, test

Development cycle

Commits as Units of Work

Enter pytest-testmon

Installation

Hello pre-commit my old friend

Configure pre-commit

Package Upgrades

Add pytest-testmon to package dependencies

Test It Out

Summary

Reuse

Citation

Emacs Tips & Tweaks (2025-09-05)- vundo, regex builder and more

vundo

Regex Builder

Yank from kill ring

use-package

Emacs pre-commit

Summary

Reuse

Citation

Debugging GitHub Actions with tmate

Setup and Configuration

SSH Keys

Testing on multiple OS’s and versions

tmate GitHub Action

Usage

Conclusion

Links

SSH and GitHub

Footnotes

Reuse

Citation

Emacs Yasnippet

Yasnippet

Installation

Inserting templates

Using Key-bindings to insert

Creating your own templates

More Templates

Summary

Links

Reuse

Citation

Version Control with Jujutsu

Installation

Gentoo

NixOS

Arch Linux

Shell Integration

Bash

ZSH

Initial Configuration

Usage

New commits

Diffing

Remotes

Bookmarks (aka branches)

Merging Branches

Conflicts

Revisions and Revsets

Symbols

Operators

Functions

Configuration

Workflow

Squash Workflow

Edit Workflow

Editing older commits

IDE/Interfaces

Conclusion

Links

Tutorials

Emacs Packages

Blogs

Reuse

Citation

Enter `pytest-testmon`

Add `pytest-testmon` to package dependencies

`vundo`

`use-package`

`tmate` GitHub Action

`scatter()`

Update `pytest` options

`--mpl-generate-summary`