TopoStats Tutorial : Virtual Environments & Git Branches

Neil Shephard

View these slides…

https://ns-rse.github.io/topostats-tutorial

Introduction

• TopoStats has undergone significant revision.
• Versioned releases available via PyPI
• TopoStats is still undergoing significant development.
• Also available via GitHub.

TopoStats has undergone significant revision and is, we hope, significantly easier to use these days. Its easy to install via PyPI as we make versioned releases there, but there is on-going development to refine image processing under the hood and add more features and flexibility. These features are, until a new version is released, only available if TopoStats is installed via GitHub.

Introduction (cont.)

• User feedback is really, really important.
  • Bugs! When things break.
  • Features - New functionality.
• Best to capture bugs/issues early in development cycle before PyPI releases

It is really useful for developers to have feedback as it helps identify “bugs” where things aren’t working as expected and as a source of new functionality which can be incorporated.

Ideally it is preferable to have that feedback earlier in the development cycle before releases are made to PyPI.

Content

• Python Virtual Environments (Conda).
• GitKraken for cloning TopoStats GitHub repository.
• Installing TopoStats from cloned repository in Virtual Environment.
• Switching branches and testing.
• Reporting issues.

Python Virtual Environments

• Comprehensive : Introduction to Conda for (Data) Scientists
• Recommend Miniconda, comes with Python which is all that is required.
  • Linux
  • Windows (64-bit)
  • OSX (Intel)
  • OSX (M1)

You’ve hopefully all managed to go through the Setup instructions from the Introduction to Conda for (Data) Scientists and have installed Miniconda on your computers.

We recommend Miniconda as its a minimal install that includes Python. If you’ve not there are direct links to install here.

Command Line Environments

• Command line programmes are searched for in order looking along the paths in $PATH.
• A “virtual environment” is an isolated set of commands that take precedence over system commands.
• You can have multiple virtual environments and switch between them.

When you’re using a command line environment programmes are searched for in order along the paths stored in a special variable known as PATH. A virtual environment is an isolated set of commands that take precedence over system commands and they do this by modifying the PATH variable. Its possible to have multiple virtual environments installed and switch between them, doing so changes which environment is listed first in your PATH.

Create and Activate Virtual Environment

echo $PATH    # Locations programmes are searched for BEFORE
which python  # Show version of python used BEFORE
conda create --name topostats-git python=3.10
conda activate topostats-git
echo $PATH    # Show locations programmes are searched for AFTER
which python  # Show version of python used AFTER

We’re now going to create a virtual environment, but first if you type echo $PATH it will show you the location programmes are searched for, these are delimited by colons.

If you type which python or in Windows where.exe python it will show you which version of Python will be invoked.

You can now create a virtual environment using conda create --name topostats-git python=3.10 this creates a new environment called topostats-git and ensure Python version 3.10 is installed within it. But its not activated yet, as the output show, and to activate it you need to type conda activate topostats-git.

Once activated you can repeat the commands, and I would recommend getting in the habit of using the up-arrow to go back through your command history rather than retyping commands, to echo $PATH and you should see the first thing listed is the path to the topostats-git Miniconda environment.

If you now type which python or in Windows where.exe python it should point to a Python executable that is within the Miniconda environment we have just created.

Virtual Environments

This is a simple graphical overview of how you can have multiple virtual environments co-existing within each other.

There are three environments each with different versions of Python installed, although you should never use Python 2.7 these days. Within each environment there are different Python packages installed, 1 has Python 2.7 along with Numpy 1.14.4 and Matplotlib 2.2.5. 2 Python 3.6 along with the packages Pandas 1.0.1 and PySpark 2.4.8. 3 has Python 3.10 and the package Pandas 1.3.5 and Matplotlib 3.5.1.

These all work in isolation, it just depedns which one has been activated.

Version Control… a digression

Hopefully you have already…

• Setup GitHub account (https://github.com) using University email address.
• Downloaded and installed GitKraken using GitHub account associated with University email address.

Comprehensive : Git, GitHub and GitKraken : From Zero to Hero.

The Research Software Engineering group run a comprehensive Git, GitHub and GitKraken course on a regular basis and I would highly recommend taking it (but then I would as I help run it!)

It is the sort of course I wish I had taken when I started out using Git as it covers the basics of version control and then moves onto collaborative work and how to use GitHub to report Issues and dealing with some of the problems that might arise when working on code collaboratively.

Some of these aspects aren’t necessary for what we are covering today but I feel they are useful to be aware of so I’d encourage you to take the course.

Hopefully you have already setup a GitHub account and installed GitKarken.

That said we’re going to make a lightening digression into version control, what it is, why it matters to TopoStats development and how it impacts testing new features.

Getting Started with Git, GitHub and GitKraken

• Git is a programme that runs on your computer.
• GitHub is an online service which hosts software projects and facilitates…
  • Distributed software development
  • Tracking bugs/features
  • Project management
• GitKraken is a client that interfaces to Git and GitHub and communicates between the two.

Its important to make a distinction between Git, GitHub and GitKraken.

Git is a piece of software that lives on your computer and is used to manage what are known as repositories, a fancy name for a copy of software that is under Git Version control.

GitHub is an online service which hosts software projects, its not the only one there are GitLab, BitBucker, Codeberg and others, but they all facilitate distributed software development, bug and feature tracking and project management.

Git & GitHub

Cloning TopoStats

We’ll now clone TopoStats using GitKraken.

• File > Clone Repository (Ctrl + N)
• GitHub.com
• Select directory to clone to.
• Under Repository to Clone start typing TopoStats
• Select TopoStats
• Clone the repo!

We’re now going to clone the GitHub repository using GitKraken, these are the steps to take, but we’re going to walk through them together to make sure everyone successfully clones the repository.

Start up or switch to GitKraken and from the menu select File > Clone Repository (you can also use the keyboard shortcut Ctrl + N).

We want to clone from GitHub so select that.

Then select a directory to clone to, I’m going to clone to tmp but you are welcome to choose any location you want.

Next you need to select what repository to clone, you can start typing Topo and should see an autocompleted list of options. Select TopoStats from under AFM-SPM organisation.

Once that is all set up Clone the repository using the green button. This will take a while once its done you should see a message saying “Successfully clonded repo ‘TopoStats’” and the option to “Open Now”. But we’ll take a break whilst that is downloading for everyone. Does anyone have any questions about anything so far?

Installing TopoStats under Virtual Environments

We have…

• Setup and activated a Virtual Environment.
• Cloned the TopoStats repository.

Next…

• Install TopoStats from cloned repository.

So far we have setup and activated a Virtual Environment and cloned the TopoStats repository its time to install it.

Now we’re going to install TopoStats that we have cloned from Git into this environment.

Installing TopoStats under Virtual Environments

cd ~/tmp/topostats-tutorial/TopoStats
which python
conda activate topostats-git
git status
pip install -e .[notebooks]  # Install additional requirements e.g. .[tests]
run_topostats -h

Check you have activated your Conda environment if not activate it.

Use git status and check that you are on the main branch its shown on the first line of output.

Now install TopoStats with the command pip install -e .[notebooks] and it should start downloading a bunch of packages that TopoStats depends on then install all of them. The [notebooks] tells pip to also install some extra packages we’ve defined that are useful for running sample Notebook. The . means to install from the current directory whilst the -e flag tells Pip to install the package in what is known as “editable” format, and so rather than copying the files to a location under your virtual environment they are referenced from the current location.

There are additional requirements that can be installed, for example if you wanted to include the dependencies require for running tests you would use .[tests].

run_topostats -h

❱ run_topostats -h
usage: run_topostats [-h] [-c CONFIG_FILE] [--create-config-file CREATE_CONFIG_FILE] [-s SUMMARY_CONFIG] [-b BASE_DIR] [-j CORES] [-l LOG_LEVEL] [-f FILE_EXT]
                     [--channel CHANNEL] [-o OUTPUT_DIR] [--save_plots SAVE_PLOTS] [-m MASK] [-q QUIET] [-v] [-w WARNINGS]

Process AFM images. Additional arguments over-ride those in the configuration file.

options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config_file CONFIG_FILE
                        Path to a YAML configuration file.
  --create-config-file CREATE_CONFIG_FILE
                        Filename to write a sample YAML configuration file to (should end in '.yaml').
  -s SUMMARY_CONFIG, --summary_config SUMMARY_CONFIG
                        Path to a YAML configuration file for summary plots and statistics.
  -b BASE_DIR, --base_dir BASE_DIR
                        Base directory to scan for images.
  -j CORES, --cores CORES
                        Number of CPU cores to use when processing.
  -l LOG_LEVEL, --log_level LOG_LEVEL
                        Logging level to use, default is 'info' for verbose output use 'debug'.
  -f FILE_EXT, --file_ext FILE_EXT
                        File extension to scan for.
  --channel CHANNEL     Channel to extract.
  -o OUTPUT_DIR, --output_dir OUTPUT_DIR
                        Output directory to write results to.
  --save_plots SAVE_PLOTS
                        Whether to save plots.
  -m MASK, --mask MASK  Mask the image.
  -q QUIET, --quiet QUIET
                        Toggle verbosity.
  -v, --version         Report the current version of TopoStats that is installed.
  -w WARNINGS, --warnings WARNINGS
                        Whether to ignore warnings.

If everything has worked you should be able to now see the help for the command run_topostats. These are command line options or “flags” and the help tells you what each is and how to use it.

run_topostats -q

(topostats) ❱ run_topostats --log_level warning
[Mon, 13 Mar 2023 14:01:01] [INFO    ] [topostats] Updated config config[log_level] : info > warning
Processing images from /home/neil/work/projects/topostats/TopoStats, results are under output: 100%|████████| 1/1 [00:00<?, ?it/s]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ COMPLETE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TopoStats Version           : 2.0.1.dev384+g365c6dd.d20230310
  Base Directory              : /home/neil/work/projects/topostats/TopoStats
  File Extension              : .spm
  Files Found                 : 1
  Successfully Processed^1    : 1 (100.0%)
  Configuration               : output/config.yaml
  All statistics              : output/all_statistics.csv
  Distribution Plots          : output/summary_distributions

  Email                       : topostats@sheffield.ac.uk
  Documentation               : https://afm-spm.github.io/topostats/
  Source Code                 : https://github.com/AFM-SPM/TopoStats/
  Bug Reports/Feature Request : https://github.com/AFM-SPM/TopoStats/issues/new/choose
  Citation File Format        : https://github.com/AFM-SPM/TopoStats/blob/main/CITATION.cff

  ^1 Successful processing of an image is detection of grains and calculation of at least
     grain statistics. If these have been disabled the percentage will be 0.

  If you encounter bugs/issues or have feature requests please report them at the above URL
  or email us.

  If you have found TopoStats useful please consider citing it. A Citation File Format is
  linked above and available from the Source Code page.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You are now ready to run_topostats. If you cd, change directory, to a location you have some scans you can simply invoke run_topostats and it will use the default configuration to scan for .spm files and process any that are found within your current working directory. If you use the -l warning or --log_level warning flag it will be considerably less verbose, but it can often be useful to be verbose, particularly when submitting issues which we will come to in a bit.

You may encounter some errors, if you’re lucky you won’t and on completion TopoStats reports how many files it found, how many it processed and where output is stored.

run_topostats --create-config-file

run_topostats --create-config-file dummy_config.yaml
[Fri, 10 Mar 2023 20:09:26] [INFO    ] [topostats] The YAML configuration file is valid.
[Fri, 10 Mar 2023 20:09:26] [INFO    ] [topostats] A sample configuration has been written to : dummy_config.yaml
[Fri, 10 Mar 2023 20:09:26] [INFO    ] [topostats] Please refer to the documentation on how to use the configuration file :

https://afm-spm.github.io/TopoStats/usage.html#configuring-topostats
https://afm-spm.github.io/TopoStats/configuration.html
❱ cat ~/tmp/dummy_config.yaml
# Sample configuration file auto-generated : 2023-03-10 20:09:26
base_dir: /home/neil/work/git/hub/design-patterns
output_dir: output
log_level: info
cores: 2
file_ext: .spm
loading:
  channel: Height
filter:
  run: true
  row_alignment_quantile: 0.5
  threshold_method: std_dev
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    lower: 10.0
    upper: 1.0
  threshold_absolute:
    lower: -1.0
    upper: 1.0
  gaussian_size: 1.0121397464510862
  gaussian_mode: nearest
  remove_scars:
    run: true
    removal_iterations: 2
    threshold_low: 0.25
    threshold_high: 0.666
    max_scar_width: 4
    min_scar_length: 16
grains:
  run: true
  threshold_method: std_dev
  otsu_threshold_multiplier: 1.0
  threshold_std_dev:
    lower: 10.0
    upper: 1.0
  threshold_absolute:
    lower: -1.0
    upper: 1.0
  direction: upper
  smallest_grain_size_nm2: 50
  absolute_area_threshold:
    upper:
    - 300
    - 3000
    lower:
    -
    -
grainstats:
  run: true
  edge_detection_method: binary_erosion
  cropped_size: 40.0
dnatracing:
  run: true
plotting:
  run: true
  save_format: png
  image_set: core
  zrange:
  -
  -
  colorbar: true
  axes: true
  cmap: nanoscope
  mask_cmap: blu
  histogram_log_axis: false
  histogram_bins: 200
summary_stats:
  run: true
  config:

You may want to customise the configuration TopoStats is run with. You can do this by first generating a configuration file using the --create-config-file flag and giving a file name. It can be anything you want but should end in .yaml.

You can then open this file in a text editor (NB Word is not a text editor, its a word-processor), edit parameters, save and then use it.

run_topostats --config_file

run_topostats --config_file dummy_config.yaml

To use the configuration file you have just created you use the --config_file flat and give it the name of your file.

Testing New Features - Git Branches

%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true,'showCommitLabel': true, 'rotateCommitLabel': true}} }%%
gitGraph
    commit
    commit
    branch bug1
    checkout main
    commit
    checkout bug1
    commit
    commit
    checkout main
    branch feature1
    checkout feature1
    commit
    commit
    checkout bug1
    commit
    checkout main
    merge bug1 tag: "v0.1.1"
    checkout feature1
    commit
    commit
    checkout main
    merge feature1 tag: "v0.1.2"
    commit

We now return to Git because in order to test new features you need to be able to switch between what is known in Git parlance as branches.

Git allows concurrent development of features and to do so uses the concept of branches which are shown in this diagram.

A branch “forks” from the main branch at a certain point, features are developed and stored in the Git version control system via commits which are represented by each of these strange alpha-numeric characters which are known as commits (they typically have human readable comments associated with them so they are more intelligible).

Once a feature is ready a “Pull Request” is created to be reviewed by others and once approved its merged back into the main branch.

Its useful to bear this mental model of branches in mind as we are about to switch branches.

GitKraken Changing Branches

You can use GitKraken to switch branches use switch to topostats-git.

Make sure you re-install TopoStats after switching branches (sometimes pip install -e . doesn’t pick up the changes Git makes).

cd ~/tmp/TopoStats
pip install -e .

Switch branches using Git Kraken. There are several ways of doing this but the easiest is to find the commit in the Branch/Tag column on the left, right-click on it and select “Checkout origin/” in this case topostats-git. This pulls down all changes from that branch and checks it out. You see that there is a laptop which indicates that your local computer has a copy of the branch at that particular commit and the tick indicates that you are checked out there. The other icon is the AFM-SPM GitHub organisation logo and indicates where that branch is on Git Hub.

Once you’ve done this at the top you should see you are Repository > topostats-git and if you want to get back to main you can select it from the drop down menu under Branch.

In the background Git changes the files so they match those on the branch, you don’t need to look for these changes, just be aware that they happen.

After switching branches its sometimes required to reinstall TopoStats, in theory this shouldn’t be required the -e flat used when installing means editable and it should just work with the files Git changed, but I’ve found in the past this didn’t always appear to be the case so I’m now in the habit of reinstalling and it doesn’t do any harm.

Re-run run_topostats

Re-run run_topostats

Traceback (most recent call last):
  File "/home/neil/.virtualenvs/topostats/bin/run_topostats", line 8, in <module>
    sys.exit(main())
  File "/home/neil/work/projects/topostats/TopoStats/topostats/run_topostats.py", line 213, in main
    print(f"Can we divide 5 by 0? {5/0}")
ZeroDivisionError: division by zero

If you now re-run run_topostats as you previously did you should get a different result, can anyone tell me what happens?

You should have seen that the last message was ZeroDivisonError: division by zero.

Ok, that is good, and by design I set this branch up so that it would purposefully fail. Python raises such errors because you can not divide any number by zero is undefined because you can not multiply any number by zero to get 2.

This isn’t a great example but it shows how switching branches can result in failures, and it is hoped that after this tutorial you are able to test new feature branches or bug fixes, but what would you do with such a failure if you encounter one whether that is on the `main** branch or a feature branch?

GitHub - Reporting Issues

• Reporting bugs helps with feature development.
• Removes them not just for you but for all users.
• Can seem cumbersome at first but it gets easier with practice.

SOLUTION : GitHub Issues…https://github.com/AFM-SPM/TopoStats/issues

Reporting bugs and indeed new feature requests is really important, it helps those who are developing the code deal with issues and scenarios they might not have anticipated and without feedback the problems persist for all users so its not just for your benefit but for everyones that such problems get identified early, reported and fixed.

The process can seem cumbersome at first but with practice it gets easier and we have written templates to ease the process.

You can do this from within GitKraken but doing so means you can’t take advantage of the Issue Templates we have setup.

Instead we’re going to go through the process on GitHub.

GitHub - Reporting Issues

In the repository on GitHub there are tabs for “Code”, “Issues” and so forth, you want to select the “Issues” tab, and you will see a list of existing issues.

On the top right is a button to report a “New Issue”, click on this and you are presented with some options to use some pre-defined templates to report issues.

GitHub - Reporting Issues (cont.)

Markdown Links

• Getting started with writing and formatting on GitHub - GitHub Docs
• Basic writing and formatting syntax - GitHub Docs
• Markdown Cheat Sheet | Markdown Guide

We have setup templates to report Bugs, make a new feature request and you may or may not see the one for reporting a vulnerability. For this exercise we want to use the Bug Report form so clock on the “Get Started” button for that.

You get a page which has a box for a title where you should put a short but descriptive title for the bug report that captures the essence of what you are reporting.

There is then a box that is pre-populated with text in Markdown Format. If you haven’t come across Markdown before don’t worry its pretty simple and there are some links to get you started in using the formatting.

The text is descriptive and walks you through the various steps for providing different aspects of information about the error you have encountered.

At any stage you can click on the “Preview” tab and see what the page looks like.

GitHub - Reporting Issues (cont.)

• Describe the bug.
• Include the configuration file.
• Copy of the output.
• Exact command that failed.
• TopoStats version
• Operating System and Python Version
• Optional Python packages that are installed.

You are asked to tick off various tasks as completed and there is a section for each describing in a bit more detail how to complete it and with a Markdown code section to paste the output.

You are asked to describe the bug, although typically most of this information is covered in the subsequent sections in greater detail.

You are asked to paste a copy of your YAML configuration file rather than taking a picture of it. Its much more useful to paste such content because a screenshot does not allow anyone investigating the problem to copy, paste and reuse.

A copy of the output is then requested to be pasted in, this should include as much information as possible about the failure.

The exact command used, typically I would expect this to be run_topostats --config_file <your_config>.yaml

Then the TopoStats command, the operating system and Python version and finally, optionally the full Python environment with all package versions included.

Exercise 1 - Tests

1. Create a new Conda environment called test.
2. Checkout topostats-test branch.
3. Install TopoStats and include the test dependencies.
4. Run the test suite with pytest

Exercise 1 - Tests (Solution)

conda create --name test python=3.10
git checkout topostats-test # This is the commandline version, use GitKraken to switch
pip install -e .[tests]  # Install the test dependencies as well
pytest

You create the test environment, making sure to include Python 3.10 in the installation. I’ve shown the command line version to switch branches using Git but you can use GitKraken to do this if you don’t have Git installed at the command line.

Next you make sure TopoStats is installed along with the test dependencies.

Finally you run the test suite which checks most of the code is running correctly. This can take a while so cancel with Ctrl+C to break out of the tests if you want.

Exercise 2 - Surprise

1. Create a new Conda environment called surprise.
2. Checkout topostats-surprise branch.
3. Install TopoStats.
4. Try running pytest.
5. Run TopoStats.

Exercise - 2 Surprise (Solution)

conda create --name surprise python=3.10
git checkout topostats-surprise # This is the commandline version, use GitKraken to switch
pip install -e .  # Install the test dependencies as well
pytest
run_topostats
      ___           ___           ___           ___
     /\  \         /\  \         /\  \         /\  \
     \:\  \       /::\  \       /::\  \       /::\  \
      \:\  \     /:/\:\  \     /:/\:\  \     /:/\:\  \
      /::\  \   /:/  \:\  \   /::\~\:\  \   /:/  \:\  \
     /:/\:\__\ /:/__/ \:\__\ /:/\:\ \:\__\ /:/__/ \:\__\
    /:/  \/__/ \:\  \ /:/  / \/__\:\/:/  / \:\  \ /:/  /
   /:/  /       \:\  /:/  /       \::/  /   \:\  /:/  /
   \/__/         \:\/:/  /         \/__/     \:\/:/  /
                  \::/  /                     \::/  /
                   \/__/                       \/__/
      ___           ___           ___           ___           ___
     /\  \         /\  \         /\  \         /\  \         /\  \
    /::\  \        \:\  \       /::\  \        \:\  \       /::\  \
   /:/\ \  \        \:\  \     /:/\:\  \        \:\  \     /:/\ \  \
  _\:\~\ \  \       /::\  \   /::\~\:\  \       /::\  \   _\:\~\ \  \
 /\ \:\ \ \__\     /:/\:\__\ /:/\:\ \:\__\     /:/\:\__\ /\ \:\ \ \__\
 \:\ \:\ \/__/    /:/  \/__/ \/__\:\/:/  /    /:/  \/__/ \:\ \:\ \/__/
  \:\ \:\__\     /:/  /           \::/  /    /:/  /       \:\ \:\__\
   \:\/:/  /     \/__/            /:/  /     \/__/         \:\/:/  /
    \::/  /                      /:/  /                     \::/  /
     \/__/                       \/__/                       \/__/

....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................,,,,,,,,,,,,,,,,,.,,,,,,,,,,,,..................................
....................................,Y$$$Y$,$$YYYY$$$,$YY$$$$$$Y$Y,.................................
.......,YYY$+DDDDDDDJJDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD$.........
........,,,.*@@@@@@@J%%*@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@JDD@@*D@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@JPJ@JD%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@DD%*PPD@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@P%%DDD@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@%DD@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@J@@@@@@@@@@@@@@@@$.........
............*J%PY%+@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@PYD@@@@@@@@@@@@@@@$.........
........$$$,***PDDD@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@YYD@@@@@@@@@@@@@@@@$.........
........$$$,P@J@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%P*P@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@JD*J@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%**DD*P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%DJD%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
........,,,,P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@D%@@@@J+%@@@@@@@@@@@@@@@@@@@@@@@@$.........
........$YY$P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@JYD*D@@J+$J@@@@@@@@@P+PJ@@@@@@@@@@$.........
.....$$.....P@@@@@@@@@@@@@@@@@@@@@@@@@@@@+PJ@@@@@@@@@@@J*D@JP@@@@D$*D@@@@@@@@D,%@@@@@@@@@@$.........
.....Y$.....P@@@@@@@@@@@@@@@@@@@@@@@@@@J%*JP@@@@@@@@@@@**DJJ@@@@@@D**@@@@@@@@J+P@@@@@@@@@@$.........
.....Y$.....P@@@@@@@@@@@@@@@@@@@@@@@@@JPJ@%*@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%@@@@@@@@@@$.........
.....Y$.....P@@@@@@@@@@@@@@@@@@@@@@@@@@*%P%@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@D*J@@@@@@@@@@@@@@$.........
.....Y$.....P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@J@JYPP@@@@@@@@@@@@@@$.........
.....Y$.....P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@P*+Y*D@@@D%D@@@@@@@@@$.........
.....YY.....P@@@@@@@@@@@@@@@@@@@J@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@J%%@@@@J*$%%P@@@@@@@@$.........
.....,,.$$$,P@@@@@@@@@@@@@@@@DD*,D@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@+P*D%J@@@@@@@@$.........
........,,,,P@@@@@@@@@@@@@@@P%%YD@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@JD@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@+JPJ@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@%*J@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
............P@@@@%J@@D*+%@@@@@@@@@@@@@@@@@@@@@@@@%%%J@@@@@@@@@@@@@@@@@@%**%@@@@@@@@@@@@@@@$.........
............P@@@@P+Y+P%*J@@@@@@@@@@@@@@@@@@@@@@@@%PDPJ@@D@@@@@@@@@@@@@+%JDP+%@@@@@@@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%%*J@@+%%DJ@@@@@@@@@%D%DJDD@@@@@@@@@@@@@$.........
........,,,,P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@***%@@%D@J+J@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
........$$$$P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@D%%*@@@DD@@@D*J**@@@@@@@D%DJJ*%@@@@@@@@****%@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%PPP%@@@@@@@@@JDD@@@@@@@J+@JD%P*@@@@@%+PP%%%*@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@DP*@@@@@@@@@@@@@@@@@@@@@@%*DD%%@@@@@@@DD@@@@@@@$.........
............P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@D$J@@@@@@@@@@@@@@@@@@@@@@@@@@@@D*%%PJJ+%%*@@@@@$.........
............P@@@@@@@@@@%PD@@@@@@@@@@@@@@@@@@J@@@@@@@@@@@@@@@@@@@@@@@@@@@@@+D@DPJ@D%D*Y%@@@$.........
............P@@@@@@@@@@D+$D@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@J+J+@@@@@@@%+D@@$.........
............*J*D@@@@@@@@@P**P@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@D+D@@@@@@@@@@@@$.........
............*J*YPP%**@@@@@D%D@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@$.........
..........,$PJJJ%P*P%JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ$.........
..........,,Y,,,,,,,,,,,,,$$Y$,,,,,,,,,,,$YYY,,,,,,,,,,,,YYY,,,,,,,,,,,,YYY$,,,,,,,,,,,$YY$.........
............,.............,,,,...........,,,,,.......,..,,,,............,,,,...........,,,,.........
.............................................YYY$$$$YY$$$,..........................................
....................................................................................................
....................................................................................................
....................................................................................................
  ____                            _         _       _   _                 _ _
 / ___|___  _ __   __ _ _ __ __ _| |_ _   _| | __ _| |_(_) ___  _ __  ___| | |
| |   / _ \| '_ \ / _` | '__/ _` | __| | | | |/ _` | __| |/ _ \| '_ \/ __| | |
| |__| (_) | | | | (_| | | | (_| | |_| |_| | | (_| | |_| | (_) | | | \__ \_|_|
 \____\___/|_| |_|\__, |_|  \__,_|\__|\__,_|_|\__,_|\__|_|\___/|_| |_|___(_|_)
                  |___/

You create the surprise environment, making sure to include Python 3.10 in the installation. Again I’ve shown the command line version to switch branches using Git but you can use GitKraken to do this if you don’t have Git installed at the command line.

Next you make sure TopoStats is installed but no additional dependencies are required here.

You should find if you try running the test suite that it fails saying pytest is not found. This is because you didn’t install the test dependencies that are required to run the test suite.

Finally you run_topostats and on this branch I have changed the code so that instead of running properly it prints out this message which is some fancy ASCII-art for TopoStats, the test-image called minicircle.spm that we use in development and a congratulations!! message.

This is another non-sensical example but hopefully demonstrates how Git branches can be used to change the code base without affecting the main/master branch.

Exercise 3 - Notebooks

1. Create a new Conda environment called notebooks.
2. Checkout ns-rse/504-plotting-arrays branch.
3. Install TopoStats with notebooks dependencies.
4. Run TopoStats.
5. Start a Notebook

Exercise 3 - Notebooks (Solution)

conda create --name test python=3.10
git checkout topostats-test # This is the commandline version, use GitKraken to switch
pip install -e .[tests]  # Install the test dependencies as well
jupyter notebook

Jupyter Notebooks

• Interactive execution of Python Code.
• Early development.
• Walk-through for running TopoStats (00-Walthrough-minicircle.ipynb)
• Summary Statistics and Plotting (02-Summary-statistics-and-plots.ipynb)
• Plotting Scans (03-Plotting-scans.ipynb)
• Try going through them to process images (some require run_topostats) to have been run.

Any Questions?