Python stuff

A development scaffold

Goals:

• All dependencies are isolated in a virtualenv.
• Typical dev tasks–such as running tests and linters–can be performed with a single, simple command.

I’ve found a Makefile to be helpful for this. Here’s a more or less standard template I use for all Python projects:

# Use the same shell anywhere for consistency
SHELL := bash
# Pass separate lines to the same shell
.ONESHELL:
# Clean stuff up if a recipe fails
.DELETE_ON_ERROR:

# Python executable used for initial virtualenv creation
python = python3
# Make sure to add this to the .gitignore
# (Or have it outside of the project directory, if we prefer)
venv = ./.venv
# Update the following as needed
src = ./lib
tests = ./tests

# Use as a prerequisite for any recipe that runs in the virtualenv
venv: $(venv)

# Create the virtualenv and install dependencies
$(venv):
	$(python) -m venv $(venv)
	$(venv)/bin/pip install -U pip
	$(venv)/bin/pip install -Ur requirements.txt

# Run this after adding or updating packages
deps: venv
	$(venv)/bin/pip install -Ur requirements.txt

default: run

# Clean up transient files
clean:
	rm -rv $(venv)
	find $(src) -name __pycache__ -type d -exec rm -rv {} +

# It's convenient to bind this to some key in your editor (I use F3 in vim)
fmt: venv
	$(venv)/bin/isort $(src) $(tests)
	$(venv)/bin/black $(src) $(tests)

lint: venv
	$(venv)/bin/pylint --rcfile=setup.cfg $(src)
	$(venv)/bin/mypy --ignore-missing-imports $(src)

# Same idea can be applied to database shells like psql
shell: venv
	PYTHONPATH=. $(venv)/bin/ipython

test: venv
	$(venv)/bin/python -m pytest $(tests)

Converting this to a Dockerized setup is pretty straightforward: we mostly have to wrap commands in docker-compose exec (or equivalent) as needed. This can be useful for running database shells like psql.

We now have a low-friction prototyping workflow:

• Create requirements.txt with standard packages like pytest, black, etc.
• Run make deps
• Write some code, run make test, and so on

And of course, this is transferrable to other project types, though it tends to be less useful for languages with good, standardized tooling (cargo for Rust, mix for Elixir, etc.).