How I set up my Python

isotopp image Kristian Köhntopp -
June 29, 2020
a featured image

Because Martin wanted some starting point, here is how I set up my Python. There are a lot of other things one can do, but this is supposed to be just a starting point.

For a new project, make a project directory, usually not with a local git repository.

kk:Python kris$ mkdir project
kk:Python kris$ cd project
kk:project kris$ git init
Initialized empty Git repository in /Users/kris/Python/project/.git/

We need a virtual environment to keep our modules apart from the system python.

kk:project kris$ python3 -m venv ~/.venv/project

In my case, I have a bash PROMPT_COMMAND installed that automatically activates the venv when I enter a directory. From the .bashrc, some highly insecure code:

autoenv() {
  if [ -f ./.venv ]; then
    nv=$(cat ./.venv)
    if [ -f "$nv/bin/activate" ]; then
      if [ "$VIRTUAL_ENV" != "$nv" ]; then
        source $nv/bin/activate
      fi
    fi
  fi
}

PROMPT_COMMAND=autoenv

And then:

kk:project kris$ echo /Users/kris/.venv/project > .venv
(project) kk:project kris$

As can be seen from the prompt, the venv already active. Usually, after install the pip is outdated.

(project) kk:project kris$ pip install --upgrade pip
...
Successfully installed pip-20.1.1

We also need wheel for C-Language Extensions, and black, mypy and pre-commit.

(project) kk:project kris$ pip install wheel
...
Successfully installed wheel-0.34.2
(project) kk:project kris$ pip install black mypy pre-commit
...
Installing collected packages: click, attrs, pathspec, regex, toml, appdirs, typed-ast, black, mypy-extensions, typing-extensions, mypy, cfgv, filelock, six, zipp, importlib-metadata, distlib, virtualenv, nodeenv, pyyaml, identify, pre-commit
Successfully installed appdirs-1.4.4 attrs-19.3.0 black-19.10b0 cfgv-3.1.0 click-7.1.2 distlib-0.3.1 filelock-3.0.12 identify-1.4.20 importlib-metadata-1.7.0 mypy-0.782 mypy-extensions-0.4.3 nodeenv-1.4.0 pathspec-0.8.0 pre-commit-2.5.1 pyyaml-5.3.1 regex-2020.6.8 six-1.15.0 toml-0.10.1 typed-ast-1.4.1 typing-extensions-3.7.4.2 virtualenv-20.0.25 zipp-3.1.0

The last three collect a large number of additional dev-dependencies.

We then need to set-up the pre-commit environment:

(project) kk:project kris$ cat .pre-commit-config.yaml
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
    - id: check-yaml
    - id: end-of-file-fixer
    - id: trailing-whitespace
-   repo: https://github.com/psf/black
    rev: ''
    hooks:
    - id: black
-   repo: https://github.com/pre-commit/mirrors-mypy
    rev: ''
    hooks:
    - id: mypy
      additional_dependencies: [ ]
(project) kk:project kris$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

Pre-commit is running mypy and black, but in a separate environment, so it will be installed again, elsewhere, and we will also need to declare our mypy dependencies here again, unfortunately.

On the other hand, all our Python will be type-checked and blacken’ed automatically on commit.

(project) kk:project kris$ cat probe.py
#! /usr/bin/env python

import sys

print(f"{sys.version}")

And here we go:

(project) kk:project kris$ git add probe.py
(project) kk:project kris$ git commit -m 'testing'
Check Yaml...........................................(no files to check)Skipped
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
black....................................................................Passed
mypy.....................................................................Passed
[master (root-commit) c7734e4] testing
 1 file changed, 5 insertions(+)
 create mode 100755 probe.py

Now, I usually collect additional project dependencies in requirements.txt and freeze them into requirements-frozen.txt for deployment into a Python Container.

For example, in one project:

(bridge) kk:bridge kris$ cat requirements.txt
paho-mqtt
influxdb
(bridge) kk:bridge kris$ cat requirements-frozen.txt
paho-mqtt==1.5.0
influxdb==5.3.0
## The following requirements were added by pip freeze:
certifi==2020.4.5.1
chardet==3.0.4
idna==2.9
msgpack==0.6.1
python-dateutil==2.8.1
pytz==2020.1
requests==2.23.0
six==1.15.0
urllib3==1.25.9
(bridge) kk:bridge kris$ cat Dockerfile
FROM python:3.8-alpine

LABEL maintainer="isotopp" \
      description="MQTT to InfluxDB Bridge"

COPY requirements-frozen.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt

COPY . /app
WORKDIR /app

CMD [ "python3", "-u", "bridge.py" ]

This is part of a larger deployment in a docker-compose.yml then:

---
version: "3"

services:
...
  bridge:
    build: "./bridge"
    image: "isotopp/mqttbridge"
    container_name: "mqttbridge"
    hostname: "mqttbridge"
    user: "1000"
    depends_on:
      - "influxdb"
      - "mosquitto"
    restart: "always"
...
Share