Python dependency management by pip and pip-tools

2021-10-26 960 words 5 minutes

Contents

For surviving hell of dependency management in python project, it’d good to use python toolkit libraries for dependency management like pip, pip-tools, setuptools (pytest, and tox). At this post, I write about pip, pip-tools and setuptools(setup.py).

Environment

MacOS Catalina 10.15.7
Python 3.9.7

Guides

Python packaging user guide | tool recommendation is fine document.

Tool

pyenv can install multiple python versions and switch python interpreter
venv can create and switch to a python virtual environment for the package package project
pip : pip is the package installer for Python. You can use it to install packages from the Python Package Index and other indexes.
pip-tools : A set of command line tools to help you keep your pip-based packages fresh.
setuptools : python packaging tool.

Setup

pip and pip-tools must be installed in each of project’s virtual environment.

venv : Embedded in python later than python 3.3.
pip : Embedded in python later than python 3.4.
pyenv : If OSX, install via brew (see document)
pip-tools : Install via pip. pip install pip-tools
setuptools : Install via pip. pip install setuptools
pytest : Install via pip. pip install pytest

Before switch python interpreter like below pytenv local 3.9.7, it’s necessary to install python by pyenv like pyenv install 3.9.7.

1
2
3
4
5
6
7


$ cd <your project>
$ pyenv local 3.9.7 # this switches python interpreter
$ python -m venv venv # this create python virtual environment
$ source venv/bin/activate # switch to the virtual environment
$ (venv) pip install setuptools pip-tools pytest
...
Successfully installed click-8.0.3 pep517-0.12.0 pip-tools-6.4.0 tomli-1.2.2 wheel-0.37.0

Installed libraries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


$ pip list
Package            Version   
------------------ --------- 
attrs              21.2.0
certifi            2021.10.8
charset-normalizer 2.0.7
click              8.0.3
idna               3.3
iniconfig          1.1.1
packaging          21.0
pep517             0.12.0
pip                21.2.3
pip-tools          6.4.0
pluggy             1.0.0
py                 1.10.0
pyparsing          3.0.1
pytest             6.2.5
setuptools         57.4.0
toml               0.10.2
tomli              1.2.2
wheel              0.37.0

Dependency management workflow with pip and pip-tools

Make requirements.in (describe necessary libraries for the package)
pip-compile requirements.in (generate requirements.txt)
pip install -r requirements.txt -e . (Install libraries and the package itself)

Directory sturcture:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


concurrent-api-client
├── .python-version
├── README.md
├── MANIFEST.in
├── requirements.in
├── requirements.txt
├── setup.py
├── src
│   └── api_client
│       ├── __init__.py
│       ├── api_client.py
│       └── ...
└── tests
    ├── __init__.py
    ├── test_api_formatter.py
    └── ...

1. Make `requirements.in`

Just describe necessary package name for the project. In sample case,

requirements.in

1
2
3
4


--index-url https://pypi.org/simple

requests
requests-toolbelt

Tip

If local machine has pip.conf file (e.g. ~/Library/Application\ Support/pip/pip.conf and it has customized index-url setting for example below,

1
2


[global]
index-url = https://pypip.your-company/simple

Delete it, then pip can see default url of https://pypi.org/simple, or
run pip-compile or pip with option --index-url=https://pypi.org/simple

2. Make `requirements.txt` by `pip-compile`

pip-tools consolidate version dependency automatically.

1
2


$ source venv/bin/activate
$ (venv)pip-compile --index-url=https://pypi.org/simple requirements.in`

Then you can get requirements.txt like:

requirements.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile requirements.in
#
certifi==2021.10.8
    # via requests
charset-normalizer==2.0.7
    # via requests
idna==3.3
    # via requests
requests==2.26.0
    # via
    #   -r requirements.in
    #   requests-toolbelt
requests-toolbelt==0.9.1
    # via -r requirements.in
urllib3==1.26.7
    # via requests

3. Install libraries and package itself

Before running pip install, you need to update setup.py.

setup.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


#!/usr/bin/env python
# -*- encoding: utf-8 -*-
from glob import glob
from os.path import basename
from os.path import splitext

from setuptools import find_packages, setup

def parse_requirements(filename):
    """ Given a filename, strip empty lines and those beginning with # """
    output = []
    with open(filename, 'r') as f:
        for line in f:
            sline = line.strip()
            if sline and not line.startswith('#'):
                output.append(sline)
    return output

setup(
    name='python_api_client',
    version='1.0',
    license='MIT',
    description='An example for API client using python request library',
    author='tatoflam',
    author_email='tatoflam@gamil.com',
    url='https://github.com/tatoflam/concurrent-api-client',
    packages=find_packages('src'),
    package_dir={'': 'src'},
    py_modules=[splitext(basename(path))[0] for path in glob('src/*.py')],
    include_package_data=True,
    zip_safe=False,
    classifiers=[
        # complete classifier list: http://pypi.python.org/pypi?%3Aaction=list_classifiers
        'Development Status :: 4 - Beta',
        ...
    ],
    project_urls={
        'Issue Tracker': 'https://github.com/tatoflam/concurrent-api-client/issues',
    },
    python_requires='>=3.7',
    install_requires=parse_requirements("requirements.txt"),
    setup_requires=[
        'pytest-runner',
    ],
)

Tip

If you have used pyenv, you might need to check python path by which python. If the python directs to pyenv interpreter directory, you would run pyenv rehash, then the path is switched to your virtual environment.

1
2
3
4
5
6
7


$ which pytest
/Users/a-user/.pyenv/shims/pytest

$ pyenv rehash

$ which python
/Users/a-user/repo/tatoflam/python_api_client/venv/bin/python

Then run pip install with specified version.

1

$ (venv) pip install --index-url=https://pypi.org/simple -r requirements.txt - e .

Installed libraries and package

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


$ pip list
Package            Version   Location
------------------ --------- --------------------------------------------------------
attrs              21.2.0
certifi            2021.10.8
charset-normalizer 2.0.7
click              8.0.3
idna               3.3
iniconfig          1.1.1
packaging          21.0
pep517             0.12.0
pip                21.2.3
pip-tools          6.4.0
pluggy             1.0.0
py                 1.10.0
pyparsing          3.0.1
pytest             6.2.5
python-api-client  1.0       /Users/a-user/repo/tatoflam/python_api_client/src
requests           2.26.0
requests-toolbelt  0.9.1
setuptools         57.4.0
toml               0.10.2
tomli              1.2.2
urllib3            1.26.7
wheel              0.37.0

Then you can run command like pytest on venv using required packages.

Sources

github