Pytest set up - How to solve ModuleNotFound error with recommended directory structure

2021-10-25 635 words 3 minutes

Contents

Problem

On running pytest with configuring strongly recommended directory structure in good practices in docs.pytest.org,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


setup.py
src/
    mypkg/
        __init__.py
        app.py
        view.py
tests/
    __init__.py
    foo/
        __init__.py
        test_view.py
    bar/
        __init__.py
        test_view.py

In my sample case, the directory is like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


concurrent-api-client
├── README.md
├── log
├── setup.py
├── src
│   └── api_client
│       ├── __init__.py
│       ├── api_client.py
│       ├── api_formatter.py
│       ├── exceptions.py
│       └── ...
│       └── ...
└── tests
    ├── __init__.py
    ├── test_api_formatter.py
    └── test_weather_api.py

However, pytest cannot find the source package directory like:

1

E   ModuleNotFoundError: No module named 'api_client'

Solution

Refering docs.pytest.org | good practices and linked blog post (ionl’s codelog | Packaging a python library) it seems writing setup.py appropriately and run pip install -e . solves this.

setup.py for project (tweaked some code for a simple example)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


#!/usr/bin/env python
# -*- encoding: utf-8 -*-
from glob import glob
from os.path import basename, splitext

from setuptools import find_packages
from setuptools import setup

def get_readme():
    """ Get the README from the current directory. If there isn't one, return an empty string """
    all_readmes = sorted(glob("README*"))
    if len(all_readmes) > 1:
        warnings.warn("There seems to be more than one README in this directory. Choosing the "
                      "first in lexicographic order.")
    if len(all_readmes) > 0:
        return open(all_readmes[0], 'r').read()

    warnings.warn("There doesn't seem to be a README in this directory.")
    return ""

setup(
    name='concurrent-api-client',
    version='1.0',
    license='MIT',
    description='An example for API client using python request library',
    long_description="\n" + get_readme(),
    author='tatoflam',
    author_email='tatoflam@gamil.com',
    url='https://github.com/tatoflam/concurrent-api-client',
    packages=find_packages('src'),
    package_dir={'': 'src'},
    py_modules=[splitext(basename(path))[0] for path in glob('src/*.py')],
    include_package_data=True,
    zip_safe=False,
    classifiers=[
        # complete classifier list: http://pypi.python.org/pypi?%3Aaction=list_classifiers
        'Development Status :: 4 - Beta',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Operating System :: MacOS :: MacOS X',
        'Programming Language :: Python :: 3',
        'Topic :: Utilities',
        'Topic :: Software Development :: Libraries :: Python Modules'
    ],
    project_urls={
#        'Changelog': 'https://github.com/tatoflam/concurrent-api-client/blob/master/CHANGELOG.rst',
        'Issue Tracker': 'https://github.com/tatoflam/concurrent-api-client/issues',
    },
    python_requires='>=3.7',
    setup_requires=[
        'pytest-runner',
    ]
)

After running pip insatll - e ., sys.path gives the path to the package directory and pytest can retrieve modules under src directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


$ python
Python 3.9.7 (default, Sep 10 2021, 10:49:24) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/Users/a-user/.pyenv/versions/3.9.7/lib/python39.zip',
 '/Users/a-user/.pyenv/versions/3.9.7/lib/python3.9',
 '/Users/a-user/.pyenv/versions/3.9.7/lib/python3.9/lib-dynload',
 '/Users/a-user/.pyenv/versions/3.9.7/lib/python3.9/site-packages',
 '/Users/a-user/repo/tatoflam/python_api_client/src']

The other ways that I tried

Make __init__.py to src or test directory
Set environment variable: PYTHONPATH (e.g. export PYTHONPATH=.:~/repo/tatoflam/python_api_client:$PYTHONPATH)
- Or append path . and ~/repo/tatoflam/python_api_client' to python to sys.path by sys.path.append()
pip insatll - e . (without setting setup.py)

This can add path to the package directory, but did not solve the ModuleNotFoundError under the modules in src directory. Even if locating modules without src directory like below structure, pytest works, that’s not the case for recommended directory structure. So, I believe above solution is better.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


python_api_client
├── README.md
├── log
├── setup.py
├── api_client
│   ├── __init__.py
│   ├── api_client.py
│   ├── api_formatter.py
│   ├── exceptions.py
│   └── ...
│   └── ...
└── tests
    ├── __init__.py
    ├── test_api_formatter.py
    └── test_weather_api.py

Sources

github