PyPI packaging, directory structure and publishing our Python software

2021-09-03

(last time edited: 2021-09-12)

tags: python, software, development

Packaging Python packages for PyPI is not a difficult task. Extensive documentation can be found in the official Python docs.

Sometimes when we are creating a package we don't follow the same exact directory structure standard. Or we completely dislike how other people package their programs. So in this case I will be explaining how I choose to package my software.

The structure I always follow is this one:

project_name
├── CHANGELOG.md
├── LICENSE.md
├── README.md
├── setup.py
├── virt
└── src
    └── module_or_app
        ├── _version.py
        ├── __init__.py
        ├── __main__.py
        └── some_functionality.py

Create the setup.py file. I recommend using a setup.py file instead of a setup.cfg file.

from setuptools import setup
from src.example_package._version import __version__

with open("README.md", "r", encoding="utf-8") as f:
    long_desc = f.read()

setup(
    name="example_package",
    version="__version__",
    author="vnrdnnfg",
    author_email="asdgreg@gdsgrteger.com",
    description="Some program.",
    long_description=long_desc,
    long_description_content_type="text/markdown",
    url="https://somegitwebsite.com/vnrdnnfg/example_package",
    license="MIT License",
    install_requires=["some_dependency>=2.0.1", "another_dependency>=0.10.2"],
    packages=["src/example_package"],
    package_data={"": ["config/something_something.txt"]},
    entry_points={
        "gui_scripts": "any_run_command = src.module_or_app.__main__:runfunction",
    },
)

Quick explanation of Entry Points

We want our program to be accessible from console shell using the tab completion.

If the program does not make use of graphical libraries then use console_scripts instead of gui_scripts. Though in UNIX systems does not make any significant change but in Windows does, it seems.

This is an example of the __main__.py file.

from module_or_app.app_or_module import runfunction

if __name__ == "__main__":
    runfunction()

Building, TestPyPI repository and PyPI repository

First activate the virtual environment.

$ . virt/bin/activate

Install twine.

This tool simplifies uploading to the Python Package Index repositories.

$ pip install twine

Create a package only of source code, also known as source distribution (sdist).

$ python setup.py sdist

We can opt to create a wheel package (.whl) too. It depends on your needs.

$ python setup.py bdist_wheel

If we take a look all distributions were created in a new dist directory at root of the project.

$ ls /path/to/project_name/dist

Configure the ~/.pypirc configuration file for repositories. Get the password for your TestPyPI account and the password for your PyPI account and add them in the file.

[testpypi]
username = __token__
password = heregoestheapitoken

[pypi]
username = yourusername
password = yourpassword

Since we already created a distribution we can test it locally before uploading to TestPyPI.

Run this command inside the root directory of your project. Some people run it with the -e argument. I don't recommend doing so.

$ pip install .

Test if it executes correctly.

$ any_run_command

TestPyPI

Upload a created source distribution (or the wheel distribution) to the TestPyPI repository using twine.

$ twine upload -r testpypi /path/to/project_name/dist/project_name*

Now we want to test locally. Create a different virtual environment somewhere else and install the package we uploaded to TestPyPI.

$ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ project_name

PyPI

If our package works fine locally and from TestPyPI we are ready to upload it to PyPI.

The process is the same as before, except we change the target repository.

$ twine upload -r testpypi /path/to/project_name/dist/project_name*

Some tips

For local development testing it's recommended that you test your program inside the virtual environment before and after uploading to TestPyPI.

Run this command in the root directory of your package.

$ pip install -e .

Also, always make use of absolute imports instead of relative imports. Absolute imports are crucial if you wanna publish a package. The setuptools module in setup.py will relay on absolute imports. Also, they are useful to make things clear, even if they are spaghetti long.