Even if you're not a Python developer, you've probably experienced working on a Python project...

Essential Python packaging tools and best practices for efficient deployment. Streamline development, distribution, and ensure smooth code delivery.

Python Packaging: Tools and Best Practices for Efficient Deployment

Packaging in Python: Tools and Best Practices

There are a lot of ready-made templates for Python packages out there. You can simply clone them, make your changes, and get ready for deployment, but in this article, we're going to talk about the very basic files and routes you need to have to make a very basic Python package.

Our package is going to be used in two ways. You can either use it inside your CLI or import it within your Python source code. All it does is it shows a hello-world message so I'm going to name this project ‘Hello’. Keep in mind that if you want to publish your package to PyPI, you must choose a unique name meaning "pypi.org/project/<package-name>" has to lead you to a 404 page.

Open a fresh terminal tab and create a directory called Hello. In this directory, we'll keep the source code and metadata showcasing our package.

Each Python package has a setup.py file that contains all the metadata about our package including its version, name, entry points, etc. I'm going to write down this file based on a basic greeting package.

from setuptools import setup


setup(
    name="hello",
    entry_points={
        "console_scripts": [
            "hello = hello.cli:main"
        ]
    }
)

In the first line, we've imported the main library that we need for package initialization which is setuptools. Then, we call the setup() function with the metadata of our package. Here I've filled up the required parameters only.

Keep in mind that some parameters and metadata keys are nice to have. I've included them here.

from setuptools import setup, find_packages    # new

setup(
    name="hello",
    version="0.1.0",    # new
    packages=find_packages(),   # new
    entry_points={
        "console_scripts": [
            "hello = hello.cli:main"
        ]
    }
)

The find_packages() function selects the subdirectories of the package and counts them as the packages that need to be included within the distribution cleverly. One handy parameter of this function is nothing but the exclude param that allows you to ignore some subdirectories that you don't want your distribution to have such as docs/, tests/, or benchmarks/, and so on as they're not part of your actual practical package and your package does not require these to work perfectly fine.

We also have an entry_points parameter inside the setup() function that is responsible for adding CLI commands that point to some of the functions/methods of our package. As you can see, we only have a hello command that points to the main() function inside the cli.py file of the hello directory. A quick tree-like shape of our package would be like this so far.

hello
├── hello
│   └── cli.py
└── setup.py

Inside the cli.py file, we need a function called main() that plays the role of the CLI command.

import sys


def main():
    try:
        stdin_name = sys.argv[1]
        print(f"Hello {stdin_name}!")
    except IndexError:
        sys.exit("Provide a name please!")

If you want to, you can create a virtual environment inside the root directory of your package and install the package in editable mode via the -e option. Otherwise, you can run the same command outside of any virtual environment.

The reason that we've used the -e is because we want to see the changes that we make at every execution of our package. It helps us not to install the package on every change that we make.

To check if the package works fine, simply run the hello command:

$ hello Sadra
hello Sadra!

$ hello
Provide a name please!

Structuring & Package Initialization

To implement the internally used function for our hello package, all we need to do is create a new internal.py file that contains a greet(name: str = "World") function that returns the greeting phrase. For that matter, I'll create it inside the hello/ directory right next to the main.py file.

def greet(name: str = "World"):
    return f"Hello {name}!"

A tree-look of our package would be as follows:

hello
├── hello
|   ├── internal.py
│   └── cli.py
└── setup.py

Easily open a Python REPL and use the greet function as follows:

from hello.internal import greet

greet("Sadra")
# Hello Sadra

greet()
# Hello World

Now, let's talk about the tools and best practices for creating a Python package.

Internal Usage

Poetry is an advanced Python environment manager that helps you manage your package dependencies and create Python packages as well. This tool automates almost all phases of your package development- creation, dependency management, and publishing.

In this example, we’ll install the Poetry package first, create a new package, and add some dependencies afterward.

Once you have the tool installed on your machine, create a fresh package via the init command and keep prompting until your package skeleton is created.

It creates a pyproject.toml file for you, which contains all the metadata of your package. (Everything that we had passed to the setup function from the setuptools library)

To add a new dependency, use the command- ‘add’.

Although we’ve not talked about the publishing phase of your package, but keep in mind that the commands built and published using Poetry are responsible for doing so.

The build command creates a new distribution file for you. It’s in the format of an archive file (source distribution or sdist) like .tar.gz or you could even specify whether you need a wheel distribution or the default one.

Using Poetry

Python Package Index (PyPI) is the place where almost all the public Python libraries are stored. It’s the server that pip tries reaching out to, by default, whenever you want to install a new package.

In this tutorial, we’re focusing on publishing your package on PyPI. You can publish your package to a private infrastructure, such as a private repository on GitHub.

What is PyPI

If you want to publish your package to PyPI, you need to follow some steps.

Make sure your package is healthy and installable.

The above points are the fundamental steps you need to consider to make your package available on PyPI. In this section, we’ll go through this subject in detail.

Head to pypi.org and create a new account. Log into your dashboard and go to your account settings. Scroll down and you’ll see the API tokens section. Generate a new token with all the accesses granted. Don’t forget to copy the token into your clipboard so that you can create a simple push configuration where it’s going to use your API token to push the changes.

Once you’ve copied the token, create the $HOME/.pypirc file and fill it with the following keys and values:

[pypi]
username = __token__
password = <the token value, including the `pypi-` prefix>

Make sure to paste the token as the value for the password key. Now, let’s publish our ‘hello’ package to PyPI. To do so, we need another third-party package called Twine. It helps us push our distribution file to PyPI so install it on your machine.

Since our package is working fine with no issues, let’s create a new distribution archive. We’re going with a simple tar.gz archive for now.

It immediately creates a new directory called dist/ which contains your source-distribution archives.

Notice that we’re trying to push all the distributions. PyPI is clever enough to not allow the duplicate releases to get overridden. So it’ll simply ignore the duplicate distributions.

If your .pypirc is filled with the correct values, your package must be up there by now. If so, you’ll see the success message.

Uploading distributions to https://upload.pypi.org/legacy/
Uploading hello-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 kB • 00:00 • ?

View at:
https://pypi.org/project/hello/0.1.0/

Now, if you run the following command, it’ll reach the hello==0.1.0 version and install the latest release on your machine.

Now, you can easily run the hello CLI command that we designed earlier.

Publish Your Package

As a quick tip, you can privately store your entire project on platforms like GitHub. Throughout this process, we’ll no longer need PyPI or anything related to this service. Simply push your project to a private/public GitHub repository and install it with the following command:

$ pip install git+https://github.com/username/hello.git

Notice that you should have Git installed on your machine.

No More PyPI

Versioning your package is a crucial phase as you can publish thousands of releases in different numeric patterns. All you need to have is a proper strategy for keeping them. A good versioning strategy is Semantic Versioning which Python highly suggests for some reasons. If you’re using Poetry, you can use the poetry version command that helps you bump up your package’s version.

Otherwise, if you’re comfortable with the simple procedure, I highly suggest you use a third-party package as it’s safer. You might’ve stored the version number of your package in multiple files and those kinds of packages are smart enough to find them and update them at the same time.

There is a popular package called bump2version that does the same thing. It’s responsible for taking care of your package versioning. Make sure you’re using the LTS fork as a few people have tried working on a stable version of this package. (The original one is deprecated)

Therefore, I prefer this as a proper version-bumping tool in Python. It also accepts a custom config file that allows you to specify the files and patterns.

Versioning

This section is for those who maintain their packages on GitHub. To clear this up, GitHub offers you a free tool called Actions. It’s a nice solution for CI automation and CD pipelines. Imagine you could hit the release button on GitHub, release a new version of your package and there was a task trying to publish your package to PyPI at the same time. Thus, you won’t need to do it on your own as it might not be quite delightful in some circumstances.

There is an open-source action that builds your package, creates a new sdist archive, gets your API token, and uses Twine to push your newly released distribution. The following steps are relevant and needed if you want to automate this procedure.

Create your repository and push your package to GitHub.

Configure the workflow and make sure that it gets triggered on a release action.

Create a new .github/workflows/publish.yml file. It’ll contain your CD pipeline.

name: Upload Python Package

on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  deploy:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v3
      with:
        python-version: '3.10'
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build
    - name: Build package
      run: python -m build
    - name: Publish new distributions to PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

In the last step, we will “publish new distributions to PyPI”. This step takes a parameter called password and as you can see, it’s trying to read the API token that we’ve been talking about. As you can see, it’s being stored as an action secret meaning you have to create a new secret for this workflow execution named PYPI_API_TOKEN and put the API token that you’ve copied from pypi.org as the value of this secret key.

When you’re creating a new API token on your PyPI dashboard, make sure not to set its scope to ALL as anyone who has access to this token will be able to modify or add new releases to the packages being maintained on the same account specifically when you’re working on a platform like GitHub whereas lots of maintainers might have access to the highly-secured parts of your repository.

Publish Automatically

In this tutorial, we took a quick look over the structure of a Python package and immediately we dived through the best practices, tips, and tools that would simplify this procedure for you. We also talked about Poetry which is a popular tool that enables you to maintain your Python package from the very beginning step toward deployment and publishing to a third-party service such as PyPI.

Conclusion

Even if you're not a Python developer, you've probably experienced working on a Python project where you needed to install additional libraries and packages to make your piece of code work. Making reusable packages is another cool aspect of working as a Python developer.

In this article, we're going to talk about developing Python packages from an initial stage, to delivering the releases to PyPI (Python Package Index). We'll have a bonus section at the end of this article that helps you automate your delivery phase using GitHub Actions if you're developing your package inside a repository located on GitHub.

Generally, your Python package can be distributed in two types. Either an archived file or a wheel distribution. We'll go through each one separately.

We'll get our hands on a quick walk-through hello-world example throughout this article. In this case, I'm not trying to publish the package to the PyPI so its name is not a big deal.

A popular packaging tool is setuptools which doesn’t come with the standard libraries anymore. A good substitution for this tool is Poetry which gives you more automation accessibility.

Wheel distribution is made of the bytecodes, compiled source codes, and already-built components which means, they’re faster in terms of installation and execution. On the other hand, sdists contain pure Python source codes that need to be interpreted on each call.

Yes, PyPI has a testing instance running alongside its real one. It’s called TestPyPI. You can refer to this instance if you want to ensure any deployment and safety before you publish your package into the real market.

Frequently Asked Questions

What’s up with Turing?
Get the latest news about us here.

Press

Know more about remote work.
Checkout our blog here.

Blog

Have any questions?
We’d love to hear from you.

FOR DEVELOPERS

Packaging in Python: Tools and Best Practices

Author

Sadra Yahyapour

Introduction to Pytest Framework

How to Check for NaN Values in Python?[With Examples]

Top Python Libraries for Automation

Master React User Authentication: A Comprehensive Guide

Feature Engineering in Machine Learning With Python: A Guide

Frequently Asked Questions

Press

Blog

Contact

Hire remote developers