How to Package Your Python Code. A simple guide to packaging your python… | by Nacho Vargas | Apr, 2022

A simple guide to packaging your python code and deploying it to PyPi or a private index like JFrog Artifactory

If you just want the step by step guide, please skip ahead.

Packaging your code is basically making it available to be used by others. Or making it ready to be shared Basically…

What does that actually mean? Well in Python it means the creation of a tar.gz file with everything your project needs (called the source archive) and more importantly, a wheel distribution file (called the built distribution) which is what we call the actual distribution.

the result of packaging a python code

Why 2 files?

The main distribution is the .whl file (wheel build distribution). The tar.gz file is used as a fall back by pip if needed when installing the .whl file.

What is Wheel and what is the Build distribution?

A build distribution is a file containing everything your project need to run in another computer including metadata such as name, version, license, etc.

Wheel is a build distribution format introduced by PEP 427 (meant to replace the Egg format).

Note that the version is always included in the file names. This is because a distribution is versioned, meaning several versions of the same package can and should exist. These are called releases.

This is the format of the distribution file name:
{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl

Ok! we now know what packaging is, very cool, but…

If you want your package to be accessible to everyone, then we use the Python Package Index (PyPi).

If you wish for your package to be used internally by yourself, your friends, or coworkers, you need a private index such as JFrog’s artifactory.

I will explain how to upload our package to both of these…

Once uploaded to any of these indices, your package will be available to be downloaded (using pip for example) and used by others.

To put things into perspective, there are 3 main components in place when distributing a Python package:

  • Package format: The source distribution we talked about before. The wheel is an example of a format.
  • Package index: A central place where packages are stored and managed. There are public ones and private ones.
    Public: PyPI
    Private: JFrog has one for example, but you can also host your own.
  • Package manager: A software that is in charge of getting the requested distribution from an index and installing the wheel into you computer. A classic example is pip, but there are others like build.

Ok, enough theory…

Let’s package a small sample code!

We will use setuptools so everything will be recommended according to their recommendations.

setuptools is a library which is built on top of distutils(that has been deprecated). The package offers a wide range of functionality that facilitates packaging of Python projects.

Ok, first the project structure:

project structure

The target code here lives inside my_awesome_module. The name of the module is important because it is what will be imported by others.

Yup… Simple enough…

Now we need a way to tell build tools some information about our package. To do this, setuptools uses 3 files:

  • pyproject.toml: A file that tells build tools (like pip and build) what is required to build your project.
  • setup.cfg: The configuration file for setuptools. It tells setuptools about your package (such as the name and version) as well as which code files to include.
  • setup.py: The build script for setuptools. It can hold the same metadata as setup.cfg but its dynamic not static, and enables the execution of packaging commands.

PyProject.toml

This article uses setuptools, so open pyproject.toml and enter the following content:

[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"

Setup.cfg

Together with the setup.pythis is the main file where your configuration is stored.

Setuptools allows using configuration files (usually setup.cfg) to define a package’s metadata and other options that are normally supplied to the setup() function (declarative config).

[metadata]
name = my_own_custom_package
version = 0.0.1
author = You
author_email = its_you@example.com
description = A small example package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://some_url_to_the_code_usually_github.com
classifiers =
Programming Language :: Python :: 3
Operating System :: OS Independent
license_files = LICENSE.txt
install_requires =
requests
[options]
package_dir == src
packages = find:
python_requires = >=3.6
[options.packages.find]
where = src

Key items:

  • name — The name of your package. This is what you will pip install <package_name>
  • versionCurrent version of your package. Very important to increase this on every change
  • descriptions — Description of the package. You can refer to your README.md file also like in the example.
  • packager_dir and packages — The packages that your package include. In this example, include everything inside the src module.
  • install_requires — Defines the packages your package needs to run, in this example I’ve added a classic Python package: requests

A full list of available items can be found here.

Setup.py

The setup.py file is probably the most important file that is supposed to be included at the root of your Python project directory and it mostly serves two primary purposes:

  1. It can contain various information relevant to your package including options and metadata, such as package name, version, author, license, minimal dependencies, entry points, data files and many more. (This is recommended to be contained in the setup.cfg as we saw).
  2. Serves as the command line interface that enables the execution of packaging commands.

As is recommended by setup-tools, we keep the metadata in the setup.cgf so our setup.py looks like this:

from setuptools import setupif __name__ == '__main__':
setup()

simple right?

Lets package it…

Make sure you have the latest version of PyPA’s build installed:

[Unix/MacOs]
python3 -m pip install --upgrade build
[Windows]
py -m pip install --upgrade build

Now let’s build:

[Unix/MacOs]
python3 -m build
[Windows]
py -m build
building the package

Done! you should end up with something like this:

notice the new dir folder

We have learned what it means to package our code, and if you followed the last part, we have now the packaged files for our awesome Python code.

Great, now let’s upload it so it can actually be used by others…

PyPi

In this article, we will use the testing server of PyPi which is a separate instance of the package index intended for testing.

To register an account, go here and complete the steps on that page. You will also need to verify your email address before you’re able to upload any packages. For more details, see Using TestPyPI.

Now that you are registered, you can use twine to upload the distribution packages. You’ll need to install Twine:

[Unix/MacOs]
python3 -m pip install --upgrade twine

[Windows]
py -m pip install --upgrade twine

Note

The twine command to push the package will ask for authentication obviously. There are 3 ways of doing this:

1. Use your email and password (fine for testing)

2. Create an API Token and use it directly.
Create one heresetting the Scope to Entire account. Don’t close the page until you have copied and saved the token — you won’t see that token again.
Username and password are the same as in the section below.

3. Create an API Token and store it in your twine conf (recommended)
After you have the token, set up a$HOME/.pypirc file like this:

[testpypi]
username = __token__
password = <your-token-including-the-pypi-prefix>

Ok, let’s run Twine to upload all of the archives under dist .

[Unix/MacOs]
python3 -m twine upload --repository testpypi dist/*
[Windows]
py -m twine upload --repository testpypi dist/*

If you did not setup your twine conf, you will be prompted for a username and password. Authenticate accordingly.

After the command completes, you should see an output similar to this:

And if you go to your projects in TestPyPi, you will se your new package!

You are done! Anyone can now use the package against TestPyPi running:

pip install my-own-custom-package

* Note that pip must be configured to point to TestPyPi.

The steps to deploy to actual PyPi are exactly the same just deploy to pypi instead of testpypi :

[Unix/MacOs]
python3 -m twine upload --repository pypi dist/*
[Windows]
py -m twine upload --repository pypi dist/*

Done!

You have your very own Python package and ready to be used in PyPi! Go check it out!

nice

Here you can see my own deployment of this test code into PyPi.

Import it and use it

I will use the one I published, but please try and use the one you just published into PyPi or TestPyPi.

pip install my_awesome_package

Now anyone can use it in their code:

from my_awesome_module import helloif __name__ == "__main__":
hello.say_hello("Nacho")

Note that the import package is my_awesome_module regardless of what name you gave your distribution package in setup.cfg or setup.py (in this case, my_awesome_package).

output of the example above.

Private Index (JFrog Artifactory)

First create a JFrog Account and add a new Python Artifactory repo.

Note: JFrog offers a free cloud solution to host packages as well as other things like docker images, etc.
That being said, it also has paid plans, so see what plan fits your needs. For this article (and in some work scenarios as well) I went for a free plan.

After you have your repository in place in JFrog, create a .pypirc file in your root folder and add this to it:

[distutils]
index-servers = local
[local]
repository: http://localhost:8081/artifactory/api/pypi/pypi
username: <user_name>
password: <password>

local can be changed to anything. Be sure to change it in the following command.

Then run this to deploy your package:

[Unix/MacOS]
python3 -m twine upload -r local dist/* --config-file ~/.pypirc
[Windows]
py -m twine upload -r local dist/* --config-file ~/.pypirc

That’s it! GO and see your package in your repo.

Find the full details for these on the JFrog manual.

Leave a Comment