Python 3.10 Upgrade, Torch Not Found
I run Arch GNU/Linux, and the maintainers recently upgraded the Python version to version 3.10. Great! With this new version I can finally use all of its newest features.
However, something sinister was waiting for me. I was working on one of my open
source repos, my 1D, 2D, and 3D positional
encodings after
recently having done a full upgrade, which brought with it Python 3.10.
Incidentally, during this time, I decided to spin up a new virtual environment
with this latest version. I ran pip install -r requirements.txt to install
the required dependencies, when I got this error:
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
Oh no! torch was nowhere to be found for Python 3.10, meaning that I couldn’t
run my tests or develop any further. And trying to downgrade any package in Arch
is more of a hassle than what it is worth, so it looked like I just had to wait until
torch was available for Python 3.10. Or so I thought.
After a lot of painstaking effort (and help from a
friend), I ended up finding a solution. I learned a
lot about Python dependencies and why they really suck the way that they are
implemented now. I also found out how other packages, mainly
pipenv and
pyenv, solved this issue by allowing me to
use a virtualenv with an older version of Python (3.9) where torch was
supported. Here is a step by step guide so that you don’t spend so much time as I did.
To start, let’s quickly go over each of these packages. What are they, and what do they do?
pipenv
pipenv aims to bring a modern package world into Python. Per the github
page,
It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever-important Pipfile.lock, which is used to produce deterministic builds.
What is nice about pipenv is that it allows you to specify an environment and
a Python version with one very simple command.
pyenv
This is the tool that allows me to switch between multiple versions of Python
easily. pipenv uses this if the version of Python that you specify isn’t the
one on your system.
Arch install
The one thing to note is that you want to make all of these packages available
system-wide. To do this, simply install them with pacman:
$ sudo pacman -S python-pipenv pyenv
Now that we know what these do, lets see how we can link these together to
resolve our dependency problems and how to use Python 3.9 to install torch
(again!). We start by navigating to our project directory:
$ cd my-awesome-repo
Let’s start with pipenv. We have to get the proper Python version, which in
our case will be Python 3.9. This is as simple as running:
$ pipenv --python 3.9
Here, it asks for a backend. This is where the pyenv will come into
use, as this will automatically pull the proper version of Python.
After this finishes downloading and installing, there will be a Pipfile. This
file specifies which version of Python to use for this virtual environment. The
Pipfile might look something like this:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
[dev-packages]
[requires]
python_version = "3.9"
To enter a virtual environment specified by this pip version, we need to run
$ pipenv shell
From the folder where the Pipfile is located. After that, you should be in
your environment running Python 3.9! To make sure that this is the case, we open up
a Python shell to confirm that this works:
((my-awesome-repo) ) me@host ~ $ python
Python 3.9.9 (main, Dec 30 2021, 18:14:31)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
Now, it is as simple as running pip install -r requirements.txt to get all of your
dependencies, and then, you’re able to run pytest just fine.
There are several other Python packages that ensure proper Python versioning
(so that this problem doesn’t happen again), one of which is poetry. We will
explore this in another blog post.
