Managing Python Versions and Virtual Environments
The idea for this blog post is about how Python can be used in different environments. These are so-called “virtual environments” in Python, and there have been many tools that provide this capability with more or less additional functionalities.
Some of you may remember venv and the virtualenv as one of the first tools that enabled developers to create specific virtual environments and not pollute the system. However, a lot was left to be desired. So then there were virtualenvwrapper and pipenv. These tried to address the lack of usability in some use cases. However, the main points that all of these tools want to address are:
- deal with multiple Python interpreter versions,
- where to install a virtual environment, or have some consistent way of doing it,
- how to logically connect the virtual environment with your project if it was installed somewhere else and not in the project directory,
- activate the virtual environment when you are about to work on the project, and then deactivate it,
- keep the lists of installed packages that were used for project development and for using the project as a library in another project,
- how to pack your project and publish it in a consistent way.
There is a slight chance I forgot some pain points that developers had to solve. Nevertheless, you can see that there are already enough concerns to deal with once you start working on your Python projects.
Those of you who know what I’m talking about will shout (silently) “Hey, you forgot pyenv and poetry!” And you would be wrong, since these two were exactly the reasons why I wanted to write this blog post. And yes, you can use just one of these. However, I like to use them both as complementary tools to achieve all of the previously addressed points in the list above.
Pyenv
This tool is really handy if you want to have more than one Python interpreter version available on your system. I will not go over the installation and basic usage tutorials here. My idea is to give you a general overview of how pyenv works.
On a vanilla system, you will probably have just one Python interpreter version installed. This will be enough for simple fiddling with Python or even working on your project, as long as the project’s Python version is exactly the same as the one on your system. Of course, you would have to install packages on a system level and pray that you wouldn’t work on two different projects with different sets of packages (and versions) at the same time.
Once you start being more serious with your development and research, you will find yourself using different Python interpreter versions with different sets of packages and their versions on each project. Using system-installed Python will probably get messy, and there is a high chance you will end up “devopsing” your time instead of focusing on Python’s Zen.
Here, pyenv comes to the rescue. It gives you a way to install as many Python interpreter versions as you want. And not just CPython ones, but others like PyPy, Pyston, and Stackless. You can really enjoy your options. And even though this part is really handy and superb, the next thing is even better. You can change the interpreter versions used on three different levels: global, local, and shell.
The global level is just the default interpreter version on your system. This way, whenever and wherever you type python in shell or run your project with python main.py you will use this specific version of the interpreter. You set this by executing pyenv global <version> in your shell.
Local level is even better because you set the interpreter version for the specific directory. This should be the project’s directory, so once you change your path to this project’s directory, the interpreter version used should be updated. The same interpreter version is set for all subdirectories. You can set this by executing pyenv local <version> in the shell, and you end up with .python-version file in the current directory, that specifies desired version.
Shell level is even more granular and sets the interpreter version just for the current shell that you are in. This is the least persistent setup because once you exit the shell and open it again, you end up with the interpreter version set up locally (if it is) or globally (which is set by default).
This makes your life a lot easier.
One more thing that is really handy is that you can manage virtual environments with pyenv. By having multiple interpreter versions and implementations, you are able to instantiate many virtual environments. These are installed at the specific system location set up by default. You can just write each down on a piece of paper and remember which project it belongs to, or you can just run pyenv local <virtual env name> in the project directory. This way, you get the automatic virtual environment activation once you position yourself in the project directory in shell.
In essence, pyenv solves 1.-4. points from the list given above. You could deal with the other two points for sure, but you would have to use additional tools. I see that pyenv was not created with that goal. And here we can slide into our next part.
Poetry
While pyenv gives the ability to manage different Python interpreter versions and many different virtual environments, poetry is good for managing package versions for each project (and virtual environment). poetry has its own way of managing virtual environments, which, in my opinion, should not be mixed with pyenv way.
Obviously, I will not be able to squeeze all of the information on how to use poetry here, but in general, it uses meta files that describe your project and specify versions of packages used. Or, to make it more eloquent, it is a tool to deal with dependency management in your project. Those who are seasoned developers can find this extremely important because of so-called dependency hell, while the newcomers will learn the importance of this very soon, if not already.
And while the user manual goes into depth describing how poetry can and should be used, it is interesting how it can be composed with pyenv. More precisely, my advice would be to use pyenv to make different Python interpreter versions available, and then use poetry to install a virtual environment for your project. Creating projects in poetry is as easy as running poetry new <project name>. From there you are able to install packages, update their versions, enter a virtual environment for easier interaction with the code, and so on.
Additionally, poetry gives you a way of structuring your project so you can publish it as a package/library. This way, it can be publicly available on PyPi or in private repositories.
Essentially, poetry solves the main points 2.-.6. So it’s almost all of them. This is why I think it is a good choice to use it with pyenv to achieve a smooth development process.
In the end, I want to make it clear that it is really up to you which set of tools you are going to use for your project, but you have to be aware of the scaling development process and make it smooth not just for you but for the whole team. You even have to make sure your project, which is going to be an integral part of some other project, is well structured with a logical package structure and good versioning. What could make your life easier as a contributor to a library could make library users lives harder. Have that in mind.
Anyway, have a nice day and share this blog post if you liked it!