pyenv is a Python version manager, but it doesn’t manage virtual environments, so you can’t manage them out of the box like conda. pyenv, however, has the ability to manage virtual environments with the pyenv-virtualenv plugin. This article provides a brief introduction to how Python’s virtual environments work under pyenv-virtualenv.

Here, I have installed Python version 3.10.9 via pyenv and created a virtual environment called play with the pyenv-virtualenv plugin. See the pyenv homepage and the pyenv-virtualenv homepage for more information on how to install pyenv and pyenv-virtualenv and create an environment.

How does Python search for modules?

This section can be found in the Python documentation.

First, when using import, Python searches for modules from the paths recorded in the list of sys.path, much like the logic of how the environment variable PATH works in a GNU/Linux environment. So the purpose of the virtual environment is to make the contents of sys.path for running Python distinct from the system-wide Python environment, so that you can search for modules and install modules in different locations.

The question then becomes how sys.path is determined, and the answer is in the documentation, and is briefly summarized as follows:

Determination of sys.path

If there is an actual running Python script file, such as main.py, then the directory where the script is located is the first entry in sys.path, otherwise (e.g. running the command with python -c) the current directory will be the first entry in sys.path.

The value of the environment variable PYTHONPATH is then appended to sys.path, if PYTHONPATH exists.

Then you add the directories that depend on the standard Python modules and the extension modules, which are related to prefix and exec_prefix.

Finally, site-packages is added, whose path is generated by the site module.

Determination of prefix and exec_prefix

In this section, X denotes the major version number of Python and Y denotes the minor version number of Python, e.g., for Python 3.11, X is 3 and Y is 11.

On Linux, ${prefix}/lib/pythonX.Y contains platform-independent standard Python modules, and ${exec_prefix}/lib/pythonX.Y/lib-dynload contains platform-dependent extension modules, so these two paths will be added to sys.path, so you need to determine the values of prefix and exec_prefix first.

extension modules are modules written in C or C++ that interact with user code via the C API, such as .pyd files for Windows platforms and .so files for other platforms

If the environment variable PYTHONHOME is present, then prefix and exec_prefix will be taken directly from the variable.

If the environment variable PYTHONHOME does not exist, prefix and exec_prefix will start at home and work their way up to the “landmark” files and directories to determine this. The default value for home is the real path to the directory containing the Python binary executable (symbolic links are dereferenced, so only the real location will be used as a starting point).

If the environment variable PYTHONHOME does not exist and a pyvenv.cfg file exists in the same directory as the Python binary executable or in a directory one level above it, then home will obey the configuration in pyvenv.cfg, which is related to the Python virtual environment.

Starting from home, prefix will be determined by looking for pythonXY.zip, which on Windows will be directly under home, but on Unix will be under lib/. Note that even if this zip file is not found at all, it will eventually be added to sys.path, and the explanation for this behavior is here. If this .zip file is not found, Windows will continue to look for Lib\os.py to determine prefix, which corresponds to lib/pythonX.Y/os.py on Unix.

On Windows prefix and exec_prefix are the same, but other platforms will continue to search lib/pythonX.Y/lib-dynload to determine exec_prefix.

The search process from home is similar to the following, and usually succeeds on the second check, or throws an error if the “landmark” file is never found.

1
2
3
4

Python executable path is /usr/bin/python3
check /usr/bin/lib/pythonX.Y/os.py  -> if exists, prefix=/usr/bin
check /usr/lib/pythonX.Y/os.py      -> if exists, prefix=/usr
check /lib/pythonX.Y/os.py          -> if exists, prefix=/

Determination of site-packages

The site module is used to generate the path to site-packages when preifx and exec_prefix have already been determined.

The logic is to use prefix and exec_prefix as the first half of the path, and lib/site-packages on Windows and lib/pythonX.Y/site-packages on Linux as the second half, combining the two paths and The non-existent path is ignored and the existing path is added to sys.path.

Start our validation

Use a simple program to print the message, in my environment it is saved in ~/Desktop/tmp/main.py.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

#!/usr/bin/env python3

import sys

def main():
    print("base_prefix:", sys.base_prefix)
    print("base_exec_prefix:", sys.base_exec_prefix)
    print("prefix:", sys.prefix)
    print("exec_prefix:", sys.exec_prefix)
    for i, j in enumerate(sys.path):
        print("sys.path", i, ":", j)

if __name__ == "__main__":
    main()

Under the system global, the program output is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ python ~/Desktop/tmp/main.py
base_prefix: /usr
base_exec_prefix: /usr
prefix: /usr
exec_prefix: /usr
sys.path 0 : /home/leafee98/Desktop/tmp
sys.path 1 : /usr/lib/python310.zip
sys.path 2 : /usr/lib/python3.10
sys.path 3 : /usr/lib/python3.10/lib-dynload
sys.path 4 : /usr/lib/python3.10/site-packages

In one of my virtual environments, the program output is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ python ~/Desktop/tmp/main.py
base_prefix: /home/leafee98/.pyenv/versions/3.10.9
base_exec_prefix: /home/leafee98/.pyenv/versions/3.10.9
prefix: /home/leafee98/.pyenv/versions/diffusion-webui
exec_prefix: /home/leafee98/.pyenv/versions/diffusion-webui
sys.path 0 : /home/leafee98/Desktop/tmp
sys.path 1 : /home/leafee98/.pyenv/versions/3.10.9/lib/python310.zip
sys.path 2 : /home/leafee98/.pyenv/versions/3.10.9/lib/python3.10
sys.path 3 : /home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/lib-dynload
sys.path 4 : /home/leafee98/.pyenv/versions/diffusion-webui/lib/python3.10/site-packages

In the system global environment, prefix and exec_prefix are controlled by setting the environment variable PYTHONHOME, and the output looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ PYTHONHOME=/home/leafee98/.pyenv/versions/3.10.9:/usr python ~/Desktop/tmp/main.py
base_prefix: /home/leafee98/.pyenv/versions/3.10.9
base_exec_prefix: /usr
prefix: /home/leafee98/.pyenv/versions/3.10.9
exec_prefix: /usr
sys.path 0 : /home/leafee98/Desktop/tmp
sys.path 1 : /home/leafee98/.pyenv/versions/3.10.9/lib/python310.zip
sys.path 2 : /home/leafee98/.pyenv/versions/3.10.9/lib/python3.10
sys.path 3 : /usr/lib/python3.10/lib-dynload
sys.path 4 : /home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/site-packages
sys.path 5 : /usr/lib/python3.10/site-packages

From the above example, you can see that the first element of sys.path is where the script is located. From the third example, you can see that the path to the standard Python module is set with prefix and the extension module is set with exec_prefix, and finally combining prefix and exec_prefix with lib/pythonX.Y/site-packages gives you two paths and These two paths all exist, so they are all added to sys.path.

Workflow under pyenv

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ python ~/Desktop/tmp/main.py
base_prefix: /home/leafee98/.pyenv/versions/3.10.9
base_exec_prefix: /home/leafee98/.pyenv/versions/3.10.9
prefix: /home/leafee98/.pyenv/versions/diffusion-webui
exec_prefix: /home/leafee98/.pyenv/versions/diffusion-webui
sys.path 0 : /home/leafee98/Desktop/tmp
sys.path 1 : /home/leafee98/.pyenv/versions/3.10.9/lib/python310.zip
sys.path 2 : /home/leafee98/.pyenv/versions/3.10.9/lib/python3.10
sys.path 3 : /home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/lib-dynload
sys.path 4 : /home/leafee98/.pyenv/versions/diffusion-webui/lib/python3.10/site-packages

In the above output, the tool used by the virtual environment is pyenv, so this section will describe the source of each value in sys.path in this output.

The 0 in sys.path is the location where the script is located.

In a virtual environment, pyenv adds /home/leafee98/.pyenv/plugins/pyenv-virtualenv/shims and /home/leafee98/.pyenv/shims to the PATH, so that when python is typed out, what is actually run is A script like the following.

1
2
3
4
5
6
7
8
9

$ cat /home/leafee98/.pyenv/shims/python
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x

program="${0##*/}"

export PYENV_ROOT="/home/leafee98/.pyenv"
exec "/usr/share/pyenv/libexec/pyenv" exec "$program" "$@"

Not wanting to analyze the logic of the script run, you can see the debugging information output by pyenv using PYENV_DEBUG=1 python3 (some of the output is below), where you can learn that the path to the last run of binary Python is /home/leafee98/.pyenv/versions/diffusion- webui/bin/python3, which brings us back to the process of initializing Python sys.path above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

$ PYENV_DEBUG=1 python ~/Desktop/tmp/main.py
+ program=python
+ export PYENV_ROOT=/home/leafee98/.pyenv
+ PYENV_ROOT=/home/leafee98/.pyenv
+ exec /usr/share/pyenv/libexec/pyenv exec python /home/leafee98/Desktop/tmp/main.py
+(/usr/share/pyenv/libexec/pyenv:23): enable -f /usr/share/pyenv/libexec/../libexec/pyenv-realpath.dylib
......
+(/usr/share/pyenv/libexec/pyenv-exec:46): PATH=/home/leafee98/.pyenv/versions/diffusion-webui/bin:/usr/share/pyenv/libexec:/home/leafee98/.pyenv/plugins/pyenv-virtualenv/bin:/usr/share/pyenv/plugins/python-build/bin:/home/leafee98/.pyenv/plugins/pyenv-virtualenv/shims:/home/leafee98/.pyenv/shims:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/leafee98/.local/bin:/home/leafee98/.yarn/bin:/home/leafee98/go/bin
+(/usr/share/pyenv/libexec/pyenv-exec:48): exec /home/leafee98/.pyenv/versions/diffusion-webui/bin/python /home/leafee98/Desktop/tmp/main.py
......

First of all /home/leafee98/.pyenv/versions/diffusion-webui/bin/python3 has pyvenv.cfg at a higher level, and its content is as follows.

1
2
3
4

$ cat /home/leafee98/.pyenv/versions/diffusion-webui/bin/../pyvenv.cfg
home = /home/leafee98/.pyenv/versions/3.10.9/bin
include-system-site-packages = false
version = 3.10.9

So following the configuration of that virtual environment, /home/leafee98/.pyenv/versions/3.10.9/bin is used as home to start looking for the landmark files lib/python310.zip and lib/python3.10/os.py to determine the prefix.

1
2
3
4
5
6
7
8

$ echo_if_exist() { [[ -e "$1" ]] && echo "$1 exist" ; }
$ home=/home/leafee98/.pyenv/versions/3.10.9/bin
$ echo_if_exist ${home}/lib/python3.10.zip
$ echo_if_exist ${home}/lib/python3.10/os.py
$ home=$(dirname $home)
$ echo_if_exist ${home}/lib/python3.10.zip
$ echo_if_exist ${home}/lib/python3.10/os.py
/home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/os.py exist

/home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/os.py exists, so prefix is determined to be home/leafee98/.pyenv/versions/3.10.9.

Start over at home and look for the landmark file lib/python3.10/lib-dynload to determine exec_prefix.

1
2
3
4
5

$ home=/home/leafee98/.pyenv/versions/3.10.9/bin
$ echo_if_exist ${home}/lib/python3.10/lib-dynload
$ home=$(dirname $home)
$ echo_if_exist ${home}/lib/python3.10/lib-dynload
/home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/lib-dynload exist

/home/leafee98/.pyenv/versions/3.10.9/lib/python3.10/lib-dynload exists, so exec_prefix is determined to be /home/leafee98/.pyenv/versions/3.10.9.

So add the paths of the standard and extended modules to sys.path according to prefix and exec_prefix, i.e. 1, 2, 3 in sys.path in the output of the virtual environment.

Then there is the site module to add site-packages to sys.path, but in the virtual environment, prefix and exec_prefix will point to the path of the virtual environment instead of the path found by the landmark file:

When a Python interpreter is running from a virtual environment, sys.prefix and sys.exec_prefix point to the directories of the virtual environment, whereas sys.base_prefix and sys.base_exec_prefix point to those of the base Python used to create the environment. It is sufficient to check sys.prefix == sys.base_prefix to determine if the current interpreter is running from a virtual environment.

Source

Here it points to /home/leafee98/.pyenv/versions/diffusion-webui, so it splices in lib/python3.10/site-packages and removes the non-existent paths and duplicate paths to get /home/leafee98/. pyenv/versions/diffusion-webui/lib/python3.10/site-packages, which is the 5 in sys.path in the output of the virtual environment.

Ref

  1. https://github.com/pyenv/pyenv
  2. https://github.com/pyenv/pyenv-virtualenv
  3. https://docs.python.org/3/library/sys_path_init.html#sys-path-init
  4. https://stackoverflow.com/questions/34822593/why-does-sys-path-have-c-windows-system-python34-zip/49293544#49293544
  5. https://docs.python.org/3/library/site.html#module-site
  6. https://docs.python.org/3/library/venv.html#how-venvs-work