Tongfamily (Richtong.com Sync)

Ok, I don’t this is well documented and it has been causing me headaches for months, but here it is:

1. If you are using a virtual environment, a very common one is Anaconda or miniconda. This let’s you install python packages on your machine and switch between environments.
2. Another package like this is pipenv which does much the same thing.
3. The problem is that there are have been many times when I get the wrong package and I can’t figure out why in Anaconda.

The answer is pretty subtle but in short:

1. Anaconda is a virtual environment except that it enables USER_SITE. This means that you can override the environment and this is set as ENABLE_USER_SITE=true. The problem is that if you have a ~/.local/lib then you will silently override the conda environment with those packages.
2. Now this is not normally a problem, because the only way to do this is to run pip install --user in your bare metal environment. So first of all, don’t do this!
3. But the problem is that a VIM package, I’m forgetting the name, maybe Black creates a virtual python environment and it installs everything into that .local directory. As a result, you will pick old packages there.
4. This creates all kinds of strangeness, the net is that if you want to figure out what packages are loading run python -m site which is a module that manages all this complexity

The solution

Well the hack solution is just to delete the ~/.local/lib but this does mean no local or user installed packages ever.

The real solution to this thought is to completely seal up Anaconda. You can do this permanently with a configuration file pyvenv.cfg and this works because there are some magic here about environment configuration. It is complex, but it is hard to know where to put this file. The documentation that its goes into the parent of one directory sys.executable, sys.prefix and sys.exec_prefix.

For Anaconda with a homebrew installation, this appears to be /usr/local/Caskroom/miniconda so put it there this pyvenv.cfg and the line to disable it is but this didn’t seem to work even adding:

include-system-site-packages=false

This by the way is an open bug is Anaconda which is how to isolate from the so called USER_SITE which is normally ~/.local

The fixes appear to be to run python with a -s flag or to set an environment variable PYTHONNOUSERSITE to false and you can check this by looking at site.ENABLE_USER_SITE in python for running python -m site from the command line.

The hacks to get around this are pretty ugly:

1. Don’t allow any user packages (ugh!)
2. Whenever you start conda, you have to remember to set that environment variable.
3. After you start Anaconda, clone a virtualenv in a special way, in short, you create python venv and then use conda create --clone of that environment since virtualenv shuts this off. In this way it works much like pipenv which is sensitive to the directory it is created in. However this creates a complete copy so suddenly in your git repo you will have a ./bin and a ./lib and have to manage all those files
4. Reach deep into the guts of Anaconda at /usr/local/Caskroom/miniconda/base which is hard to capture when you do a new installation and defeats the purpose of using Home-brew so you can create a script that does this in /usr/local/Caskroom/miniconda/base/etc/conda/activate.d
5. Reach deep into the python installation and find site.py which you can get run running import site; site.__file__ and hand editing the file to set ENABLE_USER_SITE = False

As a really strange aside, this ~/.local problem does *not* happen with Home-brew Python. That’s because the USER_SITE there is set to a home-brew directory and not ~/.local that is only true with Anaconda

So net, net, an obscure bug, but widely discussed since some users are relying of this silent import of packages. Personally, I think all packages should be completely explicit as pipenv does, but glad they have that environment variable.