Understanding Python installation mess

Summary

Useful ramblings about the difficulties to understand Python installation on macOS.

In this Tutorial-like post we’ll see:

– Where is Python installed, and which version
– Upgrading to a new Python.org version, without replacement of previous version
– Adding an alternate Python: Anaconda
– Could Docker help?

Python mess?

The problem with Python is that it can be installed in multiple versions from multiple sources, and figuring out which Python is in use can be crucial. In an older (Mac) computer I had up to 5 different pythons installed, even with combinations of versions 2 and 3, from various sources, and it was hard to figure that out!

While Python is useful, it is rather difficult as a new user to not get lost with these issues, and that is not yet counting with special ways to create “environments” that allow concurrent installations of Python in various configurations to avoid conflicts.

Python on macOS

My experience in on macOS, and the following is pertinent to understanding the installation of “Python” onto the macOS computer in the Biochemistry classroom.

Is Python installed?

By default Python version 3.x is installed on current macOS. Older versions of macOS used Python 2.7.x. If you have a Windows or Linux system there is no Python installed by default.

On macOS the command to call Python from a terminal is python3 as it allowed at one time to afford versions 2 and 3 to coexist.

But where is it?

On a fresh macOS computer the answer will be given by the command within a Terminal:
which python3 and the answer would be /usr/bin/python3.

Which version?

% python3 --version
Python 3.9.6

OK good… it’s here!

It is important to note that macOS is using Python, and therefore messing with that installation is not recommended!

“Intelligent” Python3 update

Users can install a hand-full of extra software without (for most cases) the intervention of IT Admin by using the”Workspace ONE Intelligent Hub” (in fact not so smart in my opinion.)

Installing Python (from Python.org) is one option. For this software this will not require Admin privilege and will install for all users on that Mac. Once installed the button changes to Reinstall.

So now, the question is: was the previous version removed and replaced by this version?

We can use the Terminal to inquire:

% python3 --version
Python 3.10.4

The version matches what was proposed (see image above.)  But where is it?

% which python3
/usr/local/bin/python3

So, this is the new default if we type python3 on the Terminal. However, the “old” one is still here, as we can see if we now use the full path:

% /usr/bin/python3 --version
Python 3.9.6

In other words, we now have 2 versions installed: 3.9.6 and 3.10.4. By default the new one will be used. How is this possible?

The answer can be found within one of the most important “preference” in a Unix-like system (macOS, Unix, Linux) called PATH that is simply a list of directories containing software. On the classroom Mac this would be see with the command printenv PATH:

/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

Since  /usr/local/bin/ appears first on this long list, and since this is where the new version is installed, the new version will be started when python3 is requested.

Jupyter Notebook, Lab

OK. Great we have Python. But I heard that it is really nice to use Notebooks with a system called Jupyter Notebook, or the newer Jupyter Lab.

Jupyter is an extension to Python and can be installed with the pip (or pip3)  (1) utility to install additional extensions within Python. I installed this based on on-line information:

• How to Install Jupyter Notebook on MacOS? [Archived]
• JupyterLab Read the Doc Installation [Archived]

In summary the commands were:

pip3 install --upgrade pip
pip3 install jupyter
pip install jupyterlab

I also had a warning about the PATH that a casual user might not recognize as important. For each installed component I had the warning:

... is installed in '/Users/jsgro/Library/Python/3.10/bin' which is not on PATH. So, I added this to the PATH as otherwise commands would not be found (see above!)

export PATH=/Users/jsgro/Library/Python/3.10/bin:$PATH

To make this permanent this command would have to be added to ~/.bash_profile or ~/.zsh_profile on the Mac.

Testing installation

The Jupyter Lab version adds an easier navigation to the file system, but is otherwise very similar. Either are started on the command line on a Terminal:

% jupyter lab

which starts a local web browser and automatically opens a web page for the local server as

localhost:8888/lab within which Python3 is available either on a Terminal-like “console” or within a Notebook, that makes use of markdown (see posts “Do yourself a favor: learn Markdown…”) – It may be important to read the information within the Terminal to access the Notebook with “Tokens” as described within.

The local web server running within the Terminal needs to be terminated when finished, this is accomplished with Ctrl C and then aswering the resulting question:

Shutdown this Jupyter server (y/[n])?

Ok, this seems to work.

Enter Anaconda

Since the classroom is meant for multiple users, it may be easier to use a more integrated system as is proposed by Anaconda, which can allegedly be installed without Admin password. Anaconda is yet another Python installation so now we’ll have 3 of them on this Mac!

The steps are:

1. Download Anaconda installation: https://www.anaconda.com/products/distribution (As of this writing it is Python 3.9, installer is 688 Mb)
2. Installer is saved by default in “Downloads” directory
3. Run Installer, this should not require Admin password if keeping option as “install just for me”

At the end, the installer told me that “Installation Failed” but when I tried to install again it told me that it was “already installed” and I should simply use the command:

conda update anaconda

but that resulted in  zsh: command not found: conda   which puzzled me. But, PATH to the rescue! Or at least its understanding (see above.) I also knew from previous experience that this would be installed in the Users directory rather than /usr/local/bin and so I found that I had to give the full path of the command, or the more useful universal shortcut replacing /Users/jsgro with ~.

~/opt/anaconda3/condabin/conda update anaconda

This command will work for all users.

We can also conclude that the installation resides within ~/opt for optional software and all software can be listed with ls ~/opt/anaconda3 and contains an “App” called Anaconda-Navigator.app. The App can be started with the Terminal command:

% open ~/opt/anaconda3/Anaconda-Navigator.app

or can be found by navigating to that folder with the mouse by first openign Macintosh HD and then Users , then your username (jsgro in my case) and finally opt.

JupyterLab and Jupyter Notebooks are icons to be clicked. The ensuing web server is not visible as it was in the Terminal (see above) and will be terminated when quitting (exiting) the Navigator.

Using trick commands from the page “Python System Command” [Archived] I was able to determine that the Python used was located in `/Users/jsgro/opt/anaconda3/bin/`

~/opt/anaconda3/bin/python3.9 and the version: Python 3.9.13

The code was:

import os
cmd = "which python3"
returned_value = os.system(cmd)  # returns the exit code in unix
print('returned value:', returned_value)

It is possible to have multiple version of Python, but being aware of what and where they are can have significant impact on what happens next when we want to add more packages with pip or conda package managers:  To Be Continued!

Docker to the rescue?

All of these problems with installation could be somewhat avoided by using Docker (which the “Intelligent Hub” cannot install properly as Admin password is required to finalize installation.) (Tutorials: Docker – Beginner for Biologists)

When and If Docker is available, we could launch a Jupyter Lab Notebook with the following command that also shares the current directory:

% docker run -p 8888:8888 --name notebook -v "${PWD}":/home/jovyan/work -e JUPYTER_ENABLE_LAB=yes -it jupyter/datascience-notebook

Within the Docker container we would use the Linux version and we could install anything we want. The container would be deleted when done, thus avoiding any conflict with the future.

Hopefully this will be possible soon.