Augmented Reality of Molecules – episode 1

What is Augmented Reality?

Augmented Reality (AR) is an exciting technology that overlays virtual elements on top of the real world, allowing for a unique and immersive user experience. The Apple AR platform only uses the web browser “Safari” and does not require any plug-in or specialized software, only that portable electronics (iPhone, iPad) have at a minimum iOS12.

On March 16, 2023 I started to be interested in this technology to illustrate molecules.

File Format

One of the key components of creating AR experiences for the Apple AR platform is the creation of USDZ files. For molecules I tried using PyMOL as well as ChimeraX. As it turns out, the PyMOL-derived files either “bombed” or never provided satisfactory results (e.g. loosing color) but I found later that they are suitable for the “other” platform, Android. A separate episode will be dedicated to that.

The file format for AR viewing is called USDZ. It is a non-compressed ZIP file containing a “Universal Scene Description” created by Pixar (later purchased by Apple.)

The Apple’s command line tool usdz_converter is used to convert files. As it happens, not having Admin privileges on my Mac, I was stuck. After lots of search and frustration, I was able to finally create a Docker container that contains the necessary commands, which are in fact Python scripts. This container is online jysgro/usdzconvert and the README file has some details on its creation. Since the container has an “entrypoint” I used it with or without the entry point, for example:

Using the software within directly:

docker run -it  --rm --mount 'type=bind,source=/Users/jsgro/Desktop/,target=/mnt/models,ro' -w /mnt/models  jsgro/usdzconvert:0.66-usd-22.05b ./2biw2.vrml.wrl.obj ./2biw.usdz -iOS12 -diffuseColor 1,1,1

Or using it with the “entrypoint” option to “dive” into the container

docker run -it  --rm --mount 'type=bind,source=/Users/jsgro/Desktop/,target=/mnt/models' -w /mnt/models  --entrypoint "/bin/bash"  jsgro/usdzconvert:0.66-usd-22.05b

At this early stage I used the multi-step information provided in the Supplemental material of the paper  by Babak Sanii, 2020. (Supplemental: PDF – Archived.) This required to start with PyMOL to save a WRL file (invented in the 1990’s) and then convert it to an OBJ file by the free program MeshLab.

Finally, right after midnight, I was able to “float” this cartoon structure (without colors yet) in my living room:

The next steps were to figure out how to keep the color…

But that will be for another episode.

Quick Look

This Apple page provides 3D models to play with: https://developer.apple.com/augmented-reality/quick-look/

Note that the files can contain animation… e.g. the Cosmonaut.

REFERENCES