Scanning Rocks

Experiments with volumetric scanning
It was an honor to exhibit my first augmented reality work at Gray Area in San Francisco. As art installations are fleeting, I thought I’d document the piece and share some insights from its creation.

So what is this?

These are three dimensional portraits captured with the depth sensor of an iPhone. As visitors investigate a mantlepiece crowded with star charts, the portraits are revealed in augmented reality.

Beginner’s Mind

The delightful uncertainty and curiosity of learning. It's a reassuring concept for someone with no experience creating in augmented reality. As I stumbled through tutorials and inscrutable documentation, I realized I’m not alone with this uncertainty. Everyone creating in this nascent medium – from artsy tinkerers to massive corporations – is a beginner. There are no models to follow, we’re making it up as we go along. My early sketches were far more complex than the finished piece. Fortunately, a deadline is a powerful muse for clarity and focus. Three ideals emerged as I radically simplified the concept: * Make it seamless: don’t break the illusion with abstract QR codes or wandering 3D models. * Scale the Experience: the piece should intrigue from afar and reward close inspection. * Keep it grounded: unify the physical and digital, don’t use one as a canvas for the other. I didn’t have these lines at the time, but I knew how I wanted the piece to feel. I could ditch animations and interactivity, but couldn’t budge on these basics.

The Beauty of Point Clouds

How to make three dimensional portraits? It’s remarkably easy, then really complex… Our devices are capable of far more than photos and video; they can capture the topology of our faces and the geometry of our surroundings. Products like the iPhone X and Microsoft Kinect are able to model the environment by projecting and analyzing infrared light. The resulting data can be saved as a set of coordinates called a point cloud – the raw material for a 3D model. Visualized, they look to us like delicate constellations; to our devices they're just a big list of numbers. There are several apps and third party sensors that make it easy to scan things this way (I used Capture for iPhone). The journey from point cloud to usable model, however, is a Norse saga of file conversion and application switching. There are dozens of 3D file types, and editing applications often use different formats than AR platforms. My convoluted workflow – sussed though trial, error and coffee – looked like this: When I asked an experienced friend if there were more efficient ways to work, she just smiled knowingly. This is the world of 3D asset creation. An ancillary note on point clouds: these beautiful constellations of data allow for exciting new forms of expression and storytelling. They also enable oppressive regimes as they track dissidents, and massive corporations as they sell our attention.

An Unexpected Collaborator

The most challenging aspect of this project wasn’t technical, it was optical. To function, AR experiences need to be aligned with something in the real world. It can be the plane of a tabletop, a unique object, or an image. This often results in a two step process for users: first scan an anchor, then see something interesting. While exploring ways to render these AR portraits as constellations (point clouds evoke a stellar mindset), I realized a star chart might serve as an anchor and a texture for the models. Merged this way, the act of scanning is integrated into the experience. It becomes (I hoped) more like looking through a magic window, less like checking out at the grocery store. But there’s a catch: as experienced AR creators know, not every image makes for a good anchor. Apple’s development tools use a machine learning algorithm that issues a warning when an image isn’t suitable for tracking. But as the color and texture of the charts would determine the aesthetic of the piece, these warnings felt like creative criticism from an unwelcome collaborator. Heedless of its protests, I tried all the charts. I was lucky. The chart most acceptable to my machine learning partner was also what I liked best. With the installation only days away, the project was coming together. Hubris. Anchors must be individually trackable and distinct from each other. While a star chart may be unique, it’s not sufficiently different from other star charts. Seen through the video feed of a moving iPad, they all look like speckled blue rectangles. I learned this the hard way, as my confused app attempted to place multiple portraits on the same anchor – a terrible experience. You can see my crest fall in this video: This was…deflating. A night before the exhibition and the piece wasn’t just glitchy, it was unusable. But! The muse of deadlines also inspires lateral thinking. With my last sheets of paper (for real) I printed a new set of anchors as distinct as I could make them. Rather than full star charts, I reduced the density of the images and focused on the unique areas of each. They worked, even in the dim lighting of my apartment. Seen side by side, it seems obvious that greater contrast and simpler patterns would make for better anchors. But I couldn't have intuited this at the outset; integrating anchors requires a give and take with the AR platform itself.

What's Next?

This is a proof of concept, not a finished piece. I’d love to develop it into a room scale experience and bring it to life with short animations – a laugh or a smile – rather than static models. I’m also deeply curious how the experience will feel in AR headsets like the Magic Leap and HoloLens. More than anything, this was a profound learning experience. I’m not at all certain of what to explore next, or where these rapidly evolving technologies will lead. Could there be a better time to create? Give me my Romeo; and, when he shall die,
Take him and cut him out in little stars,
And he will make the face of heaven so fine
That all the world will be in love with night
And pay no worship to the garish sun.
– Shakespeare, Romeo and Juliet