Malleable World

A new city has emerged. This may be a bit of an obtuse thing to say, but it is true. It is a city of portal, of data, of wireless and network; it is both a fulfilment of decades of promise and deeply problematic. Sometimes Utopic and Dystopic elements arise in tandem in periods of great transition and flux.

This may be the convergence of the many threads of augmentation, space, communication, human need for expression and the rising digi space; Utopic and Dystopic constructions and dilemmas must fall away and in their place arise a paradigm shift and cohesion. . The city now for many is a city of one.

We must also stop and consider where our current paradigms have progressed from and where they may be going. We must also stop and think about the pitfalls and dangers we are currently facing in our age of cartography, augmentation and smart phones. We need to look closely at trends and events of the present, to consider near future possibilities, and to look back.


There is an ongoing tradition, a whole sort of hieroglyphic, textual system of markings: where to find food, where dangers are, etc. Historically, markings have long been left for other travelers along trails, train paths, and in other spaces. There also are augmentations of experience and woe such as the poems carved into the walls of the Angel Island immigration detention center which paint vivid portraits of pain, confusion, anger and sadness of many individuals stuck there between 1910 and 1940. These two examples both show a progression and thread in time and a dichotomy. Augmentation historically been used pragmatically and expressively, often in tandem. Creativity and deep necessity have worked in tandem.

hieroglyphics immigration detention center

The desire and need to augment spaces is as old as cities themselves and beyond. The cave paintings of Lascaux and other places were pictographic annotations as well as stylized data recorded in a space.

cave paintings

Some historians see some such works as accounting methods of hunting and others more of a pictorial narrative of a people. Is this artistic innovation and aesthetics working for a needed documentation? Yes. Could it be a scene witnessed and held in a space as a reminder? Sure. Is this series of images and inherent symbolism something that not only now gives deeper information into these space and the people, patterns and life there? Yes. Cathartic expression? Measure ?

The ruins of many ancient civilizations such as ancient Rome and Greece have included graffiti tags of groups of youth as well as dissent. The thing about letterforms is that they alone are sculptures, abstractions, pure form and color, yet they hold voice within. It is almost like a room full of dull jars that when lined together phonetically spoke. This is true of all written and spoken language, it is pictographic, it is sculptural, and it is systematized for functions ranged from clipped, practical; deeply complex and urgent amongst chaos, pain or crisis. Text is still quite an esoteric concept when taken out of day to day context and usage. We also have an ability to convey and read symbolism with deep acuity in images: singular or in sequence. Road signs illustrate the deep and base power of metaphor within image or image and text or numerical measure. Some even argue that in sleep we defrag our daily data dump of recall, stress, incompletion, sensory stimuli, tasks completed or emergent through a sieve of metaphor: r.e.m state and dreams. So, we process and communicate through typographic and pictorial symbolism and readings and we have long used them to augment spaces as well as to guide though them and raise commentary about them and people and events therein; so we see an emergent thread of augmentation integrated into interactions in spaces.

The desire to augment spaces often in the past has simply arisen from a need to survive and place warnings for others.

hobo code

The markings placed on the rails that formulated the hobo alphabet (also referred to as “hobo code” or hobo marks”) included notes/signs/symbols of danger, shelter, food and commentary that are a direct lineage to geotagging, and AR. Iconography quite similar to a system of hieroglyphs was marked by others near railroads as an added layer of information and icons as indicators of both options and hazards to consider before moving on from that place. It also clearly was a form of graffiti in the sense of being abstracted typographic forms as well as shaped with acknowledged and systemized symbolism/portent as well as being created at a grassroots level as communication and survival. Much like the world of graf art, hobo code to has come to be appreciated for its form and stylistic emergence from bare necessity. The pictorial;/typographic lineage back to cave paintings also arguably can be tied to the origins of written language itself as a system of augmentation and forms connoting ideas and components of meaning arising out of necessity and survival.


These notations were symbols that created an important dialogue and use of iconography and its inherent metaphorical portent borne out of necessity especially in the time when those riding the rails was at its highest during the great depression when it is estimated that as many as 1 to 2 million in America took this chance and needed these augmentations along the way.

We can take this even further back.


A practice used by some when riding by carriage at night on nights with moonlight was to pour coloured sand or rocks behind to form a linear coloured mapping along the trail to hopefully not get lost. This evocative practice allowed a form of mapping to align along one’s path and on the ground. Before streetlights became common, it was quite difficult and dangerous to travel at night. Augmentation was for some the path where there then were no roads.

The roots of AR can also be taken back another thread in history and time. Morton Helig unfortunately is another classic archetype in the history of technology and innovation; he did not get his due and his ideas were not fully appreciated in their time and are not as well known now as deserved. In his 1955 paper, “The Cinema of the Future” and “experience theatre, Helig laid out his innovative ideas of theatre as immersive, multi media and mixed reality in space.

proto ar sensorama  Morton Helig

His invention in 1960/61 of Sensorama is referenced by some as a precedent to V.R, which is true, but also limiting. Sitting in a small partially open booth watching films with sensory augmentations (smells, air blasts, movement matching those in the film) also predates mixed reality and AR. The work left the person still aware of being in two places at once and mixed the physical and the immersive/virtual (seeing a film in a semi goggle section of the booth). The fact that the funding failed to materialize for this brilliant work both in theory and in prototype is tragic, but unfortunately far from isolated. The desire for fusing experience and data moves beyond the sitting VR model to the desire for mixed reality, for awareness in the physical world and movement and moving though another experiental form and/or datascape. This leads back strongly to what he was interested in. This work now also sits at that precipice of the static body in the physical while moving through an augmented other; the current dystopic thread we must address.

Myron Krueger in 1974 developed a space of VR and AR called “Videoplace” that allowed users in different rooms to interact with one another as digital “shadows”.

Videoplace Videoplace diagramme

The use of video cameras, projections and interactions of mixed spaces and forms was a key precedent to the feedback system of head mounted displays (HDM) and mixed reality. Now we see the active person in a space with an interactive data scape correlated to them, their actions and the space, this experimental work is a key point in the progression and again illustrates the power of a dual awareness, data space, interactivity, physical/digital interface and active movement and spatial awareness.

arrows up

At this point, the line splits…

arrows down

The first line drawn in time leads to works such as Kit Galloway and Sherrie Rabinowitz’s “Hole in Space”.

hole in space

This work connected people and spaces in two cities simultaneously as well as connecting the physical space and an image augmenting a connected space. The work saw people in New York and Los Angeles “seeing” not themselves reflected in window glass ( Lincoln Center for Performing Arts in N.Y and a Broadway store in L.A), but an image of the crowd in the other space. This 1980 work was part of an area of works that explored spaces, image, expectation, telepresence and the real in relation to the virtual. Other works did similar tech savvy interruptions and collisions of the semiotic of spaces such as elevators, hallways, storefronts, windows, mirrors etc… This crucially also led to working with spaces, information and adding (or in some cases actually subtracting) by augmentation. The wonder at the time was of the “magic mirror” of perception, space and simultaneity as well as of connectivity; a pragmatic space and its understood base function had through technology been transformed and interactions in two physical spaces (through the interface and digital) were linked. This also is not of passivity, of the removal from the physical into the digital, nor of intrusiveness.

Let’s take the other end of this splitting line a bit further back. The mid 1950’s saw the emergence of System Dynamics.

system dianamics

This key development looked at feedback loops, data, function and processes over time. Jay Forrester helped develop key core concepts while at M.I.T starting in 1956. The input/output model of data, interaction and process theory can be traced back to the late 1920’s and earlier, but System Dynamics moves this into more complex aspects of data, feedback loops and analysis of multi-faceted information and processes. In the early 70’s this moved also into global socioeconomics and later to many disparate areas of data analysis, study and visualization needs in process, thus moving from more management focus to global feedback loops and modeling. This is extremely important to the development of AR as both it led work into feedback loops and their visualizations and a new way of “reading” data and processes beyond static and setting the stage for later works with ecological data, historical data etc and locations in both Locative Media and Augmented Reality. Also, the feedback loop is the basic line of functionality of geo-spatial augmentation (map based AR) as well as head gear and cam based AR seeing the visual field and looping back spatial placement of graphical objects and information. This also leads to the growing field of screen based AR. Also we see data presented in flow, connectivity, flux, problem solving and tensions beyond pie charts etc. The nature of data is not static. It only appears so when clipped, isolated. It also needs to be analyzed from multiple valences, points of entry, contexts and in the sense of dual spaces be it literally so or in the case of the image above, in contexts, and between the hypothetical (another data space even if in the human mind and our calculations and speculative pattern formations and analysis) and the present physical location.

Buckminster Fuller is best known for his work with Geodesic domes, related mathematics and the Dymaxion Car and Dymaxion House, but he laid a key notion in place for spatial or geo-spatial AR. In 1962 Bucky proposed the Geoscope. This idea is so revolutionary that teams are working on its concepts and possible permutations now. He had the incredible foresight to understand the need for not just real time data visualization but also geo spatial augmentation, global connectivity, malleable visualization and information design as a mode for critical problem solving. Fuller had already developed the dymaxion house and car as well as his most well known work, that of geodesic domes. He also had a profound understanding of elemental forms (triangles), tensile strength, the need for adaptable architecture shown in geodesic domes and a desire to find “simple” solutions for complex problems in the world.

Concept drawing for Geoscope
Concept drawing for Geoscope
Geoscope conceptual test model
Geoscope conceptual test model

This was something he had been thinking of for many years. His proposal essentially was of a huge functional globe with what we today would call clustered servers or even hive computing running augmentations live on the surface of this globe. The notion was that it was a way to see interconnected data and information from local to global on a geo-spatial platform.

We are seeing Fuller’s moment. We are seeing the emergence of multi layered live and stored data laid across maps, we are seeing more and more apps that allow greater augmentation. This is arguably the greatest period of cartographic awareness in human history. The move toward geo spatial global connectivity is exploding. The speed of signal and data access is layering spaces and allowing geo spatial searches as well as smarter information mapping. This also is the time of growing pains that comes with periods like this; the gaping dystopic maw of “the city of one” of people losing the physical for the data space is screaming the need for radical new considerations in city planning, architecture, data architectures, augmentation in relation to physical world distraction of passivity. The same paradigm that ignited locative media as a move beyond the passivity of virtual reality ten years ago is upon us. Now it is more urgent and more multi faceted as things push quickly.

Spatialized AR is now exploding as so many people have mapping software on their cell phones and companies are seeing the huge potential of what many google maps hacks have long been doing as well as works of Locative Media, Experimental Mapping and spatially aware AR. The move now is into multiple layers of options on these maps, of interactive and even open source functionality (build layers of augmentation and upload as others do to build more data and also communities of those interested in this).

What has exploded in the last year is the middle ground between AR that requires HMD which has long been the cutting edge of AR and the digital lines shown behind hockey pucks. One could compare it to a goalpost forming as two poles stand apart from each other. The screen based AR does not need the extreme speed and memory of head mounted displays as the feedback loop is far less complex . It also takes the desire to tag spaces and moves it to representations of spaces as well as of spaces as trigger points on devices (GPS). The commercial explosion is exciting, but also a bit troubling as the hype centred massively initially on the danger zone of all new cutting edge technology, especially as it enters quickly into the culture at large, the gadget factor of dancing hallmark card characters held up to computer cameras in the face of the hundreds of other deeper elements and applications already existing and being developed. This myopic sense can stigmatize and worse, freeze a long progression and deeper possibilities into a false singularity, a “oh, that thing” sense of one aspect alone.

The upside is that what has so many parts in decades of cutting edge research may indeed be finding its time. AR in all forms is now being recognized and from HMD’s becoming lighter and lighter and faster, screen based apps moving into greater layering of functionality and processing sophistication to the spectre arising of AI elements of relative intuition coming into the fore (smart searches, learning curves with use).

gps utopia

Texting and driving is now in many areas the number one cause of accidents. Reports are emerging of an increasing data withdrawal disorder for lack of a better word being compared to going cold turkey from drugs. Studies are evaluating rising issues of continual data engagement affecting socialization skills as well as modes of communication. Much of this is of course likely overstated, but there is a root core that must be considered especially in light of the deep and logical progression of augmentation and data visualization that has led to this point from those cave paintings, hieroglyphs, hobo marks etc.

We also must reconsider how we move in cities. We must reconsider how we build cities. We must reconsider how data is experienced. We need advanced AR now. We need data aware architecture and city planning on a scale and scope never seen before. We need these things now. The screen is, for many now, a primary space. Once at a panel at Cal Arts the philosopher Jalal Toufic interrupted the discussion on photography, space, collective and singular memory and spaces to point out the camera whirring beside him, shooting video. He shouted to turn the camera off immediately. Heads turned and many were taken aback by this obtuse turn and utterance. He then pointed out that the one video (this was 1996) the sculptural slab of plastic in that machine was already deemed to be the artifact, the recall of this event. This meant that it superseded all else, held precedence as the actual moment superseded this recording of it, this arguably secondary data.

The comments, questions, discussions, physical space and people in that moment were functioning for the tape. His logic then was not so obtuse. The link could be made to the classic concept of a man with a camera filming an animal in the wild and the animal being on some level aware of being filmed and thus making these moments veined in an unreality, a change in awareness and action. Also in time the video would be the one thing (along with its reproductions and any secondary quotations or documentation of it from there on out) of these moments in its tangibility and the semantic/semiotic sense of authority so often given to objects, artifacts, “evidence” [not quite sure what you mean].

Reports are emerging of an increasing number of events where a tragedy unfolds in a public space and no one does anything to help, but many shoot cell phone video. This includes auto deaths and celebrities collapsing on stage doing lectures. Is this not the point Toufic was making? If the recording holds precedence, then the event (and its participants, willing or not) are deemed secondary.

This is only a portion of the people of the world but it is troubling. Our sense of space never has stayed static. Our modes of communication and expression have progressed over time. . The time is for deeper AR. The time is to (re)consider how we move in cities in an increasing simultaneity of digital space and physical space at once. The time is for apps to use varied forms of media, for projection technology to facilitate a new second space between HDM/Goggle AR and screen based with such things as partial projections in the field of vision or onto surfaces not just of screen. The time is to see more sophisticated AR apps on screen that allow great multi tasking live on screen. This may also be the moment of Architecture utilizing physical and data as a fused architecture that those art works like “hole in space” in the late 70’s and 80’s touched upon in their avant garde explorations. This is the time of the digital skinned city, of the data city and physical in tandem, but must it be a city of one at the cost of digital distraction into a singularity at times dangerously unaware of the physical world that surrounds us?


Jeremy Hight is a locative media/new media artist and a writer. He is credited with inventing locative spatial narrative in the first locative narrative project 34 north 118 west. His essay Narrative Archaeology was recently named one of the 4 primary texts in locative media. A retrospective look at his work and a look at "reading" the landscape is in volume 14 issue 08 of Leonardo. He has published over 20 theoretical essays and exhibited work in festivals and museums internationally.