Loss of Sonnet 18

This project is an explorative digital art installation that delves into the phenomenon of generation loss in digital signal processing: the inevitable degradation of information quality through copying and propagation. It transcends the technical realm to probe how this concept applies to the transmission and transformation of words and meanings through time and technology.

It consists of a series of videos illustrating the loss of words themselves and their meaning, combined with real-time interpretation from both humans and AI. The project allows the audience to find a border of loss, beyond which the information of words might change.

At the heart of the piece lies an acknowledgment of the inherent loss in information storage and spreading, echoing through the corridors of human history and our methods of preserving and transmitting words. Although there is loss in every transfer, the thought behind the stored information hardly changes.

Drawing from David Elliott’s JPEG compression for 600 times [1], Alvin Lucier’s I’m Sitting in a Room [2], and Dietmar Offenhuber’s Autographic Visualization [3], the work focuses on the generation loss of text through the image medium. The last two lines of Shakespeare’s Sonnet 18, “So long as men can breathe or eyes can see, / So long lives this, and this gives life to thee,” are the main text. The sonnet promises immortality through verse; the piece asks what happens when the medium of that immortality, the words themselves, undergoes generational loss.

Through a blend of traditional poetry and modern technology, we simulate the blurring and compression of text, like the distortion experienced in social media and digital communication. In a nod to the pre-digital era, the text is presented on images, recalling ink on paper, prints, and etchings on stone, all of which can be read as images.

The audience becomes witness to a slow metamorphosis of language. Familiar words erode into ambiguity and reinterpretation, and each viewer looks for a border of recognizable loss within the process.

The major medium is a series of videos that record the loss of words being stored and spread. Each video is a sequence of 999 images arranged by compression iteration. In every iteration, programmed noise and blur degrade the image; the output of one iteration becomes the input for the next, so each frame aggregates all preceding distortion.

The background sound has two layers: the Morse code of the two lines, an echo of machine signals set against the echo of human thought, and ambient music chosen to match the idea of the piece. As loss deepens, the sound also becomes distorted and noisy, producing a synesthesia of sound and picture.

Images are processed in Python with Pillow using low-quality JPEG compression and blur. Optical character recognition runs on Microsoft Azure [4], and the video presentation is rendered as an HTML5 application.

The installation features two screens and a web camera. The primary screen shows the iteration of text loss; the secondary screen presents OCR results from AI, including word bounding boxes, to simulate a new understanding of the degraded text. Bounding-box opacity encodes the model’s confidence, a visual metaphor for the clarity of machine understanding.

The audience also becomes a recognition agent. With no gesture detected, the loss plays automatically. When a hand wave is recognized, the earliest hand takes control and scrubs the loss to a specific moment, so viewers can find the frame after which they can no longer read the text and compare it with what the AI recognized. The number of people present changes the playback speed: the more interpretations, the more information changes in transfer, and the more loss. The version exhibited at SIGGRAPH 2024 had no interaction module.

The Ship of Theseus asks whether an object is still the same after all of its parts have been replaced one by one. Here the information in the original poem is the ship, and each interpretation replaces one part. After many rounds of communication, propagation, and interpretation, the same question appears: is the information still the original one, and if not, when did it change? Everyone has their own answer, and the machine has its own as well.

Taking the term apart, generation loss is generation and loss. During communication, information is not only lost but also generated, and both compose what is current. Loss is ongoing rather than a single event: it leaves traces of the past and opens new understandings.