Synesthesia

2023, Art Tech Installation, Hong Kong

Exploring the Intersection of Senses with ArtTech Installation, Synesthesia

fig 1. A 3D rendering of Synesthesia's main component, where multicolored ink pod valves introduce a new look to the sculptural installation over time.

COURSE

SD3555 Digital & Interactive Spaces

TEAM

Lily So - 3D Art
Grace Moon - Creative Coding, Fabrication

deliverables

Functioning installation artefact, Process Book

Table of Contents

1_The Idea
3_The Build
4_The Installation

Building a collective (synth) perception (esthenia) of the world around us

Synesthesia explores the age-old enigma of how humanity’s collective perception indirectly and directly shape the natural world. We currently live in a society where two dichotomous ways of interacting with the planet exists: one where pop science experiments like identical plants showing drastically different growth results based on whether the owner uses kind words or harsh words to communicate makes viral headlines, and where man-made waste concerns are brushed under the rug to undermine its true severity. How can we bring conversation about our seeming need to selectively hear and see our universal impact?

Inspired the Art Nouveau movement and the unique approach of our human interpretation of—and interaction with—nature, we created Synesthesia, about a physical, interactive installation to spark conversations about our cyclical imprint and impact on the environment around us.

fig 2. Functioning fabrication of Synesthesia

The Challenge

Translating an abstract concept like "collective perceptions" into a tangible experience can be complex. The key challenges were making the installation intuitive to interact with (while conveying an obvious surface-level message); incorporating a social commentary element within; and considering the limited space in Hong Kong, where the installation would be showcased. We sought to address these concerns in hopes to create an engaging and thought-provoking experience, effectively communicating the intended question to the audience.

The Process

With a tight three-week timeline from conceptualization to final prototype, we optimized our design and fabrication workflow by working on multiple aspects simultaneously, rather than following a linear process. Lily focused on preparing the 3D models for printing and cutting, while I handled the computational side by coding the installation's sensors and actuators—all to be brought together a week before installation.

The Outcome

Synesthesia is a custom-made machine that features three or more valve-controlled ink droppers that operate based on connected movement sensors. Each sensor monitors a specific zone and determines the duration for which its corresponding valve will remain open.
Its bottom vessel functions as a stamp, allowing installation viewers to imprint a unique pattern on their paper souvenirs—one that is created only once by the installation.

Communal Perceptions & Memories

fig 3-4. (from left to right): The observational study that inspired the keyword of Synesthesia (propagation), a visualizer overlay of reaction-diffusion patterns.

During a winter trip to Seoul, I had the opportunity to spend my final Christmas with both of my grandmothers (할머니 or halmuni) from each side of the family. In retrospect, and following the recent loss of my maternal grandmother, I dedicate this installation to her memory.

On a cold, snowy day, three generations of ladies—my two halmunis, umma, and I—ventured out to the national botanical garden. Our goal was to enjoy the fresh air outside the polluted city center and to immerse ourselves in the greenery after a long stretch of grey winter.

As we strolled through the large greenhouse, I opened my trusty iPhone 11's camera to record the conversations and interactions between my loved ones and the beautiful plants surrounding us (fig 3.). Upon reflection, I noticed a recurring pattern of shared and individual memory-making. When diverted to different areas of the grenehouse, we created individual memories of interacting with the plants. In contrast, when two or more people came together to discuss and converse, shared memories of the space were formed. This observation later served as inspiration for Synesthesia.

As we increasingly capture memories made in physical spaces using digital tools like cameras, I began to question: is there a way to explore the reverse process, where digital objects create new memories? To merge the digital dimension as a mediator in the built environment, we sought to build an evolving installation that computes our interaction with the installation space to generate a novel visual experience—an experience that can then be taken home as a physical memento.

Building Synesthesia

fig 5-8. Synesthesia's 3D model progression. From left to right: initial idea sketch, exploration of main component mechanisms, ink pod mechanism blowout, final model with all components present.

As the installation's starting point "spark" was an observation made at a botanical garden, we sought to pay homage to some of the plants that were present in the greenhouse. That is how we came to the conclusion of building a ever-changing sculptural piece that lets viewers "propagate" a the experience to take home via a unique imprint on a pamphlet.
This custom-made machine features three valve-controlled ink droppers that operate based on connected movement sensors. Each sensor monitors a specific zone and determines the duration for which its corresponding valve will remain open.
For example, if Zone A is linked to the blue valve, the duration of the valve's opening depends on how long one or more individuals are present in that area.

As the various colors of ink droplets travel down to the bottom half of the installation, they converge in a vessel that contains a gel to suspend the inks. This gel creates a slow trickle of fluid through the perforated bottom of the vessel.

The bottom functions as a stamp, allowing installation viewers to imprint a unique pattern on their paper souvenirs—one that is created only once by the Synesthesia installation. This version breaks down the information into clearer segments, making it easier to understand the flow of interaction within the installation.

fig 9-14. The fabrication process of building the Synesthesia installation (captions available upon clicking image)

From Code to Construct

Our work included prototyping and experimenting with the nature and limits of our university's material resource lab through 3D-printing and laser cutting most of our bespoke hardware pieces.

A few modifications to the digital 3D model in the fabrication process are as follows: 1) we planned to use a single thermal camera to detect the density of the installation's space. However, we quickly pivoted to using multiple distance sensors to simplify the code that controls the opening and closing of the ink pod valves via servo motors. And 2), a few material swap-outs were made to accommodate for successful weight-bearing (e.g., 3D print material and infill density).

fig 15. Anatomy of Synesthesia. Left: 3D Rendering. Right: working physical prototype with all sensor-actuator components.

Visualizing vs. Materializing

The 3D printing process required several iterations to balance structural integrity and aesthetic appeal. To enhance production efficiency, we substituted a few flat components with laser cutting instead of 3D printing, resulting in cost and time savings.The suspension system for the installation's components utilized metal chains, effectively securing the computational parts, ink pods, funnels, and vessel in a streamlined, linear arrangement.

fig 16-27. The completed fabricated prototype and 3D rendering of Synesthesia, captured in action at Tai Kwun (HK), and the Hong Kong Polytechnic University.

Expanding Synesthesia

While we initially envisioned a single installation with multiple vessels, we began to explore the idea of placing individual vessels throughout the city, which could be equally, if not more, meaningful and thought-provoking.

During our final presentation to a panel of guest judges, we proposed color-coding various iconic spots around Hong Kong. In this concept, the visitor movement at Location A would influence the experiences of the vessel at Location B, and vice versa—fostering an interconnected dialogue between people and their environment, with digital objects acting as mediators for conversation and the creation of new memories.

CLOSING THOUGHTS

On Our Ephemeral Imprint

Synesthesia is not reliable or sustainable in that it requires constant monitoring and refilling for it to stay in a space for a prolonged period of time. It's also one that lacks definite rule sas to how one can interact with it. The viewer's physiology, observation of others and their way of interacting, and creative approach determines their interactions. We note the ephemeral nature of the installation alongside its open-ended approach to interacting with passerby, but still stay grounded in our belief that our creative process contributed to the art tech community by inviting a sense of wonder and play to the conversation.