Auditory Augmented Reality for Well-Being

1.1 Definitions: AR, VAR, AAR

In a broad sense, augmented reality (AR) can be defined as a simulated sensory environment seamlessly superposed upon ordinary reality, rather than replacing it entirely, as in virtual reality (VR), such that the user perceives this environmental fusion through sensory channels. While the human senses are more numerous than the Aristotelian five,¹ AR encompasses primarily visual and auditory channels; olfactory, haptic, and gustatory channels are rarely addressed due to technological obstacles. But for the most part, AR combines possibilities offered by visual augmented reality (VAR) and auditory augmented reality (AAR).² AAR and VAR may aim for coordinated, congruent sensory fusion, providing auditory and visual perspectives on a common multisensory simulated environment (e.g., matching sights and sounds of wild animals, a moving vehicle, or a trumpet player), or they may generate independent auditory and visual environments.

Frequently, AR is narrowly equated (sometimes tacitly) with VAR. As Yang, Barde, and Billinghurst state in their review, referencing earlier authors: “…the overwhelming majority of AR research has been focusing on visual augmentation” [6, 9, 10, 11]. Prevailing definitions center on computer technology generating graphical images augmenting natural vision, using optical see-through displays to superimpose virtual visual objects over reality [12]. The following entry in a technical dictionary is typical of the ocularcentric bias:

Modern VAR is predicated upon high-tech devices developed only in recent decades—such as stereographic (binocular) head-mounted or head-up displays (HMDs, HUDs), AR eyeglasses and contact lenses, and virtual retinal displays (VRDs)—to present virtual images to the user [14, 15]. Indeed, though AR dates to Morton Heilig’s 1962 Sensorama, which combined visual, acoustic, and olfactory stimulation [16, 17], and Sutherland’s early HMD [18], the term “augmented reality” was not coined until 1992, with the development of digital HMD technologies [19], which came to be intimately associated with the concept [20]. Textbook treatments of AR generally pay short shrift to sound, limiting it to a support role in the service of virtual images [15] (for instance, both virtual avatars and NPCs may need to speak) despite the fact that AAR is, in many respects, simpler, less expensive, and more emotionally powerful.

Actually, Mavor and Durlach’s foundational 1995 edited volume on virtual environments devotes its entire third chapter to the auditory channel, following prefatory remarks, presciently echoing our own observations on low-tech AAR:

There follows a discussion of how to communicate information to the user through the auditory channel, including (on p. 136) the related but distinct techniques of data audification and sonification (these terms are elaborated below), as well as spatial sound, all within the broader field of auditory display technology [22], while noting that “…relatively little attention has been given to augmented reality in the auditory channel” [21].

However, despite this broad “vision” for AAR early in AR’s history, in practice most subsequent AR research has focused on the visual field, as well as practical informational applications (medical, robotic, etc.), rather than creative or therapeutic ones, at least until the more recent era of AR gaming. Several widely cited overviews of AR exclude the auditory field entirely. Thus, Dey et al., citing Azuma: “Augmented Reality (AR) is a technology field that involves the seamless overlay of computer generated virtual images on the real world, in such a way that the virtual content is aligned with real world objects, and can be viewed and interacted with in real time (Azuma, 1997)” [9], or Zhou et al.: “Augmented Reality (AR) is a technology which allows computer generated virtual imagery to exactly overlay physical objects in real time” [10].

Yet, in his pioneering article, Ronald Azuma himself defined AR broadly as a system with three characteristics: “1) Combines real and virtual; 2) Interactive in real time; 3) Registered in 3-D” ([23], section 1.2). And while nearly all his examples are visual, he also emphasizes AR’s multisensory nature, specifically mentioning the possibility of AAR:

Azuma’s definition of AR emphasizes an important ingredient noted by many other authors (e.g., [20]), namely interactivity: the simulated sensory environment (whatever its modalities) is required to respond to the subject’s spatio-temporal environment: location, orientation, movement, time, and potentially internal physiological variables (e.g., heart rate) as well. While perhaps “not easy to do” at the time of his writing (1997), interactive augmentation in the auditory field is readily available today. For instance, earbuds can directionalize sound depending on head orientation or cancel unwanted noises, enabling greater focus on desired sounds. Hearing aids are a case in point. AAR may also interact with the visual field, for instance, reading visible texts aloud [8] or providing an audible dimension to virtual images. Several projects have developed sophisticated AAR for the blind and visually impaired, rendering the world as sound [24, 25, 26, 27, 28, 29]. But AAR need not be technologically complex to be effective. Unfortunately, nearly 30 years after Azuma’s speculations, comprehensive explorations of AAR concepts, technologies, applications, taxonomies, and potentials remain scarce, though several recent articles and one book (by a sound designer) herald an emerging disciplinary maturity (see [5, 6, 7, 30]).

1.2 AAR and VAR compared

VAR generally entails the technological complexity and cost of a mobile, head-mounted visual display in order to effectively fuse the simulated visual field with the real one. But when AR is defined as a computer-generated overlay occupying any sensory modality [31], a far lower-tech version is possible in the acoustic domain, where “…virtual auditory content is blended into the physical world to augment the user’s real acoustic environment.” Yet, “audio augmented reality (AAR) remains relatively under-explored” [6] as well as under-theorized, though the recent grounded theory taxonomy by Dam et al. [5, 30] represents a salutary step toward establishing AAR on a more solid foundation. As in feature film or video game soundtracks, auditory channels, though universally acknowledged as critical, tend to be subordinate to the visual ones. But, like VAR, AAR is powerful, though in a different way, and presents a much lower technological bar.

In its broadest form, AAR is familiar as recorded audio content played over a loudspeaker. Such AR offers a potentially enormous range of content without the need for more advanced technology, and thus at a much lower cost, with fewer barriers to entry. No digital technology is required for AAR; open-backed or bone-conduction headphones suffice, allowing the audio stream to blend seamlessly with the real environment. Indeed, any acousmatic³ sound fulfills AR’s broad sense definition. With the advent of portable, personal listening devices starting with the cassette-based Walkman (1979), through the portable CD and MP3 players, to the contemporary smartphone and the latest crop of smart glasses and lightweight speakers (headphones or earbuds), the radical impact of acousmatic environmental augmentation became portable too [33, 34, 35, 36].

Played through loudspeakers, sound is broadcast relatively omnidirectionally, providing a collective augmented reality by diffusing, reflecting, and diffracting throughout a space, circumnavigating solid objects that would block light, and thereby affecting all who inhabit it, as well as drawing them together through shared auditory experience. Played through personal listening devices, sound becomes mobile and can be personalized. Either way, acousmatic sound can transform mood and color other perceptual modalities far more readily than in the visual domain. The low-tech VAR equivalent to the loudspeaker, for instance, projecting images on one wall of a room,⁴ is far less immersive. Even when the projected image is large enough to encompass peripheral vision, the user is required to look in a particular direction. But augmentation of visual reality via smart eyewear, the equivalent of a personal listening device, requires the complexity and expense of an HMD in order to blend seamlessly with visual reality.⁵ Furthermore, at its simplest, the auditory modality is emotionally powerful, widely viewed as more affective, unmediated, soulful, or embodied than the visual, as numerous philosophers and neuroscientists have variously discussed across the ages ([40, 41, 42, 43]; [44] book 3, section 402). Even newer AR technologies have begun to recognize the advantages of AAR. Indeed, Meta’s first foray into the world of AR glasses, the Ray-Ban Meta smart glasses, has resulted in a system that is mostly geared toward sound and lacks an HMD due to current constraints of technology and cost.⁶

One technology critic has noted that music is the simpler, original AR: “Augmenting our visual field is a new feat, but augmenting our sense of sound is one of the oldest human tricks… When music plays in the background, it provides an emotional undertone for everyone in its presence.” And it does so without the need for intrusive, distracting devices strapped to one’s body. “Having music at our fingertips gives us something many of us lack but crave–control over our environment as well as control of our emotions… Even Google may have realized that augmented reality solutions don’t need a screen. They just need sound” [46].

AAR’s relative simplicity and lower cost depend in part on the fact that providing an immersive auditory field does not require bringing auditory objects into spatial focus, unlike the visual field, where reality augmentation requires spatial focusing on the retina. Indeed, the power of acoustic AR lies partly in the unfocused nature of sound, which tends to permeate enclosed spaces without casting sharp shadows. The tonotopic cochlea performs frequency analysis, and while auditory spatialization follows in the auditory cortex, sonic spatialization is relatively crude compared to spatialization of the visual field. Acousmatic sound takes advantage of these facts while radically widening the range of sonic possibility to anything that has been recorded. Acoustically transparent headphones are thus a simple, relatively low-tech way to augment acoustic reality without blocking it.

However, simply playing music or other sounds in the background does not satisfy everyone’s definition of AR. As stated earlier, in a narrower sense, AR (and VR) require a simulated interactive sensory environment responsive to, as well as superposed upon, the real one. To meet this definition, both AR and VR must respond to the user’s spatial attributes, location, and orientation, using sensors and processors to track GPS position, as well as head, hand, eye, or body movements. Interactivity makes both VAR and AAR more difficult and costly: simulated visual and acoustic fields must be continuously recomputed with corresponding changes in the user position. Such requirements entail additional input and output technology: cameras, accelerometers, magnetometers, gyroscopes, microphones, and GPS to map the environment and detect body and eye location, orientation, and movement; binocular displays and binaural speakers to render the virtual sensible. AR can thereby adjust the projected scene according to the real scene, combined with subject movement.

But even subject to this narrower definition, AAR is simpler than VAR. Given a user trajectory within an AV environment, each AR audio channel is a map from time to amplitude, while each video channel is a map from time and 2D raster pixel space to the four channels defining each pixel (e.g., R, G, B, alpha). And displaying visual information is far more complex than displaying acoustical information, as has been noted. Furthermore, when performing realistic AR modeling through binaural and binocular representations, auditory channels are less variable than the visual due to the more omnidirectional propagation of sound compared to light in the real domain and therefore also in the virtual one. Vision is highly directional, while audition is more omnidirectional and diffused, even when fully modeling binaural hearing by computing a head-related transfer function (HRTF). When the subject moves or turns, VAR necessitates rapid computations corresponding to the quickly changing spatial visual field; in contrast, the spatial auditory field remains more constant, even when computing binaural sound as a function of position, at least when simulating realistic sensory environments.

Interactive sonic technologies have been developed with far fewer technological machinations than required for their visual counterparts. To quote the same technological critic: “Further augmenting its effect, the music we hear can now be responsive to our environments. Spotify allows users to discover popular tracks in their cities or even sync their music to their run. Technology also exists to sync music with gaming experiences or to soundtrack your surroundings” [46, 47, 48, 49, 50].

But interactivity may also entail responsiveness to a user’s physiological attributes by means of biometric data, including heart rate (HR, indicating physiological arousal), heart rate variability (HRV, reflecting stress and autonomic nervous system activity), galvanic skin response (GSR, measuring emotional arousal), respiration rate (RR, related to stress and relaxation), and skin temperature, as well as data from electroencephalography (EEG, indicating brain activity) and electromyography (EMG, indicating muscle activation). This sort of interactivity is crucial in several AAR applications we discuss in this chapter.

VAR and AAR also differ in relation to cognitive (informational) communication versus emotional (affective) evocation: mentation vs. mood. The visual field superimposed in VAR is highly effective for the former, for instance, conveying airspeed to a pilot via a HUD, rendering critical data as a colored plot, or indicating positions of monsters in Pokémon Go [51]. AAR may function in the same way, carrying cognitive content via language (for instance, a synthesized voiceover describing a scene, virtual or real), as well as nonverbal auditory displays, including sonification (data-controlled sound) and audification (data rendered as sound) [22]. Using sonification, one might detect patterns in a dataset (perhaps a lengthy DNA sequence) by hearing it, while audification can render bounded oscillations (seismological data, say) audible. Whole research labs are dedicated to such topics [52]. Likewise, both VAR and AAR evoke mood and arouse emotion. But here the latter is more effective. Perhaps the affective power of sound is most widely recognized in the form of music. However, in our projects, we focus not on music as usually understood, but rather on “soundscapes.”

The soundscape concept first came to prominence through the work of Canadian composer and environmentalist R. Murray Schafer, who in the early 1970s wrote an influential text, The Tuning of the World, in which he introduced the term, encouraging the reader to consider the world soundscape as “a macrocosmic musical composition” [53]. With this text, along with a series of research projects out of Simon Fraser University, Schafer and his colleagues founded the field of acoustic ecology, along with the World Forum for Acoustic Ecology, an organization that still exists today, with its own journal, entitled The Soundscape. Schafer’s project, part activism and part artistic inquiry, inspired a new generation of listeners who combined this “world soundscape” sensibility with Deep Listening methodologies of Pauline Oliveros, an American composer and improviser who cultivated an immersive life practice of listening that complemented many of Schafer’s ideas [54].

Our projects center on soundscapes not as occurring naturally in the world, but rather soundscapes that are deliberately fashioned for aesthetic-educational-therapeutic purposes and generated acoustically. Therefore, our definition is much narrower. For our purposes, we succinctly define a soundscape as a stochastically steady-state, nonperiodic (and hence extensible), non-distractive (hence non-denotative⁷) acousmatic sound. Such sounds include collages of static natural sounds, such as rain, wind, surf, insects, or dawn choruses; sounds of human activity (e.g., highway traffic, cafe babble); synthetic “colored” noise (white, brown, pink, etc.; see [55]); as well as ambient and minimalist music, especially drones, and so-called sonic or acoustic “wallpaper” [56, 57]. Brian Eno, for example, who coined the term ambient music in the 1970s, conceived of room ambiences enhanced through musical processes, by which spaces could be suffused with “an atmosphere, or a surrounding influence: a tint” [58]. A great variety of sounds fit, or can be made (through judicious editing) to fit, our definition of soundscape, including various combinations of natural sounds, synthetic sounds, and musical sounds in genres such as ambient, drone, or minimalism, though not the broader category of music as usually conceived (and especially not song!). Following from their definition, it is easy to see that the set of all soundscapes (quite unlike music) is additively closed: mixing two soundscapes produces a third. This closure property—not shared with music—is quite useful, as we will observe. Furthermore, again, unlike most music, soundscapes tend to be non-distractive, fading into the perceptual background, rendering them more suitable for inducing mental calm and focus.

While visual content may trigger emotional responses, as in the visual arts, and especially via emotionally disturbing imagery, the auditory channel is widely regarded as far more potent in this regard [59, 60], generating an affective response by indexing environment features, carrying paralinguistic emotion, evoking affective memory, or setting a mood via musical properties like timbre, melody, and harmony. Our response to natural soundscapes is probably shaped by evolution, stimulating the autonomic nervous system directly, triggering a stress response (sympathetic nervous system) for some sounds (e.g., roaring lion) and a relaxation response (parasympathetic nervous system) for others (e.g., chirping crickets) [61, 62, 63, 64, 65, 66, 67]. Further, audition is naturally immersive; unlike light, sound waves suffuse an enclosed space, and our auditory system is, correspondingly, always “open.” There is no auditory counterpart to “looking away” (“hearing away”?); as Schafer famously remarked, “there are no earlids” [68].

Yet another contrast between VAR and AAR lies in the possibilities for a collective social media experience. VAR can, of course, be synchronized across users, but for the purposes of realism, the technology must uniquely tailor visual imagery to each user in order to achieve perspectival interactivity; unless similarly positioned, no two people will receive the same signal (though they may access different perspectives on the same virtual space, by which the technology can nevertheless induce mediated socialization). However, AAR opens the possibility of a realistic collective social experience in which a group of people simultaneously hears the same sounds. This collective experience enables AAR to generate broad social connections in a way that VAR does not, just as a concert, which may gather thousands to participate in a synchronous communal aural event, is a more effective generator of collective experience than a museum exhibit, in which shared experiences are scarce and largely unsynchronized.

1.3 Auditory augmented reality for well-being (AAR4W)

In this chapter, we are primarily interested in the use of AAR to generate an independent simulated auditory environment, without any coordinated VAR analog, to enhance well-being. Compared to VAR, the relative technical simplicity, immersiveness, emotional power, and collective experience of AAR render it more effective, as well as more practical and economical, toward the promotion of well-being.

Well-being admits a wide array of definitions; here we focus on its broadest psychological and sociological dimensions: human health in a general sense—physical, mental, and social—particularly stress reduction, serenity, focus, insight, and interpersonal connection. Soundscapes appear to be ideal for well-being in this general sense. In our work, we therefore focus upon soundscapes, either as intentionally therapeutic tools or as sound art compositions inducing well-being as a side effect of an intended aesthetic experience. Being steady-state and non-distractive, soundscapes are extensible to arbitrary length, fading into the sensory background while strongly, if subliminally, shaping mood. The closure property mentioned earlier enables sonic mixing and collaging, opening new possibilities for their creation, deployment, and optimization: effective soundscapes can be combined and composited.

Such soundscapes serve to alter the intersensory perception of one’s environment, powerfully and unobtrusively shaping consciousness as surely as other environmental factors, such as lighting, temperature, and wall color. While both sounds and the AR devices that produce them may become objects of perceptual focus, they can also recede into the background realm of utility as naturalized practical instruments for interacting with the world, what Heidegger called “ready-to-hand” [69]. Just as a seasoned pianist, for whom the piano is a virtual bodily extension, needn’t think about her musical tool, neither must the wearer of a portable acousmatic playback device, which likewise becomes a bodily extension.

There is a large literature on the value of music and sound for well-being [63, 70, 71, 72, 73, 74], including among students [75, 76, 77, 78, 79, 80]. Many of these studies focus on soundscapes using a definition comparable to ours (steady-state aperiodic), though typically they do not describe these possibilities as AR. Sounds that activate the parasympathetic autonomic nervous system and thus elicit a relaxation response are widely regarded as therapeutic in a broad as well as specific sense, from accelerating convalescence in the hospital to enhancing focus in the classroom and generally improving mood [81, 82, 83]. Soundscapes (by our definition) tend to have this effect. The literature suggests that colored noise [84, 85] and predictable natural soundscapes are particularly effective [86, 87, 88, 89, 90, 91, 92]. No doubt human beings have evolved over millennia to interpret predictable (hence low-information) nature soundscapes featuring stochastically steady-state sounds resulting from the sonic summation of myriad unthreatening agents (such as chirping crickets, croaking frogs, rustling leaves, light rain, and gentle surf) as auditory streams requiring minimal perceptual attention, to which we therefore respond with decreased stress, increased relaxation, and improved mood. Such sounds contrast with unpredictable high-information impulse sounds emitted by potentially dangerous agents, like the roar of a wild animal or unexpected footsteps in one’s home, which trigger a high perceptual alert, requiring rapid processing and stimulating a stressful fight-flight response⁸ [86, 94, 95, 96].

Another immersive practice, soundwalking, adds to the already recognized practice of walking (for physical and mental health): a method of listening and attunement to the sense of place through a specific, often artistically curated pathway. Soundwalks can be self-directed or curated and guided, and at a basic level involve moving attentively through an environment and practicing what Pauline Oliveros called Deep Listening, or listening with attention to analytical detail, intentionally [54]. Hildegard Westerkamp—an original member of the aforementioned acoustic ecology movement of R. Murray Schafer, Barry Truax, and others—has written extensively on soundwalking as a practice, describing it as an “intense introduction to the experience of uncompromised listening” [97].

Soundwalking can also be mediated through technology, and this is where it leans into an AR practice. For example, Janet Cardiff’s audio walks, a huge part of her oeuvre since the 1990s, require the listener to wear open-backed headphones, which allow the local soundscape to leak through a mediated version of that same soundscape Cardiff recorded earlier, alongside her reflections, instructions, musings, and storytelling [98]. In another example, Christina Kubisch’s Electrical Walks uses special headphones and instructions for listening to sounds that are created through magnetic induction noises in modern electronics spread throughout cities, including ATMs, security electronics, and other electronic equipment found in everyday urban environments. For this work, the listener is hearing sounds that normally would not be possible to hear, bringing into sonic awareness a whole layer of existence that we normally would not notice [99]. These two examples differ in their approaches to the normative soundscape of the environments in which they happen, Cardiff’s being more attuned to the sense of place and its potential for narrative and historical reflection, whereas Kubisch’s work brings us into a hidden layer where our imaginations can interpret the experience in personal ways. Additionally, both of these examples involve solo reflections, although often they are presented in groups of individual listeners who walk together.

Soundwalks can be mediated by older technologies as well, such as musical instruments or other sounds. One of the current authors experienced such a soundwalk in Koli National Forest in Finland, led by Pessi Parvianinen in 2010. In this walk through the forest, Parvianinen secretly populated the trail with musicians, each in a solo environment playing excerpts of music by Finnish composers or improvising with birds. It was a lovely way to bring an additional sonic layer into the natural sounds of the environment, inviting the participants to consider generations of musicians whose music was influenced by this environment.

Whether intended as aesthetic, therapeutic, or wellness interventions, and whether deliberately combined with real sounds (as in soundwalks) or not, AR soundscapes can contribute to well-being.

2.1 Coronium 3500 (Lucie’s Halo)

In 2014, the second author (Smallwood) created an environmental, site-specific sound installation called Coronium 3500 (Lucie’s Halo) on the site of Caramoor Center for Music and the Arts in Katonah, New York [101, 102, 103]. Part of a larger group of sound installations by various artists, this piece sought to tie the soundscape directly to the environment by (a) creating sounds inspired by the natural sounds of the site’s birds and insects and (b) creating sounds that were entirely dependent on available sunlight (see Figure 1). The resulting “solarsonic” voices, which used custom electronics, played through several small speakers distributed around a grassy area. These speakers were powered by solar panels without batteries, which thus produced a variable level of real-time energy, affecting both the intensity and tempo of musical phrases (see [101] for technical details).

Although it has since been exhibited elsewhere, the original intention was to create a piece tied directly to Caramoor, incorporating its sounds into a mix of other sounds. Field recordings of the installation site were used to create the sound-making circuitry in order to engineer a soundscape inclusive of the local one. Then, when the piece was installed, the environment itself “played” the piece through a combination of sunlight and shadows created by large deciduous trees surrounding the site. Indeed, even the changing seasons played a role in the resulting soundscape, since the sun’s angle, the presence or lack of leaves on the trees, and other environmental factors had a direct bearing on the sounds of the electronic “critters.” This was a risky proposition, since the piece was exhibited on-site for 2 years, and the visitor’s aural experience of the piece could not be controlled or determined. On a cloudy day, for example, the piece would be very quiet, with some voices never sounding at all. It was fun to watch people experiencing the piece, especially when the sun would suddenly come out from behind clouds to awaken previously silent voices, to the surprise and delight of the visitors.

2.2 Emergences/Expanded Anatomies

In another work from 2024, entitled Emergences/Expanded Anatomies, Smallwood collaborated with visual artist Sean Caulfield on a site-specific installation on the historic townsite of Malvina, Quebec, a colonial-era village near the New Hampshire border [104]. In this work, combining multichannel audio with Caulfield’s prints and drawings, many of which were made on-site, the piece captured the historic and current reality of Malvina, inclusive of a garden with its wooden fence and small garden cottage (see Figure 2). The sound piece included sounds of found objects, including many historical objects from the original town, loaned by local residents, and worked into drones and textures created on-site by the artist. In addition to these sounds, the idea was to also include the remarkable sounds of Malvina’s birds, the wind and rain, the creek running by, and occasional tractors driving down the gravel road. Due to the lack of highway noise, these sounds were ever present in this “high fidelity” environment, and so rather than attempting to record them, the piece simply left room for them by including sparse spaces and silences through which they could be heard.

In both of these examples, the “AR” is clear: the pieces incorporate the sounds of the place where the piece happens, making the place itself integral to the work. This work owes some inspiration to many of Schafer’s concert works, in which he eschewed the “temples of silence” that are concert halls in favor of natural environments that become part of the music and stories told in the work, often requiring the audience to travel to remote locations in northern Ontario and elsewhere [105]. Schafer’s work reflects a more general shift towards inclusion of space and place as active parameters, due, in part, to the rise of multichannel sound, but also to the opening up of sonic materials to include our human soundscape. In fact, the rise of sound art in the 1980s, a genre of art born in visual art culture in which sound is an artistic medium, introduced ways of thinking about the artistic uses of sound beyond music, and this includes sound in the context of site-specificity.

2.3 Sounding the Garden

Sounding the Garden [106] is an interdisciplinary, educational AR sound art project situated at the intersection of fine arts, humanities, social sciences, artificial intelligence, and engineering. It draws on student contributors, including electroacoustic music composers, world music performers, designers, and ethnomusicologists, aiming to support their development as artists and scholars.

In 2013, the Canadian Centre for Ethnomusicology (CCE) [107] and the Aga Khan Trust for Culture [108] collaborated on an international event entitled “I am a bird from Heaven’s Garden: Music, Sound, and Architecture in the Muslim World,” heralding the creation of a new Islamic garden (later named the “Aga Khan Garden”) to be funded by the Aga Khan and located on the site of the University of Alberta’s Botanic Garden [109, 110]. Out of this initiative emerged an AR project funded by the Canadian government’s Social Science and Humanities Research Council (SSHRC) entitled “Evolving the Botanic Garden,” which provided AR overlays as educational and aesthetic enhancements of the garden experience via visitors’ mobile devices [111].

As CCE director, the first author (Frishkopf) led construction of an AAR layer, Sounding the Garden [106], aiming to enhance the garden experience with poetry, music, and other sound, with a Sufi interpretation: the garden stroll becoming a mystical-aesthetic path. A team was recruited, including both authors along with graduate students in composition, ethnomusicology, computer science, design, and drama.

Sounding the Garden is a spatial mapping of mystical language, projecting a famous poem by the Persian Sufi poet Farid al-Din Attar (c. 1145–c. 1221) entitled “Language of the Birds” (Mantiq al-Tayr), with its seven-stage spiritual journey toward enlightenment, onto the physical garden. We associated each stage—or “valley” (wādi), to adopt Attar’s metaphor—with a section of the garden. An ensemble comprising students, alumni, and faculty of our music program then collectively created and recorded ambient improvisatory music in a Persian dastgah (melodic mode) corresponding (according to prominent Iranian music scholar, performer, and composer Majid Kiani) to each such “valley.” We also embedded relevant tracks from a CD series, “Music of Central Asia” [112], at specific locations in the garden’s augmented reality space, along with bird calls and poetic recitations of Attar’s masterpiece in Persian, with English translations and interpretations.

Traversing the garden counterclockwise (aligning with Islam’s spiritual direction), one progresses through the seven valleys in order while experiencing associated poetry, sounds, and music (see Figure 3). User position is tracked via GPS (or signaled by tapping on-screen icons), triggering the soundscape. In this way, the piece provides visitors a new sonic layer overlaid upon the remarkable soundscape of the garden itself, deepening affective understanding and enhancing well-being through sound.

2.4 Autonomous adaptive soundscapes

In 2012, sound artist and composer Yoko Senn found herself in the emergency room of a hospital in New York, and the sonic hellscape she described and sought to escape was almost more traumatic than her medical reason for being there. As an electronic musician and composer, Sen started to imagine ways that such a hospital soundscape could be “composed” by sound artists to create a calming space, rather than one of chaos. Her project to redesign hospital alarm sounds and noises has caught the attention of hospital administrators and researchers, and experiments are now underway to collaborate with equipment manufacturers to create a more supportive sonic environment [113].

Hospital environments are excellent candidates for augmenting a stressful reality with soothing sounds. High stress levels and anxiety, associated with delirium and sleep deprivation, are very common in critically ill patients and may compromise recovery and survival, as well as increase the length and costs of hospital stays [114, 115, 116]. Pharmacological approaches typically used to treat these conditions have non-negligible expense, limited effectiveness, and serious side effects. As a means to counter high stress levels and related effects, music and sound therapies are low-cost and noninvasive, with far more limited side effects than medication. Research (including our own systematic review) has shown them to be highly effective if customized to the patient [117, 118]. However, critically ill patients cannot be expected to communicate effectively with music therapists, who are scarce, frequently unavailable, and costly. Linguistic or cultural differences between the patient and therapist may also limit their effectiveness. Furthermore, as discussed above, soundscapes (as we define them) present many advantages over music.

Building on and integrating knowledge from multiple disciplines (music, music therapy, computer science, nursing, and critical care medicine), we developed autonomous adaptive soundscapes (AAS), an intelligent bio-algorithmic system generating therapeutic soundscapes for critically ill patients, using machine learning and biofeedback to induce relaxation, improve sleep, and reduce agitation, anxiety, and delirium [81]. AAS seeks to optimize a patient’s sonic environment by dynamically selecting, tuning, and mixing soundscape files (currently around 10) drawn from an audio library spanning a wide range of recordings (natural, musical, and synthetic). A reinforcement learning approach [119] developed by computer scientist Martha Steenstrup guides the search of the soundscape space based on autonomic biosignals indicating the patient’s current state, thereby delivering a customized soundscape to the patient (see Figure 4). No conscious, active engagement with the system is required from the patient. Our objective is a system suitable for the intensive care unit (ICU), one that is highly effective, always available, simple to operate, minimally intrusive, and low risk.

AAS also promises enormous potential beyond the ICU, as stress, anxiety, and insomnia are pervasive social problems. Recent evidence-based research supports the use of music and sound for sleep [121] and has led to the specification of sonic criteria (rhythm, pitch, frequency, volume, genre, and duration) suitable for relaxation [122]. However, this research has not yet been operationalized in autonomous adaptive soundscapes. Experimentation with AAS was difficult to conduct in practice due to restrictions surrounding ICU access, particularly during the pandemic, but we ran experiments using a medical simulation laboratory [123]; results are pending.

2.5 Mindful Social Listening (MSL)

We decided to modify AAS to provide adaptive soundscapes for relatively healthy subjects outside medical facilities but nevertheless in need of support, starting with those closest to us: our students. For Mindful Social Listening (MSL), we removed the constraint that no conscious, active engagement with the system is required. However, we also imposed a new one: the MSL system should select soundscapes suitable for group rather than individual listening in order to catalyze student socialization and reduce isolation, exacerbated by the use of headphones and earbuds while studying in the library.

The motivation for such a system stemmed from a simple observation: Academic life is replete with exploration, discovery, learning, creativity…and stress. How can soundscapes enhance student well-being and academic success? Across all disciplines, postsecondary students suffer from the stress that inevitably accompanies the harried occupation of university life, impeding learning and jeopardizing well-being. This dire situation has become a veritable mental health crisis. As university mental health programs are woefully under-resourced, students rely on various forms of self-care. Our pilot study revealed that over 75% of students experience high levels of stress, 94% use music and sound to cope, and about 80% find this use highly effective, wearing personal listening devices (headphones, earbuds) to enhance concentration, mask distractions, and reduce stress. However, students may not be optimizing their sonic environments for stress reduction and focus; often, their selected soundscapes, or preferred genres of music, are actually distracting them from work by demanding attention. Furthermore, personal listening devices can not only be deleterious to hearing but also contribute to social isolation. We sought instead to develop an autonomously evolving, shared, non-distracting augmented sonic reality: a soothing sonic factor analogous to such common (but nonadaptive) ergonomic factors as room temperature, lighting, seating, and interior design.

We address these limitations by developing an autonomous system that generates a responsive soundscape environment in a social listening space. Building on AAS, combining data mining, machine learning, and algorithmic composition, and drawing on field recordings, applied ethnomusicology, health science research, and creative sound art, we have designed a responsive soundscape environment that can autonomously adapt to feedback from socially interacting users, enhancing mindful calm and supporting academic well-being. Installed in a library study room on the University of Alberta campus, this octophonic system¹⁰ operates by gathering conscious feedback from students, then compiling this information to update a “value” for each soundscape, as shown in Figure 5.

MSL deploys a “multi-armed bandit” reinforcement learning algorithm, balanced between exploration (seeking the optimal soundscape) and exploitation (making use of current knowledge), quite similar to that of AAS, and requiring calibration of three critical parameters:α, ε, and T. Each soundscape has a value ∈ [−1,1], adjusted over time according to user feedback, comprising a vector. Initially, all values are set = 1. After playing a soundscape for T minutes, we compute its reward as the average user response: reward ∈ [−1,1]. Next, we recompute its value as a weighted average between the new reward (weight = α) and the old value (weight = 1−α). Finally, we select the next soundscape: with probability ε, select at random (exploration); with probability 1−ε select the highest-valued soundscape (“greedy” exploitation). Currently, we explore a small set of 10 (like AAS) but plan to develop the algorithm to enable exploration of larger sets (see below). Adaptation is difficult due to change: users may enter or exit, and their preferences may shift. Simulations show that with proper parameter settings, such a system can effectively adapt to play effective sounds addressing user preferences, provided the change is not too rapid.