// Initialising experience

HACKED:
VR

Sound Design Case Study  ·  ASU Spring 2024

DSC 551
Immersive Experience Design
Spring 2024

HACKED:
VR

Immersive VR Sound Design

Case Study — ASU Dreamscape Learn × Meteor Studio

Sound Design
Arsh + William Mar
Unity XR / VR
MIXibition, May 2024
// Project Context

Setting the Stage

ASU Collaboration

The Immersive Experience Design cohort at Arizona State University partnered with Dreamscape Learn and Meteor Studio to develop two parallel VR environments. Each was built by one half of the class over the course of the spring semester.

Project A
Urban Inferno
Project B — This Study
Hacked: VR

A four-person Sound Design team split 2+2 across projects. Arsh and William Mar focused exclusively on Hacked: VR — responsible for every audio decision from concept through public exhibition.

Sound Design Contributions

Interaction Sounds

Tactical UX audio feedback tied to every physical prop interaction — the digital button, floor shakers, fans, the antivirus stick, and the lever return to the server room.

Background Music

Four adaptive mood loops — one per act — shifting emotionally from soothing ambient through distorted glitch to triumphant resolution.

Voice-over Narration

A scripted, eight-frame narrative arc synthesised using AI voice tools and delivered through 3D spatial audio anchored to scene positions within Unity.

Hacked VR — data stream environment, Act I
// Act I — The Serenity of the Realm

Data flows like water.
The network is in harmony.

// Public Exhibition

MIXibition — Spring 2024

DateMay 1, 2024
Time6:00 PM – 8:30 PM
VenueASU MIX Center
Address50 N. Centennial Way, Mesa, AZ 85201
HostASU Herberger Institute for Design and the Arts

A Night of Immersive Exhibits

Hacked: VR was publicly exhibited at the MIXibition showcase — an event open to the Mesa community, featuring immersive exhibits and artwork from ASU faculty and students.

The experience was presented alongside Urban Inferno, giving the public a live demonstration of the full dual-project initiative in completed form.

The sound design — voices, music loops, and interaction SFX — was heard by a live public audience for the first time at this event. This was the culminating validation of every design decision made across the semester.
// The Design Brief

What Sound Had to Solve

"Design audio that makes a virtual computer world feel real — narrate a cybersecurity story, trigger precise feedback on every physical interaction, and shift the emotional register from serenity to threat to triumph across four acts."

01
Narrative Audio

Script and synthesise four voice-over narrations — one per act — guiding the observer through the cybersecurity storyline from calm to chaos to resolution.

02
Ambient Music

Compose four background mood loops shifting emotionally across acts: soothing ambient → ominous shift → distorted glitch → triumphant resolution score.

03
Interactive SFX

Produce ten interaction sound effects tied to physical props: the digital button, floor shakers, fans, the antivirus stick, and the lever return to the server room.

// Audio Design Framework

How We Mapped Sound to the World

Virus Sounds

Audio representing the threat — entering, spreading, corrupting.

Birth of the virus — analogous to a phishing email
Propagation: how the virus spreads through the network
Glitch-out moments and active cyber attack sequences
Photos and data being erased or corrupted
Anti-Virus Sounds

Audio representing the response — fighting back, restoring order.

Calming of the anarchy — the attack beginning to subside
Resolution — system integrity fully restored
Protection engaging against this specific virus
Firewall activating, code rains, CPU fans spinning up
Haptic Connection

Audio synchronised to physical actuators — floor, fans, vibration.

Floor shakes: indicating the virus has arrived
Fans directing air in the direction of the flying deck
Floor escalation matched to the chip transition
Gradual calm as the system is restored
Interactivity

Audio cues triggered by direct user actions in the VR environment.

Initiating the virus — analogous to clicking a phishing link
If colour is used, sound attaches to the colour cue
Button press to initiate chip transport
Lever pull to return to the server room

The audio blueprint was defined before any assets were produced — categories mapped to world objects and storyline so that every sound served a specific narrative or UX function.

Hacked VR — data streams Act II
// Act II — Exploration of the Chip

User data, system files,
financial records — each interconnected.

// Narrative Arc

Soundscaping a Four-Act Digital Journey

Act I
// SYS_STABLE
The Serenity of the Realm
Voice
"Welcome, observer, to Hacked. A serene network in perfect harmony."
Audio
Ambient, soothing — reinforcing system stability.
Interaction
Button press to initiate chip transport.
Act II
// NAVIGATING
Exploration of the Chip
Voice
"Data flows like water — user data, system files, each element vital."
Audio
City-like ambient hum, data stream tones — the world feels alive.
Interaction
Floor shakes escalate; fans activate toward the flying deck.
Act III
// ERR_CRITICAL
The Virus Attack
Voice
"A rogue element slips through. System files corrupted. Chaos begins."
Audio
Distorted glitch sounds escalating into high-stakes dramatic music.
Interaction
Fans disperse, floor shakes; stick used to pop virus bubbles.
Act IV
// RESTORED
Restoration of Integrity
Voice
"In the digital world, vigilance is our greatest ally. Stay safe."
Audio
Climactic resolution building to a triumphant score.
Interaction
Floor calms; lever pulled to return to the server room.
// Listen — Audio by Act
Act I — Serene
Background Music
Voice Narration
Act II — Exploration
Background Music
Voice Narration
Act III — Attack
Background Music
Voice Narration
Act IV — Restored
Background Music
Voice Narration
Hacked VR — virus world with dome structure
// Act III — The Virus Attack

A rogue element slips through.
The battle ensues.

// Interaction Sound Effects

Sound as UX Feedback

Every physical interaction in the experience had a paired audio cue. Sound was not decorative — it was functional feedback that confirmed each action and reinforced the narrative state the observer was in.

Interactivity
Button Press

Triggered when the observer presses the digital button to initiate chip transport from the server room.

Haptic — Virus Arrival
Ominous Rumble

Loopable low-frequency rumble synchronised to floor shakes — indicating the virus has entered the environment.

Haptic — Transport
Blowing Fans

Fan audio matching the physical fans that activate in the direction of the flying deck during chip transport.

Virus — Attack
Analog Glitch 1

Distorted analog glitch — used during active cyber attack sequences and data corruption moments.

Virus — Attack
Glitch Tickle

A secondary glitch texture layered over the primary attack audio to create a sense of pervasive system corruption.

// Production Pipeline

AI-Powered Audio — Fully Digital

04
Voice Narrations
One per act, AI-synthesised
04
Music Loops
Adaptive mood per act
10
Sound Effects
Mapped to physical interactions

Single-Laptop Digital Pipeline

The full audio production pipeline — voice synthesis, music composition, and SFX sourcing — was operated entirely from a single laptop. This consolidated, portable architecture enabled rapid iteration and seamless asset deployment throughout the project without hardware bottlenecks.

The approach proved that immersive VR audio for a publicly exhibited experience can be produced with minimal cost of entry, provided the tooling choices are deliberate.

Pipeline Status: Operational // Live_v1.0.4
"Generative AI not as a replacement for creativity — but as a tool to accelerate it. Rapid iteration, novel exploration, reduced cost of entry."
// Integrated Toolset
▶
Google Cloud Text-to-Speech
Voice-over synthesis for all four narration scripts. AI voice generation allowed rapid iteration on tone, pacing, and emotional register across the eight-frame arc.
▶
Ableton + Spitfire Labs
Text-to-music composition for adaptive mood loops. Four background tracks produced — each calibrated to the emotional register of its act.
▶
Freesound Library
Interactive SFX sourcing for all ten prop-triggered sound effects — selected, edited, and mapped to specific physical interactions within Unity.
// Technical Integration

Unity XR and 3D Spatial Audio

Unity 2019.4.40f1

Audio assets were integrated using Unity 2019.4.40f1, the ARCore XR Plugin, and the Dreamscape Learn SDK. The Unity XR Interaction Toolkit was used to map audio cues to real-time 3D interactions within each scene.

Scene Structure
Scene 1ServerRoom
Scene 1.5Transcending (transition)
Scene 2MainEnvironment (chip world)
Scene 3EndingServerRoom

3D Spatial Sound System

Spatially placed 3D audio fields
Environmental audio anchored to scene positions — sound shifts as the observer moves through the world.
Innate object emitters
Prop-embedded sound sources on interactive objects — each element has its own audio identity.
Prop-triggered SFX
Sound fires on physical interaction — button press, stick contact, and lever pull each trigger their mapped cue.
Controller haptic and audio feedback
Audio and haptics synchronised per controller event — every tactile moment paired with a corresponding sound.
Haptic Sync —

Floor vibrations escalate progressively into the chip transition. Fans activate in the direction of the flying deck. During the virus attack, sudden floor shakes and fan dispersion mirror the spread of the malware. Audio was tightly synchronised to each physical actuator cue throughout the experience.

Hacked VR — orb foreground, data streams Act IV
// Act IV — Restoration of Integrity

System restored.
Vigilance is our greatest ally.

// Takeaways and Reflections

What We Learned

01
Rapid Iteration

A fully digital pipeline — cloud voice synthesis, AI music tools, and open SFX libraries — allowed the team to iterate audio ideas quickly without hardware bottlenecks or studio access requirements.

02
Workflow Shift

Music and voice production can be entirely digital with minimal cost of entry. Generative AI tools significantly reduced the learning curve while expanding the range of what was achievable within a semester timeline.

03
Audio as UX

Sound was not decorative — it was functional feedback. Every physical prop interaction had a paired audio cue. The haptic-audio synchronisation proved critical to the experience feeling grounded and real.

04
Cross-Team Coordination

Delivering audio assets to Unity across technical teams required disciplined asset naming, act-level tagging (A1–A4, S1–S4), and timeline placement documentation to ensure correct integration.

// Status Log
[DONE]Voice scripts written and synthesised
[DONE]4 background music loops produced
[DONE]10 SFX mapped to physical interactions
[DONE]Unity 3D spatial audio deployed
[DONE]Haptic sync coordinated with tech team
[DONE]Publicly exhibited at MIXibition
[NEXT]Real-time generative audio (in progress)
"Human creative potential unleashed — not replaced — through AI-assisted audio production."