INTERPRET

INTERPRET is a performance system that places computational interpretations of dance within an evolving audiovisual exchange. Camera-based movement analysis interacts with five composed sound resources, neural audio reconstruction, and generative visuals. A performer can respond through movement while the artist redirects compositional choices or negotiates bounded AI proposals. The project asks whether interpretations formed from limited movement evidence can become material for continued improvisation.

MediaPipe Pose follows body landmarks, and the pipeline derives geometric and temporal movement features. Learned representations and short-term forecasts describe how movement develops. Affinity to Emergence, Flow, Fragmentation, Tension, and Transformation informs compositional routing. These estimates describe relationships to recorded material and remain distinct from the dancer’s intended meaning.

The sound has three layers: prepared reconstructions provide continuity, a live neural decoder gives the main response, and a companion shares that decoded sound with changes in color and stereo position.

Five related source works share a timbral foundation while exploring different compositional contexts. Their encoded trajectories form a Raw Latent Atlas. Prepared reconstructions provide Memory, while a continuous live RAVE decoder produces the Primary Response. The Context Companion shares that decoded audio with bounded changes of color and panning. INTERPRET also draws on many of NeuMu’s functions and materials in the development of its generative-audio system.

Prepared Memory carries a transition while the live response leaves, settles, and rejoins. Shared timing, gain ramps, and conditional recovery shape how layers enter and recede. These mechanisms support a continuous performance setting, while the musical relationship between the layers remains a listening judgment.

Five Origin states and ten pairwise Transition states provide a composed visual vocabulary. Context routing determines their relative presence, and selected movement-derived controls modulate parameters within the visual networks. Projected responses become part of what the performer can perceive and develop through further movement.

The artist can choose a visual Origin, a target context, and transition progress independently of the movement estimate. Manual direction provides a separate source of authority and enters the context available to the language model. This keeps a disagreement between observation and artistic intention available for a decision.

The language model receives structured movement summaries, recent context, and artistic direction, and proposes bounded control trajectories. Off, Observe, Negotiate, and Bounded autonomous modes determine whether a proposal can act. The artist can reject a suggestion or withdraw its influence. The model has no authority to select source material, change RAVE latents, or choose visual Origin and Transition states.

Movement features can become material for generating sound and visuals, but they cannot be equated directly with the dancer’s artistic intention. Retaining manual direction so that the artist can assess, revise, or reject the system’s interpretation is an important design choice in this work. The next step is to bring together synchronized records of movement, changes in control, and audiovisual output to examine when the system’s interpretations support improvisation and when human intervention is needed.