controlXY — Multitouch XY Control & Score Animation System

controlXY transforms your score into an interactive animation canvas. It defines persistent, multitouch XY control surfaces that enable composers to create complex choreographed animations directly within the score — the score-as-instrument, instrument-as-score.

Unlike traditional time-based cues, controlXY provides continuous spatial control that can be:

• Pre-programmed as animated sequences (choreography)
• Performed live as a tactile control surface
• Hybrid — switching between automation and manual control

As a bonus, all movements can simultaneously transmit OSC for external synthesis and media control.

Oscilla OSC controller design in Inkscape

Core Concept: Score Animation Through Spatial Control

Think of controlXY as a spatial modulation source embedded in your score:

1. Define control pads with draggable handles
2. Bind handle positions to visual/sonic parameters
3. Animate via presets & sequences OR perform live
4. Record/playback spatial gestures as part of the composition

This inverts the traditional "playhead reads static notation" model — instead, notation becomes dynamic, responding to spatial control in real-time.

Syntax

controlXY(
  uid: <string>,
  handle: <element-id> | [<id1>, <id2>, ...],
  bounds: <element-id> | "self",
  label: <bool>,
  osc: <bool|number>,
  oscAddr: <string>
)

The cue expression is attached to the bounding element (or any element if using explicit bounds).

Parameters

Coordinate System

• X axis

  • Left = 0.0
  • Right = 1.0

• Y axis (musical convention)

  • Bottom = 0.0
  • Top = 1.0

All values are normalized to the bounding box, making animations resolution-independent.

Published Signals

Single Handle

controlXY:<uid>.x        // 0.0 — 1.0
controlXY:<uid>.y        // 0.0 — 1.0
controlXY:<uid>.handle   // handle element id

Multiple Handles

controlXY:<uid>.<handleId>.x   // 0.0 — 1.0
controlXY:<uid>.<handleId>.y   // 0.0 — 1.0

These signals can be bound to any parameter in the system, creating direct connections between spatial control and visual/sonic outcomes.

Setup Examples

Basic Single Handle

<rect id="pad1" x="100" y="100" width="400" height="300" 
      fill="#222" stroke="#666"
      cue="controlXY(uid:pad1, handle:dot1)"/>

<circle id="dot1" cx="0" cy="0" r="12" fill="#ff4444"/>

Multi-Handle (Multitouch)

<rect id="mixer" x="100" y="100" width="600" height="400"
      fill="#111"
      cue="controlXY(uid:mixer, handle:[fader1,fader2,fader3,fader4], label:true)"/>

<circle id="fader1" cx="0" cy="0" r="10" fill="#ff4444"/>
<circle id="fader2" cx="0" cy="0" r="10" fill="#44ff44"/>
<circle id="fader3" cx="0" cy="0" r="10" fill="#4444ff"/>
<circle id="fader4" cx="0" cy="0" r="10" fill="#ffff44"/>

With OSC Output

<rect id="synthControl" x="50" y="50" width="300" height="300"
      cue="controlXY(uid:synth, handle:dot1, osc:true, oscAddr:synth/xy)"/>

Animation Workflow: Creating Score Choreography

The true power of controlXY emerges when you pre-program spatial animations as part of your composition.

Step 1: Save Spatial States as Presets

Using the Preset UI:

1. Press Alt+Shift+P to open the preset manager (or click ⚙ on the pad)
2. Move handles to desired positions
3. Type a preset name (e.g., "intro_position")
4. Click 💾 Save

Via DSL (triggered by playhead or buttons):

<!-- Save current state when playhead passes -->
<rect x="100" y="0" width="2" height="600" 
      cue="ui(action:'controlXYSave', preset:'stateA')"
      fill="red" opacity="0.3"/>

<!-- Button to save state -->
<g cue="button(
        trigger:ui(action:'controlXYSave', preset:'verse1'),
        style(label:'Save Verse', x:10, y:10)
      )"/>

Via Console (for experimentation):

// Save all controlXY instances
window.controlXYPresets.save('stateA');

// Save specific pad only
window.controlXYPresets.save('stateB', 'pad1');

Step 2: Recall States with Animation

Instant recall (no tween):

<rect x="500" y="0" width="2" height="600"
      cue="ui(action:'controlXYRecall', preset:'stateA')"
      fill="blue" opacity="0.3"/>

Smooth tween (2 seconds, easing):

<rect x="1000" y="0" width="2" height="600"
      cue="ui(action:'controlXYRecall', preset:'stateB', dur:2, ease:'easeInOutSine')"
      fill="green" opacity="0.3"/>

Via button:

<g cue="button(
        trigger:ui(action:'controlXYRecall', preset:'chorus', dur:3, ease:'easeOutElastic'),
        style(label:'▶ Chorus', x:10, y:50)
      )"/>

Step 3: Define Sequences (Choreographed Animation)

Sequences are playlists of presets that play automatically.

Define sequence via DSL:

<!-- Define a 3-state sequence -->
<rect x="100" y="0" width="2" height="600"
      cue="ui(action:'controlXYDefineSequence', 
              name:'intro_dance', 
              steps:'stateA,stateB,stateC')"
      fill="yellow" opacity="0.3"/>

Play the sequence:

<!-- Auto-play when playhead reaches this point -->
<rect x="200" y="0" width="2" height="600"
      cue="ui(action:'controlXYSequence', 
              seq:'intro_dance', 
              dur:2, 
              ease:'easeInOutSine', 
              loop:false)"
      fill="cyan" opacity="0.3"/>

Stop sequence:

<rect x="1500" y="0" width="2" height="600"
      cue="ui(action:'controlXYSequenceStop')"
      fill="red" opacity="0.5"/>

Via buttons (interactive control):

<!-- Define sequence button -->
<g cue="button(
        trigger:ui(action:'controlXYDefineSequence', 
                    name:'verse_pattern', 
                    steps:'v1,v2,v3,v1'),
        style(label:'Define Pattern', x:10, y:10)
      )"/>

<!-- Play button -->
<g cue="button(
        trigger:ui(action:'controlXYSequence', 
                    seq:'verse_pattern', 
                    dur:1.5, 
                    loop:true),
        style(label:'▶ Play', x:10, y:50)
      )"/>

<!-- Stop button -->
<g cue="button(
        trigger:ui(action:'controlXYSequenceStop'),
        style(label:'■ Stop', x:10, y:90)
      )"/>

Complete Animation Example: Verse-Chorus-Bridge

Here's a full composition workflow showing how to choreograph spatial animation:

<!-- ===== SETUP: Define the control pad ===== -->
<rect id="controlPad" x="100" y="100" width="600" height="400"
      fill="#111" stroke="#333"
      cue="controlXY(uid:mixer, handle:[dot1,dot2,dot3], label:true)"/>

<circle id="dot1" cx="0" cy="0" r="12" fill="#ff4444"/>
<circle id="dot2" cx="0" cy="0" r="12" fill="#44ff44"/>
<circle id="dot3" cx="0" cy="0" r="12" fill="#4444ff"/>

<!-- ===== PLAYHEAD TRIGGERS: Auto-save states ===== -->
<!-- Measure 1: Save intro position -->
<rect x="100" y="0" width="1" height="600"
      cue="ui(action:'controlXYSave', preset:'intro')"
      fill="transparent"/>

<!-- Measure 5: Save verse position -->
<rect x="500" y="0" width="1" height="600"
      cue="ui(action:'controlXYSave', preset:'verse')"
      fill="transparent"/>

<!-- Measure 13: Save chorus position -->
<rect x="1300" y="0" width="1" height="600"
      cue="ui(action:'controlXYSave', preset:'chorus')"
      fill="transparent"/>

<!-- Measure 21: Save bridge position -->
<rect x="2100" y="0" width="1" height="600"
      cue="ui(action:'controlXYSave', preset:'bridge')"
      fill="transparent"/>

<!-- ===== PLAYHEAD AUTOMATION: Recall with tweens ===== -->
<!-- M9: Tween to verse over 2 seconds -->
<rect x="900" y="0" width="2" height="600"
      cue="ui(action:'controlXYRecall', preset:'verse', dur:2, ease:'easeInOutSine')"
      fill="blue" opacity="0.4"/>

<!-- M17: Quick snap to chorus -->
<rect x="1700" y="0" width="2" height="600"
      cue="ui(action:'controlXYRecall', preset:'chorus', dur:0.5, ease:'easeOutQuad')"
      fill="green" opacity="0.4"/>

<!-- M25: Elastic bounce to bridge -->
<rect x="2500" y="0" width="2" height="600"
      cue="ui(action:'controlXYRecall', preset:'bridge', dur:3, ease:'easeOutElastic')"
      fill="purple" opacity="0.4"/>

<!-- ===== SEQUENCES: Complex multi-step animations ===== -->
<!-- M29: Define and play verse pattern -->
<rect x="2900" y="0" width="1" height="600"
      cue="ui(action:'controlXYDefineSequence', 
              name:'verse_pattern', 
              steps:'verse,intro,verse')"
      fill="transparent"/>

<rect x="2950" y="0" width="2" height="600"
      cue="ui(action:'controlXYSequence', 
              seq:'verse_pattern', 
              dur:1, 
              loop:false)"
      fill="yellow" opacity="0.4"/>

<!-- M40: Stop all automation, return to intro -->
<rect x="4000" y="0" width="2" height="600"
      cue="ui(action:'controlXYSequenceStop')"
      fill="red" opacity="0.5"/>

<rect x="4020" y="0" width="2" height="600"
      cue="ui(action:'controlXYRecall', preset:'intro', dur:4, ease:'easeInOutSine')"
      fill="cyan" opacity="0.4"/>

<!-- ===== MANUAL OVERRIDE: Buttons for live performance ===== -->
<!-- Performers can override automation at any time -->
<g cue="button(
        trigger:ui(action:'controlXYRecall', preset:'intro', dur:2),
        style(label:'⏮ Intro', x:10, y:500, width:80)
      )"/>

<g cue="button(
        trigger:ui(action:'controlXYRecall', preset:'verse', dur:1.5),
        style(label:'V Verse', x:100, y:500, width:80)
      )"/>

<g cue="button(
        trigger:ui(action:'controlXYRecall', preset:'chorus', dur:1),
        style(label:'C Chorus', x:190, y:500, width:80)
      )"/>

<g cue="button(
        trigger:ui(action:'controlXYSequenceStop'),
        style(label:'■ Stop Auto', x:280, y:500, width:100)
      )"/>

Binding to Score Elements: Creating Visual Animations

This is where spatial control becomes visual transformation.

Position Control

<!-- Dot X position controls rectangle X position -->
<rect id="box1"
      cue="scale(uid:box1, tx:mixer.dot1.x[-200,200])"/>

Size/Scale Control

<!-- Dot Y controls vertical scale -->
<rect id="box2"
      cue="scale(uid:box2, sy:mixer.dot1.y[0.5,2.0])"/>

Rotation Control

<!-- Map X position to rotation angle -->
<g id="spinner"
   cue="rotate(uid:spinner, values:mixer.dot1.x[0,360])"/>

Color Control

<!-- Y position controls hue -->
<circle id="colorDot"
        cue="color(uid:colorDot, hue:mixer.dot1.y[0,360])"/>

Opacity Control

<!-- X position fades element -->
<rect id="fader"
      cue="fade(uid:fader, opacity:mixer.dot1.x)"/>

Multi-Parameter Complex Animation

<!-- Dot1 controls rotation, Dot2 controls scale, Dot3 controls opacity -->
<g id="complexShape">
  <rect cue="rotate(uid:r1, values:mixer.dot1.x[0,360])
            scale(uid:s1, sx:mixer.dot2.x[0.5,2], sy:mixer.dot2.y[0.5,2])
            fade(uid:f1, opacity:mixer.dot3.y)"/>
</g>

Spatial Navigation

<!-- Two handles define start/end points of a loop region -->
<g cue="o2p(path:loopA, start:mixer.dot1.x, end:mixer.dot2.x)"/>

Advanced: Programmatic Animation via Console

For complex choreography, you can script animations directly:

Tweening Without Presets

// Move ALL handles to center over 2 seconds
window.controlXYPresets.tweenTo({ x: 0.5, y: 0.5 }, 2, 'easeInOutSine');

// Move to bottom-left corner
window.controlXYPresets.tweenTo(0, 0, 3, 'easeOutElastic');

// Multi-handle choreography (by index)
window.controlXYPresets.tweenTo([
  { x: 0.2, y: 0.8 },  // Handle 0
  { x: 0.5, y: 0.5 },  // Handle 1
  { x: 0.8, y: 0.2 }   // Handle 2
], 2, 'easeInOutBack');

Complex Sequences with Per-Step Timing

// Define sequence with variable timing
window.controlXYPresets.defineSequence('complex_dance', [
  { preset: 'position1', dur: 2 },
  { preset: 'position2', dur: 1 },
  { preset: 'position3', dur: 3 },
  { preset: 'position1', dur: 2 }
]);

// Play with custom options
window.controlXYPresets.playSequence('complex_dance', {
  loop: true,
  onStep: (step, preset) => console.log(`Now: ${preset}`)
});

Per-Handle Timing Control

// Staggered animation - each handle moves independently
window.controlXYPresets.recall('myPreset', {
  handles: {
    dot1: { dur: 2, delay: 0, ease: 'easeInOutSine' },
    dot2: { dur: 1.5, delay: 0.5, ease: 'easeOutQuad' },
    dot3: { dur: 1, delay: 1, ease: 'easeOutElastic' }
  }
});

Easing Functions

Choose from 15 easing functions to shape your animations:

Use by name or number in DSL:

<!-- By name -->
cue="ui(action:'controlXYRecall', preset:'state1', dur:2, ease:'easeOutElastic')"

<!-- By number -->
cue="ui(action:'controlXYRecall', preset:'state1', dur:2, ease:14)"

Complete DSL Action Reference

All controlXY preset actions work through ui(action:...) syntax.

1. Save Preset

<!-- Save all pads -->
ui(action:"controlXYSave", preset:"stateName")

<!-- Save specific pad -->
ui(action:"controlXYSave", preset:"stateName", uid:"pad1")

Examples:

<!-- Playhead trigger -->
<rect x="500" cue="ui(action:'controlXYSave', preset:'verse1')"/>

<!-- Button -->
<g cue="button(
        trigger:ui(action:'controlXYSave', preset:'myState'),
        style(label:'Save State', x:10, y:10)
      )"/>

2. Recall Preset

<!-- Instant -->
ui(action:"controlXYRecall", preset:"stateName")

<!-- With tween -->
ui(action:"controlXYRecall", preset:"stateName", dur:2, ease:"easeInOutSine")

Examples:

<!-- Playhead trigger with animation -->
<rect x="1000" 
      cue="ui(action:'controlXYRecall', preset:'chorus', dur:3, ease:'easeOutElastic')"/>

<!-- Button with quick snap -->
<g cue="button(
        trigger:ui(action:'controlXYRecall', preset:'breakdown', dur:0.5),
        style(label:'⚡ Breakdown', x:10, y:50)
      )"/>

3. Define Sequence

ui(action:"controlXYDefineSequence", name:"seqName", steps:"preset1,preset2,preset3")

Examples:

<!-- Define on load -->
<rect x="100" 
      cue="ui(action:'controlXYDefineSequence', 
              name:'intro_pattern', 
              steps:'a,b,c,a')"/>

<!-- Button to create sequence -->
<g cue="button(
        trigger:ui(action:'controlXYDefineSequence', 
                    name:'verse_loop', 
                    steps:'verse1,verse2,verse1'),
        style(label:'Create Loop', x:10, y:90)
      )"/>

4. Play Sequence

ui(action:"controlXYSequence", seq:"seqName", dur:2, ease:"easeInOutSine", loop:true)

Examples:

<!-- Auto-play when playhead hits -->
<rect x="2000" 
      cue="ui(action:'controlXYSequence', 
              seq:'intro_pattern', 
              dur:1.5, 
              loop:false)"/>

<!-- Button with looping -->
<g cue="button(
        trigger:ui(action:'controlXYSequence', 
                    seq:'verse_loop', 
                    dur:2, 
                    loop:true),
        style(label:'▶ Loop Verse', x:10, y:130)
      )"/>

5. Stop Sequence

ui(action:"controlXYSequenceStop")

Examples:

<!-- Playhead trigger -->
<rect x="4000" cue="ui(action:'controlXYSequenceStop')"/>

<!-- Button -->
<g cue="button(
        trigger:ui(action:'controlXYSequenceStop'),
        style(label:'■ Stop', x:10, y:170)
      )"/>

OSC Output (Bonus Feature)

While spatial animation is the primary use case, all handle movements can simultaneously control external software.

Enable OSC

<rect id="pad1" 
      cue="controlXY(uid:synth, handle:dot1, osc:true)"/>

Custom Address & Throttle

<rect id="pad1"
      cue="controlXY(uid:synth, handle:dot1, osc:50, oscAddr:'max/xy')"/>

OSC Messages Sent

Single handle:

/controlXY/synth 0.42 0.87

Multiple handles:

/controlXY/synth/dot1 0.42 0.87
/controlXY/synth/dot2 0.15 0.63

Custom address:

/max/xy 0.42 0.87

Works with Max/MSP, Pure Data, SuperCollider, TouchOSC, and any OSC-compatible software.

Preset Panel UI

Opening the Panel

1. Click ⚙ button in top-right of any controlXY pad
2. Keyboard: Alt+Shift+P
3. Console: window.controlXYPresetUI.toggle()

Panel Features

• Save Section: Type name, click 💾
• Presets List: Click ▶ to recall, 🗑 to delete
• Recall Options: Set duration (seconds) and easing
• Sequences: Play/stop defined sequences
• Import/Export: Save/load preset files

Keyboard Shortcut

Alt+Shift+P toggles the panel (changed from Ctrl+Shift+P to avoid Chrome conflict).

Storage & Persistence

Presets auto-save to scores/<project>/controlxy-presets.json:

{
  "presets": {
    "intro": {
      "pad1": {
        "dot1": { "x": 0.25, "y": 0.75 },
        "dot2": { "x": 0.80, "y": 0.30 }
      }
    }
  },
  "sequences": {
    "verse_pattern": ["verse1", "verse2", "verse1"]
  },
  "updatedAt": 1704067200000,
  "projectId": "myProject"
}

Loaded automatically when project loads.

CSS Styling

Handle Classes

.controlxy-handle {
  cursor: grab;
  transition: opacity 0.15s;
}

.controlxy-handle:hover {
  opacity: 0.8;
}

.controlxy-handle--active {
  cursor: grabbing;
  filter: drop-shadow(0 0 10px rgba(100, 200, 255, 0.9));
}

Label Styling

.controlxy-label {
  font-family: 'SF Mono', 'Monaco', monospace;
  font-size: 11px;
  fill: #fff;
  filter: drop-shadow(0 1px 2px rgba(0, 0, 0, 0.8));
}

Toggle Button

The ⚙ button is automatically added to each pad's top-right corner. Style via:

.controlxy-toggle-btn {
  background: rgba(30, 30, 30, 0.9);
  border: 1px solid rgba(255, 255, 255, 0.3);
}

.controlxy-toggle-btn:hover {
  background: rgba(30, 30, 30, 1);
  transform: scale(1.1);
}

Complete API Reference

JavaScript Console API

// ===== PRESETS =====
controlXYPresets.save(name, uidFilter?)
controlXYPresets.recall(name, options?)
controlXYPresets.delete(name)
controlXYPresets.list()                    // Returns array of preset names
controlXYPresets.get(name)                 // Returns preset data

// ===== TWEENING =====
controlXYPresets.tweenTo(positions, dur, ease)
controlXYPresets.tweenTo({ x: 0.5, y: 0.5 }, 2)        // All handles
controlXYPresets.tweenTo(0.8, 0.3, 1.5, 'easeOutElastic')  // Shorthand
controlXYPresets.stopAllTweens()

// ===== SEQUENCES =====
controlXYPresets.defineSequence(name, steps)
controlXYPresets.playSequence(name, options)
controlXYPresets.stopSequence()
controlXYPresets.getActiveSequence()       // Returns current sequence info

// ===== PERSISTENCE =====
controlXYPresets.init(projectId)           // Called automatically
controlXYPresets.export()                  // Returns JSON string
controlXYPresets.import(json, merge?)
controlXYPresets.importFromProject(projectId, merge?)

// ===== UI =====
controlXYPresetUI.show()
controlXYPresetUI.hide()
controlXYPresetUI.toggle()
controlXYPresetUI.refresh()

Compositional Patterns

Pattern 1: Verse-Chorus Automation

1. Save state at verse start
2. Save state at chorus start  
3. Tween between them at transitions
4. Add button overrides for live performance

Pattern 2: Looping Textures

1. Define 3-4 related states
2. Create sequence with varied timing
3. Loop sequence with `loop:true`
4. Bind to visual parameters for evolving texture

Pattern 3: Build-Tension-Release

1. Start at calm state (center, low values)
2. Sequence through increasingly tense positions
3. Climax: rapid sequence or elastic bounce
4. Release: slow tween back to calm

Pattern 4: Call-Response

1. Manual control for "call" phrase
2. Automated sequence for "response"
3. Alternate between live and programmed

Pattern 5: Polytemporal Layers

1. Multiple pads, each with own sequence
2. Different loop lengths (3, 5, 7 steps)
3. Creates phasing/evolving relationships
4. Each pad controls different visual layer

Technical Notes

• controlXY is a control-plane cue, not temporal
• Registered during assignCues(), not via playhead
• Handles initialized at center of bounds
• Uses Pointer Events API (works with touch, mouse, stylus)
• All tweening uses requestAnimationFrame for smooth 60fps
• Event propagation stopped to prevent score dragging
• Multi-touch: each pointer controls nearest available handle

Summary

controlXY fundamentally transforms the score from static notation into dynamic canvas. By combining:

1. Spatial control surfaces (the pads)
2. Parameter binding (connecting space to parameters)
3. Preset/sequence system (programming choreography)
4. Live performance (manual override)

...composers gain the ability to create complex, evolving visual/sonic animations directly within the score itself. The score becomes both instrument and notation, collapsing the traditional divide between composition and performance.

As a bonus, OSC output allows the same spatial gestures to control external synthesis and media systems, making controlXY a unified interface for all aspects of a multimedia performance.

Score-as-instrument. Instrument-as-score.

Quick Start Checklist

1. Add CSS: <link rel="stylesheet" href="controlxy-preset-ui.css">
2. Define pad: <rect cue="controlXY(uid:pad1, handle:dot1)"/>
3. Open UI: Press Alt+Shift+P or click ⚙
4. Save states: Move handles, name them, click 💾
5. Animate: Use ui(action:'controlXYRecall', ...) on playhead triggers
6. Choreograph: Define sequences, play/loop them
7. Bind: Connect handle positions to visual parameters
8. Perform: Override automation with buttons or manual control

Now go create some animated score-instruments!

Tip: use ← → or ↑ ↓ to navigate the docs