MP4E Studio - Interactive Video Editor

Overview#

Actions are the building blocks of interactivity in MP4E. They can be triggered by user interactions, rules, or executed programmatically from plugins. Actions control video playback, UI elements, variables, and more.

Playback

Navigation

Variable

Dialog

Plugin

Media

Integration

Interaction

Advanced

Action Structure#

Every action has a type property and action-specific parameters:

typescript

// Actions are JSON objects executed by the runtime (Studio + Player).
// Canonical fields live in player/src/shared/actionExecutor.ts
interface Action {
  type: string
  // action-specific fields...
}

// Example: Seek to 30 seconds
const action: Action = { type: 'seek', time: 30 }

// Example: Set a variable
const action: Action = { type: 'setVariable', variableId: 'score', value: 100 }

Template Variables

Many action parameters support template variables using double curly braces. For example, {{object.id}} or {{event.data.price}}.

Playback Actions#

Control video playback state, position, and speed.

seek

Jump to a specific time or frame in the video

Parameters

timenumberrequired- Target time in seconds

frameintegerrequired= 0- Target frame number

Example

{"type":"seek","time":15.5,"frame":450}

pause

Pause video playback

Parameters

reasonstring- Optional reason for pause

Example

{"type":"pause","reason":"Please review the product details"}

play

Resume video playback

Example

{"type":"play"}

togglePlayback

Toggle between play and pause

Example

{"type":"togglePlayback"}

toggleFullscreen

Toggle fullscreen mode

Example

{"type":"toggleFullscreen"}

setPlaybackRate

Change video playback speed

Parameters

ratenumberrequired= 1- Playback speed multiplier

Example

{"type":"setPlaybackRate","rate":1.5}

setVolume

Change video volume level

Parameters

volumenumberrequired= 1- Volume level (0-1)

Example

{"type":"setVolume","volume":0.5}

mute

Mute video audio

Example

{"type":"mute"}

unmute

Unmute video audio

Example

{"type":"unmute"}

seekRelative

Seek relative to current position

Parameters

offsetSecondsnumber= 5- Seconds to skip (can be negative)

offsetFramesinteger- Frames to skip (can be negative)

Example

{"type":"seekRelative","offsetSeconds":5,"offsetFrames":150}

setPlaybackStart

Set the earliest time viewers can seek to

Parameters

timenumberrequired= 0- Start time in seconds

Example

{"type":"setPlaybackStart","time":0}

setPlaybackEnd

Set the latest time viewers can play to

Parameters

timeany= null- End time in seconds (null to remove restriction)

Example

{"type":"setPlaybackEnd"}

Navigate within the video or to external URLs.

openUrl

Navigate to a URL in specified target

Parameters

urlstringrequired= ""- URL to navigate to

target"blank" | "self" | "parent" | "top"required= "blank"- Browser target for navigation

Example

{"type":"openUrl","url":"https://example.com","target":"blank"}

goToScene

Navigate to a specific scene

Parameters

sceneIdstringrequired= ""- Target scene ID

seekToStartboolean= true- Seek to scene start time

timenumber- Optional time offset within scene

Example

{"type":"goToScene","sceneId":"scene_intro"}

UI Actions#

Control overlays, layers, plugins, and visual elements.

showOverlay

Make an overlay visible

Parameters

overlayIdstringrequired= ""- ID of overlay to show

Example

{"type":"showOverlay","overlayId":"product-details"}

hideOverlay

Make an overlay invisible

Parameters

overlayIdstringrequired= ""- ID of overlay to hide

Example

{"type":"hideOverlay","overlayId":"product-details"}

setOverlayStyle

Change an overlay's position, size, transform, or visual style at runtime. Optionally animate the transition.

Parameters

overlayIdstringrequired- Target overlay

positionobject- Position { x, y, xUnit, yUnit }

sizeobject- Size { width, height, widthUnit, heightUnit }

transformobject- Transform { scale, rotation, flipX, flipY }

styleobject- Visual Style { opacity, borderRadius, boxShadow, filter, backdropFilter, backgroundColor, zIndex, pointerEvents }

animateobject- Animation { duration, easing, delay }

Example

{
  "type": "setOverlayStyle",
  "overlayId": "card_overlay",
  "style": { "opacity": 0.8, "borderRadius": 12, "boxShadow": "0 4px 12px rgba(0,0,0,0.3)" },
  "transform": { "scale": 1.1, "rotation": 5 },
  "animate": { "duration": 300, "easing": "ease-out" }
}

resetOverlayStyle

Reset an overlay to its original metadata-defined style

Parameters

overlayIdstring- Target overlay (leave empty to reset all)

Example

{"type":"resetOverlayStyle"}

pluginHide

Hide a plugin overlay (stealth mode). The plugin script continues running and can listen to events. Routed through the engine for sandbox/gating checks.

Parameters

pluginIdstringrequired= ""- ID of the plugin to hide

Example

{"type":"pluginHide","pluginId":""}

pluginShow

Show a previously hidden plugin overlay. Restores visibility after a pluginHide action.

Parameters

pluginIdstringrequired= ""- ID of the plugin to show

Example

{"type":"pluginShow","pluginId":""}

showCustomModal

Display a modal plugin dialog, optionally pausing the video

Parameters

pluginIdstringrequired= ""- ID of the modal plugin to display

pauseVideoboolean= true- Whether to pause video while modal is open

configobject- Override plugin config values

inputsobject- Override plugin input values

showBackdropboolean= true- Whether to show a backdrop behind the modal

Example

{
  "type": "showCustomModal",
  "pluginId": "plugin_checkout",
  "pauseVideo": true,
  "config": { "title": "Complete Purchase" }
}

showCustomOverlay

Display a custom plugin as a positioned overlay on the video

Parameters

pluginIdstringrequired= ""- ID of the overlay plugin to display

overlayIdstring= ""- Overlay ID (if different from plugin ID)

positionany- Screen position — named position (e.g., 'bottom-right') or { x, y } coordinates

marginnumber- Margin from edges in pixels

configobject- Override plugin config values

Example

{ "type": "showCustomOverlay", "pluginId": "plugin_tooltip", "position": "bottom-right", "margin": 16 }

hideCustomOverlay

Hide a custom overlay plugin

Parameters

pluginIdstringrequired- ID of the overlay plugin to hide

overlayIdstring= ""- Overlay ID (if different from plugin ID)

Example

{"type":"hideCustomOverlay","pluginId":"pluginId_value"}

highlightObject

Apply visual highlight effect to detected object

Parameters

objectIdstringrequired= ""- ID of object to highlight

durationintegerrequired= 1000- Highlight duration in milliseconds

style"glow" | "pulse" | "outline" | "shadow"required= "glow"- Highlight visual style

Example

{"type":"highlightObject","objectId":"obj_1","duration":1000,"style":"glow"}

showNotification

Display toast notification

Parameters

titlestring= ""- Notification title (optional)

messagestringrequired= ""- Notification message

variant"info" | "success" | "warning" | "error"required= "info"- Notification type

durationintegerrequired= 3000- Display duration in milliseconds

animation"slide" | "fade" | "bounce" | "none"= "slide"- Entry/exit animation style

backgroundColorstring- Custom background color

textColorstring- Custom text color

borderColorstring- Custom border color

Example

{ "type": "showNotification", "message": "Added to cart!", "variant": "success", "duration": 2000, "position": "top-right" }

activateLayer

Force a layer to be active

Parameters

layerIdstringrequired= ""- Layer ID

Example

{"type":"activateLayer","layerId":"layer_1"}

deactivateLayer

Force a layer to be inactive

Parameters

layerIdstringrequired= ""- Layer ID

Example

{"type":"deactivateLayer","layerId":"layer_1"}

toggleOverlay

Toggle overlay visibility

Parameters

overlayIdstringrequired= ""- Overlay ID

Example

{"type":"toggleOverlay","overlayId":"overlay_1"}

enterFullscreen

Enter fullscreen mode

Example

{"type":"enterFullscreen"}

exitFullscreen

Exit fullscreen mode

Example

{"type":"exitFullscreen"}

focus

Set focus to an overlay

Parameters

overlayIdstringrequired- Overlay ID to focus

Example

{"type":"focus","overlayId":"overlay_1"}

Variable Actions#

Modify variables and manage state. MP4E supports 15 variable types, each with dedicated control actions.

setState

Set a key-value pair in engine state

Parameters

keystringrequired= ""- State key

valueanyrequired= ""- State value

Example

{"type":"setState","key":"quiz_score","value":5}

setVariable

Set or modify a variable value

Parameters

variableIdstringrequired= ""- Variable ID

valueanyrequired= ""- New value

operation"set" | "increment" | "decrement" | "toggle" | "append"= "set"- Operation type

Example

{"type":"setVariable","variableId":"","value":""}

timerControl

Start, stop, or reset a timer variable

Parameters

variableIdstringrequired= ""- Timer variable ID

operation"start" | "stop" | "reset"required= "start"- Timer operation

resetValuenumber- Value to reset to

Example

{"type":"timerControl","variableId":"","operation":"start","resetValue":0}

counterControl

Increment, decrement, or reset a counter

Parameters

variableIdstringrequired- Counter variable ID

operation"increment" | "decrement" | "reset"required- Counter operation

amountnumber- Amount to increment/decrement

Example

{"type":"counterControl","variableId":"variableId_value","operation":"increment","amount":1}

stateTransition

Transition a state machine variable

Parameters

variableIdstringrequired- State variable ID

targetStatestringrequired- Target state name

Example

{"type":"stateTransition","variableId":"variableId_value","targetState":"targetState_value"}

arrayControl

Manipulate array variable

Parameters

variableIdstringrequired- Array variable ID

operation"push" | "pop" | "clear" | "removeAt" | "insertAt"required- Array operation

valueany- Value to push/insert

indexinteger- Index for removeAt/insertAt

Example

{"type":"arrayControl","variableId":"variableId_value","operation":"push","index":0}

accumulatedControl

Add to or reset accumulator variable

Parameters

variableIdstringrequired- Accumulator variable ID

operation"add" | "reset"required- Accumulator operation

valuenumber- Value to add

Example

{"type":"accumulatedControl","variableId":"variableId_value","operation":"add","value":1}

objectControl

Manipulate object variable

Parameters

variableIdstringrequired= ""- Object variable ID

operation"set" | "setField" | "deleteField" | "merge"required= "set"- Object operation

valueobject= [object Object]- Object value for set

fieldstring- Field name for setField/deleteField

fieldValueany- Field value for setField

partialobject- Partial object for merge

Example

{
  "type": "objectControl",
  "variableId": "userProfile",
  "operation": "setField",
  "field": "email",
  "fieldValue": "user@example.com"
}

mapControl

Manipulate map variable

Parameters

variableIdstringrequired= ""- Map variable ID

operation"set" | "setKey" | "deleteKey" | "incrementKey" | "clear"required= "setKey"- Map operation

valueobject- Map value for set

keystring= ""- Key for setKey/deleteKey/incrementKey

keyValueany= ""- Value for setKey

amountnumber- Amount for incrementKey

Example

{
  "type": "mapControl",
  "variableId": "voteCounts",
  "operation": "incrementKey",
  "key": "option_a",
  "amount": 1
}

setControl

Manipulate set variable

Parameters

variableIdstringrequired= ""- Set variable ID

operation"set" | "add" | "remove" | "toggle" | "clear"required= "add"- Set operation

valueany[]- Array value for set

itemany= ""- Item for add/remove/toggle

Example

{"type":"setControl","variableId":"","operation":"add"}

dateControl

Manipulate date variable

Parameters

variableIdstringrequired= ""- Date variable ID

operation"set" | "setNow" | "clear" | "add" | "subtract"required= "setNow"- Date operation

timestampnumber- Unix timestamp for set

daysnumber- Days

hoursnumber- Hours

minutesnumber- Minutes

secondsnumber- Seconds

millisecondsnumber- Milliseconds

Example

{"type":"dateControl","variableId":"","operation":"setNow"}

listObjectControl

Manipulate list variable containing objects

Parameters

variableIdstringrequired= ""- List variable ID

operation"findByField" | "updateByField" | "removeByField" | "sortByField"required= "findByField"- List object operation

matchFieldstring= ""- Field name to match

matchValueany= ""- Value to match

updatesobject- Fields to update for updateByField

ascendingboolean- Sort order for sortByField

Example

{
  "type": "listObjectControl",
  "variableId": "cartItems",
  "operation": "updateByField",
  "matchField": "sku",
  "matchValue": "SKU-001",
  "updates": { "quantity": 3 }
}

Dialog Actions#

Show system dialogs and prompts.

alert

Display an alert dialog to the viewer

Parameters

titlestring= ""- Dialog title

messagestringrequired= ""- Dialog message

pauseVideoboolean= true- Whether to pause video while dialog is open

buttonTextstring= "OK"- Button text

Example

{"type":"alert","message":""}

confirm

Display a confirmation dialog with confirm/cancel actions

Parameters

titlestring= ""- Dialog title

messagestringrequired= ""- Dialog message

confirmTextstring= "Yes"- Confirm button text

cancelTextstring= "No"- Cancel button text

pauseVideoboolean= true- Whether to pause video while dialog is open

onConfirmAction[]- Actions to execute on confirm

onCancelAction[]- Actions to execute on cancel

Example

{
  "type": "confirm",
  "title": "Remove Item",
  "message": "Remove this item from cart?",
  "onConfirm": [{ "type": "arrayControl", "variableId": "cart", "operation": "pop" }]
}

prompt

Display a text input prompt dialog and store the result in a variable

Parameters

titlestring= ""- Dialog title

messagestringrequired= ""- Dialog message

variableIdstringrequired= ""- Variable ID to store the input value

placeholderstring= ""- Input placeholder text

inputType"text" | "number" | "email" | "url" | "password"= "text"- HTML input type

pauseVideoboolean= true- Whether to pause video while dialog is open

onSubmitAction[]- Actions to execute on submit

onCancelAction[]- Actions to execute on cancel

defaultValuestring= ""- Default input value

Example

{"type":"prompt","message":"","variableId":""}

Plugin Actions#

Invoke plugin methods and service plugin actions.

callPlugin

Invoke a method on a plugin

Parameters

pluginNamestringrequired= ""- Plugin name

methodstringrequired= ""- Method name to call

paramsobject- Method parameters

Example

{
  "type": "callPlugin",
  "pluginName": "shopify",
  "method": "addToCart",
  "params": { "productId": "12345", "quantity": 1 }
}

pluginAction

Execute a service plugin action

Parameters

pluginIdstringrequired= ""- Service plugin ID

actionIdstringrequired= ""- Action ID from plugin manifest

configobject- Action configuration

Example

{
  "type": "pluginAction",
  "pluginId": "cart",
  "actionId": "addItem",
  "config": {
    "objectId": "{{object.id}}",
    "productName": "{{object.data.title}}",
    "price": "{{object.data.price}}"
  }
}

Media Actions#

Control audio tracks, subtitles, and embedded media playback.

setAudioTrack

Switch to different audio track

Parameters

trackIdstring- Audio track ID

trackIndexinteger- Audio track index

linkedSubtitles"true" | "false"= true- Auto-switch subtitles to match

Example

{"type":"setAudioTrack","trackId":"audio_en","trackIndex":0}

setAudioVolume

Set audio volume level

Parameters

volumenumberrequired= 1- Volume level (0-1)

Example

{"type":"setAudioVolume","volume":0.5}

setAudioMuted

Mute or unmute audio

Parameters

muted"true" | "false"required= true- Mute state

Example

{"type":"setAudioMuted","muted":true}

setSubtitlePosition

Reposition subtitle overlay

Parameters

overlayIdstringrequired= ""- Subtitle overlay ID

positionobjectrequired- Position (JSON) { x, y, xUnit, yUnit }

Example

{
  "type": "setSubtitlePosition",
  "overlayId": "subtitle-overlay",
  "position": {
    "x": 50,
    "y": 80,
    "xUnit": "%",
    "yUnit": "%"
  }
}

setSubtitlesEnabled

Enable or disable subtitles

Parameters

enabled"true" | "false"required= true- Subtitle enabled state

Example

{"type":"setSubtitlesEnabled","enabled":true}

setSubtitleTrack

Switch to different subtitle track

Parameters

trackIdstring- Subtitle track ID

Example

{"type":"setSubtitleTrack","trackId":"subtitle_en"}

setActiveSubtitleTrack

Set the active subtitle track

Parameters

trackIdstring- Subtitle track ID

Example

{"type":"setActiveSubtitleTrack","trackId":"subtitle_en"}

setActiveSubtitleTracks

Set multiple active subtitle tracks

Parameters

trackIdsstring[]required- Array of subtitle track IDs

Example

{"type":"setActiveSubtitleTracks","trackIds":["subtitle_en","subtitle_es"]}

downloadMedia

Download a media file

Parameters

mediaIdstringrequired- Media asset ID

filenamestring- Download filename

Example

{"type":"downloadMedia","mediaId":"brochure_pdf","filename":"Product Brochure.pdf"}

openMedia

Open an embedded media file in a viewer modal or new tab

Parameters

mediaRefstringrequired- Media reference (media:// URL or raw media ID)

mode"modal" | "new-tab"= "modal"- How to display the media

titlestring- Title for the viewer

Example

{"type":"openMedia","mediaRef":"media://floor_plan"}

playMedia

Play embedded media in fullscreen, picture-in-picture, or audio mode

Parameters

mediaRefstringrequired- Media reference (media:// URL or raw media ID)

mediaVideoIdstring- Media video ID from metadata.mediaVideos (for tracked objects/events)

mode"fullscreen-replace" | "pip" | "audio"= "pip"- Playback mode: fullscreen-replace (replaces main video), pip (overlay video), audio (background audio)

onEnd"dismiss" | "pause"= "dismiss"- What happens when media ends

positionobject- For mode=pip: position and size of PiP overlay

pauseMainboolean= false- Pause main video during playback

controlsboolean= true- Show playback controls (PiP mode)

autoCloseboolean= true- Close overlay when media ends

volumenumber- Playback volume (0-1)

loopboolean= false- Loop audio playback (audio mode)

Example

{
  "type": "playMedia",
  "mediaRef": "media://product_demo",
  "mode": "pip",
  "position": { "x": 70, "y": 10, "width": 25 },
  "pauseMain": true,
  "autoClose": true
}

Integration Actions#

Communicate with external services, track analytics, and synchronize state.

trackEvent

Send analytics event with properties

Parameters

eventNamestringrequired= ""- Name of the event to track

propertiesobject- Event properties

Example

{
  "type": "trackEvent",
  "eventName": "product_viewed",
  "properties": {
    "productId": "{{object.id}}",
    "productName": "{{object.userLabel}}",
    "timestamp": "{{video.currentTime}}"
  }
}

api

Make HTTP API request

Parameters

method"GET" | "POST" | "PUT" | "DELETE"required= "GET"- HTTP method

urlstringrequired= ""- API endpoint URL

headersobject- HTTP headers

bodyany- Request body

storeResultInstring- Variable ID to store response

onSuccessAction[]- Actions to execute on success

onErrorAction[]- Actions to execute on error

Example

{
  "type": "api",
  "method": "POST",
  "url": "https://api.example.com/track",
  "body": { "event": "video_complete", "videoId": "{{video.id}}" },
  "storeResultIn": "apiResponse"
}

syncState

Synchronize state with other viewers

Parameters

keysstring[]required- Variable IDs to sync

channelstring- Sync channel name

Example

{"type":"syncState","keys":[]}

Interaction Actions#

Programmatically simulate user interactions on objects and overlays.

triggerInteraction

Programmatically simulate user interactions (click, hover, etc.) on objects, overlays, or plugin inner elements. Supports label-based fuzzy matching for objects and text-based element matching within plugin iframes.

Parameters

target"object" | "overlay"- Target type to interact with

targetIdstring- Direct ID of the target object or overlay

labelstring- Object label for fuzzy matching against visible objects (case-insensitive)

textstring- Text to search for within plugin iframe DOM elements (for clickText interaction)

interaction"click" | "doubleClick" | "hover" | "hoverEnd" | "clickText"required= "click"- Type of interaction to simulate

sourcestring- Origin of the interaction for analytics tracking (e.g., mouse, touch, keyboard, voice, plugin:pluginId, rule:ruleId)

Example

{
  "type": "triggerInteraction",
  "target": "object",
  "label": "red chair",
  "interaction": "click"
}

Advanced Actions#

Tracking visualization, event control, and debugging utilities.

showTracking

Display object tracking visualization

Parameters

target"triggeringObject" | "all" | "group" | "object"required= "object"- Target to visualize

targetIdstring= ""- Group or object ID

styleobject- Style (JSON) { fillColor, strokeColor, strokeWidth, opacity, animated }

durationinteger= 3000- Duration in milliseconds

zIndexinteger= 100- Render order

Example

{
  "type": "showTracking",
  "mode": "polygon",
  "target": "object",
  "targetId": "obj_123",
  "style": {
    "strokeColor": "#00ffff",
    "strokeWidth": 2,
    "opacity": 0.8,
    "animated": "shimmer"
  },
  "duration": 3000
}

hideTracking

Hide object tracking visualization

Parameters

target"all" | "group" | "object"required= "all"- Target to hide

targetIdstring- Group or object ID

Example

{"type":"hideTracking","target":"all","targetId":"group_shoppable"}

emitEvent

Emit a custom event for other rules to listen to

Parameters

eventNamestringrequired= ""- Custom event name

dataobject- Event data payload

Example

{"type":"emitEvent","eventName":"myCustomEvent"}

log

Log message to console for debugging

Parameters

messagestringrequired= ""- Log message

level"debug" | "info" | "warn" | "error"= "info"- Log level

dataobject- Additional data to log

Example

{"type":"log","message":"Debug: value is {{myVar}}"}

wait

Delay execution of subsequent actions

Parameters

durationMsintegerrequired= 1000- Wait duration in milliseconds

Example

{"type":"wait","durationMs":1000}

resetEvents

Reset event occurrence flags

Parameters

eventNamesstring[]- Event names to reset

Example

{"type":"resetEvents"}

Action Chaining

Actions can be chained in arrays. They execute sequentially, with wait actions creating delays between them. This enables complex multi-step behaviors.

Chained Actions Example

{
  "actions": [
    { "type": "pause" },
    { "type": "showOverlay", "overlayId": "overlay_quiz" },
    { "type": "wait", "durationMs": 500 },
    { "type": "timerControl", "variableId": "quizTimer", "operation": "start" },
    { "type": "trackEvent", "eventName": "quiz_started" }
  ]
}

Tracking + Replacement Zone Pattern

Combine tracking visualization with replacement zones for interactive product selection workflows:

Interactive Selection Example

{
  "actions": [
    { "type": "showTracking", "mode": "polygon", "target": "triggeringObject", "style": { "animated": "shimmer" }, "duration": 2000 },
    { "type": "showOverlay", "overlayId": "fabric_picker" },
    { "type": "setVariable", "variableId": "selectedObjectId", "value": "{{object.id}}" }
  ]
}

Actions Reference

Overview#

Action Structure#

Playback Actions#

Navigation Actions#

UI Actions#

Variable Actions#

Dialog Actions#

Plugin Actions#

Media Actions#

Integration Actions#

Interaction Actions#

Advanced Actions#