Plugin System

Build interactive plugins with HTML, CSS, and JavaScript. 6 plugin types across 12 categories.

Overview#

Plugins add interactive UI elements to your video. Each plugin is a self-contained component with HTML, CSS, and JavaScript that runs in a sandboxed iframe. Plugins communicate with the player through the mp4e bridge API.

You can create plugins directly in the Studio Plugin Editor (a built-in code editor with live preview), install them from the Marketplace, or build them externally and import them.

6
Plugin Types
12
Categories
65+
Bridge API Methods

Plugin Types#

MP4E has 6 plugin types, each designed for different interaction patterns:

TypeDescriptionDefault SizeHas UI
overlayFixed position UI element placed in video layers400 x 300Yes
object-displayShows when user interacts with detected objects (hover, click)300 x 200Yes
modalFullscreen dialog that pauses video playback420 x 600Yes
subtitleTime-synced subtitle text with special rendering800 x 100Yes
controlsVideo player controls (play/pause/seek)400 x 80Yes
serviceAlways-alive plugin that provides custom actions and variablesHiddenNo (background)

Overlay

Fixed position UI element placed in video layers

Trigger
showCustomOverlay action or timeline
Position
Fixed screen position (top-left, bottom-right, etc.)
Video
Continues playing
Use Case
CTAs, persistent buttons, banners, notifications
Overlay example
// Show overlay via action
mp4e.executeActions([{
  type: 'showCustomOverlay',
  pluginId: 'plugin_cta',
  position: 'bottom-right',
  margin: 16
}]);

// Hide overlay
mp4e.executeActions([{
  type: 'hideCustomOverlay',
  pluginId: 'plugin_cta'
}]);

Object Display

Shows when user interacts with detected objects (hover, click)

Trigger
User clicks/hovers a detected object
Position
Relative to object or fixed screen position
Video
Continues playing
Use Case
Product cards, tooltips, info popups
Object Display example
// Configured in objectGroup displaySettings:
{
  "displaySettings": {
    "eventBindings": [{
      "event": "clicked",
      "enabled": true,
      "display": {
        "pluginType": "mp4e:product-card",
        "config": {
          "title": "{{object.userLabel}}",
          "price": "{{object.data.price}}"
        },
        "position": { "position": "above", "offsetY": 10 }
      }
    }]
  }
}

Fullscreen dialog that pauses video playback

Trigger
showCustomModal action
Position
Centered with backdrop
Video
Pauses (configurable)
Use Case
Forms, confirmations, checkout, detailed views
Modal example
// Show modal from JavaScript
mp4e.executeActions([{
  type: 'showCustomModal',
  pluginId: 'plugin_checkout',
  pauseVideo: true,
  config: { title: 'Complete Purchase' }
}]);

// Close modal from inside the plugin
mp4e.close();

Subtitle

Time-synced subtitle text with special rendering

Trigger
Active subtitle track with word-level timing
Position
Bottom of video (full-width)
Video
Continues playing
Use Case
Custom subtitle renderers, karaoke, boxed words
Subtitle example
// Subtitle plugins receive word-level timing data:
// - Words with start/end times for highlighting
// - Current word index for karaoke effects
// Access via mp4e.getData() or the data context

// Listen for word clicks
mp4e.onSubtitleWordClick((wordData) => {
  console.log('Clicked word:', wordData.text);
});

Controls

Video player controls (play/pause/seek)

Trigger
Always visible when video is loaded
Position
Bottom-docked, full-width
Video
Controls playback directly
Use Case
Custom play/pause/seek UI, progress bars, volume controls
Controls example
// Controls plugin receives video state via events:
mp4e.on('mp4e:video:timeupdate', (state) => {
  updateProgressBar(state.currentTime / state.duration);
});

mp4e.on('mp4e:video:play', () => setPlayIcon('pause'));
mp4e.on('mp4e:video:pause', () => setPlayIcon('play'));

// Control playback
document.getElementById('play-btn')
  .addEventListener('click', () => mp4e.playVideo());

Servicebackground

Always-alive plugin that provides custom actions and variables

Trigger
Always running in background (no UI)
Position
Hidden (0x0)
Video
No effect
Use Case
Cart state, analytics, API integrations, shared logic
Service example
// Service plugins provide variables, actions, and events
// to other plugins — no visible UI.

// Example: Shopping cart service
// Variables: cart:items, cart:itemCount, cart:total
// Actions: addItem, removeItem, clearCart
// Events: cartUpdated, itemAdded

// Other plugins call service actions:
mp4e.executeActions([{
  type: 'pluginAction',
  pluginId: 'cart',
  actionId: 'addItem',
  config: { productId: '{{object.id}}', price: '{{object.data.price}}' }
}]);

Plugin Editor#

The Studio includes a built-in Plugin Editor — a full code editor with live preview for creating and editing plugins. Open it from any overlay's settings.

The editor has a two-panel layout with 12 tabs you can arrange between the left and right panels:

Code
HTML
Template with {{variable}} interpolation. Supports Mustache syntax for conditionals and loops.
CSS
Scoped styles. Use CSS custom properties for theming and --s/--scale for responsive sizing.
Script
JavaScript code that runs in the sandboxed iframe. Access the mp4e bridge API here.
Data
Inputs
Define configuration fields (configSchema) using JSON Schema. The Studio auto-generates form UI from this.
Outputs
Declare variables the plugin can write to. Maps plugin events to setVariable actions.
Emits
Define custom events the plugin can emit. These become available as rule triggers.
Capabilities
Actions
Register actions other plugins or rules can call via pluginAction.
Variables
Auto-created project variables when plugin is installed. Prefixed with plugin namespace.
Display
Configure display settings: position, animation, auto-close behavior.
Meta
Settings
Plugin metadata: name, version, type, category, author, license, tags.
Preview
Live iframe preview of your plugin. Changes update in real-time.
API Docs
Quick reference for the mp4e bridge API, available right inside the editor.
Two-Panel Workflow
Hold Cmd/Ctrl and click a tab to open it in the right panel. This lets you edit HTML on the left while seeing the live Preview on the right, or write Script while referencing the API Docs.

Plugin Structure#

Every plugin consists of HTML, CSS, JavaScript, and a configuration schema:

Complete plugin structure
{
  id: 'plugin_product_card',
  name: 'Product Card',
  type: 'object-display',         // overlay | object-display | modal | subtitle | controls | service

  // Content — the actual plugin code
  content: {
    html: '<div class="card"><h3>{{object.userLabel}}</h3></div>',
    css: '.card { padding: 16px; background: white; border-radius: 8px; }',
    script: 'const config = mp4e.getConfig(); /* ... */'
  },

  // Configuration schema — generates form UI in Studio
  configSchema: {
    type: 'object',
    properties: {
      showPrice: { type: 'boolean', title: 'Show Price', default: true },
      accentColor: { type: 'string', format: 'color', title: 'Accent Color', default: '#3b82f6' }
    }
  },

  // Events the plugin can emit (available as rule triggers)
  emits: {
    purchased: {
      label: 'Item Purchased',
      description: 'Fired when user completes purchase',
      dataSchema: { type: 'object', properties: { productId: { type: 'string' } } }
    }
  },

  // Actions other plugins/rules can call on this plugin
  actions: [
    { id: 'highlight', label: 'Highlight', description: 'Flash highlight effect' }
  ],

  // Variables auto-created when plugin is installed
  variables: [
    { id: 'viewCount', name: 'View Count', type: 'number', default: 0 }
  ]
}

Plugin API#

The mp4e global provides methods for interacting with the player. All methods communicate via postMessage from the sandboxed iframe.

mp4e API quick reference
// ═══════════════════════════════════════════
// CONFIG & DATA (Sync)
// ═══════════════════════════════════════════
const config = mp4e.getConfig();       // Plugin configuration values
const data = mp4e.getData();           // Data object (object data for group-bound)
const pluginId = mp4e.getPluginId();   // This plugin's ID

// ═══════════════════════════════════════════
// VARIABLES (Async)
// ═══════════════════════════════════════════
const score = await mp4e.getVariable('score');
await mp4e.setVariable('score', score + 10);
mp4e.onVariableChange((variables, changedIds) => {
  // changedIds = array of changed variable IDs, or undefined for full broadcast
  if (!changedIds || changedIds.includes('score')) {
    updateScoreUI(variables.score);
  }
});

// ═══════════════════════════════════════════
// PLAYBACK (Sync getters, Async setters)
// ═══════════════════════════════════════════
const time = mp4e.getCurrentTime();    // Current time in seconds
const frame = mp4e.getCurrentFrame();  // Current frame number
const dur = mp4e.getDuration();        // Video duration
const playing = mp4e.isPlaying();      // Playback state

await mp4e.playVideo();                // Play
await mp4e.pauseVideo();               // Pause
await mp4e.seekToTime(30);             // Seek to 30s
await mp4e.setVolume(0.5);            // Volume (0-1)
await mp4e.setPlaybackRate(1.5);      // Playback speed

// ═══════════════════════════════════════════
// EVENTS (Sync)
// ═══════════════════════════════════════════
mp4e.emit('purchased', { productId: '123' });  // Emit custom event
mp4e.on('mp4e:video:play', () => { /* ... */ });
mp4e.on('mp4e:video:pause', () => { /* ... */ });
mp4e.on('mp4e:video:timeupdate', (data) => { /* ... */ });

// ═══════════════════════════════════════════
// ACTIONS
// ═══════════════════════════════════════════
mp4e.executeActions([
  { type: 'showNotification', message: 'Added!', variant: 'success' },
  { type: 'setVariable', variableId: 'cartCount', operation: 'increment', value: 1 }
]);
mp4e.close();                                   // Close this overlay
await mp4e.openUrl('https://...', '_blank');    // Open URL

// ═══════════════════════════════════════════
// INTER-PLUGIN COMMUNICATION
// ═══════════════════════════════════════════
mp4e.onAction('highlight', (config) => { /* handle action from rules */ });
await mp4e.callAction('cart-plugin', 'addItem', { sku: '123' });

// ═══════════════════════════════════════════
// STORAGE (Async, sandboxed per plugin)
// ═══════════════════════════════════════════
await mp4e.storage.set('lastSeen', Date.now());
const val = await mp4e.storage.get('lastSeen');

// ═══════════════════════════════════════════
// SCALING & VIEWPORT (Sync)
// ═══════════════════════════════════════════
const scale = mp4e.getScaleFactor();
const px = mp4e.scale(16);            // Scale-adjusted pixels
const vp = mp4e.getViewport();        // { width, height, scaleFactor, hasTouch }
Full API Reference
See the API Reference for complete documentation of all 65+ methods including media access, menu control, thumbnail previews, and the postMessage protocol.

Categories#

Plugins are organized into 12 semantic categories for the marketplace:

Commerce
Shopping, payments, and e-commerce
Education
Learning, courses, and quizzes
Engagement
Polls, surveys, and user interaction
Analytics
Tracking, stats, and insights
Creative
Effects, filters, and artistic elements
Lead Generation
Forms, signups, and contact capture
Social
Sharing, comments, and social features
Navigation
Menus, chapters, and scene navigation
Information
Info cards, tooltips, and help text
Accessibility
Captions, audio descriptions, and accessibility features
Controls
Video player control interfaces
Entertainment
Games, easter eggs, and fun interactions

Lifecycle#

Understanding the plugin lifecycle helps write robust plugins:

Plugin lifecycle
// 1. IFRAME CREATED
// - Engine generates srcDoc with bridge script + your HTML/CSS/JS
// - {{config.field}} values interpolated in HTML/CSS at generation time
// - {{variable}} patterns preserved for runtime interpolation

// 2. BRIDGE READY
// - mp4e global object available
// - mp4e.getConfig() returns current configuration
// - window.object available (for group-bound overlays)
// - mp4e.ready() auto-called on DOMContentLoaded

// 3. INITIAL DATA
// - mp4e-variable-update message delivers all variable values
// - mp4e-interpolated message resolves remaining {{variable}} patterns
// - onVariableChange callbacks fire with changedIds = undefined (full broadcast)

// 4. RUNTIME
// - onVariableChange fires when variables change (with changedIds array)
// - onConfigUpdate fires when overlay config changes
// - Video events (play, pause, timeupdate) delivered via mp4e.on()
// - Plugin can call any mp4e API method

// 5. CLEANUP
// - Plugin closes via: mp4e.close(), backdrop click, hideOverlay action,
//   or overlay visibility change
// - Iframe is destroyed — no manual cleanup needed
// - Tip: clear intervals/timeouts to prevent errors during teardown