Skip to content

Add native score-to-video sync#33323

Draft
ToranadoMusic wants to merge 13 commits into
musescore:mainfrom
ToranadoMusic:score-to-video
Draft

Add native score-to-video sync#33323
ToranadoMusic wants to merge 13 commits into
musescore:mainfrom
ToranadoMusic:score-to-video

Conversation

@ToranadoMusic
Copy link
Copy Markdown

@ToranadoMusic ToranadoMusic commented May 8, 2026

Summary

This draft adds a native score-to-video workflow to MuseScore Studio. Users can attach one reference video to a score, open a dockable Video panel, sync the video to MuseScore playback, and control the video audio from the Mixer as a dedicated Video channel.

Motivation

Composers working on film, games, animation, theater, dance, and other picture-locked media often need to write notation against video. Today that workflow usually requires running MuseScore beside a separate video player or DAW, which makes playhead sync, offset handling, save/load, and audio control fragile.

Current Scope

Implemented in this draft:

  • Adds per-score video attachment settings, stored in videosettings.json inside the score package.
  • Persists video path, offset in milliseconds, volume, balance, mute, and solo state.
  • Adds a native dockable Video panel.
  • Embeds a QtMultimedia video player in the panel.
  • Syncs video play/pause/seek to MuseScore playback.
  • Handles positive and negative offsets, including the pre-roll case where video should wait at 0 instead of looping/jumping.
  • Adds Ctrl+Alt+V to open/toggle the Video panel.
  • Adds a real Video channel to the Mixer model, positioned before Metronome.
  • Uses the normal mixer sections for pan/balance, fader, mute, solo, title strip, and alignment.
  • Keeps irrelevant instrument/source controls blank for Video.
  • Moves Video solo behavior into PlaybackController, so it no longer depends on the Mixer panel being open.
  • Keeps the Video panel layout stable when docked or floating: preview on top, controls underneath.

Known Limitations / Follow-up Questions

  • Video audio currently plays through QtMultimedia, not through MuseScore's full audio graph.
  • Video solo mutes MuseScore's master playback path at runtime while leaving Qt video audio audible. This is intentionally scoped as a v1 behavior, not full audio-graph routing.
  • The Mixer balance value is persisted and represented in the UI, but QtMultimedia-backed video audio may need a deeper routing path before balance/pan can be guaranteed across platforms.
  • Current implementation supports one attached video per score.
  • Export/render behavior for video audio is intentionally out of scope for this draft.
  • Cross-platform media codec behavior needs testing on macOS and Linux.

Tester Checklist

Please test on Windows first:

  • Open a score and attach an .mp4/.mov video from the Video panel.
  • Save, close, reopen, and confirm the video settings persist.
  • Play from the beginning and confirm the video follows MuseScore playback.
  • Seek/play from the middle of the score.
  • Test pause/resume.
  • Test positive and negative offset values.
  • Confirm negative offset does not make the video jump repeatedly back to 0.
  • Open Mixer and confirm Video appears before Metronome.
  • Confirm Video mute mutes video audio.
  • Confirm Video solo silences score playback while video audio remains audible.
  • Dock Video at the bottom, float/undock it, and resize it.
  • Confirm the Video panel layout keeps the preview above the controls.

Validation

Local validation so far:

  • Windows Debug build/install: passing.
  • Windows runtime launch: passing.
  • Runtime log check: no new VideoPanel, MixerPanel, or PlaybackController errors after the latest changes.
  • Project video settings serialization test updated for persisted balance/mute/solo fields.

Still needed:

  • Optimized Windows tester build.
  • Manual feature pass on an optimized build.
  • macOS/Linux build and media playback checks.
  • Maintainer feedback on whether v1 solo behavior is acceptable before deeper audio-graph routing.

@ToranadoMusic
Copy link
Copy Markdown
Author

ToranadoMusic commented May 8, 2026

Updated the mixer integration for the score-to-video patch.

What changed:

  • Video is now represented as a real MixerChannelItem::Video in MixerPanelModel, inserted before the Metronome channel.
  • Removed the custom hand-built Video strip from MixerPanel.qml, so the existing mixer sections now render Video with the same row heights/alignment as the other channels.
  • Kept Sound/Aux rows visually empty for Video since the media source comes from the attached video rather than an instrument/reverb path.
  • Added persisted video balance so the normal mixer pan/balance control has a model value alongside volume, mute, and solo.
  • Kept the title strip color distinct for Video.

Verification:

  • Debug build and install completed successfully.
  • Launched MuseScoreStudio5Development; process opened and responded.
  • Runtime log check showed no new VideoPanel/MixerPanel QML errors or missing-type errors.

@ToranadoMusic
Copy link
Copy Markdown
Author

Follow-up fix for bottom-docked Video panel clipping:

  • Removed the hard 360px minimum width from the Video controls column.
  • Switched the panel from a fixed horizontal RowLayout to a responsive GridLayout.
  • When the dock gets narrow, the preview and controls stack vertically instead of letting Browse/Clear run off the right edge.
  • Build/install succeeded and the app launched with no new VideoPanel QML runtime errors.

@ToranadoMusic
Copy link
Copy Markdown
Author

Follow-up video panel sizing change:

  • Removed the remaining thumbnail-style cap from the video preview.
  • The preview now fills the available panel width and height, with the controls taking only their implicit space.
  • In compact/bottom-docked layouts, the video area grows as the dock or floating panel gets taller instead of staying at a fixed 96px height.

Verification:

  • Debug build/install succeeded.
  • Launched MuseScoreStudio5Development; process opened and responded.
  • Runtime log check showed no new VideoPanel QML errors or binding-loop warnings.

@ToranadoMusic
Copy link
Copy Markdown
Author

Fixed two playback behavior bugs from testing:

  • Video solo now mutes MuseScore's master playback path at runtime, so soloing the Video channel leaves the video audio audible while silencing score playback. Disabling Video solo restores the master control state from project audio settings.
  • Negative video offsets no longer clamp-and-play at video time 0. If scorePlaybackPositionMs + offsetMs is still negative, the video seeks/stays at 0 and pauses until the score catches up to the video start.

Verification:

  • Debug build/install succeeded.
  • Launched MuseScoreStudio5Development; process opened and responded.
  • Runtime log check showed no new VideoPanel/MixerPanel QML errors or binding-loop warnings.

@ToranadoMusic
Copy link
Copy Markdown
Author

Video panel layout follow-up:

  • Kept the Video panel in a stable vertical layout: preview on top, transport/file/offset/clear controls underneath.
  • Removed the width-based switch back to the side-by-side layout so floating/undocked panels keep the same layout as the tested compact view.
  • The preview still expands into the available remaining space above the controls.

Verification:

  • Debug build/install succeeded.
  • Launched MuseScoreStudio5Development; process opened and responded.
  • Runtime log check showed no new VideoPanel QML errors or binding-loop warnings.

@ToranadoMusic ToranadoMusic changed the title Add native video sync proposal Add native score-to-video sync May 8, 2026
@ToranadoMusic
Copy link
Copy Markdown
Author

PR readiness update: refreshed the description to reflect the current implementation, v1 scope, known limitations, and a concrete tester checklist. Also moved Video solo behavior into PlaybackController so it is no longer dependent on the Mixer UI being open.

@igorkorsukov igorkorsukov changed the base branch from master to main May 14, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants