wxGUI/animation: Fix freeze and descriptor exhaustion on large datasets - #7811
Open
saket0187 wants to merge 2 commits into
Open
wxGUI/animation: Fix freeze and descriptor exhaustion on large datasets#7811saket0187 wants to merge 2 commits into
saket0187 wants to merge 2 commits into
Conversation
ninsbl
reviewed
Aug 8, 2026
ninsbl
left a comment
Member
There was a problem hiding this comment.
Thanks @saket0187 for the thorough analysis. Your changes look good to me. You may consider using enumerate instead of manually handling counter variables. But that is not crucial.
Anna is much more knowledgable regarding the GUI code so I leave it to here approving and merging the PR. It is good to see further temporal related GUI improvements as part of your GSoC.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Loading a space-time dataset with many maps (like ~1000) in the Animation Tool
exhausted system resources and locked up the machine.
There were several performance issues like -
OSError: [Errno 24] Too many open files, since each process carries aqueue pipe and its own render subprocess pipes
make the desktop unresponsive
closing the GUI
progress signal is emitted
AI Summary
The `animation:nprocs` setting defaults to `-1` (autodetect) and is resolved to a real number only when the animation preferences are opened. Users who never open them pass `-1` straight to `BitmapProvider.Load()`, where the batch gate was an equality test:proc_countis always >= 1 there, so the comparison against-1never held andthe batch was never drained. Every map was started before a single one was
waited for, giving one live process per map, which produced all of the above.
Three further problems were reachable once the batch did drain: a queue was
allocated per map (a pipe and POSIX semaphores each) instead of per parallel
slot, the queue was read with a blocking
get()which froze the GUI and madeCancel ineffective, and the process was joined before its result was read,
which the multiprocessing docs call out as a deadlock.
A fourth, unrelated to the freeze, showed up while testing on a dataset whose
STRDS still registered deleted maps: both failure paths call
os.remove()on afile the failed command never created, so an already-reported warning turned
into an unhandled
FileNotFoundErrorinside the render process.Changes
nprocs < 1togetCpuCount()inLoad(), so the-1placeholdercan never reach the scheduling logic.
RenderandComposewith one sharedscheduler,
_renderInParallel(). It keepsnprocsslots and refills a slotas soon as the process in it finishes, so a slow map no longer holds back the
ones queued behind it and no batch barrier remains. It also clamps to at
least one slot, so no value of
nprocscan spin or fan out without a bound.map, so the number of pipes and semaphores stays bounded by
nprocs._takeRenderedFile()) and report progressbetween polls, so the dialog repaints and Cancel is honoured. Each result is
read before its process is joined, avoiding the documented deadlock.
instead of waiting for them.
the loop raises, so none outlive the run.
failed.
DictRefCounter.__delitem__andMapFilesPool.GetSizenow toleratekeys that were never stored, which a cancelled or failed render leaves behind.
deletion is still done, so a partially written image is not left behind to be
mistaken for a good render later.
Tested on macOS with a 1008-map STRDS.
Claude assistance was used to diagnose the root cause and draft the changes.
Other Suggestions from Claude (Out of Scope for this PR)
BitmapPoolretains oneuncompressed
wx.Bitmapper frame andMapFilesPoolkeeps a PPM/PGM pair permap on disk. At 1008 maps that is roughly 1.2 GB of RAM at 640x480 and
considerably more at larger window sizes. That is architectural (every frame is
cached to allow scrubbing) and wants its own issue, most likely an LRU bound on
BitmapPoolor a warning above a map-count threshold.@petrasovaa @ninsbl