Skip to content

wxGUI/animation: Fix freeze and descriptor exhaustion on large datasets - #7811

Open
saket0187 wants to merge 2 commits into
OSGeo:mainfrom
saket0187:Fix-performance-issue-in-animation
Open

wxGUI/animation: Fix freeze and descriptor exhaustion on large datasets#7811
saket0187 wants to merge 2 commits into
OSGeo:mainfrom
saket0187:Fix-performance-issue-in-animation

Conversation

@saket0187

@saket0187 saket0187 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Loading a space-time dataset with many maps (like ~1000) in the Animation Tool
exhausted system resources and locked up the machine.

There were several performance issues like -

  • OSError: [Errno 24] Too many open files, since each process carries a
    queue pipe and its own render subprocess pipes
  • unbounded CPU and memory use from ~1000 concurrent interpreters, enough to
    make the desktop unresponsive
  • the processes were non-daemonic and never terminated, so they survived
    closing the GUI
  • progress appeared to stop one map short, because the drain runs before the
    progress signal is emitted
AI Summary The `animation:nprocs` setting defaults to `-1` (autodetect) and is resolved to a real number only when the animation preferences are opened. Users who never open them pass `-1` straight to `BitmapProvider.Load()`, where the batch gate was an equality test:
if proc_count == nprocs or count == mapNum:

proc_count is always >= 1 there, so the comparison against -1 never held and
the batch was never drained. Every map was started before a single one was
waited for, giving one live process per map, which produced all of the above.

Three further problems were reachable once the batch did drain: a queue was
allocated per map (a pipe and POSIX semaphores each) instead of per parallel
slot, the queue was read with a blocking get() which froze the GUI and made
Cancel ineffective, and the process was joined before its result was read,
which the multiprocessing docs call out as a deadlock.

A fourth, unrelated to the freeze, showed up while testing on a dataset whose
STRDS still registered deleted maps: both failure paths call os.remove() on a
file the failed command never created, so an already-reported warning turned
into an unhandled FileNotFoundError inside the render process.

Changes

  • Resolve nprocs < 1 to getCpuCount() in Load(), so the -1 placeholder
    can never reach the scheduling logic.
  • Replace the per-batch loops in Render and Compose with one shared
    scheduler, _renderInParallel(). It keeps nprocs slots and refills a slot
    as soon as the process in it finishes, so a slow map no longer holds back the
    ones queued behind it and no batch barrier remains. It also clamps to at
    least one slot, so no value of nprocs can spin or fan out without a bound.
  • Give each slot one queue reused for the whole run instead of one queue per
    map, so the number of pipes and semaphores stays bounded by nprocs.
  • Take results without blocking (_takeRenderedFile()) and report progress
    between polls, so the dialog repaints and Cancel is honoured. Each result is
    read before its process is joined, avoiding the documented deadlock.
  • Mark the render processes daemonic, so closing the GUI terminates them
    instead of waiting for them.
  • Terminate and join any process still running when the user cancels or when
    the loop raises, so none outlive the run.
  • Report progress per finished map rather than per started map.
  • Do not store a map which failed to render; the composition reports it as
    failed. DictRefCounter.__delitem__ and MapFilesPool.GetSize now tolerate
    keys that were never stored, which a cancelled or failed render leaves behind.
  • Do not fail when removing a file the failed command never created. The
    deletion is still done, so a partially written image is not left behind to be
    mistaken for a good render later.

Tested on macOS with a 1008-map STRDS.

Claude assistance was used to diagnose the root cause and draft the changes.

Other Suggestions from Claude (Out of Scope for this PR)

  • Memory is still unbounded in the number of maps: BitmapPool retains one
    uncompressed wx.Bitmap per frame and MapFilesPool keeps a PPM/PGM pair per
    map on disk. At 1008 maps that is roughly 1.2 GB of RAM at 640x480 and
    considerably more at larger window sizes. That is architectural (every frame is
    cached to allow scrubbing) and wants its own issue, most likely an LRU bound on
    BitmapPool or a warning above a map-count threshold.

@petrasovaa @ninsbl

@github-actions github-actions Bot added GUI wxGUI related Python Related code is in Python labels Aug 8, 2026

@ninsbl ninsbl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @saket0187 for the thorough analysis. Your changes look good to me. You may consider using enumerate instead of manually handling counter variables. But that is not crucial.

Anna is much more knowledgable regarding the GUI code so I leave it to here approving and merging the PR. It is good to see further temporal related GUI improvements as part of your GSoC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

GUI wxGUI related Python Related code is in Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants