Skip to content

Make the Overlay Algorithm Multithreadable#12

Open
samf25 wants to merge 4 commits intoMuonColliderSoft:mainfrom
samf25:main
Open

Make the Overlay Algorithm Multithreadable#12
samf25 wants to merge 4 commits intoMuonColliderSoft:mainfrom
samf25:main

Conversation

@samf25
Copy link

@samf25 samf25 commented Feb 23, 2026

This is the last piece of the multithreading story. I've tested this and it's good to go 👍 . Now all of digi and reco can be multithreaded over events

@samf25 samf25 closed this Feb 23, 2026
@samf25 samf25 reopened this Feb 23, 2026
@samf25
Copy link
Author

samf25 commented Feb 23, 2026

The main issue was something with ROOT file UUID. By forcing all of the ROOT I/O onto a single thread, while the other threads handle the actual overlaying, this was fixed. I set it up as a queue system since this allows each working thread to request a new file in a thread-safe manner and the I/O thread can handle these requests in order.

In theory, there will be some waiting early on, but due to this waiting the overlay threads should start to stagger their requests. It depends on the file opening time. There will be an upper limit on the number of effective threads since the bottleneck will always be the I/O thread.

I did consider implementing a caching feature, but with the amount of BIB samples, that's a high memory cost. If we were okay with this memory cost, then things would dramatically speed up as we lazy-load the BIB files.

A different speed-up could be gained if we combined BIB sample files. Right now the bottleneck is file opening. So if there were fewer files to open, we could implement a cache because there would be more frequent requests for the same files. As it stands, with 6666 files in each group, repeated requests are unlikely.

@samf25
Copy link
Author

samf25 commented Feb 24, 2026

This error with the refitter as exists outside of the multithreading framework. It comes up when there's BIB which I didn't test at first. Basically, if there are tracks that aren't linked to any MCParticles, it will look somewhere that doesn't exist in the map. Now it checks to make sure it exists first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant