Skip to content

Improve benchmark installation speed#2683

Open
benjaminglass1 wants to merge 1 commit intopytorch:mainfrom
benjaminglass1:improve_install_speed
Open

Improve benchmark installation speed#2683
benjaminglass1 wants to merge 1 commit intopytorch:mainfrom
benjaminglass1:improve_install_speed

Conversation

@benjaminglass1
Copy link
Copy Markdown
Contributor

This solves a problem noticed while installing benchmarks and watching htop. A significant amount of time per installation is wasted checking package version that have already been checked. This PR caches those versions.

I also fixed one or two minor Ruff lints in the files I touched.

@benjaminglass1
Copy link
Copy Markdown
Contributor Author

Test failures look the same as those on main.

This solves a problem noticed while installing benchmarks and watching
htop.  A significant amount of time per installation is wasted checking
package version that have already been checked.  This commit caches
those versions.
@benjaminglass1 benjaminglass1 force-pushed the improve_install_speed branch from 22ab4a1 to b97ab5a Compare April 16, 2026 18:24
@benjaminglass1 benjaminglass1 requested a review from xmfan April 16, 2026 18:24
@benjaminglass1
Copy link
Copy Markdown
Contributor Author

@xmfan Please take a second look, I had to rewrite this after realizing this function is also used to validate that package versions haven't changed after installing benchmarks (which caching would invalidate). Currently getting some timing runs, but I still expect roughly the same speedup.

@benjaminglass1
Copy link
Copy Markdown
Contributor Author

benjaminglass1 commented Apr 16, 2026

On my machine, installing all benchmarks on main takes roughly 35 minutes. With this change, installing all benchmarks takes roughly 12 minutes. These timings are obviously super variable, but that hopefully gives a sense of how impactful this change is.

The main reason for this is that we checked the version of several dependencies twice per benchmark. We got the version by importing the package in a new subprocess and querying __version__. For torch, this takes 2-3 seconds (on my machine), so that's a minute wasted every 10-15 packages. The new version queries PEP 566 metadata from the package in less than a millisecond, effectively eliminating the delay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants