MAINT: Excluding large datasets from automatic scenario-default loading - #2484
MAINT: Excluding large datasets from automatic scenario-default loading#2484jbolor21 wants to merge 3 commits into
Conversation
| "garak_crates_packages", | ||
| "garak_npm_packages", | ||
| "garak_pypi_packages", | ||
| "garak_rubygems_packages", |
There was a problem hiding this comment.
curious how you found out that these are "large"? is there a way that we can systematically quantify what "large enough" means so that future datasets are in here? We have SeedDatasetLoadTime but can't tell if we actually use that for anything.
There was a problem hiding this comment.
This isn't super extensible. Nobody is going to maintain this list. I'd be tempted to do it anyway since it's just an initializer and it will make load_defaults return fast, but an issue is that if we run the scenarios notebook (garak.ipynb) these will still load and it'll still take forever.
I think my fav fix is
- revert these changes
- Make Rust the default for package hallucination and remove python/ruby/etc. That should reduce the size 20x. Then this function will return fast, and the basic scenario in scanner docs will run fast
There was a problem hiding this comment.
Just the number of entries in the dataset! PyRIT puts this definition
SeedDatasetSizeCategory = Literal["tiny", "small", "medium", "large", "huge"]
# tiny (<10), small (10-99), medium (100-499), large (500-4999), huge (5000+) ```
There was a problem hiding this comment.
oh okay yeah that sounds good! then I'll just close this PR Richard Lundeen (@rlundeen2) ?
3 participants
Description
Prevent
LoadDefaultDatasetsfrom exhausting integration-test memory by excluding large package registries from automatic scenario-default loading.A previous PR introduced four large Garak datasets (npm, PyPI, RubyGems, and crates.io) whose registries contain roughly 4.19 million records combined. They remain registered and available when explicitly selected through dataset_names or tags , but are no longer fetched during a default initialization. The integration test previously discovered and loaded every dataset required by registered scenarios.
Tests and Documentation:
Add unit coverage for:
• Excluding opt-in registries from implicit loading.
• Loading an opt-in registry when explicitly requested.
• Confirming all opt-in registries remain discoverable.