Parallelize bundle audit for NetworkPolicy objects#161
Conversation
Also switch from using docker save -> extract approach to opm render -> declCfg.Load() approch, which is much faster and memory efficient. Performance improvements while aduiting catalogs: 1. redhat-operators v4.13 took ~1 hour 20 mins 2. redhat-operators v4.14 took ~1 hour 30 mins Down from ~4 hours.
Pull Request Test Coverage Report for Build 16891364393Details
💛 - Coveralls |
| os.MkdirAll(workerCacheDir, 0755) | ||
|
|
||
| for attempt := 1; attempt <= maxRetries; attempt++ { | ||
| cmd := exec.Command("opm", "alpha", "bundle", "unpack", job.image, "-o", bundleDir) |
There was a problem hiding this comment.
FYI, this failed for me with one of the bundles in the community catalog, since it does some validation. What that validation was is lost to 10K+ lines in my console buffer, but I shifted to oc image extract as a foolproof solution.
| // This avoids image registry dependencies (gpgme) while giving us structured data | ||
|
|
||
| // Execute opm render command as external process | ||
| cmd := exec.Command("opm", "render", "-o", "yaml", indexImage) |
There was a problem hiding this comment.
You could avoid the shell/read-from-disk by doing this in go like https://github.com/operator-framework/operator-registry/blob/3212635c5e3847e5e9f4b5db3f87b147405274fa/cmd/opm/render/cmd.go#L62C3-L88C1.
Also, since you are using opm render already without any migration option, you could grab all the manifests by just walking the olm.bundle.object properties on each olm.bundle, base64 decode them and string search them. That would allow you to skip the unpacking later in the flow completely!
3 participants
Also switch from using docker save -> extract approach to opm render -> declCfg.Load() approch, which is much faster and memory efficient.
Performance improvements while aduiting catalogs:
Down from ~4 hours.