Export more metadata from export_model() - #684
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## mainline #684 +/- ##
============================================
- Coverage 84.52% 83.93% -0.60%
============================================
Files 75 75
Lines 6823 6790 -33
Branches 557 557
============================================
- Hits 5767 5699 -68
- Misses 1056 1091 +35 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@JulienAu Can you review this pull request? |
JulienAu
left a comment
There was a problem hiding this comment.
@chyunsu3 Thanks for the quick turnaround on this. I reviewed the diff and ran the branch's exporter against the compiled treelite 4.7.0 runtime (the changed files are pure Python), on both sklearn round trips and cuML model bytes.
The TreeliteModelBuilderDataCount fix is correct, and the sklearn round trip works as advertised: data_count populated (root equals max_samples_), correct estimator classes, n_node_samples preserved exactly.
Two things worth considering:
-
The exported IsolationForest cannot score: only
estimators_,n_outputs_,n_features_in_and_sklearn_versionare set, soscore_samplesraisesAttributeError: '_max_features'. Most of the missing state is derivable from the trees, and I verified that populating it plus the defaultoffset_ = -0.5gives exact score parity on the round trip.offset_itself is not recoverable from the serialized model (base_scores stays 0), so contamination-fitted models would predict differently. Whichever contract you prefer (populate with a documented -0.5 assumption, or keep the export structural with an explicit error on scoring), ascore_samplescall in test_iforest_round_trip would pin it down. -
Present-but-zero data_count flows through silently. cuML's isolation forest export currently emits exactly that (checked on nightly bytes: every node present with value 0, task_type kRegressor), which would export as n_node_samples equal to 0 everywhere and break scoring downstream. Since a reachable node can never hold zero samples, failing loudly on non-positive present values seems safer. It also means NVIDIA/cuml#8483 can only adopt
exported.estimators_once the cuML C++ export writes real counts and tags kIsolationForest; happy to coordinate that follow-up.
Minor: np.bool is missing on numpy 1.24 through 1.26 and the package floor allows them; np.bool_ works everywhere.
|
@JulienAu Can you take another look?
offset = treelite_model.attributes.get("sklearn_iforest_offset", -0.5)
|
JulienAu
left a comment
There was a problem hiding this comment.
Took another look.
Storing offset_ as a model attribute is a nice solution, I confirmed it survives serialize_bytes and deserialize_bytes, which is the path cuML will use. Scoring parity is exact on my side for the default and contamination cases.
One real issue: with bootstrap=True the scores drift (about 1e-3 in my runs). sklearn fits the bagged trees with sample weights instead of repeating rows, so the root n_node_samples counts distinct rows (63 in my run) and the derived _max_samples comes out wrong. The root weighted_n_node_samples is exactly max_samples on every tree, so I put the fix as a suggestion: with it I get exact parity in every configuration I tested (bootstrap on and off, with and without user sample_weight).
Small note: the public max_samples_ attribute is not set. Scoring only needs the private one, but users may look for it.
|
Forgot to answer your data_count point: that matches what csadorf already scoped in NVIDIA/cuml#8420, populate data_count in the Treelite export, then simplify the Python side. |
Co-authored-by: Julien Audibert <audibert.julien.pro@gmail.com>
Co-authored-by: Julien Audibert <audibert.julien.pro@gmail.com>
|
@JulienAu Can you look at it one last time? I addressed all your feedback. |
JulienAu
left a comment
There was a problem hiding this comment.
Ran my test bench once more on the latest head: exact parity everywhere including bootstrap, offset recovery survives serialize_bytes, and max_samples_ is set. Looks good to me, thanks for the quick turnaround.
n_node_samplesandweighted_n_node_samplesfields when exporting Treelite model as a scikit-learn model.data_count()setter in the model builder API.offset_in the Treelite model object.