-
Notifications
You must be signed in to change notification settings - Fork 548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix potential CUDA context poison when negative (invalid) categories provided to FIL model [21.10] #4315
Fix potential CUDA context poison when negative (invalid) categories provided to FIL model [21.10] #4315
Conversation
…inference: now equivalent to non-matching. FIL now converts dummy nodes to numerical on import and never generates max_matching == -1 categorical features in test.
Codecov Report
@@ Coverage Diff @@
## branch-21.12 #4315 +/- ##
=============================================
Coverage 86.04% 86.04%
=============================================
Files 231 231
Lines 18691 18715 +24
=============================================
+ Hits 16083 16104 +21
- Misses 2608 2611 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
I think this is targeting the wrong branch? Looks to be targeting 21.12, but it's labeled 21.10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! As is, this should address the problem of out-of-bounds access for categorical features.
The main comment is about more testing for different kinds of values for categorical features. Other comments are mostly technical.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great!
Approved provided that the comments (mostly technical) are addressed.
@dantegd should I close this PR since there's the https://github.com/rapidsai/cuml/tree/hotfix-21.10-pr-4315 branch is open? |
@levsnv we still need this PR, since we want to release a full conda/docker hotfix. The only issue is that it is targetted to branch-21.12 as opposed to 21.10, could you change the target (or I can help doing that as well). After that, we can start the hotfix process later in the week. |
fix edge case of reeeeeeeally large models give more room by not being pedantic about the name extra test and technical questions +1
@@ -510,11 +526,9 @@ def to_categorical(features, n_categorical): | |||
for icol in range(n_categorical, n_features): | |||
df_cols[icol] = pd.Series(features[:, icol]) | |||
# shuffle the columns around | |||
seed(42) | |||
new_idx = sample(range(n_features), k=n_features) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see review in #4326 - re: determinism
…provided to FIL model (rapidsai#4315) Fix potential CUDA context poison due to invalid global read when negative categories provided at inference: now equivalent to non-matching. Same for `+-Inf` categories. NAN categories are still `!def_left`, and fractional categories are truncated (via a typecast), as this is what Treelite does FIL now converts dummy nodes to numerical on import and never generates max_matching == -1 categorical features in test. FIL will still generate empty categorical nodes in test (a non-empty bits vector which contains only zeros), export them as dummy numerical nodes and import again as dummy numerical nodes. If a feature only contains dummy numerical nodes, it will be deemed a numerical feature (same as for non-dummy numerical nodes or a mix thereof). Therefore, categorical feature max_matching == -1 is still prevented.
Fix potential CUDA context poison due to invalid global read when negative categories provided at inference:
now equivalent to non-matching. Same for
+-Inf
categories. NAN categories are still!def_left
, and fractional categories are truncated (via a typecast), as this is what Treelite doesFIL now converts dummy nodes to numerical on import
and never generates max_matching == -1 categorical features in test.
FIL will still generate empty categorical nodes in test (a non-empty bits vector which contains only zeros), export them as dummy numerical nodes and import again as dummy numerical nodes. If a feature only contains dummy numerical nodes, it will be deemed a numerical feature (same as for non-dummy numerical nodes or a mix thereof). Therefore, categorical feature max_matching == -1 is still prevented.