-
Notifications
You must be signed in to change notification settings - Fork 1.8k
refactor: refactor spark like function to use datafusion like #19324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@Jefffrey I need some clarification about this change, according to your issue #17964, you wanted to use the DF's equivalent function to the Spark. But DataFusion doesn’t expose LIKE as a reusable |
|
|
||
| fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> { | ||
| match (arg_types.first(), arg_types.get(1)) { | ||
| (Some(lhs), Some(rhs)) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This arm will match even if there are more than two arguments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. Thanks!
|
In this case it seems like there is quite minimal duplication since the Spark version uses the arrow |
|
I didnt' use the |
If that's the case then I think we can leave the Spark Like code on main as is without any changes 👍 |
|
Good call. |
Which issue does this PR close?
Rationale for this change
There are several Spark functions which have the equivalent Datafusion functions, our goal is to reduce this duplication.
What changes are included in this PR?
ike_coercionand delegates execution toarrow::compute::likeAre these changes tested?
All previouse tests pass
Are there any user-facing changes?