-
Notifications
You must be signed in to change notification settings - Fork 28.7k
[SPARK-9014][SQL] Allow Python spark API to use built-in exponential operator #8658
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -91,6 +91,17 @@ def _(self): | |
return _ | ||
|
||
|
||
def _bin_func_op(name, reverse=False, doc="binary function"): | ||
def _(self, other): | ||
sc = SparkContext._active_spark_context | ||
fn = getattr(sc._jvm.functions, name) | ||
jc = other._jc if isinstance(other, Column) else _create_column_from_literal(other) | ||
njc = fn(self._jc, jc) if not reverse else fn(jc, self._jc) | ||
return Column(njc) | ||
_.__doc__ = doc | ||
return _ | ||
|
||
|
||
def _bin_op(name, doc="binary operator"): | ||
""" Create a method for given binary operator | ||
""" | ||
|
@@ -151,6 +162,8 @@ def __init__(self, jc): | |
__rdiv__ = _reverse_op("divide") | ||
__rtruediv__ = _reverse_op("divide") | ||
__rmod__ = _reverse_op("mod") | ||
__pow__ = _bin_func_op("pow") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So this doesn't quite match the scala API (which I suppose isn't the end of the world), but would it possible make sense to have a similar functions.py file to match the scala API? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We already have For here, it's easy to do like this:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @davies, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see, I'd like to go with current approach. We could change to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can rename, but what if later we would implement something like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That make sense, but _bin_func_op expect that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree. What if I replace
with
Would it still worth renaming to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This sounds better, go for it, thanks! |
||
__rpow__ = _bin_func_op("pow", reverse=True) | ||
|
||
# logistic operators | ||
__eq__ = _bin_op("equalTo") | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we are just looking at adding pow then it might make more sense to add pow to column, but if we are looking at making it easy to access the rest of the functions in functions.scala this sounds good (although maybe add a tiny comment explaining the purpose).