New filter for orphan afterglows #403
Conversation
JulienPeloton
left a comment
JulienPeloton
left a comment
There was a problem hiding this comment.
Thanks @marinamasson ! I have done a first pass on the code, and left some comments. I will perform a profiling of the code, and paste the results here later this week.
JulienPeloton
left a comment
JulienPeloton
left a comment
There was a problem hiding this comment.
Thanks @marinamasson -- sorry for the delay in getting back to you. There are still inconsistencies to use this properly with Apache Spark (I can see that your code for Elasticc was using purely pandas). I will let you handle my last comment (see below), and if you are ok, I will make the modifications myself to integrate it with Spark.
| from fink_filters.tester import spark_unit_tests | ||
|
|
||
|
|
||
| def orphan_grb(ctimemjd, cabmags, cabmagserr, cfilts): |
There was a problem hiding this comment.
This should have a decorator for Spark, namely:
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import FloatType
@pandas_udf(FloatType())
def orphan_grb(ctimemjd, cabmags, cabmagserr, cfilts):
...There was a problem hiding this comment.
Thanks @JulienPeloton for your review! I'm completely ok if you can make the modifications to integrate it with Spark. Don't hesitate if more things need to be modified from my side.
JulienPeloton
merged commit 01e7412
into
astrolabsoftware:orphans_integration
* New filter for orphan afterglows (#403) * new orphans module * Create README.md * add machine learning model for orphans * modify model path * corrected and updated orphan module * added decorator for Spark * Fix bug with mean_colours * PEP8 * PEP8 * Rename Y in y for rubin * Update with fink-filters new path. Closes #526 * Add __init__ * Update paths * Fix missing imports * Expand test suite. Fix empty batch after quality cuts. * PEP8 --------- Co-authored-by: marinamasson <[email protected]>
2 participants
Linked to issue(s): Closes #402
If this is a new release, did you issue the corresponding schema in fink-client?
This is a new module, not a new release.
What changes were proposed in this pull request?
A new science module.
How is the issue this PR is referenced against solved with this PR?
The PR solves the issue.
How was this patch tested?
Not tested within Fink.