SPARK-1150: fix repo location in create script (re-open) #52
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
|
Thanks, merged |
jhartlaub
referenced
this pull request
in jhartlaub/spark
May 27, 2014
Add an optional closure parameter to HadoopRDD instantiation to use when creating local JobConfs. Having HadoopRDD accept this optional closure eliminates the need for the HadoopFileRDD added earlier. It makes the HadoopRDD more general, in that the caller can specify any JobConf initialization flow. (cherry picked from commit 9979690) Signed-off-by: Reynold Xin <[email protected]>
clockfly
added a commit
to clockfly/spark
that referenced
this pull request
Aug 30, 2016
## What changes were proposed in this pull request?
This is the second part of closure translation feature, which translates the Node tree returned from ByteCodeParser to Spark sql expressions.
For example, input Node tree for filter operation:
```
Arithmetic[Z](>)
Argument[I]
Constant[I](0)
```
is translated to expression:
```
GreaterThan
ColumnField("value")
Literal(0)
```
After translation, the expression may be further flattened to Seq[Expression] if its type
contain sub-fields. This is consistent with the behavior of Dataset typed Map operation.
```
// If the type U is a case class, then all fields of type U are flattened. The result Dataset
// may contains multiple fields.
dataset.map(func: T => U)
```
### Design doc
https://docs.google.com/document/d/1JZ0-lZfjGTMZto7Oxg_yOkZh3G6VDo4lznVMSD0EM6k/edit
## How was this patch tested?
Unit tests.
Author: Sean Zhong <[email protected]>
Closes apache#52 from clockfly/closure_parser_part2.
robert3005
pushed a commit
to robert3005/spark
that referenced
this pull request
Jan 12, 2017
tnachen
pushed a commit
to tnachen/spark
that referenced
this pull request
Jan 27, 2017
* Use "extraTestArgLine" to pass extra options to scalatest. Because the "argLine" option of scalatest is set in pom.xml and we can't overwrite it from the command line. Ref apache-spark-on-k8s#37 * Added a default value for extraTestArgLine * Use a better name. * Added a tip for this in the dev docs.
lins05
added a commit
to lins05/spark
that referenced
this pull request
Apr 23, 2017
* Use "extraTestArgLine" to pass extra options to scalatest. Because the "argLine" option of scalatest is set in pom.xml and we can't overwrite it from the command line. Ref apache-spark-on-k8s#37 * Added a default value for extraTestArgLine * Use a better name. * Added a tip for this in the dev docs.
erikerlandson
pushed a commit
to erikerlandson/spark
that referenced
this pull request
Jul 28, 2017
* Use "extraTestArgLine" to pass extra options to scalatest. Because the "argLine" option of scalatest is set in pom.xml and we can't overwrite it from the command line. Ref apache-spark-on-k8s#37 * Added a default value for extraTestArgLine * Use a better name. * Added a tip for this in the dev docs.
jlopezmalla
pushed a commit
to jlopezmalla/spark
that referenced
this pull request
Nov 3, 2017
gczsjdy
pushed a commit
to gczsjdy/spark
that referenced
this pull request
Jul 30, 2018
…pache#52) * Fix exception: Child of ShuffleQueryStage must be a ShuffleExchange * top ShuffleExchange of QueryStage should not be removed anyway * remove unecessary parentheses * check top shuffle exchange for ShuffleQueryStage only * minor comments * improve topShuffleCheck * simplfy codes
luzhonghao
pushed a commit
to luzhonghao/spark
that referenced
this pull request
Dec 11, 2018
…pache#52) * Fix exception: Child of ShuffleQueryStage must be a ShuffleExchange * top ShuffleExchange of QueryStage should not be removed anyway * remove unecessary parentheses * check top shuffle exchange for ShuffleQueryStage only * minor comments * improve topShuffleCheck * simplfy codes
hejian991
pushed a commit
to growingio/spark
that referenced
this pull request
Jun 24, 2019
…pache#52) * Fix exception: Child of ShuffleQueryStage must be a ShuffleExchange * top ShuffleExchange of QueryStage should not be removed anyway * remove unecessary parentheses * check top shuffle exchange for ShuffleQueryStage only * minor comments * improve topShuffleCheck * simplfy codes
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Add new credentials to Orange
wangyum
pushed a commit
that referenced
this pull request
May 26, 2023
* index synax * index build * index prune * index metrics * index ut * [CARMEL-3157] index pruning - upgrade to 3.0 * remove ut for index treated as unsupport feature * fix conflict * fix conflict * fix style
panbingkun
pushed a commit
that referenced
this pull request
Nov 22, 2024
…ead pool ### What changes were proposed in this pull request? This PR aims to use a meaningful class name prefix for REST Submission API thread pool instead of the default value of Jetty QueuedThreadPool, `"qtp"+super.hashCode()`. https://github.com/dekellum/jetty/blob/3dc0120d573816de7d6a83e2d6a97035288bdd4a/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L64 ### Why are the changes needed? This is helpful during JVM investigation. **BEFORE (4.0.0-preview2)** ``` $ SPARK_MASTER_OPTS='-Dspark.master.rest.enabled=true' sbin/start-master.sh $ jstack 28217 | grep qtp "qtp1925630411-52" #52 daemon prio=5 os_prio=31 cpu=0.07ms elapsed=19.06s tid=0x0000000134906c10 nid=0xde03 runnable [0x0000000314592000] "qtp1925630411-53" #53 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=19.06s tid=0x0000000134ac6810 nid=0xc603 runnable [0x000000031479e000] "qtp1925630411-54" #54 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=19.06s tid=0x000000013491ae10 nid=0xdc03 runnable [0x00000003149aa000] "qtp1925630411-55" #55 daemon prio=5 os_prio=31 cpu=0.08ms elapsed=19.06s tid=0x0000000134ac9810 nid=0xc803 runnable [0x0000000314bb6000] "qtp1925630411-56" #56 daemon prio=5 os_prio=31 cpu=0.04ms elapsed=19.06s tid=0x0000000134ac9e10 nid=0xda03 runnable [0x0000000314dc2000] "qtp1925630411-57" #57 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=19.06s tid=0x0000000134aca410 nid=0xca03 runnable [0x0000000314fce000] "qtp1925630411-58" #58 daemon prio=5 os_prio=31 cpu=0.04ms elapsed=19.06s tid=0x0000000134acaa10 nid=0xcb03 runnable [0x00000003151da000] "qtp1925630411-59" #59 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=19.06s tid=0x0000000134acb010 nid=0xcc03 runnable [0x00000003153e6000] "qtp1925630411-60-acceptor-0108e9815-ServerConnector1e497474{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #60 daemon prio=3 os_prio=31 cpu=0.11ms elapsed=19.06s tid=0x00000001317ffa10 nid=0xcd03 runnable [0x00000003155f2000] "qtp1925630411-61-acceptor-11d90f2aa-ServerConnector1e497474{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #61 daemon prio=3 os_prio=31 cpu=0.10ms elapsed=19.06s tid=0x00000001314ed610 nid=0xcf03 waiting on condition [0x00000003157fe000] ``` **AFTER** ``` $ SPARK_MASTER_OPTS='-Dspark.master.rest.enabled=true' sbin/start-master.sh $ jstack 28317 | grep StandaloneRestServer "StandaloneRestServer-52" #52 daemon prio=5 os_prio=31 cpu=0.09ms elapsed=60.06s tid=0x00000001284a8e10 nid=0xdb03 runnable [0x000000032cfce000] "StandaloneRestServer-53" #53 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=60.06s tid=0x00000001284acc10 nid=0xda03 runnable [0x000000032d1da000] "StandaloneRestServer-54" #54 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=60.06s tid=0x00000001284ae610 nid=0xd803 runnable [0x000000032d3e6000] "StandaloneRestServer-55" #55 daemon prio=5 os_prio=31 cpu=0.09ms elapsed=60.06s tid=0x00000001284aec10 nid=0xd703 runnable [0x000000032d5f2000] "StandaloneRestServer-56" #56 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=60.06s tid=0x00000001284af210 nid=0xc803 runnable [0x000000032d7fe000] "StandaloneRestServer-57" #57 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=60.06s tid=0x00000001284af810 nid=0xc903 runnable [0x000000032da0a000] "StandaloneRestServer-58" #58 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=60.06s tid=0x00000001284afe10 nid=0xcb03 runnable [0x000000032dc16000] "StandaloneRestServer-59" #59 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=60.06s tid=0x00000001284b0410 nid=0xcc03 runnable [0x000000032de22000] "StandaloneRestServer-60-acceptor-04aefbaa8-ServerConnector44284d85{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #60 daemon prio=3 os_prio=31 cpu=0.13ms elapsed=60.05s tid=0x000000015cda1a10 nid=0xcd03 runnable [0x000000032e02e000] "StandaloneRestServer-61-acceptor-148976251-ServerConnector44284d85{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #61 daemon prio=3 os_prio=31 cpu=0.12ms elapsed=60.05s tid=0x000000015cd1c810 nid=0xce03 waiting on condition [0x000000032e23a000] ``` ### Does this PR introduce _any_ user-facing change? No, the thread names are accessed during the debugging. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #48924 from dongjoon-hyun/SPARK-50385. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: panbingkun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
reopen for https://spark-project.atlassian.net/browse/SPARK-1150