Skip to content
This repository was archived by the owner on May 9, 2024. It is now read-only.

Add dynamic allocation to the Spark coarse-grained scheduler. #5

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
6845cb2
[SPARK-7681] [MLLIB] remove mima excludes for 1.3
mengxr May 19, 2015
32fa611
[SPARK-7704] Updating Programming Guides per SPARK-4397
daisukebe May 19, 2015
fb90273
[SPARK-7047] [ML] ml.Model optional parent support
jkbradley May 19, 2015
7b16e9f
[SPARK-7678] [ML] Fix default random seed in HasSeed
jkbradley May 19, 2015
3c4c1f9
[SPARK-7726] Fix Scaladoc false errors
dragos May 19, 2015
68fb2a4
[SPARK-7586] [ML] [DOC] Add docs of Word2Vec in ml package
yinxusen May 19, 2015
c12dff9
[SPARK-7652] [MLLIB] Update the implementation of naive Bayes predict…
viirya May 19, 2015
4de74d2
[SPARK-7738] [SQL] [PySpark] add reader and writer API in Python
May 19, 2015
bcb1ff8
[SPARK-7662] [SQL] Resolve correct names for generator in projection
chenghao-intel May 19, 2015
2bc5e06
[SPARK-6246] [EC2] fixed support for more than 100 nodes
oleksii-sliusarenko May 19, 2015
3860520
[SPARK-7744] [DOCS] [MLLIB] Distributed matrix" section in MLlib "Dat…
dusenberrymw May 20, 2015
60336e3
[SPARK-7656] [SQL] use CatalystConf in FunctionRegistry
scwf May 20, 2015
b3abf0b
[SPARK-7663] [MLLIB] Add requirement for word2vec model
yinxusen May 20, 2015
09265ad
[SPARK-7320] [SQL] Add Cube / Rollup for dataframe
chenghao-intel May 20, 2015
3ddf051
[SPARK-7533] [YARN] Decrease spacing between AM-RM heartbeats.
zzvara May 20, 2015
589b12f
[SPARK-7654] [MLLIB] Migrate MLlib to the DataFrame reader/writer API
mengxr May 20, 2015
98a46f9
[SPARK-6094] [MLLIB] Add MultilabelMetrics in PySpark/MLlib
yanboliang May 20, 2015
b631bf7
[SPARK-7713] [SQL] Use shared broadcast hadoop conf for partitioned t…
yhuai May 20, 2015
2ad4837
[SPARK-7537] [MLLIB] spark.mllib API updates
mengxr May 20, 2015
829f1d9
[SPARK-7579] [ML] [DOC] User guide update for OneHotEncoder
sryza May 20, 2015
6338c40
Revert "[SPARK-7320] [SQL] Add Cube / Rollup for dataframe"
pwendell May 20, 2015
191ee47
[SPARK-7511] [MLLIB] pyspark ml seed param should be random by defaul…
holdenk May 20, 2015
9b84443
[SPARK-7237] [SPARK-7741] [CORE] [STREAMING] Clean more closures that…
May 20, 2015
3c434cb
[SPARK-7767] [STREAMING] Added test for checkpoint serialization in S…
tdas May 20, 2015
7956dd7
[SPARK-7698] Cache and reuse buffers in ExecutorMemoryAllocator when …
JoshRosen May 20, 2015
f2faa7a
[SPARK-7251] Perform sequential scan when iterating over BytesToBytesMap
JoshRosen May 20, 2015
c330e52
[SPARK-7762] [MLLIB] set default value for outputCol
mengxr May 21, 2015
5196eff
[SPARK-7719] Re-add UnsafeShuffleWriterSuite test that was removed fo…
JoshRosen May 21, 2015
a70bf06
[SPARK-7750] [WEBUI] Rename endpoints from `json` to `api` to allow fu…
harishreedharan May 21, 2015
895baf8
[SPARK-7777] [STREAMING] Fix the flaky test in org.apache.spark.strea…
zsxwing May 21, 2015
42c592a
[SPARK-7320] [SQL] Add Cube / Rollup for dataframe
chenghao-intel May 21, 2015
ddec173
[SPARK-7774] [MLLIB] add sqlContext to MLlibTestSparkContext
mengxr May 21, 2015
d0eb9ff
[SPARK-7746][SQL] Add FetchSize parameter for JDBC driver
viirya May 21, 2015
04940c4
[SPARK-7389] [CORE] Tachyon integration improvement
May 21, 2015
8ddcb25
[SPARK-7606] [SQL] [PySpark] add version to Python SQL API docs
May 21, 2015
947ea1c
[SPARK-7753] [MLLIB] Update KernelDensity API
mengxr May 21, 2015
1ee8eb4
[SPARK-7745] Change asserts to requires for user input checks in Spar…
brkyvz May 21, 2015
feb3a9d
[SPARK-7320] [SQL] [Minor] Move the testData into beforeAll()
chenghao-intel May 21, 2015
a25c1ab
[SPARK-7565] [SQL] fix MapType in JsonRDD
May 21, 2015
13348e2
[SPARK-7752] [MLLIB] Use lowercase letters for NaiveBayes.modelType
mengxr May 21, 2015
8730fbb
[SPARK-7749] [SQL] Fixes partition discovery for non-partitioned tables
liancheng May 21, 2015
4b7ff30
[SPARK-7787] [STREAMING] Fix serialization issue of SerializableAWSCr…
tdas May 21, 2015
6e53402
[SPARK-6416] [DOCS] RDD.fold() requires the operator to be commutative
srowen May 21, 2015
699906e
[SPARK-7394][SQL] Add Pandas style cast (astype)
kaka1992 May 21, 2015
4f57200
[SPARK-7793] [MLLIB] Use getOrElse for getting the threshold of SVM m…
coderxiang May 21, 2015
f6c486a
[SQL] [TEST] udf_java_method failed due to jdk version
scwf May 21, 2015
15680ae
[SPARK-7775] YARN AM negative sleep exception
May 21, 2015
6d75ed7
[SPARK-7585] [ML] [DOC] VectorIndexer user guide section
jkbradley May 21, 2015
cdc7c05
[SPARK-7498] [MLLIB] add varargs back to setDefault
mengxr May 21, 2015
311fab6
[SPARK-7722] [STREAMING] Added Kinesis to style checker
tdas May 21, 2015
30f3f55
[SPARK-7763] [SPARK-7616] [SQL] Persists partition columns into metas…
yhuai May 21, 2015
3d0cccc
[SPARK-7478] [SQL] Added SQLContext.getOrCreate
tdas May 21, 2015
6b18cdc
[SPARK-7711] Add a startTime property to match the corresponding one …
holdenk May 21, 2015
5287eec
[SPARK-7718] [SQL] Speed up partitioning by avoiding closure cleaning
May 21, 2015
5a3c04b
[SPARK-7800] isDefined should not marked too early in putNewKey
viirya May 21, 2015
147b6be
[BUILD] Always run SQL tests in master build.
yhuai May 21, 2015
347b501
[SPARK-7737] [SQL] Use leaf dirs having data files to discover partit…
yhuai May 21, 2015
d68ea24
[SPARK-7776] [STREAMING] Added shutdown hook to StreamingContext
tdas May 22, 2015
17791a5
[SPARK-7783] [SQL] [PySpark] add DataFrame.rollup/cube in Python
May 22, 2015
f5db4b4
[SPARK-7794] [MLLIB] update RegexTokenizer default settings
mengxr May 22, 2015
85b9637
[SPARK-7219] [MLLIB] Output feature attributes in HashingTF
mengxr May 22, 2015
956c4c9
[SPARK-7657] [YARN] Add driver logs links in application UI, in clust…
harishreedharan May 22, 2015
e4136ea
[DOCS] [MLLIB] Fixing broken link in MLlib Linear Methods documentation.
dusenberrymw May 22, 2015
8f11c61
[SPARK-7535] [.0] [MLLIB] Audit the pipeline APIs for 1.4
mengxr May 22, 2015
2728c3d
[SPARK-7578] [ML] [DOC] User guide for spark.ml Normalizer, IDF, Stan…
jkbradley May 22, 2015
f6f2eeb
[SPARK-7322][SQL] Window functions in DataFrame
chenghao-intel May 22, 2015
4e5220c
[MINOR] [SQL] Ignores Thrift server UISeleniumSuite
liancheng May 22, 2015
3b68cb0
[SPARK-6743] [SQL] Fix empty projections of cached data
marmbrus May 22, 2015
f490b3b
[SPARK-7404] [ML] Add RegressionEvaluator to spark.ml
May 22, 2015
c63036c
Revert "[BUILD] Always run SQL tests in master build."
pwendell May 22, 2015
509d55a
[SPARK-7574] [ML] [DOC] User guide for OneVsRest
May 22, 2015
eac0069
[SPARK-7766] KryoSerializerInstance reuse is unsafe when auto-reset i…
JoshRosen May 22, 2015
31d5d46
[SPARK-7758] [SQL] Override more configs to avoid failure when connec…
WangTaoTheTonic May 22, 2015
e4aef91
[SPARK-7724] [SQL] Support Intersect/Except in Catalyst DSL.
smola May 22, 2015
126d723
[SPARK-7270] [SQL] Consider dynamic partition when inserting into hiv…
viirya May 22, 2015
821254f
[SPARK-7760] add /json back into master & worker pages; add test
squito May 22, 2015
3c13051
[SPARK-7834] [SQL] Better window error messages
marmbrus May 23, 2015
3d8760d
[SPARK-7771] [SPARK-7779] Dynamic allocation: lower default timeouts …
May 23, 2015
1c388a9
[SPARK-7788] Made KinesisReceiver.onStart() non-blocking
tdas May 23, 2015
8014e1f
[SPARK-7224] [SPARK-7306] mock repository generator for --packages te…
brkyvz May 23, 2015
63a5ce7
[SPARK-7830] [DOCS] [MLLIB] Adding logistic regression to the list of…
dusenberrymw May 23, 2015
a163574
[SPARK-7795] [CORE] Speed up task scheduling in standalone mode by re…
coolfrood May 23, 2015
017b340
[MINOR] Add SparkR to create-release script
shivaram May 23, 2015
baa8983
[SPARK-7838] [STREAMING] Set scope for kinesis stream
tdas May 23, 2015
368b8c2
[HOTFIX] Add tests for SparkListenerApplicationStart with Driver Logs.
harishreedharan May 23, 2015
4583cf4
[SPARK-5090] [EXAMPLES] The improvement of python converter for hbase
GenTang May 23, 2015
7af3818
[SPARK-6806] [SPARKR] [DOCS] Fill in SparkR examples in programming g…
May 23, 2015
a40bca0
[SPARK-6811] Copy SparkR lib in make-distribution.sh
shivaram May 23, 2015
ad0badb
[SPARK-7777][Streaming] Handle the case when there is no block in a b…
zsxwing May 23, 2015
8883e3e
Add dynamic allocation to the Spark coarse-grained scheduler.
dragos Mar 10, 2015
8b237ff
Reviewer’s comments.
dragos May 11, 2015
b17dbd4
Use the `SchedulerDriver` interface instead of the Mesos implementation.
dragos May 12, 2015
5005cc2
[WIP] Add persistent shuffle directory
dragos May 21, 2015
d50249a
Call “applicationRemoved” when the application ends and delete extern…
dragos May 23, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions R/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ The SparkR documentation (Rd files and HTML files) are not a part of the source
SparkR comes with several sample programs in the `examples/src/main/r` directory.
To run one of them, use `./bin/sparkR <filename> <args>`. For example:

./bin/sparkR examples/src/main/r/pi.R local[2]
./bin/sparkR examples/src/main/r/dataframe.R

You can also run the unit-tests for SparkR by running (you need to install the [testthat](http://cran.r-project.org/web/packages/testthat/index.html) package first):

Expand All @@ -63,5 +63,5 @@ You can also run the unit-tests for SparkR by running (you need to install the [
The `./bin/spark-submit` and `./bin/sparkR` can also be used to submit jobs to YARN clusters. You will need to set YARN conf dir before doing so. For example on CDH you can run
```
export YARN_CONF_DIR=/etc/hadoop/conf
./bin/spark-submit --master yarn examples/src/main/r/pi.R 4
./bin/spark-submit --master yarn examples/src/main/r/dataframe.R
```
5 changes: 4 additions & 1 deletion R/pkg/NAMESPACE
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# Imports from base R
importFrom(methods, setGeneric, setMethod, setOldClass)
useDynLib(SparkR, stringHashCode)

# Disable native libraries till we figure out how to package it
# See SPARKR-7839
#useDynLib(SparkR, stringHashCode)

# S3 methods exported
export("sparkR.init")
Expand Down
Loading