Where to find spark-redis 2.4.0

[thedevd]

I am using spark 2.4.0, and I do not see dependency for spark-redislabs 2.4.0 in maven central (Dependencies till 2.3.1 are only available - https://mvnrepository.com/artifact/com.redislabs/spark-redis).
And if I use spark-redis 2.3.1, my build fails as jedis-3.0.0-20181113.105826-9.jar is required by this.
How to fix both issues.

These are dependencies I am using now -
// to write spark stream in redis
compile group: 'com.redislabs', name: 'spark-redis', version: '2.3.1'
compile group: 'redis.clients', name: 'jedis', version: '3.0.0'

[fe2s]

Spark-Redis 2.4.0 should be released to Maven Central very soon.

The 2.3.1 + SBT is a known issue, the workaround is described in this comment

[thedevd]

@fe2s Please let me know here, whenever it is released.
One thing to ask, can I use 2.3.1 as of now with spark 2.4.0.

[thedevd]

@fe2s I have started facing another issue when using version 2.3.1 with spark 2.4.0, This is how I am reading from redis-

import com.redislabs.provider.redis._
val keyRdd = spark.sparkContext.fromRedisHash("tokenToSymbol")

Above lines throws this exception at runtime-

Symbol 'type .streaming.StreamingContext' is missing from the classpath.
This symbol is required by 'value com.redislabs.provider.redis.RedisFunctions.ssc'.
Make sure that type StreamingContext is in your classpath and check for conflicting dependencies with -Ylog-classpath.
A full rebuild may help if 'RedisFunctions.class' was compiled against an incompatible version of .streaming.
val keyRdd = spark.sparkContext.fromRedisHash("tokenToSymbol")

[fe2s]

Do you run it as a standalone application or submitting to spark cluster? Do you have spark-streaming_2.11 declared in your maven/sbt? Can you provide your maven/sbt here please?

[thedevd]

@fe2s Running it as a standalone currently.
No I dont have that. This is my gradle dependencies -

group 'myapp'
version '1.0'

apply plugin: 'scala'
apply plugin: 'idea'
apply plugin: 'eclipse'

repositories {
mavenLocal()
mavenCentral()
maven { url "https://mvnrepository.com" }
maven { url "https://oss.sonatype.org/content/repositories/snapshots" }
}

idea {
project {
jdkName = '1.8'
languageLevel = '1.8'
}
}

dependencies {

compile 'org.scala-lang:scala-library:2.11.6'
compile (group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.0'){
    exclude group:"com.fasterxml.jackson.core"
}
compile (group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.0'){
    exclude group:"com.fasterxml.jackson.core"
}
compile group: 'org.scala-lang', name: 'scala-library', version: '2.11.11'

compile (group: 'org.apache.kafka', name: 'kafka_2.11', version: '2.0.1'){
    exclude group:"com.fasterxml.jackson.core"
}

compile group: 'com.opencsv', name: 'opencsv', version: '4.1'

compile group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1'

compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.6.7.2'


// https://mvnrepository.com/artifact/org.apache.spark/spark-streaming
compile group: 'org.apache.spark', name: 'spark-sql-kafka-0-10_2.11', version: '2.4.0'

// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.4.0'

// This is workaround, once spark-redis 2.4.0 available this should be fixed
compile (group: 'com.redislabs', name: 'spark-redis', version: '2.3.1') {
    exclude group: "com.redislabs.jedis"
}
compile group: 'com.redislabs', name:'jedis', version:'3.0.0-20181113.105826-9'
compile group: 'redis.clients', name: 'jedis', version: '3.0.0'

// https://mvnrepository.com/artifact/mysql/mysql-connector-java
compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.16'

// https://mvnrepository.com/artifact/net.debasishg/redisclient
compile group: 'net.debasishg', name: 'redisclient_2.11', version: '3.9'

}

compileScala.targetCompatibility = "1.8"
compileScala.sourceCompatibility = "1.8"

[fe2s]

@thedevd , please, include additional dependency spark-streaming_2.11

This will be fixed in Spark-Redis 2.4.0

[thedevd]

@fe2s Thanks, I was then able to fetch key-value from redis after adding spark-streaming_2.11 dependency.

      val spark = SparkSession.builder()
      .appName("Spark-Redis-Demo")
      .master("local[*]")
      .config("spark.sql.shuffle.partitions", 8)
      .config("spark.redis.host", "localhost")
      .config("spark.redis.port", "6379")
      .getOrCreate()

    import com.redislabs.provider.redis._
    val keyRDD = spark.sparkContext.fromRedisKeyPattern("tokenToSymbol*")
    var myMap = Map[String,String]()
    keyRDD.getKV().foreach(kvRDD => {
      myMap += kvRDD._1 -> kvRDD._2
    })

[fe2s]

Spark-Redis 2.4.0 has been released to Maven, so closing this issue.