Update code to use Spark 3.0 & Scala 2.12

[alexott]

Most of tests are passing, except the read with offset option - I suspect that it's caused by incorrect usage of the SparkContext inside the test, so context is stopped, leading to eviction of the blocks.

test log is attached: test.log

[areese]

@alexott I managed to get the tests to pass by increasing the timeout in src/test/scala/org/apache/spark/sql/redis/stream/RedisStreamSourceSuite.scala:231 from 50 milliseconds to 5000 milliseconds. ;)

[alexott]

@areese even 2 seconds is enough on my old laptop... I've tried to do while(!query.isActive) {Thread.sleep(50)} but it didn't help

[areese]

@alexott I used 5000 cause it was simple. ;).
I think the issue with the loop is that the code closes the SparkContext before the query has been emitted.

I bet sleeping for 1s, and then doing the query is active might work.

[jeisinge]

In some of our streaming code, we also need to wait for the state of streams. I figured I would share a bit of that code here --- maybe it would be useful for you all?

def untilStreamIsReady(name:String):Unit = {
  val queries = spark.streams.active.filter(_.name == name)

  if (queries.length == 0) {
    println("The stream is not active.")
  } else {
    while (queries(0).isActive && queries(0).recentProgress.length == 0) {
      // wait until there is any type of progress
    }

    if (queries(0).isActive) {
      queries(0).awaitTermination(5*1000)
      println("The stream is active and ready.")
    } else {
      println("The stream is not active.")
    }
  }
}

[fe2s]

Thanks @alexott, the PR looks good!
We will merge the master into branch-2.4 and merge this PR after that.

[fe2s]

Hi @alexott, sorry, it's been a long time since this PR was created, but we sorted out some other PR that blocked this one just now. Would you still be able to merge the conflict here?

[alexott]

@fe2s I've rebased this branch onto master