How to implement toRedisHASH(keyMaps: RDD[(String, Map[String, String])], ttl: Int = 0)

[zhou533]

The current implementation of toRedisHASH seems write one key per time. Now, I wanna write a RDD
[(String, Map[String, String])], is there possible? or any suggestions?

[mnarrell]

This is interesting as well. Why isn't this the RDD returned from fromRedisHash?? You lose the key-context with the current API. You cannot tell which hashes belong to which keys, without embedding the keys themselves in the hash.

Or am I missing something fundamental?

[zqhxuyuan]

current the implementation of save multi map to redis

    val wc = sc.parallelize(List(
      ("hello", "1")
    ))
    sc.toRedisHASH(wc, "map1")
    sc.toRedisHASH(wc, "map2")
    sc.toRedisHASH(wc, "map3")

what if I have an rdd like

    val wcBatch = sc.parallelize(List(
      ("map1", "hello", "1"),
      ("map2", "hello", "1"),
      ("map3", "hello", "1")
    ))

How can I batch save multi map into redis with single method call?

this batch style also work for list, zset and so on..

PS: If I use pure redis api, I can do this job with pipeline

    val rdd = sc.parallelize(List(
      ("map1", "key1", "value1"),
      ("map1", "key2", "value2"),
      ("map1", "key3", "value3"),
      ("map2", "key1", "value1")
    ))
    rdd.foreachPartition{iter =>
      val redis = new Jedis("localhost",6379,400000)
      val ppl = redis.pipelined()
      iter.foreach{row =>
        val mapKey = row._1
        val key = row._2
        val value = row._3
        ppl.hmset(mapKey, Map(key -> value))
      }
      ppl.sync()
    }

[simalince]

+1 on this issue.

[charsyam]

@gkorland @fe2s Is there any progress?

[fe2s]

@charsyam , sorry, no progress on this yet