Writing Byte Array to Hash

[MartinsGabrielC]

I'm currently writing a dataframe/RDD consisting of <Key: String, Data: Array[Byte]>.
I have the scenario where we have a hash representing a entity and some of the data is loaded through a spark job and some is loaded by an API.
The API also need to read the data that is loaded by the spark job.
We use data as byte array to reduce it's size, the data itself is compressed and encoded.

We are using Jedis to read the data and also run some tests doing direct load which worked fine.
Below the first result is doing direct load, which is then returning the correct value and the second is using Dataframe load.

From #205 I saw that byte array Lists was implemented, is it possible to have the same for Hashes? Or is there an workaround for it?

Thanks in advance.

[fe2s]

Hi @MartinsGabrielC , I think it should be possible to add DataFrame support for byte array fields. I will take a look.

[fe2s]

Hi @MartinsGabrielC ,
Adding dataframe support is not a trivial change. Instead I prototyped a function to write RDD.
Would you be able to check it out in branch https://github.com/RedisLabs/spark-redis/tree/issue-304-toRedisByteHASHes ?
The function is called toRedisByteHASHes. All arguments are represented as Array[Byte]. If you have String in your case you can convert it with .getBytes into array.

[MartinsGabrielC]

Hi @fe2s,

Thanks for the return, I built it locally and made the apropriate changes and now it's working 😄

[MartinsGabrielC]

Hi @fe2s,

Thanks again for the implementation. Would you know when it will be available in a release?

Best regards.

[fe2s]

@MartinsGabrielC ,
sorry for the delay, I've created a PR #306
Should be released in a couple of days.

[MartinsGabrielC]

Hey guys, thanks for the release,
I was checking it was only added to the Scala 2.12 version, is it possible to add it to the Scala 2.11 as well?