DataBricks: Spark runtime version

[pieterheijmans]

Cosmos DB scenario-based labs - IoT hands-on lab step-by-step
Hi,

First of all. This lab is really good!

However, I executed this lab and ran into some errors regarding Databricks and the notbook execution on the cluster.

Runtime 6.1 (Scala 2.11, Spark 2.4.4).
com.microsoft.azure:azure-cosmosdb-spark_2.4.0_2.11:1.4.1
https://github.com/AzureCosmosDB/scenario-based-labs/blob/master/IoT/Notebooks/01%20IoT.dbc

I tried this configuration set-up last week and the notebook with this spark configuration and the current notebook version is not working (anymore).

Perhaps you have a more recent version of the notebook working with a newer version of the runtime.
Please contact me about this topic which runtime, spark, notebook version to use to.

Thanks in advance.

Kind regards,

Pieter Heijmans. Azure Solution Architect.

[pieterheijmans]

Hi @joelhulen, currently I am organizing an IoT challenge based on this lab for my colleagues.
The new 2020 version of the IoT lab with Azure Synapse Workspace looks very promising!

Already 35 colleagues executed the 2019 version. Besides the Databricks issue, we got very positive feedback for this lab.
Keep up the good work!.

Some of them even replaced CosmosDB by the new serverless CosmosDB version to cut down the costs of the lap execution.
And this worked out fine. We only had to skip the RU scale code in the .net code. Perhaps you can invoke this in your latest version.

I already downloaded and installed the 2020 ARM template version and with a smaller team we will execute this version as well.
This new version is more suitable of a wider audience, because less technical skills are needed to implement batch and realtime machine learning.

Kind regards,

Pieter Heijmans.

[joelhulen]

This sounds great, @pieterheijmans. Thank you for the feedback! I'm very glad that the new version appears to be working well for you and that your attendees are finding a lot of value with the current version. Good luck with your IoT challenge!

Feel free to use the Cosmos DB IoT solution accelerator as a resource for more detailed information about the architecture and IoT aspects as well: https://github.com/AzureCosmosDB/solution-accelerator