Skip to content

Commit 08d203a

Browse files
NiharikaduttaNiharikadutta
authored
[DOC] Adding guides to explain UDF serialization and Broadcast variable usage (#464)
1 parent ba609f5 commit 08d203a

File tree

2 files changed

+263
-0
lines changed

2 files changed

+263
-0
lines changed

‎docs/broadcast-guide.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Guide to using Broadcast Variables
2+
3+
This is a guide to show how to use broadcast variables in .NET for Apache Spark.
4+
5+
## What are Broadcast Variables
6+
7+
[Broadcast variables in Apache Spark](https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html#broadcast-variables) are a mechanism for sharing variables across executors that are meant to be read-only. They allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used, for example, to give every node a copy of a large input dataset in an efficient manner.
8+
9+
### How to use broadcast variables in .NET for Apache Spark
10+
11+
Broadcast variables are created from a variable `v` by calling `SparkContext.Broadcast(v)`. The broadcast variable is a wrapper around `v`, and its value can be accessed by calling the `Value()` method.
12+
13+
Example:
14+
15+
```csharp
16+
string v = "Variable to be broadcasted";
17+
Broadcast<string> bv = SparkContext.Broadcast(v);
18+
19+
// Using the broadcast variable in a UDF:
20+
Func<Column, Column> udf = Udf<string, string>(
21+
str => $"{str}: {bv.Value()}");
22+
```
23+
24+
The type parameter for `Broadcast` should be the type of the variable being broadcasted.
25+
26+
### Deleting broadcast variables
27+
28+
The broadcast variable can be deleted from all executors by calling the `Destroy()` method on it.
29+
30+
```csharp
31+
// Destroying the broadcast variable bv:
32+
bv.Destroy();
33+
```
34+
35+
> Note: `Destroy()` deletes all data and metadata related to the broadcast variable. Use this with caution - once a broadcast variable has been destroyed, it cannot be used again.
36+
37+
#### Caveat of using Destroy
38+
39+
One important thing to keep in mind while using broadcast variables in UDFs is to limit the scope of the variable to only the UDF that is referencing it. The [guide to using UDFs](udf-guide.md) describes this phenomenon in detail. This is especially crucial when calling `Destroy` on the broadcast variable. If the broadcast variable that has been destroyed is visible to or accessible from other UDFs, it gets picked up for serialization by all those UDFs, even if it is not being referenced by them. This will throw an error as .NET for Apache Spark is not able to serialize the destroyed broadcast variable.
40+
41+
Example to demonstrate:
42+
43+
```csharp
44+
string v = "Variable to be broadcasted";
45+
Broadcast<string> bv = SparkContext.Broadcast(v);
46+
47+
// Using the broadcast variable in a UDF:
48+
Func<Column, Column> udf1 = Udf<string, string>(
49+
str => $"{str}: {bv.Value()}");
50+
51+
// Destroying bv
52+
bv.Destroy();
53+
54+
// Calling udf1 after destroying bv throws the following expected exception:
55+
// org.apache.spark.SparkException: Attempted to use Broadcast(0) after it was destroyed
56+
df.Select(udf1(df["_1"])).Show();
57+
58+
// Different UDF udf2 that is not referencing bv
59+
Func<Column, Column> udf2 = Udf<string, string>(
60+
str => $"{str}: not referencing broadcast variable");
61+
62+
// Calling udf2 throws the following (unexpected) exception:
63+
// [Error] [JvmBridge] org.apache.spark.SparkException: Task not serializable
64+
df.Select(udf2(df["_1"])).Show();
65+
```
66+
67+
The recommended way of implementing above desired behavior:
68+
69+
```csharp
70+
string v = "Variable to be broadcasted";
71+
// Restricting the visibility of bv to only the UDF referencing it
72+
{
73+
Broadcast<string> bv = SparkContext.Broadcast(v);
74+
75+
// Using the broadcast variable in a UDF:
76+
Func<Column, Column> udf1 = Udf<string, string>(
77+
str => $"{str}: {bv.Value()}");
78+
79+
// Destroying bv
80+
bv.Destroy();
81+
}
82+
83+
// Different UDF udf2 that is not referencing bv
84+
Func<Column, Column> udf2 = Udf<string, string>(
85+
str => $"{str}: not referencing broadcast variable");
86+
87+
// Calling udf2 works fine as expected
88+
df.Select(udf2(df["_1"])).Show();
89+
```
90+
This ensures that destroying `bv` doesn't affect calling `udf2` because of unexpected serialization behavior.
91+
92+
Broadcast variables are useful for transmitting read-only data to all executors, as the data is sent only once and this can give performance benefits when compared with using local variables that get shipped to the executors with each task. Please refer to the [official documentation](https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html#broadcast-variables) to get a deeper understanding of broadcast variables and why they are used.

‎docs/udf-guide.md

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Guide to User-Defined Functions (UDFs)
2+
3+
This is a guide to show how to use UDFs in .NET for Apache Spark.
4+
5+
## What are UDFs
6+
7+
[User-Defined Functions (UDFs)](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/expressions/UserDefinedFunction.html) are a feature of Spark that allow developers to use custom functions to extend the system's built-in functionality. They transform values from a single row within a table to produce a single corresponding output value per row based on the logic defined in the UDF.
8+
9+
Let's take the following as an example for a UDF definition:
10+
11+
```csharp
12+
string s1 = "hello";
13+
Func<Column, Column> udf = Udf<string, string>(
14+
str => $"{s1} {str}");
15+
16+
```
17+
The above defined UDF takes a `string` as an input (in the form of a [Column](https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Sql/Column.cs#L14) of a [Dataframe](https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Sql/DataFrame.cs#L24)), and returns a `string` with `hello` appended in front of the input.
18+
19+
For a sample Dataframe, let's take the following Dataframe `df`:
20+
21+
```text
22+
+-------+
23+
| name|
24+
+-------+
25+
|Michael|
26+
| Andy|
27+
| Justin|
28+
+-------+
29+
```
30+
31+
Now let's apply the above defined `udf` to the dataframe `df`:
32+
33+
```csharp
34+
DataFrame udfResult = df.Select(udf(df["name"]));
35+
```
36+
37+
This would return the below as the Dataframe `udfResult`:
38+
39+
```text
40+
+-------------+
41+
| name|
42+
+-------------+
43+
|hello Michael|
44+
| hello Andy|
45+
| hello Justin|
46+
+-------------+
47+
```
48+
To get a better understanding of how to implement UDFs, please take a look at the [UDF helper functions](https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Sql/Functions.cs#L3616) and some [test examples](https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark.E2ETest/UdfTests/UdfSimpleTypesTests.cs#L49).
49+
50+
## UDF serialization
51+
52+
Since UDFs are functions that need to be executed on the workers, they have to be serialized and sent to the workers as part of the payload from the driver. This involves serializing the [delegate](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/delegates/) which is a reference to the method, along with its [target](https://docs.microsoft.com/en-us/dotnet/api/system.delegate.target?view=netframework-4.8) which is the class instance on which the current delegate invokes the instance method. Please take a look at this [code](https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark/Utils/CommandSerDe.cs#L149) to get a better understanding of how UDF serialization is being done.
53+
54+
## Good to know while implementing UDFs
55+
56+
One behavior to be aware of while implementing UDFs in .NET for Apache Spark is how the target of the UDF gets serialized. .NET for Apache Spark uses .NET Core, which does not support serializing delegates, so it is instead done by using reflection to serialize the target where the delegate is defined. When multiple delegates are defined in a common scope, they have a shared closure that becomes the target of reflection for serialization. Let's take an example to illustrate what that means.
57+
58+
The following code snippet defines two string variables that are being referenced in two function delegates that return the respective strings as result:
59+
60+
```csharp
61+
using System;
62+
63+
public class C {
64+
public void M() {
65+
string s1 = "s1";
66+
string s2 = "s2";
67+
Func<string, string> a = str => s1;
68+
Func<string, string> b = str => s2;
69+
}
70+
}
71+
```
72+
73+
The above C# code generates the following C# disassembly (credit source: [sharplab.io](https://sharplab.io)) code from the compiler:
74+
75+
```csharp
76+
public class C
77+
{
78+
[CompilerGenerated]
79+
private sealed class <>c__DisplayClass0_0
80+
{
81+
public string s1;
82+
83+
public string s2;
84+
85+
internal string <M>b__0(string str)
86+
{
87+
return s1;
88+
}
89+
90+
internal string <M>b__1(string str)
91+
{
92+
return s2;
93+
}
94+
}
95+
96+
public void M()
97+
{
98+
<>c__DisplayClass0_0 <>c__DisplayClass0_ = new <>c__DisplayClass0_0();
99+
<>c__DisplayClass0_.s1 = "s1";
100+
<>c__DisplayClass0_.s2 = "s2";
101+
Func<string, string> func = new Func<string, string>(<>c__DisplayClass0_.<M>b__0);
102+
Func<string, string> func2 = new Func<string, string>(<>c__DisplayClass0_.<M>b__1);
103+
}
104+
}
105+
```
106+
As can be seen in the above decompiled code, both `func` and `func2` share the same closure `<>c__DisplayClass0_0`, which is the target that is serialized when serializing the delegates `func` and `func2`. Hence, even though `Func<string, string> a` is only referencing `s1`, `s2` also gets serialized when sending over the bytes to the workers.
107+
108+
This can lead to some unexpected behaviors at runtime (like in the case of using [broadcast variables](broadcast-guide.md)), which is why we recommend restricting the visibility of the variables used in a function to that function's scope.
109+
110+
Going back to the above example, the following is the recommended way to implement the desired behavior of previous code snippet:
111+
112+
```csharp
113+
using System;
114+
115+
public class C {
116+
public void M() {
117+
{
118+
string s1 = "s1";
119+
Func<string, string> a = str => s1;
120+
}
121+
{
122+
string s2 = "s2";
123+
Func<string, string> b = str => s2;
124+
}
125+
}
126+
}
127+
```
128+
129+
The above C# code generates the following C# disassembly (credit source: [sharplab.io](https://sharplab.io)) code from the compiler:
130+
131+
```csharp
132+
public class C
133+
{
134+
[CompilerGenerated]
135+
private sealed class <>c__DisplayClass0_0
136+
{
137+
public string s1;
138+
139+
internal string <M>b__0(string str)
140+
{
141+
return s1;
142+
}
143+
}
144+
145+
[CompilerGenerated]
146+
private sealed class <>c__DisplayClass0_1
147+
{
148+
public string s2;
149+
150+
internal string <M>b__1(string str)
151+
{
152+
return s2;
153+
}
154+
}
155+
156+
public void M()
157+
{
158+
<>c__DisplayClass0_0 <>c__DisplayClass0_ = new <>c__DisplayClass0_0();
159+
<>c__DisplayClass0_.s1 = "s1";
160+
Func<string, string> func = new Func<string, string>(<>c__DisplayClass0_.<M>b__0);
161+
<>c__DisplayClass0_1 <>c__DisplayClass0_2 = new <>c__DisplayClass0_1();
162+
<>c__DisplayClass0_2.s2 = "s2";
163+
Func<string, string> func2 = new Func<string, string>(<>c__DisplayClass0_2.<M>b__1);
164+
}
165+
}
166+
```
167+
168+
Here we see that `func` and `func2` no longer share a closure and have their own separate closures `<>c__DisplayClass0_0` and `<>c__DisplayClass0_1` respectively. When used as the target for serialization, nothing other than the referenced variables will get serialized for the delegate.
169+
170+
This behavior is important to keep in mind while implementing multiple UDFs in a common scope.
171+
To learn more about UDFs in general, please review the following articles that explain UDFs and how to use them: [UDFs in databricks(scala)](https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html), [Spark UDFs and some gotchas](https://medium.com/@achilleus/spark-udfs-we-can-use-them-but-should-we-use-them-2c5a561fde6d).

0 commit comments

Comments
 (0)