Skip to content

Conversation

Niharikadutta
Copy link
Collaborator

@Niharikadutta Niharikadutta commented Jun 18, 2020

This PR fixes a bug found when using UDFs.

The class JvmBroadcastRegistry has a variable defined that stores the list of all active Broadcast variables' jvmObjects that are to be shipped to the workers. This variable was marked as ThreadStatic to maintain a thread-specific copy of the object. The way ThreadStatic works is it only initializes the object for the very first thread that uses it, and subsequent threads are not initialized. This caused a Null reference exception to be thrown when using UDFs that used a different thread.

Solution: This was fixed by declaring it to be a ThreadLocal<> variable instead, that let's you provide an initialization function, which would define a new instance every time before it's accessed in a new thread.

@imback82
Copy link
Contributor

Is this still WIP?

@imback82
Copy link
Contributor

Can you also update the description please (make it more general instead of limiting to interactive session)?

@imback82 imback82 added the bug Something isn't working label Jun 20, 2020
@imback82 imback82 requested a review from suhsteve June 20, 2020 17:06
@Niharikadutta Niharikadutta changed the title [WIP] Broadcast variable bug fix UDF ThreadStatic bug fix Jun 21, 2020
@Niharikadutta
Copy link
Collaborator Author

@imback82 Updated the description and opened the PR for review. Thanks!

Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test? Any bug fix should come with a test that validates the fix. I think different threads just defining UDFs should suffice.

@suhsteve
Copy link
Member

Can you update the title to say it's related to Broadcast?

@Niharikadutta Niharikadutta changed the title UDF ThreadStatic bug fix UDF bug fix caused by ThreadStatic BroadcastVariablesRegistry Jun 21, 2020
@suhsteve
Copy link
Member

LGTM.

Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @Niharikadutta!

@imback82 imback82 merged commit 1cd9cca into dotnet:master Jun 22, 2020
@Niharikadutta Niharikadutta deleted the BroadcastBugFix branch August 24, 2020 01:25
@Niharikadutta Niharikadutta restored the BroadcastBugFix branch August 24, 2020 01:25
@Niharikadutta Niharikadutta deleted the BroadcastBugFix branch August 24, 2020 01:26
@suhsteve suhsteve added fixing bug Fixing a bug and removed bug Something isn't working labels Oct 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixing bug Fixing a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants