Skip to content

[SPARK-4908][SQL] Prevent multiple concurrent hive native commands #3834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-4908][SQL] Prevent multiple concurrent hive native commands #3834

wants to merge 1 commit into from

Conversation

marmbrus
Copy link
Contributor

This is just a quick fix that locks when calling runHive. If we can find a way to avoid the error without a global lock that would be better.

@SparkQA
Copy link

SparkQA commented Dec 30, 2014

Test build #24879 has started for PR 3834 at commit bf25300.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 30, 2014

Test build #24879 has finished for PR 3834 at commit bf25300.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24879/
Test PASSed.

@liancheng
Copy link
Contributor

According to my earlier investigation, this issue seems to be related to the asynchronous execution feature enabled by default in Hive 0.13.1 JDBC driver. I'm not pretty sure whether synchronizes HiveContext.runHive completely fixes this issue. But I'd vote to merge this first before we find the root cause, since doing so does no harm.

@marmbrus
Copy link
Contributor Author

Merged to master and branch-1.2

asfgit pushed a commit that referenced this pull request Dec 30, 2014
@marmbrus
[SPARK-4908][SQL] Prevent multiple concurrent hive native commands
cde8a31 
This is just a quick fix that locks when calling `runHive`.  If we can find a way to avoid the error without a global lock that would be better.

Author: Michael Armbrust <[email protected]>

Closes #3834 from marmbrus/hiveConcurrency and squashes the following commits:

bf25300 [Michael Armbrust] prevent multiple concurrent hive native commands

(cherry picked from commit 480bd1d)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in 480bd1d Dec 30, 2014
@liancheng
Copy link
Contributor

Just for reference, the root cause behind this issue is discussed in SPARK-4908.

@baishuo baishuo mentioned this pull request Jan 12, 2015
Closed
@marmbrus marmbrus deleted the hiveConcurrency branch February 17, 2015 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@marmbrus @SparkQA @AmplabJenkins @liancheng