[SPARK-33143][PYTHON] Add configurable timeout to python server and client #30389

gaborgsomogyi · 2020-11-16T15:19:09Z

What changes were proposed in this pull request?

Spark creates local server to serialize several type of data for python. The python code tries to connect to the server, immediately after it's created but there are several system calls in between (this may change in each Spark version):

getaddrinfo
socket
settimeout
connect

Under some circumstances in heavy user environments these calls can be super slow (more than 15 seconds). These issues must be analyzed one-by-one but since these are system calls the underlying OS and/or DNS servers must be debugged and fixed. This is not trivial task and at the same time data processing must work somehow. In this PR I'm only intended to add a configuration possibility to increase the mentioned timeouts in order to be able to provide temporary workaround. The rootcause analysis is ongoing but I think this can vary in each case.

Because the server part doesn't contain huge amount of log entries to with one can measure time, I've added some.

Why are the changes needed?

Provide workaround when localhost python server connection timeout appears.

Does this PR introduce any user-facing change?

Yes, new configuration added.

How was this patch tested?

Existing unit tests + manual test.

#Compile Spark

echo "spark.io.encryption.enabled true" >> conf/spark-defaults.conf
echo "spark.python.authenticate.socketTimeout 10" >> conf/spark-defaults.conf

$ ./bin/pyspark 
Python 3.8.5 (default, Jul 21 2020, 10:48:26) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
20/11/20 10:17:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/11/20 10:17:03 WARN SparkEnv: I/O encryption enabled without RPC encryption: keys will be visible on the wire.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Python version 3.8.5 (default, Jul 21 2020 10:48:26)
Spark context Web UI available at http://192.168.0.189:4040
Spark context available as 'sc' (master = local[*], app id = local-1605863824276).
SparkSession available as 'spark'.
>>> sc.setLogLevel("TRACE")
>>> sc.parallelize([0, 2, 3, 4, 6], 5).glom().collect()
20/11/20 10:17:09 TRACE PythonParallelizeServer: Creating listening socket
20/11/20 10:17:09 TRACE PythonParallelizeServer: Setting timeout to 10 sec
20/11/20 10:17:09 TRACE PythonParallelizeServer: Waiting for connection on port 59726
20/11/20 10:17:09 TRACE PythonParallelizeServer: Connection accepted from address /127.0.0.1:59727
20/11/20 10:17:09 TRACE PythonParallelizeServer: Client authenticated
20/11/20 10:17:09 TRACE PythonParallelizeServer: Closing server
...
20/11/20 10:17:10 TRACE SocketFuncServer: Creating listening socket             
20/11/20 10:17:10 TRACE SocketFuncServer: Setting timeout to 10 sec
20/11/20 10:17:10 TRACE SocketFuncServer: Waiting for connection on port 59735
20/11/20 10:17:10 TRACE SocketFuncServer: Connection accepted from address /127.0.0.1:59736
20/11/20 10:17:10 TRACE SocketFuncServer: Client authenticated
20/11/20 10:17:10 TRACE SocketFuncServer: Closing server
[[0], [2], [3], [4], [6]]
>>>

…lient

gaborgsomogyi · 2020-11-16T15:29:26Z

It's still WIP because hesitating how it should be named and would like to see how tests behaving.

SparkQA · 2020-11-16T15:30:08Z

Test build #131169 has finished for PR 30389 at commit 09b1235.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-16T16:02:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35771/

SparkQA · 2020-11-16T16:35:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35773/

SparkQA · 2020-11-16T16:36:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35771/

SparkQA · 2020-11-16T17:04:11Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35773/

SparkQA · 2020-11-16T17:58:57Z

Test build #131170 has finished for PR 30389 at commit 17d357b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-17T11:23:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35820/

SparkQA · 2020-11-17T11:53:21Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35820/

SparkQA · 2020-11-17T12:15:57Z

Test build #131217 has finished for PR 30389 at commit 424be64.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2020-11-17T12:17:53Z

retest this please

SparkQA · 2020-11-17T13:05:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35825/

SparkQA · 2020-11-17T13:26:22Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35825/

core/src/main/scala/org/apache/spark/internal/config/Python.scala

gaborgsomogyi · 2020-11-17T14:22:56Z

cc @HyukjinKwon

SparkQA · 2020-11-17T14:51:14Z

Test build #131222 has finished for PR 30389 at commit 424be64.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-17T15:02:21Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35826/

SparkQA · 2020-11-17T15:19:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35827/

SparkQA · 2020-11-17T15:31:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35826/

SparkQA · 2020-11-17T15:40:51Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35827/

SparkQA · 2020-11-20T11:05:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36021/

SparkQA · 2020-11-20T11:28:28Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36021/

SparkQA · 2020-11-20T11:31:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36023/

SparkQA · 2020-11-20T11:55:33Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36023/

SparkQA · 2020-11-20T12:07:46Z

Test build #131417 has finished for PR 30389 at commit a67acd7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-11-20T18:58:55Z

core/src/main/scala/org/apache/spark/security/SocketAuthServer.scala

    val serverSocket = new ServerSocket(0, 1, InetAddress.getByAddress(Array(127, 0, 0, 1)))
-    // Close the socket if no connection in 15 seconds
-    serverSocket.setSoTimeout(15000)
+    // Close the socket if no connection in configured seconds


in configured -> in the configured?

dongjoon-hyun

+1, LGTM (only minor comments).
Thank you, @gaborgsomogyi and @HyukjinKwon .

HyukjinKwon · 2020-11-23T00:48:53Z

@dongjoon-hyun @gaborgsomogyi, yeah I think it makes more sense to separate this change #30389 (comment). Let's exclude this in this PR.

….scala

HyukjinKwon · 2020-11-23T00:54:12Z

I just pushed some changes to remove the test and address the minor comment @gaborgsomogyi :-). Let me just merge this in first. For others, we could open a separate PR.

SparkQA · 2020-11-23T01:35:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36123/

SparkQA · 2020-11-23T02:07:39Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36123/

HyukjinKwon · 2020-11-23T02:38:00Z

core/src/main/scala/org/apache/spark/security/SocketAuthServer.scala

-    // Close the socket if no connection in 15 seconds
-    serverSocket.setSoTimeout(15000)
+    // Close the socket if no connection in the configured seconds
+    val timeout = authHelper.conf.get(PYTHON_AUTH_SOCKET_TIMEOUT).toInt


Okay, actually the test in PythonRDDSuite was related.

In python server error handling test we don't initialize SparkEnv because we don't create SparkContext or executor there by default, but SparkEnv.get was used in the previous change which ended up with NPE.

Now, I made a change to work around by using SparkConf directly here.

SparkQA · 2020-11-23T03:25:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36129/

SparkQA · 2020-11-23T03:29:01Z

Test build #131520 has finished for PR 30389 at commit ef137b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-23T03:34:41Z

Test build #131519 has finished for PR 30389 at commit d9f0a1b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-23T03:50:01Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36129/

dongjoon-hyun · 2020-11-23T04:12:44Z

Thank you for your decision, @HyukjinKwon .

SparkQA · 2020-11-23T05:58:38Z

Test build #131527 has finished for PR 30389 at commit 363f3bb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-11-23T06:18:37Z

GA passed. I will merged this.

Merged to master.

gaborgsomogyi · 2020-11-23T07:21:38Z

Just visited this back and I would like to say a big thanks for such a help!
Thanks @HyukjinKwon and @dongjoon-hyun !

gaborgsomogyi · 2020-11-23T07:23:22Z

I'm intended to solve the issue w/ SPARK_BUFFER but first I need to polish my Kafka related PR because I'm really late w/ that and it would be good to be included in 3.1.

HyukjinKwon · 2020-11-23T07:48:02Z

Thanks @gaborgsomogyi!

[SPARK-33143][PYTHON] Add configurable timeout to python server and c…

09b1235

…lient

github-actions bot added CORE PYTHON labels Nov 16, 2020

compile fix

17d357b

dongjoon-hyun marked this pull request as draft November 16, 2020 19:07

PythonRDDSuite fix

424be64

Some cleanup

d9feed8

gaborgsomogyi commented Nov 17, 2020

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/Python.scala Outdated Show resolved Hide resolved

gaborgsomogyi changed the title ~~[WIP][SPARK-33143][PYTHON] Add configurable timeout to python server and client~~ [SPARK-33143][PYTHON] Add configurable timeout to python server and client Nov 17, 2020

gaborgsomogyi marked this pull request as ready for review November 17, 2020 14:22

Additional trace

6e8e194

dongjoon-hyun reviewed Nov 20, 2020

View reviewed changes

dongjoon-hyun approved these changes Nov 20, 2020

View reviewed changes

HyukjinKwon added 5 commits November 23, 2020 09:51

Update core/src/test/scala/org/apache/spark/api/python/PythonRDDSuite…

0f9e587

….scala

Update core/src/test/scala/org/apache/spark/api/python/PythonRDDSuite…

cd2d595

….scala

Update core/src/main/scala/org/apache/spark/security/SocketAuthServer…

d9f0a1b

….scala

Update core/src/test/scala/org/apache/spark/api/python/PythonRDDSuite…

2913fb1

….scala

Update core/src/test/scala/org/apache/spark/api/python/PythonRDDSuite…

ef137b6

….scala

Fix test in PythonRDDSuite

363f3bb

HyukjinKwon reviewed Nov 23, 2020

View reviewed changes

HyukjinKwon closed this in 0bb911d Nov 23, 2020

gaborgsomogyi mentioned this pull request Mar 12, 2021

[SPARK-34726][SQL][2.4] Fix collectToPython timeouts #31818

Closed

[SPARK-33143][PYTHON] Add configurable timeout to python server and client #30389

[SPARK-33143][PYTHON] Add configurable timeout to python server and client #30389

Uh oh!

Conversation

gaborgsomogyi commented Nov 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

gaborgsomogyi commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 16, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

gaborgsomogyi commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

Uh oh!

gaborgsomogyi commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 17, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

SparkQA commented Nov 20, 2020

Uh oh!

dongjoon-hyun Nov 20, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Nov 23, 2020

Uh oh!

HyukjinKwon commented Nov 23, 2020

Uh oh!

SparkQA commented Nov 23, 2020

Uh oh!

SparkQA commented Nov 23, 2020

Uh oh!

HyukjinKwon Nov 23, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 23, 2020

Choose a reason for hiding this comment

Uh oh!

gaborgsomogyi commented Nov 16, 2020 •

edited

Loading