[SPARK-4336] [SQL] support type auto detection from json string #3202
Conversation
|
Test build #23205 has started for PR 3202 at commit
|
| @@ -131,6 +131,19 @@ protected[sql] object DataTypeConversions { | |||
| StructType(structType.getFields.map(asScalaStructField)) | |||
| } | |||
|
|
|||
| def guessTypeFromString(s:String): DataType = { | |||
| try { | |||
| val res = stringToTime(s) | |||
There was a problem hiding this comment.
This function will be called against every string in json, so it should be very fast. It should fail fast in most cases if the s is not a time, the path in catch is extremely slow, see [1].
[1] http://stackoverflow.com/questions/299068/how-slow-are-java-exceptions
There was a problem hiding this comment.
Okay, I will use regex to judge then.
There was a problem hiding this comment.
regex maybe is not fast enough, we still need some quick test to fail fast, such as check it contains some special characters?
Could you post some micro benchmark of it?
|
Test build #23205 has finished for PR 3202 at commit
|
|
Test PASSed. |
|
Thanks for working on this! Since it looks like there is some concern about speed, and since we are trying to keep the PR queue small, I propose we close this issue for now and reopen once the performance comparisons are done. |
This is for comment from @davies in #3012
By this patch, we can detect TimestampType and DateType from a json string.