[SPARK-25243][SQL] Use FailureSafeParser in from_json#22237

Closed

MaxGekk wants to merge 27 commits intoapache:masterfrom

MaxGekk:from_json-failuresafe

Commits on Oct 24, 2018

Support any type by FailureSafeParser

MaxGekk
authored and
HyukjinKwon
committed
Use FailSafeParser in from_json

MaxGekk
authored and
HyukjinKwon
committed
Fix tests

MaxGekk
authored and
HyukjinKwon
committed
Removing unused bind variable

MaxGekk
authored and
HyukjinKwon
committed
Adding tests for different modes

MaxGekk
authored and
HyukjinKwon
committed
Improve a test

MaxGekk
authored and
HyukjinKwon
committed
Fix a sql test

MaxGekk
authored and
HyukjinKwon
committed
Addressing Takeshi's review comments

MaxGekk
authored and
HyukjinKwon
committed
Restring to PERMISSIVE and FAILFAST modes so far

MaxGekk
authored and
HyukjinKwon
committed
Changing the test according to the PERMISSIVE mode

MaxGekk
authored and
HyukjinKwon
committed
Renaming schema -> dataType

MaxGekk
authored and
HyukjinKwon
committed
Throwing AnalysisException instead of IllegalArgumentException

MaxGekk
authored and
HyukjinKwon
committed
Check that the AnalysisException is thrown during producing a plan

MaxGekk
authored and
HyukjinKwon
committed
Replacing AnalysisException by IllegalArgumentException

MaxGekk
authored and
HyukjinKwon
committed
Fix a test

MaxGekk
authored and
HyukjinKwon
committed
Improving the test

MaxGekk
authored and
HyukjinKwon
committed
Removing the BadRecordException handler

MaxGekk
authored and
HyukjinKwon
committed
Updating test's title

MaxGekk
authored and
HyukjinKwon
committed
Addressing Hyukjin's review comments

MaxGekk
authored and
HyukjinKwon
committed
Produce null and put input to the corrupted column if an input array has more than 1 element for struct schema

MaxGekk
authored and
HyukjinKwon
committed
Removing special handling of empty strings

MaxGekk
authored and
HyukjinKwon
committed
Don't allow parsing arrays as structs by from_json

MaxGekk
authored and
HyukjinKwon
committed
Modified a test to check the spark.sql.columnNameOfCorruptRecord config

MaxGekk
authored and
HyukjinKwon
committed
Improving tests

MaxGekk
authored and
HyukjinKwon
committed
Fix a python test

MaxGekk
authored and
HyukjinKwon
committed
add migration guide
HyukjinKwon
committed
address comments
HyukjinKwon
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-25243][SQL] Use FailureSafeParser in from_json#22237

[SPARK-25243][SQL] Use FailureSafeParser in from_json#22237
MaxGekk wants to merge 27 commits intoapache:masterfrom
MaxGekk:from_json-failuresafe

Commits on Oct 24, 2018

Support any type by FailureSafeParser

Use FailSafeParser in from_json

Fix tests

Removing unused bind variable

Adding tests for different modes

Improve a test

Fix a sql test

Addressing Takeshi's review comments

Restring to PERMISSIVE and FAILFAST modes so far

Changing the test according to the PERMISSIVE mode

Renaming schema -> dataType

Throwing AnalysisException instead of IllegalArgumentException

Check that the AnalysisException is thrown during producing a plan

Replacing AnalysisException by IllegalArgumentException

Fix a test

Improving the test

Removing the BadRecordException handler

Updating test's title

Addressing Hyukjin's review comments

Produce null and put input to the corrupted column if an input array has more than 1 element for struct schema

Removing special handling of empty strings

Don't allow parsing arrays as structs by from_json

Modified a test to check the spark.sql.columnNameOfCorruptRecord config

Improving tests

Fix a python test

add migration guide

address comments