feat: add UDFs for json by Xuanwo · Pull Request #4577 · lance-format/lance

Xuanwo · 2025-08-27T09:03:58Z

Part of #4516

This PR will add UDFs for JSON so users can read JSON like:

ds_path = Path(tmpdir) / "nested_json_test.lance"
lance.write_dataset(table, ds_path, data_storage_version="2.2")
dataset = lance.dataset(ds_path)

# Access nested fields using json_get recursively
# First get user, then profile, then name
result = dataset.to_table(
    filter="json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'"
)
assert result.num_rows == 1
assert result["id"][0].as_py() == 1

# Or use JSONPath for deep access
result = dataset.to_table(
    filter="json_extract(data, '$.user.profile.settings.theme') = '\"dark\"'"
)
assert result.num_rows == 1
assert result["id"][0].as_py() == 1

This PR was primarily authored with Claude Code using Opus 4.1 and then hand-reviewed by me. I AM responsible for every change made in this PR. I aimed to keep it aligned with our goals, though I may have missed minor issues. Please flag anything that feels off, I'll fix it quickly.

Signed-off-by: Xuanwo <github@xuanwo.io>

codecov-commenter · 2025-08-27T09:40:23Z

Codecov Report

❌ Patch coverage is 57.99770% with 365 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.85%. Comparing base (e2f08db) to head (c59315f).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-datafusion/src/udf/json.rs	63.57%	167 Missing and 61 partials ⚠️
rust/lance-arrow/src/json.rs	48.32%	75 Missing and 2 partials ⚠️
rust/lance/src/dataset/utils.rs	23.07%	60 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4577      +/-   ##
==========================================
- Coverage   80.99%   80.85%   -0.15%     
==========================================
  Files         314      314              
  Lines      115727   116609     +882     
  Branches   115727   116609     +882     
==========================================
+ Hits        93735    94283     +548     
- Misses      18682    18952     +270     
- Partials     3310     3374      +64

Flag	Coverage Δ
unittests	`80.85% <57.99%> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

westonpace

Quick initial question. How do these compare to https://github.com/datafusion-contrib/datafusion-functions-json ?

Xuanwo · 2025-08-27T15:09:33Z

Quick initial question. How do these compare to https://github.com/datafusion-contrib/datafusion-functions-json ?

datafusion-functions-json has:

json_get::json_get_udf(),
json_get_bool::json_get_bool_udf(),
json_get_float::json_get_float_udf(),
json_get_int::json_get_int_udf(),
json_get_json::json_get_json_udf(),
json_get_array::json_get_array_udf(),
json_as_text::json_as_text_udf(),
json_get_str::json_get_str_udf(),
json_contains::json_contains_udf(),
json_length::json_length_udf(),
json_object_keys::json_object_keys_udf(),

Perhaps it would be better to implement them directly on jsonb rather than converting to JSON and using those UDFs. What do you think?

westonpace

Very cool, I have some suggestions but no major concerns.

westonpace · 2025-08-28T01:30:44Z

+        .iter()
+        .map(|arg| match arg {
+            ColumnarValue::Array(arr) => arr.clone(),
+            ColumnarValue::Scalar(scalar) => scalar.to_array().unwrap(),


It would be great if we could avoid broadcasting the scalars here. Let's create a follow-up?

westonpace · 2025-08-28T01:45:41Z

+    match raw_jsonb.get_by_name(key, false) {
+        Ok(Some(value)) => return Ok(Some(value.as_raw().as_ref().to_vec())),
+        Ok(None) => {}
+        Err(e) => {
+            return Err(datafusion::error::DataFusionError::Execution(format!(
+                "Failed to get field: {}",
+                e
+            )))
+        }
+    }
+
+    // Try as array index
+    if let Ok(index) = key.parse::<usize>() {


It might be more efficient to determine if the key is a field or an integer once instead of each value? Although I guess it could be an integer-named field and so vary from value to value 😛

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo · 2025-08-28T15:51:16Z

Catching some bugs from python side, working on it.

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo · 2025-08-28T18:16:14Z

I really don't like how I'm handling JSON and JSONB conventions in the code. Maybe I over-optimized too early, adding unnecessary complexity just to allow we can perform UDFs on JSONB instead of the entire JSON.

I'm starting to reconsider @westonpace's suggestion to keep JSONB as part of the file format instead of treating it as a logical type. While it's working for now, we might as well keep moving forward and see how it holds up in the long run. We can always make adjustments later.

I'll merge this PR first and do anyother changes in follow-up PRs. This will also unlock us to implement index over JSON.

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo · 2025-08-29T04:10:14Z

The follow-up tracked at #4595

Part of lance-format#4516 This PR will add UDFs for JSON so users can read JSON like: ```python ds_path = Path(tmpdir) / "nested_json_test.lance" lance.write_dataset(table, ds_path, data_storage_version="2.2") dataset = lance.dataset(ds_path) # Access nested fields using json_get recursively # First get user, then profile, then name result = dataset.to_table( filter="json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'" ) assert result.num_rows == 1 assert result["id"][0].as_py() == 1 # Or use JSONPath for deep access result = dataset.to_table( filter="json_extract(data, '$.user.profile.settings.theme') = '\"dark\"'" ) assert result.num_rows == 1 assert result["id"][0].as_py() == 1 ``` --- **This PR was primarily authored with Claude Code using Opus 4.1 and then hand-reviewed by me. I AM responsible for every change made in this PR. I aimed to keep it aligned with our goals, though I may have missed minor issues. Please flag anything that feels off, I'll fix it quickly.** --------- Signed-off-by: Xuanwo <github@xuanwo.io>

feat: add UDFs for json

2d79f96

Signed-off-by: Xuanwo <github@xuanwo.io>

github-actions Bot added enhancement New feature or request python labels Aug 27, 2025

Xuanwo requested review from jackye1995 and westonpace August 27, 2025 09:04

westonpace reviewed Aug 27, 2025

View reviewed changes

westonpace approved these changes Aug 28, 2025

View reviewed changes

Xuanwo mentioned this pull request Aug 28, 2025

Create docs for JSON UDFs #4590

Closed

Xuanwo added 3 commits August 28, 2025 17:08

Cleanup

2015204

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix python lint

f6759ed

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix UDF register

d8045a2

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo added 3 commits August 29, 2025 01:47

Correct tests

69ecb54

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix JSONB convert

c2c71e0

Signed-off-by: Xuanwo <github@xuanwo.io>

Merge branch 'main' into add-udf-for-json

610e0ec

Xuanwo added 5 commits August 29, 2025 03:58

Fix tests

0351d80

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix tests

864efc2

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix json behavior in null

c3173e4

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix datafusion version

c6b5fe8

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix typo

c59315f

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo merged commit a65857a into main Aug 29, 2025
30 checks passed

Xuanwo deleted the add-udf-for-json branch August 29, 2025 04:01

Xuanwo mentioned this pull request Aug 29, 2025

Refactor JSONB and JSON conversion logic #4595

Closed

ddupg mentioned this pull request Sep 19, 2025

chore: upgrade datafusion to 50.0.0 and arrow to 56.1 #4751

Merged

Conversation

Xuanwo commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Aug 27, 2025

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

westonpace Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

westonpace Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Xuanwo commented Aug 28, 2025

Uh oh!

Xuanwo commented Aug 28, 2025

Uh oh!

Uh oh!

Xuanwo commented Aug 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Xuanwo commented Aug 27, 2025 •

edited

Loading

codecov-commenter commented Aug 27, 2025 •

edited

Loading