JsonPath#set() does not create new paths

[ben-manes]

@Test
public void test() {
  String createJson = "{}";
  DocumentContext createContext = JsonPath.parse(createJson).set("$.id", 1);
  System.out.println(createContext.jsonString());

  String updateJson = "{id: 1}";
  DocumentContext updateContext = JsonPath.parse(updateJson).set("$.id", 2);
  System.out.println(updateContext.jsonString());
}

{}
{"id":2}

Is there a way to perform the addition? e.g. set the id if not present.

[ben-manes]

Sorry, somehow I skipped over put and saw only add.

[threesunshin]

@ben-manes i use add and put method, but they don't work. set method just can update the existing key. Please tell me how to add a new key using it. Thanks.

`
String createJson = "{}";
DocumentContext createContext = JsonPath.parse(createJson).add("$.name", "aaa");
System.out.println(createContext.jsonString());

    String updateJson = "{name: 1}";
    DocumentContext updateContext = JsonPath.parse(updateJson).put("$.age", "aaa",String.class);
    System.out.println(updateContext.jsonString());

{}
{"name":1}
`

[ben-manes]

Unfortunately JsonPath does not create intermediate paths for you. So you have to do,

context.put("$", "age", "aaa");

It is very frustrating to write into a context, and hopefully the authors will change that someday. I constantly shift from JsonPath to pojos depending on which mental model is more appropriate. Usually I read out into a pojo, do my work, and push back into the context for the next pipeline stage. However writing other tools is frustrating, like CSV->Json or resolving array ranges in paths to their absolutes, but its still been flexible enough despite some API limitations.

[threesunshin]

@ben-manes Thanks for your quick response.

[akanshSirohi]

@ben-manes @threesunshin Any solution you can tell for this one???
#982 (comment)

[ben-manes]

We have a utility method but it has grown a little messy over time.

static final ObjectMapper mapper = new ObjectMapper();

/**
 * Set the value a the given path, creating the parent property is necessary. This differs from
 * {@link JsonPath} methods which do nothing if the parent is absent.
 */
public static void deeplySet(DocumentContext json, JsonPath path, Object value) {
  deeplySet(json, path, value, true);
}

public static void deeplySet(DocumentContext json, JsonPath path, Object value, boolean expandArray) {
  checkArgument(path.isDefinite(), "Path must be absolute: %s", path.getPath());
  List<String> parts = Splitter
    .on('.')
    .omitEmptyStrings()
    .splitToList(path.getPath().replaceAll("(\\]\\[)|\\[|\\]", ".").replaceAll("\\$|'", ""));

  String currentPath = "$";
  String prevPath = currentPath;

  // traverse the JSON path and create parent if it doesn't already exist
  for (int i = 0; i < parts.size(); i++) {
    String node = parts.get(i);
    var isNodeArray = isDigits(parts.get(i));

    prevPath = currentPath;
    currentPath = isNodeArray
        ? currentPath + "[" + node + "]"
        : currentPath + "." + parts.get(i);

    var nodeValue = json.read(currentPath, JsonNode.class);

    /**
     * if element is an array, then do the following
     * 1. if there is existing array, check if it has enough capacity, if not expand it
     * 2. create a new array with an initial capacity if  none exists yet
     */
    if (isNodeArray && expandArray) {
      var nodeIndex = Integer.valueOf(node);
      var expandSize = nodeIndex + 1;
      ArrayNode arrayNode = null;

      var prevNodeValue = json.read(prevPath);
      if (prevNodeValue instanceof ArrayNode) {
        arrayNode = (ArrayNode) prevNodeValue;
        // re-adjust expand size to account for existing size
        expandSize = Math.max(expandSize - arrayNode.size(), 0);
      } else {
        arrayNode = mapper.createArrayNode();
      }

      // expand the array
      for (int cnt = 0; cnt < expandSize; cnt++) {
        arrayNode.addObject();
      }

      // ensure the node is non-null so we could set the children later on
      if (arrayNode.get(nodeIndex).isNull()) {
        arrayNode.insertObject(nodeIndex);
      }

      // set the array back to the json
      json.set(prevPath, arrayNode);
    } else if (nodeValue == null || nodeValue.isNull()) {
      json.put(prevPath, node, mapper.createObjectNode());
    }
  }

  // all the missing path should be created already, now just set it
  json.set(path, value);
}

[akanshSirohi]

@ben-manes can you please write a example use according to the code I provided in my issue please?
Also please write any dependencies required for this function to work, like I can see Splitter

[ben-manes]

Here's the imports for the entire utility class.

import static com.google.common.base.MoreObjects.firstNonNull;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkState;
import static com.jayway.jsonpath.Option.AS_PATH_LIST;
import static com.jayway.jsonpath.Option.SUPPRESS_EXCEPTIONS;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.util.Objects.requireNonNull;
import static java.util.function.Function.identity;
import static java.util.stream.Collectors.joining;
import static java.util.stream.Collectors.toMap;
import static org.apache.commons.lang3.StringUtils.isBlank;
import static org.apache.commons.lang3.math.NumberUtils.isDigits;

import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.function.Function;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.fasterxml.jackson.core.JsonPointer;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.google.auto.value.AutoValue;
import com.google.common.base.CharMatcher;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterators;
import com.google.common.hash.Hasher;
import com.google.common.hash.Hashing;
import com.jayway.jsonpath.Configuration;
import com.jayway.jsonpath.DocumentContext;
import com.jayway.jsonpath.InvalidPathException;
import com.jayway.jsonpath.JsonPath;
import com.jayway.jsonpath.TypeRef;
import com.jayway.jsonpath.spi.cache.CacheProvider;

import net.autobuilder.AutoBuilder;

and a unit test

@Test
public void deeplySet() {
  DocumentContext json = JsonPath.parse("{}");
  JsonPaths.deeplySet(json, JsonPath.compile("$.a.b.c"), 1);
  JsonPaths.deeplySet(json, JsonPath.compile("$.a.b.d"), 1);
  JsonPaths.deeplySet(json, JsonPath.compile("$.a.b.e.f.g"), 1);

  assertThat(json.read("$.a.b.c", Integer.class), is(1));
  assertThat(json.read("$.a.b.d", Integer.class), is(1));
  assertThat(json.read("$.a.b.e.f.g", Integer.class), is(1));

  // update existing value
  JsonPaths.deeplySet(json, JsonPath.compile("$.a.b.c"), 11);
  assertThat(json.read("$.a.b.c", Integer.class), is(11));

  // check if existing values are not modified or cleared out
  assertThat(json.read("$.a.b.d", Integer.class), is(1));
  assertThat(json.read("$.a.b.e.f.g", Integer.class), is(1));
}

I don't know if you want to wrangle it from that code, use at your own peril. 😄

[akanshSirohi]

@ben-manes Thanks for this, I will try to make use of this!
And thanks for quick reply too!

[akanshSirohi]

@ben-manes My approach is this

[
    {
      "msg": "hello",
      "_uuid": "1483f2e6-1241-4e84-9fbf-503ce5ea64df"
    },
    {
      "msg": "hello",
      "child": {
        "timestamp": "any_timestamp"
      },
      "_uuid": "dbe96d86-4542-4bbd-b64e-f31d69fa735b"
    }
]

DocumentContext collections_doc = JsonPath.parse(collectionArray.toString()); // contains the json array
JSONHelper.deeplySet(collections_doc, JsonPath.compile("$[?(@.msg == 'hi')].child.timestamp"), "new_timestamp");

But I am getting the error

Expected to find an object with property ['?'] in path $ but found 'net.minidev.json.JSONArray'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

Any fix you can tell..??

[ben-manes]

We only handled absolute paths and asserted unsupported behavior in,

checkArgument(path.isDefinite(), "Path must be absolute: %s", path.getPath());

I'm not sure the best way to extend that. Ideally the authors would support it so it would be a live evaluation, but I don't know the internals enough. You might be able to break down into segments to recursively walk and fill in from the outside. I didn't think much about it since we avoid that in our usages.

[akanshSirohi]

@ben-manes Thanks for the solution, and I have tried other ways to make it work with array, I am almost there but the code you provided is still not working at object level only. Attaching logs and code, can you check

[ben-manes]

sorry, I probably can't help too much further to debug this (no code, no time, etc). We do use the code that I provided in critical flows so it works, but was narrow to what we needed so I'd expect gaps or bugs. You'd want to step through with a debugger and inspect the documentContext's jsonString() at different points to see at what step it is failing. I would love to see this feature implemented properly and made available for everyone.

[akanshSirohi]

@ben-manes Well thanks for this much help, really appreciate your efforts. I will keep it as a limitation until the author of this library implement it. I will keep trying to find an optimized way to achieve this, and surely let you know if got something working...