Fix nested message object in API responses by ErykKul · Pull Request #12097 · IQSS/dataverse

ErykKul · 2026-01-16T20:50:38Z

What this PR does / why we need it:

Fixes the ok(String msg, JsonObjectBuilder bld) method in AbstractApiBean.java which incorrectly wraps the message in a nested object:

// Before (incorrect):
{"status":"OK","message":{"message":"actual message"},"data":{...}}

// After (correct):
{"status":"OK","message":"actual message","data":{...}}

The bug was introduced in commit f311312c34 on May 12, 2020, as part of #4813.

Which issue(s) this PR closes:

Closes API returns malformed JSON with nested message object instead of string in ok(String, JsonObjectBuilder) responses #12096
Closes Dataverse API responses inconsistent in JSON structure #11667

Special notes for your reviewer:

This is a one-line fix in AbstractApiBean.java line 1000:

// From:
.add("message", Json.createObjectBuilder().add("message",msg))
// To:
.add("message", msg)

The following 7 API endpoints are affected:

File	Line	API Endpoint
`Datasets.java`	3087	`POST /api/datasets/{id}/add` (duplicate file warning)
`Admin.java`	233	`PUT /api/admin/settings`
`Dataverses.java`	754	`PUT /api/dataverses/{id}`
`Dataverses.java`	788	`PUT /api/dataverses/{id}/inputLevels`
`SavedSearches.java`	161	`POST /api/admin/savedsearches`
`HarvestingClients.java`	309	`PUT /api/harvest/clients/{nickName}`
`HarvestingServer.java`	228	`PUT /api/harvest/server/oaisets/{specname}`

Suggestions on how to test this:

Upload a file to a dataset via API
Upload the same file again (duplicate)
Check that the response message field is a string, not a nested object:

curl -X POST "http://localhost:8080/api/datasets/:persistentId/add?persistentId=doi:..." \
  -H "X-Dataverse-key: $API_TOKEN" \
  -F "file=@test-file.txt"
# Upload again to trigger duplicate warning
curl -X POST "http://localhost:8080/api/datasets/:persistentId/add?persistentId=doi:..." \
  -H "X-Dataverse-key: $API_TOKEN" \
  -F "file=@test-file.txt"

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No

Is there a release notes update needed for this change?:

Yes, included in this PR: doc/release-notes/12096-fix-ok-message-nested-object.md

Additional documentation:

N/A

coveralls · 2026-01-16T21:02:07Z

coverage: 24.354% (+0.03%) from 24.323%
when pulling 47dece5 on 12096-fix-ok-message-nested-object
into 86796a1 on develop.

pdurbin

A couple quick comments.

Also, @poikilotherm noticed this double message bug, which lead to this API v2 proposal: https://docs.google.com/document/d/1XtXPF_PZCuhPbm4HCu28yRge_4_--ah604NhigPdjPk/edit?usp=sharing . And these related issues:

doc/release-notes/12096-fix-ok-message-nested-object.md

src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java

ErykKul · 2026-01-17T00:24:47Z

@pdurbin Thanks for the quick review and the context about the API v2 proposal!

I was a bit hesitant to make this change since it could break existing API consumers that have implemented workarounds for the nested message object. However, I think it's worth fixing. The current behavior is clearly a bug, and the affected endpoints are relatively niche (the bug was actually discovered through duplicate file uploads, while regular file additions work fine).

That said, I can see the argument for keeping this for API v2 if you think the breaking change risk is too high. Let me know your thoughts - happy to go either way.

I've fixed the integration tests and added the changelog entry as requested. The docs build should pass now too (had a small RST syntax issue with the reference).

poikilotherm

While the commits look good overall, I am reluctant to give this a pass as-is.
I discovered more problems with inconsistencies in #11667 and as @pdurbin pointed out, this led to the v2 API proposal.

I agree with @ErykKul that changing this is very useful. And for 6 of the 7 affected endpoints I am agreeing that changing them may cause minor inconveniences for administrators (fine, just deal with it).

Yet with the Dataset endpoint to add files warning about duplicates and which are duplicated - this may seriously affect some integration with Dataverse out there. This may actually be a breaking change causing trouble for users. IMHO we shouldn't do this.

Also, if we fix that, let's do it all in one sweep and fix any other inconsistencies as well.

There may be some middle ground here if we make this opt-in. Using a feature flag marking it as experimental, we can leave a choice to the admin. They can test with their integrations and see what happens and return to the default at any time. (This reminds me of mentioning https://openfeature.dev somewhere - making feature flags activate based on more criteria than mere configuration...)
A way forward may be to make this experimental for now, then deprecate it in a future version, then drop it. This way this may not even need an API v2 (only regarding this message - the mileage for the other inconsistencies may vary!)

Adds API_MESSAGE_FIELD_LEGACY feature flag that allows admins to revert to the old (buggy) nested message format if they have integrations that depend on it. The fix is now the default behavior. Set dataverse.feature.api-message-field-legacy=true to use the legacy format. This flag will be removed in a future version.

ErykKul · 2026-02-09T10:13:58Z

Hi @pdurbin, @poikilotherm,

I've addressed the feedback in this PR:

Merged the latest develop into this branch.
Added a feature flag dataverse.feature.api-message-field-legacy as @poikilotherm suggested. The fix is now the default behavior, and admins who discover issues with their integrations can set this flag to true to revert to the previous (buggy) format. This way we get the fix out without risking breakage for anyone.
Updated the release notes and API changelog accordingly.

I ran the full test suite locally (unit tests + integration tests) and everything passes. However, I'm unable to access the Jenkins server from here — it seems to be restricted to internal networks. @pdurbin, if Jenkins fails again on this latest commit, could you share the relevant log output or a screenshot? That would help me figure out what's going on. Thanks!

…se message styles

ErykKul · 2026-02-25T10:55:12Z

Overall, Oliver’s code changes look good to me and the direction is solid.

I only added a few follow-up tweaks after merge/testing:

Fixed a merge regression in JvmSettings (SCOPE_LEGACY was duplicated, causing compile errors).
Added a small robustness fix in LDN relationship parsing: after JSON-LD processing, fields may come as either plain strings or as {"@id":"..."} objects.
Stabilized CacheFactoryBeanTest by removing environment-sensitive Hazelcast/cgroup dependency in the test cache implementation.
Added extra test coverage for API message compatibility paths in AbstractApiBeanTest:
- default ok(String, JsonObjectBuilder) shape
- legacy shape with dataverse.legacy.api-response-message-style=true

Targeted tests pass:

AbstractApiBeanTest
LDNInboxTest
CacheFactoryBeanTest

… low-level details from being exposed to clients

src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java

Copilot

Pull request overview

This PR fixes a bug in the ok(String msg, JsonObjectBuilder bld) method in AbstractApiBean.java that incorrectly returned nested message objects ({"message":{"message":"..."}}) instead of plain strings ({"message":"..."}). The bug has existed since Dataverse 5.0 (May 2020) and affects 7 API endpoints. The fix includes backward compatibility options and introduces comprehensive test infrastructure for feature flags and JVM settings.

Changes:

Fixed the nested message object bug in API responses with opt-out legacy mode via dataverse.legacy.api-response-message-style
Refactored all ok() response methods in AbstractApiBean to use a unified implementation with NullSafeJsonBuilder
Added experimental opt-in feature flag UNIFY_API_RESPONSE_MESSAGE_STYLE for aligning ~230 additional API responses
Introduced test infrastructure for manipulating feature flags and JVM settings in unit tests

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java`	Core fix for nested message bug, refactored ok() methods to use unified implementation
`src/main/java/edu/harvard/iq/dataverse/api/ApiConstants.java`	Moved API constants from AbstractApiBean for better organization
`src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java`	Added LEGACY_API_RESPONSE_MESSAGE_STYLE setting for opt-out backward compatibility
`src/main/java/edu/harvard/iq/dataverse/settings/FeatureFlags.java`	Added UNIFY_API_RESPONSE_MESSAGE_STYLE feature flag and getScopedKey() method
`src/main/java/edu/harvard/iq/dataverse/engine/command/DataverseRequest.java`	Updated to use ApiConstants instead of AbstractApiBean constants
`src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java`	Improved error handling (unrelated to main PR purpose)
`src/main/java/edu/harvard/iq/dataverse/api/ldn/COARNotifyRelationshipAnnouncement.java`	Enhanced JSON-LD IRI handling (unrelated to main PR purpose)
`src/test/java/edu/harvard/iq/dataverse/api/AbstractApiBeanTest.java`	Added tests for default, legacy, and unified message styles
`src/test/java/edu/harvard/iq/dataverse/api/SearchIT.java`	Updated constant reference from AbstractApiBean to ApiConstants
`src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java`	Updated test assertions to expect plain string messages instead of nested objects
`src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java`	Updated constant reference from AbstractApiBean to ApiConstants
`src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java`	Updated test assertion to expect plain string message instead of nested object
`src/test/java/edu/harvard/iq/dataverse/util/cache/CacheFactoryBeanTest.java`	Replaced Hazelcast with ConcurrentHashMap (unrelated to main PR purpose)
`src/test/java/edu/harvard/iq/dataverse/util/testing/FeatureFlag.java`	New test annotation for setting feature flags during tests
`src/test/java/edu/harvard/iq/dataverse/util/testing/FeatureFlagBroker.java`	Interface for accessing and manipulating feature flags in tests
`src/test/java/edu/harvard/iq/dataverse/util/testing/FeatureFlagExtension.java`	JUnit5 extension for feature flag test support
`src/test/java/edu/harvard/iq/dataverse/util/testing/LocalFeatureFlags.java`	Annotation for tests that manipulate local feature flags via system properties
`doc/sphinx-guides/source/installation/config.rst`	Documentation for new legacy setting and feature flag
`doc/sphinx-guides/source/api/changelog.rst`	API changelog entry for v6.10 breaking change
`doc/release-notes/12096-fix-ok-message-nested-object.md`	Release notes explaining the fix and migration path

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java

src/test/java/edu/harvard/iq/dataverse/util/cache/CacheFactoryBeanTest.java

doc/sphinx-guides/source/api/changelog.rst

doc/release-notes/12096-fix-ok-message-nested-object.md

Copilot · 2026-02-25T11:23:40Z

src/main/java/edu/harvard/iq/dataverse/api/ldn/COARNotifyRelationshipAnnouncement.java

    private String extractField(JsonObject msgObject, String key) {
-        return msgObject.containsKey(key) ? msgObject.getString(key) : null;
+        if (!msgObject.containsKey(key)) {
+            return null;
+        }
+
+        JsonValue value = msgObject.get(key);
+        if (value.getValueType().equals(JsonValue.ValueType.STRING)) {
+            return msgObject.getString(key);
+        }
+
+        // JSON-LD expansion may represent IRI values as {"@id":"..."} objects.
+        if (value.getValueType().equals(JsonValue.ValueType.OBJECT)) {
+            JsonObject objectValue = msgObject.getJsonObject(key);
+            JsonValue id = objectValue.get("@id");
+            if (id != null && id.getValueType().equals(JsonValue.ValueType.STRING)) {
+                return objectValue.getString("@id");
+            }
+        }
+
+        return null;
    }


The changes to extractField() method in this file appear unrelated to the PR's stated purpose of fixing nested message objects in API responses. These changes handle JSON-LD expansion for IRI values. While the implementation looks correct, this should ideally be in a separate PR focused on JSON-LD handling improvements, not bundled with an API response format fix. This makes it harder to review, test, and potentially revert if issues arise with either change.

A similar change has already been made in #11983

I will wait to see if all tests pass (Jenkins is still running at this point), and then revert it.

All tests passed, reverted.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…e local only, e.g., passes on github?

…e-nested-object' into 12096-fix-ok-message-nested-object

github-actions · 2026-02-25T12:18:27Z

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:12096-fix-ok-message-nested-object

ghcr.io/gdcc/configbaker:12096-fix-ok-message-nested-object

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

ErykKul · 2026-02-25T15:49:09Z

All tests passed, I reverted the unrelated changes so tests fail again. However, the fix is in #11983

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Fix nested message object in API responses #12096

7911c63

github-actions bot added the Type: Bug a defect label Jan 16, 2026

ErykKul changed the title ~~Fix nested message object in API responses #12096~~ Fix nested message object in API responses Jan 16, 2026

This comment has been minimized.

Sign in to view

pdurbin added this to IQSS Dataverse Project Jan 16, 2026

pdurbin moved this to Ready for Triage in IQSS Dataverse Project Jan 16, 2026

pdurbin reviewed Jan 16, 2026

View reviewed changes

doc/release-notes/12096-fix-ok-message-nested-object.md Outdated Show resolved Hide resolved

src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java Outdated Show resolved Hide resolved

ErykKul added 2 commits January 17, 2026 01:13

Fix integration tests and add API changelog entry

2e43bd7

Fix RST reference syntax in changelog

0a9ce32

This comment has been minimized.

Sign in to view

scolapasta moved this from Ready for Triage to Ready for Review ⏩ in IQSS Dataverse Project Jan 20, 2026

pdurbin self-assigned this Jan 20, 2026

pdurbin moved this from Ready for Review ⏩ to In Review 🔎 in IQSS Dataverse Project Jan 20, 2026

pdurbin assigned ErykKul Jan 20, 2026

pdurbin requested a review from poikilotherm January 20, 2026 16:13

poikilotherm requested changes Jan 20, 2026

View reviewed changes

pdurbin assigned poikilotherm Jan 20, 2026

cmbz added FY26 Sprint 15 FY26 Sprint 15 (2026-01-14 - 2026-01-28) FY26 Sprint 16 FY26 Sprint 16 (2026-01-28 - 2026-02-11) labels Jan 29, 2026

pdurbin added the Waiting label Feb 4, 2026

ErykKul added 2 commits February 9, 2026 10:24

Merge develop into 12096-fix-ok-message-nested-object

0ba746a

This comment has been minimized.

Sign in to view

pdurbin removed the Waiting label Feb 10, 2026

pdurbin unassigned ErykKul Feb 10, 2026

stevenwinship removed their assignment Feb 20, 2026

pdurbin moved this from Ready for Review ⏩ to In Progress 💻 in IQSS Dataverse Project Feb 24, 2026

This was referenced Feb 24, 2026

Feature Request: Session-Cookie API Hardening with CSRF Protections and Mutating-GET Guardrails #12178

Open

Feature Request/Idea: reusing file upload page as dvwebloader IQSS/dataverse-frontend#468

Open

ErykKul added 3 commits February 25, 2026 11:05

Merge branch 'develop' into 12096-fix-ok-message-nested-object

df40e5f

Cimpilation error fix

8d06cb4

Fix JSON field extraction for nested objects and add tests for respon…

b51ae29

…se message styles

Enhance error handling in BatchImport: log I/O exceptions and prevent…

3d4e841

… low-level details from being exposed to clients

ErykKul requested review from Copilot February 25, 2026 11:15

Copilot started reviewing on behalf of ErykKul February 25, 2026 11:15 View session

This comment has been minimized.

Sign in to view

github-advanced-security bot found potential problems Feb 25, 2026

View reviewed changes

src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java Fixed Show fixed Hide fixed

Copilot AI reviewed Feb 25, 2026

View reviewed changes

ErykKul and others added 2 commits February 25, 2026 12:27

Update doc/sphinx-guides/source/api/changelog.rst

af79e3f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update doc/release-notes/12096-fix-ok-message-nested-object.md

d01b6b6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This comment has been minimized.

Sign in to view

Reverted; out of scope of this PR, and the fails in the tests might b…

a974dbb

…e local only, e.g., passes on github?

This comment has been minimized.

Sign in to view

ErykKul added 2 commits February 25, 2026 12:55

Revrted error handling in BatchImport

f882e11

Merge remote-tracking branch 'refs/remotes/origin/12096-fix-ok-messag…

2a60336

…e-nested-object' into 12096-fix-ok-message-nested-object

reverted test fix

b5a0406

ErykKul removed their assignment Feb 25, 2026

scolapasta moved this from In Progress 💻 to Ready for Review ⏩ in IQSS Dataverse Project Feb 25, 2026

Copilot AI reviewed Feb 25, 2026

View reviewed changes

cmbz added the FY26 Sprint 18 FY26 Sprint 18 (2026-02-25 - 2026-03-11) label Feb 26, 2026

Conversation

ErykKul commented Jan 16, 2026 • edited by poikilotherm Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ErykKul commented Jan 17, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

poikilotherm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

ErykKul commented Feb 9, 2026

Uh oh!

This comment has been minimized.

ErykKul commented Feb 25, 2026

Uh oh!

This comment has been minimized.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

qqmyers Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

ErykKul Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

ErykKul Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 25, 2026

Uh oh!

ErykKul commented Feb 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ErykKul commented Jan 16, 2026 •

edited by poikilotherm

Loading

coveralls commented Jan 16, 2026 •

edited

Loading

poikilotherm left a comment •

edited

Loading