feat: Optional Jackson serialization for 5x performance improvement on large batches

[jaredmixpanel]

Summary

This PR introduces optional high-performance JSON serialization using Jackson while maintaining complete backward compatibility. This addresses the performance challenges you identified when importing large batches of events through the /import endpoint.

Key Achievement: We get Jackson's performance benefits WITHOUT breaking the public API or requiring any code changes from users.

Problem Statement

As you discovered in your benchmarking:

• org.json becomes a bottleneck when serializing large batches (50+ messages)
• The /import endpoint handles up to 2000 messages per batch (40x larger than regular endpoints)
• Direct replacement with Jackson would cause breaking changes due to incompatible types (JSONObject vs ObjectNode)
• Bridge pattern attempts failed due to conversion overhead negating performance gains

Solution Approach

Instead of trying to replace org.json or create a bridge between incompatible types, this implementation:

1. Keeps org.json for the public API - All public methods continue to accept/return JSONObject
2. Uses Jackson only for internal serialization - Jackson's streaming API serializes JSONObjects directly
3. Automatic runtime detection - The library detects if Jackson is available and uses it transparently
4. Zero conversion overhead - We stream JSONObject data directly through Jackson's JsonGenerator

Architecture

Public API (unchanged)
    ↓
JSONObject messages (org.json)
    ↓
JsonSerializer Interface (new)
    ├── OrgJsonSerializer (default)
    └── JacksonSerializer (when available)
    ↓
Serialized JSON string/bytes

Performance Results (Jackson 2.20.0)

Benchmarked with 1000 iterations per test:

Batch Size	org.json	Jackson	Speedup
1 message	12ms	11ms	1.09x
10 messages	96ms	52ms	1.85x
50 messages	337ms	51ms	6.61x
100 messages	600ms	94ms	6.38x
500 messages	2,798ms	624ms	4.48x
1000 messages	5,690ms	1,116ms	5.10x
2000 messages	11,688ms	2,224ms	5.26x

The performance improvement is most significant for batches of 50+ messages, which is exactly where the /import endpoint operates.

Implementation Details

Key Components

1. JsonSerializer Interface (internal/JsonSerializer.java)

   • Defines contract for JSON serialization
   • Methods for both String and byte[] output

2. OrgJsonSerializer (internal/OrgJsonSerializer.java)

   • Default implementation using org.json
   • Maintains existing behavior

3. JacksonSerializer (internal/JacksonSerializer.java)

   • High-performance implementation using Jackson's streaming API
   • Recursively streams JSONObject/JSONArray without conversion
   • Handles all JSON types (objects, arrays, primitives, nulls)

4. SerializerFactory (internal/SerializerFactory.java)

   • Runtime detection of Jackson availability
   • Singleton pattern for efficiency
   • Logging to inform users of active implementation

Changes to Existing Code

• MixpanelAPI.dataString(): Now uses JsonSerializer instead of direct JSONArray.toString()
• pom.xml: Added Jackson as optional dependency with provided scope
• README.md: Added documentation for enabling high-performance mode

How to Enable

Users simply add Jackson to their dependencies:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.20.0</version>
</dependency>

The library automatically detects and uses it - no code changes required!

Testing

• ✅ All 81 existing tests pass
• ✅ Added comprehensive tests for both serializer implementations
• ✅ Created performance benchmark suite
• ✅ Tested with and without Jackson on classpath
• ✅ Verified correct fallback behavior

Why This Approach Works

1. No conversion overhead: Unlike the bridge pattern attempt, we don't convert between JSONObject and ObjectNode
2. Streaming efficiency: Jackson's JsonGenerator writes directly to output without intermediate objects
3. Selective optimization: We optimize only the serialization bottleneck, not the entire message pipeline
4. Runtime flexibility: Users choose whether to include Jackson based on their performance needs

Backward Compatibility

This change is 100% backward compatible:

• All public APIs unchanged
• No breaking changes to method signatures
• Existing code continues to work without modifications
• Users without Jackson see no change in behavior

Recommendations

• For regular event tracking (< 50 events): org.json is sufficient
• For /import endpoint usage: Add Jackson for 5-6x performance boost
• For high-volume applications: Jackson is highly recommended

Next Steps

After this PR is merged, users importing large batches of historical data will see dramatic performance improvements simply by adding the Jackson dependency. This solves the performance bottleneck while maintaining the stability and compatibility of the existing API.

[Copilot]

Pull request overview

This PR introduces optional high-performance JSON serialization using Jackson's streaming API to address performance bottlenecks when serializing large event batches (50-2000 messages) for the /import endpoint. The implementation maintains 100% backward compatibility by keeping org.json for the public API while transparently using Jackson for internal serialization when available.

Key changes:

• New internal serializer abstraction with factory pattern for runtime detection of Jackson
• Jackson-based streaming serialization provides 5x speedup for large batches without API changes
• Automatic fallback to org.json when Jackson is unavailable or if serialization fails

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
JsonSerializer.java	New interface defining serialization contract for array serialization
SerializerFactory.java	Factory with runtime Jackson detection and singleton instance management
OrgJsonSerializer.java	Default implementation using existing org.json library
JacksonSerializer.java	High-performance implementation using Jackson streaming API
MixpanelAPI.java	Updated dataString() to use new serializer abstraction with fallback
pom.xml	Added Jackson dependency with provided scope for optional usage
README.md	Documentation on enabling high-performance serialization
JsonSerializerTest.java	Comprehensive unit tests for both serializer implementations
SerializerBenchmark.java	Performance benchmark tool comparing implementations
.gitignore	Added .vscode/ directory exclusion

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@jaredmixpanel I've opened a new pull request, #49, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

@jaredmixpanel I've opened a new pull request, #50, to work on those changes. Once the pull request is ready, I'll request review from you.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.