Runtime rules engine ⚙️ #249

joshua-koehler · 2025-11-17T15:23:24Z

Background

Local evaluation already supports evaluating the legacy runtime rule, which was simple exact key-value matching.

This pr

Support complex runtime rules in local-evaluation using jsonLogic.

QA'd local and remote eval ✅

https://github.com/mixpanel/sdk-integration-testing/commit/fe6d0968da0d4f30ce482abf46906d82bf7cc373

Example of what they look like in the UI

Still need caseinsensitive on rule

…me-rules-engine

msiebert

tdumitrescu

Thanks for the thorough test cases. Mostly just some nitpicks on my end!

lib/flags/utils.js

package.json

test/flags/local_flags.js

tdumitrescu · 2026-01-07T23:41:34Z

test/flags/local_flags.js

+    });
+
+    it("should return fallback when runtime rule is invalid", async () => {
+      const runtimeEvaluationRule = { "=oops=": [{ var: "plan" }, "Premium"] };


Might be worth adding debug logging for this case? (at least when the rule is loaded over the network, probably too noisy to do it on every check)

Good call out, looks like I've already covered that here

this.logger?.error(`Error evaluating runtime rule: ${error.message}`);

tdumitrescu · 2026-01-07T23:46:36Z

test/flags/local_flags.js

+      });
+
+      const result = provider.getVariant(FLAG_KEY, FALLBACK, context);
+      expect(result.variant_value).not.toBe(FALLBACK_NAME);


Can we look for a specific value here rather than just "not the fallback"? Since that could pass the test for weird failure cases like if variant_value is unexpectedly null

Good thinking, just refactored to use a helper assertion that asserts both. Note that because there's a 50/50 split on the variant, I just check that the result is one of the two. The hashing logic is tested elsewhere and is an orthogonal concern to these test cases.

tdumitrescu · 2026-01-07T23:50:02Z

test/flags/local_flags.js

-    it("should return fallback when runtime evaluation not satisfied", async () => {
-      const runtimeEval = { plan: "premium", region: "US" };
-      const flag = createTestFlag({ runtimeEvaluation: runtimeEval });
+    it("should return variant when runtime evaluation rule case-insensitively satisfied", async () => {


Logically this test case isn't any different than the one on L318 is it?

They're actually different. The one on L318 is asserting case-insensitivity of rules, this one here is asserting case-insensitivity of runtime parameters. Both have to be lowercased. If only one is lowercased, then the corresponding test fails.

tdumitrescu · 2026-01-07T23:54:30Z

test/flags/local_flags.js

+      await createFlagAndLoadItIntoSDK({ runtimeEvaluationRule }, provider);
+
+      const context = userContextWithRuntimeParameters({
+        name: "d",


might be worth adding a test case where this value is a substring of one of the array values e.g. name: "all"

Good idea, just added substring and superstring cases.

test/flags/local_flags.js

tdumitrescu · 2026-01-07T23:59:00Z

test/flags/local_flags.js

-      mockFlagDefinitionsResponse([flag]);
-      await provider.startPollingForDefinitions();
+      const context = userContextWithRuntimeParameters({
+        plan: "premium",


Apologies if I missed it, but let's make sure to have a test case that specifies the behavior when the case of the key is mismatched e.g. {PLAN: "premium"}

Good catch, this isn't under test, just added a test.

Co-authored-by: teddddd <ted@mixpanel.com>

…ode into runtime-rules-engine

joshua-koehler

Thanks for the thorough review, ready for a second look @tdumitrescu

joshua-koehler · 2026-01-08T22:32:28Z

test/flags/local_flags.js

+    });
+
+    it("should return fallback when runtime rule is invalid", async () => {
+      const runtimeEvaluationRule = { "=oops=": [{ var: "plan" }, "Premium"] };


Good call out, looks like I've already covered that here

this.logger?.error(`Error evaluating runtime rule: ${error.message}`);

joshua-koehler · 2026-01-08T22:37:16Z

test/flags/local_flags.js

+      });
+
+      const result = provider.getVariant(FLAG_KEY, FALLBACK, context);
+      expect(result.variant_value).not.toBe(FALLBACK_NAME);


Good thinking, just refactored to use a helper assertion that asserts both. Note that because there's a 50/50 split on the variant, I just check that the result is one of the two. The hashing logic is tested elsewhere and is an orthogonal concern to these test cases.

joshua-koehler · 2026-01-08T22:38:57Z

test/flags/local_flags.js

-    it("should return fallback when runtime evaluation not satisfied", async () => {
-      const runtimeEval = { plan: "premium", region: "US" };
-      const flag = createTestFlag({ runtimeEvaluation: runtimeEval });
+    it("should return variant when runtime evaluation rule case-insensitively satisfied", async () => {


They're actually different. The one on L318 is asserting case-insensitivity of rules, this one here is asserting case-insensitivity of runtime parameters. Both have to be lowercased. If only one is lowercased, then the corresponding test fails.

joshua-koehler · 2026-01-08T22:44:05Z

test/flags/local_flags.js

-      mockFlagDefinitionsResponse([flag]);
-      await provider.startPollingForDefinitions();
+      const context = userContextWithRuntimeParameters({
+        plan: "premium",


Good catch, this isn't under test, just added a test.

joshua-koehler · 2026-01-08T22:46:18Z

test/flags/local_flags.js

+      await createFlagAndLoadItIntoSDK({ runtimeEvaluationRule }, provider);
+
+      const context = userContextWithRuntimeParameters({
+        name: "d",


Good idea, just added substring and superstring cases.

hans-lizihan and others added 24 commits November 13, 2025 13:34

Initial formatting

fb198e7

Initial formatting + linting

bc94992

Github actions

32ce172

no-var

fce257e

Merged conflicts

2fdf905

Correct export

d1ea57b

Fixed

bdeaa59

extract var for test values

064f4af

mark old runtime rules as legacy

29b2d3b

succint flag setup to one line

fbde335

DRY tests with helper - more succint

76d8393

DRY context creation for runtime

89fea7f

use random string for clarity on wrong values

a8033d4

DRY fallback

748324f

runtime rule negative case ❌

43b344f

add stopgap positive test case (doesn't fail)

9c31468

import jsonLogic and evaluate runtime rule ✅

1278be3

caseinsensitive to parameters

c095d50

Still need caseinsensitive on rule

case-insensitivity ✅

08cd7db

add complex rule test-cases ✅

0c9ffcf

consistent runtime rule test names

6706908

add error handling for garbled jsonlogic

6fd7958

add multi-condition case-insensitive check

558b2d9

Merge branch 'master' of github.com:mixpanel/mixpanel-node into runti…

6d17f73

…me-rules-engine

joshua-koehler requested a review from msiebert November 17, 2025 15:23

pretty

c6b4031

msiebert approved these changes Nov 17, 2025

View reviewed changes

joshua-koehler requested a review from tdumitrescu November 26, 2025 18:29

joshua-koehler mentioned this pull request Dec 2, 2025

🚢 Release runtime rules engine #250

Draft

tdumitrescu reviewed Jan 8, 2026

View reviewed changes

joshua-koehler and others added 8 commits January 8, 2026 16:23

Apply simple suggestions from code review

efff77f

Co-authored-by: teddddd <ted@mixpanel.com>

prevent randomString() from accidental matches

0339f50

Merge branch 'runtime-rules-engine' of github.com:mixpanel/mixpanel-n…

c48189c

…ode into runtime-rules-engine

consolidate lowercasing of json objects

f9bf9b4

assert that variant is returned, not just NOT fallback

5296952

dry variant-returned assertion

4c2c7e0

add test for case-insensitive runtime param keys

6b02921

add "in" negative case for substring & superstring

4c10a56

joshua-koehler commented Jan 8, 2026

View reviewed changes

prettier

4dc5321

joshua-koehler requested a review from tdumitrescu January 8, 2026 22:51

Runtime rules engine ⚙️ #249

Are you sure you want to change the base?

Runtime rules engine ⚙️ #249

Uh oh!

Conversation

joshua-koehler commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

This pr

QA'd local and remote eval ✅

Example of what they look like in the UI

Uh oh!

msiebert left a comment

Choose a reason for hiding this comment

Uh oh!

tdumitrescu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshua-koehler Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshua-koehler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshua-koehler Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

joshua-koehler commented Nov 17, 2025 •

edited

Loading

joshua-koehler Jan 8, 2026 •

edited

Loading

joshua-koehler Jan 8, 2026 •

edited

Loading