[Proposal] Share Threshold for Wildcard Rate Limiting

[xuannam230201]

Proposal: Share Threshold for Wildcard Rate Limiting

Problem Statement

Currently, rate limiting rules for wildcard values create isolated thresholds for each matching value. While this provides fine-grained control, there are scenarios where multiple values matching a wildcard pattern should share a single rate limit threshold.

Current Behavior

When a wildcard pattern like files/* is defined, each matching value (e.g., files/a.pdf, files/b.csv, files/c.txt) gets its own isolated rate limit counter. This means:

• files/a.pdf has its own quota of 100 requests/hour
• files/b.csv has its own quota of 100 requests/hour
• files/c.txt has its own quota of 100 requests/hour

Each can consume up to 100 requests/hour independently.

Use Cases Requiring Shared Thresholds

There are legitimate use cases where all values matching a wildcard should share a single threshold:

1. File-based rate limiting: When rate limiting access to files under a path pattern (e.g., api/files/{file_name}), you may want to limit total access across all files rather than per-file.

2. Resource-based rate limiting: For resources like models (e.g., api/models/{model_name}), you may want to limit total API calls across all models rather than per-model.

3. Version-based rate limiting: For API versions (e.g., api/v1/*, api/v2/*, api/v3/*), you may want to limit total requests across all versions.

4. Cost control: When different wildcard values represent resources with similar costs, sharing thresholds helps control aggregate resource consumption.

Example Scenario

Consider a rate limit configuration:

domain: api
descriptors:
  - key: files
    value: files/*
    rate_limit:
      unit: hour
      requests_per_unit: 10

With current behavior:

• Request for files/a.pdf uses 1 of 10 quota
• Request for files/b.csv uses 1 of 10 quota (separate from a.pdf)
• Both can make 10 requests each = 20 total requests

With shared threshold (desired):

• Request for files/a.pdf uses 1 of 10 shared quota
• Request for files/b.csv uses 1 of 10 shared quota (same pool)
• Total of 10 requests across all files

Proposed Solution

Add a new optional field share_threshold: true to descriptor entries with wildcard values. When enabled, all values matching that wildcard pattern will share the same rate limit threshold.

Configuration Syntax

domain: api
descriptors:
  # Shared threshold: all files/* values share the same limit
  - key: files
    value: files/*
    share_threshold: true
    rate_limit:
      unit: hour
      requests_per_unit: 10

  # Isolated threshold: each files_no_share/* value has its own limit
  - key: files_no_share
    value: files_no_share/*
    share_threshold: false  # or omit, defaults to false
    rate_limit:
      unit: hour
      requests_per_unit: 10

Behavior

1. Default behavior (backward compatible): When share_threshold is not specified or set to false, each wildcard value gets its own isolated threshold (current behavior).

2. Shared threshold: When share_threshold: true is set:

   • All values matching the wildcard pattern share the same cache key
   • All values share the same rate limit counter
   • When the shared threshold is exhausted, all matching values are rate limited

3. Validation: share_threshold can only be used with wildcard values (values ending with *). Using it with non-wildcard values will result in a configuration error.

4. Explicit value precedence: Explicit values (non-wildcard) always take precedence over wildcard matches, even when share_threshold is enabled on the wildcard.

Example Configurations

Example 1: Basic Shared Threshold

domain: file-service
descriptors:
  - key: path
    value: files/*
    share_threshold: true
    rate_limit:
      unit: hour
      requests_per_unit: 100

• files/a.pdf, files/b.csv, files/subdir/c.txt all share a pool of 100 requests/hour
• If 50 requests are made for files/a.pdf and 50 for files/b.csv, a request for files/c.txt will be rate limited

Example 2: Mixed Shared and Isolated

domain: api
descriptors:
  - key: files
    value: files/*
    share_threshold: true
    rate_limit:
      unit: hour
      requests_per_unit: 10

  - key: models
    value: models/*
    share_threshold: false
    rate_limit:
      unit: hour
      requests_per_unit: 20

• All files/* values share 10 requests/hour
• Each models/* value gets its own 20 requests/hour

Implementation Details

Cache Key Generation

When share_threshold: true is enabled, the cache key uses the wildcard pattern instead of the actual runtime value:

• Without share_threshold: domain_key_files/a.pdf_<timestamp>
• With share_threshold: domain_key_files/*_<timestamp>

This ensures all matching values use the same cache key and share the same counter.

Metrics

When share_threshold is combined with value_to_metric or detailed_metric:

• The metric key includes the wildcard prefix (the part before *) instead of the full runtime value
• This reflects that values are sharing a threshold while still providing visibility

Example:

• Request: route=api/v1, method=GET
• Metric key: domain.route_api.method_GET (includes wildcard prefix api instead of full value api/v1)

Backward Compatibility

• Fully backward compatible: Default value is false, maintaining current isolated threshold behavior
• No breaking changes: Existing configurations continue to work without modification
• Opt-in feature: Must explicitly set share_threshold: true to enable shared thresholds

Testing

The implementation includes:

• Unit tests covering basic functionality, nested descriptors, and edge cases
• Integration tests verifying end-to-end behavior with Redis
• Test cases for explicit value precedence over wildcards
• Test cases for interaction with value_to_metric and detailed_metric

[xuannam230201]

Hi @collin-lee , could you please help to take a look at this proposal? Does it make sense to apply this?

Thanks in advance!

[collin-lee]

Think it's a good idea. I left some comments.

[collin-lee]

// Instead of unconditional allocation:
var shareThresholdPatterns map[int]string // nil initially

// Only allocate when needed:
if matchedViaWildcard && nextDescriptor != nil && nextDescriptor.shareThreshold {
if shareThresholdPatterns == nil {
shareThresholdPatterns = make(map[int]string)
}
// ... rest of logic
}

[xuannam230201]

Updated

[collin-lee]

Maybe only allocate ShareThresholdKeyPattern when patterns are tracked:

if i == len(descriptor.Entries)-1 {
// Create a copy of the rate limit to avoid modifying the shared object
originalLimit := nextDescriptor.limit
rateLimit = &RateLimit{
FullKey: originalLimit.FullKey,
Stats: originalLimit.Stats,
Limit: originalLimit.Limit,
Unlimited: originalLimit.Unlimited,
ShadowMode: originalLimit.ShadowMode,
Name: originalLimit.Name,
Replaces: originalLimit.Replaces,
DetailedMetric: originalLimit.DetailedMetric,
// Initialize ShareThresholdKeyPattern to nil - will be allocated only if needed
ShareThresholdKeyPattern: nil,
}
// Apply all tracked share_threshold patterns when we find the rate_limit
// This works whether the rate_limit is at the wildcard level or deeper
// Only allocate the slice if we actually have patterns to store
if len(shareThresholdPatterns) > 0 {
rateLimit.ShareThresholdKeyPattern = make([]string, len(descriptor.Entries))
for idx, pattern := range shareThresholdPatterns {
rateLimit.ShareThresholdKeyPattern[idx] = pattern
logger.Debugf("share_threshold enabled for entry index %d, using wildcard pattern %s", idx, pattern)
}
}
}

[xuannam230201]

Updated

[xuannam230201]

@collin-lee , I updated based on your comments. Could you please take a look at this PR again, thank you!

[xuannam230201]

Hi @collin-lee , I see the build was failed after this PR was merged. But I think the failed thing doesn't relate to this change.

Could you please help to take a look? Does it make sense to re-trigger the build? I think it's flaky because I ran few times locally and it was passed.

Thanks in advance.