Skip to content

fix(engine): add anti-banding x264/x265 params for dark gradients#222

Merged
jrusso1020 merged 1 commit intomainfrom
fix/encoder-dark-gradient-banding
Apr 7, 2026
Merged

fix(engine): add anti-banding x264/x265 params for dark gradients#222
jrusso1020 merged 1 commit intomainfrom
fix/encoder-dark-gradient-banding

Conversation

@jrusso1020
Copy link
Copy Markdown
Collaborator

Description

Fixes color banding on dark gradients — eval issue #3, affecting prompts 3, 5, 10, 14.

Root cause: libx264's default aq-mode=1 doesn't allocate enough bits to dark flat gradient areas. With 8-bit yuv420p, subtle gradient steps get quantized to the same value, producing visible horizontal bands.

Fix: Add aq-mode=3 (auto-variance adaptive quantization) via -x264-params / -x265-params to both chunk and streaming encoders. This redistributes bits from bright/textured areas to dark flat areas where banding is most visible.

Preset Params
draft (ultrafast) aq-mode=3 only
standard (medium) aq-mode=3:aq-strength=0.8:deblock=1,1
high (slow) aq-mode=3:aq-strength=0.8:deblock=1,1
  • aq-mode=3 — auto-variance AQ, designed for flat dark areas
  • aq-strength=0.8 — slightly below default to avoid over-softening textures
  • deblock=1,1 — stronger deblocking smooths quantization band boundaries
  • GPU and VP9 encoders unaffected (have their own AQ implementations)

Testing

  • Rendered countdown composition with dark radial gradient (#1a1a2e → #0a0a0a) — smooth gradients, no visible banding
  • 6 new regression tests verifying anti-banding params are emitted for h264/h265 CPU, omitted for GPU/VP9, and deblock is skipped for ultrafast
  • All 32 engine tests pass

Note: pre-commit typecheck failure is pre-existing on main (linkedom types missing in core package).

Add aq-mode=3 (auto-variance adaptive quantization) to CPU H.264/H.265
encoding. This redistributes bits from bright/textured areas to dark flat
areas where color banding is most visible in 8-bit yuv420p output.

- standard/high presets: aq-mode=3 + aq-strength=0.8 + deblock=1,1
- draft (ultrafast): aq-mode=3 only (deblock too slow for ultrafast)
- GPU and VP9 encoders unaffected (have their own AQ implementations)

Adds 6 regression tests verifying the params are emitted correctly.

Fixes color banding on dark gradients (eval issue #3, prompts 3,5,10,14).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jrusso1020 jrusso1020 merged commit 7bd8939 into main Apr 7, 2026
20 checks passed
@jrusso1020 jrusso1020 deleted the fix/encoder-dark-gradient-banding branch April 7, 2026 16:49
jrusso1020 added a commit that referenced this pull request Apr 7, 2026
## Description

Adds proper BT.709 color space metadata and full→limited range conversion to H.264/H.265 encoding. Chrome captures frames in full-range sRGB (BT.709 primaries), but without explicit color tagging, players guess the wrong color space and range — causing color shifts across iOS/Android/desktop and crushed dark values that compound the gradient banding issue fixed in #222.

**What changed:**

| Setting | Before | After |
|---------|--------|-------|
| `color_space` | `bt470bg` (guessed) | `bt709` (explicit) |
| `color_primaries` | `unknown` | `bt709` |
| `color_transfer` | `unknown` | `bt709` |
| `color_range` | `pc` (full, wrong for H.264) | `tv` (limited, correct) |
| `time_base` | `1/15360` (varies by platform) | `1/90000` (fixed) |

**Approach:**
- BT.709 VUI params embedded via x264-params/x265-params (`colorprim=bt709:transfer=bt709:colormatrix=bt709`) — ensures the bitstream itself carries color info
- FFmpeg-level metadata flags (`-colorspace:v bt709`, etc.) — belt-and-suspenders
- `scale=in_range=pc:out_range=tv` filter converts Chrome's full-range output to TV/limited range
- VAAPI path chains the range filter with existing `format=nv12,hwupload`
- `-video_track_timescale 90000` for consistent cross-platform A/V timing (same as Remotion)
- VP9 and ProRes encoding unaffected

## Testing

- Verified via ffprobe: all 5 color metadata fields now correct
- Directly tested FFmpeg args produce expected output
- 40 engine tests pass (8 new: color metadata h264/h265, range filter CPU, VAAPI filter chain, GPU skip, VP9 skip, timescale)
- Builds cleanly, lint + format pass
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants