ZSTD decoder: cleanups, new sequence executor and fixes for subcomponents by magancarz · Pull Request #3636 · google/xls

magancarz · 2025-12-31T13:11:28Z

Added functionality:

Cleaned up the ZSTD decoder and cocotb tests environment
Fixed the ZSTD decoder subcomponents
Added a detailed testing environment with intermediate values checks
Added a CLI tool for running decoder test cases
Refactored the sequence executor architecture
Added fixes for parameterizing AXI buses width
Added a verilog model for 2RW ram
Added Verilator as a new dependency to improve cocotb tests runtime
Updated README
- Updated and added new information about the components of the ZSTD Decoder
- Added diagrams to present the updated structure
Added a workaround for a bug causing the xls_benchmark_ir targets to fail

Supersedes #3035

google-cla · 2025-12-31T13:11:37Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

proppy · 2026-03-03T07:14:58Z

@rw1nkler @magancarz sorry I let that slip, can you rebase?

…d channels; update flags, channel names and top names." This reverts commit 77dcc81.

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

Add cocotb testing utilities: - XLSStruct for easier handling and serializing/deserializing XLS structs - XLSChannel that serves as a dummy receiving channel - XLSMonitor that monitors transactions on an XLS channel - XLSDriver that can send data on an XLS channel - LatencyScoreboard that can measure latency between corresponding transactions on input and output buses - File-backed AXI memory python model Add cocotb tests for: - modules/zstd/memory/MemReader - modules/zstd/memory/AxiWriter - modules/zstd/memory/MemWriter Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com> Co-authored-by: Robert Winkler <rwinkler@antmicro.com> Co-authored-by: Michal Czyz <mczyz@antmicro.com> Signed-off-by: Michal Czyz <mczyz@antmicro.com> Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com> Signed-off-by: Robert Winkler <rwinkler@antmicro.com> Signed-off-by: Krzysztof Obłonczek <koblonczek@antmicro.com>

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

This reverts commit 04ad379225b706ddf492d440c673e77348d7a409.

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Co-authored-by: Robert Winkler <rwinkler@antmicro.com> Signed-off-by: Robert Winkler <rwinkler@antmicro.com> Signed-off-by: Krzysztof Obłonczek <koblonczek@antmicro.com>

ZstdDecoder: * Cocotb tests: * Move third-party verilog modules (AXI Interconnect) to external directory * Replace AXI Interconnect with AXI Crossbar that handles simultaneous AXI Read and Write transactions * Add reference memory and fill it with expected data for comparison against testbench memory at the end of the decoding Internal-tag: [#67272] Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Adjust cocotb test to: * Removal of Repacketizer proc * Removal of stream-based output channels from * SequenceExecutor * ZstdDecoder Internal-tag: [#67272] Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

* Decode multiple ZSTD frames in a single cocotb testbench * Add one cocotb testbench per type of the ZSTD frames: * Frames with RAW blocks only * Frames with RLE blocks only * Frames with Compressed blocks only (disabled) * Frames with mixed blocks (disabled) Internal-tag: [#67272] Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Caused by timeouts after rebase - looks like regression in the performance of codegen Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

…sources Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Improve formatting, wording and fix lint issues Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com> Signed-off-by: Wojciech Sipak <wsipak@antmicro.com>

Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com> Co-authored-by: Krzysztof Oblonczek <koblonczek@antmicro.com> Co-authored-by: Wojciech Sipak <wsipak@antmicro.com> Co-autored-by: Dominik Lau <dlau@antmicro.com> Co-authored-by: Szymon Gizler <sgizler@antmicro.com> Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

…calculation As defined in https://datatracker.ietf.org/doc/html/rfc8878#section-4.2.1.1 raw weights consume ceil(n_symb/2) bytes. The correct equivalent of that ceiled div is: (n_symb+1)>>1 rather than (n_symb>>1)+1 we previously used

…t weight Previous version did not take into account weight swapping.

Old test reimplemented parts of decoder and compared output of proc against reimplementation. It had very little value (as we are likely to slip same bug into both implementations) and was hard to reason about. New test compares decoder against readable explicit golden prepared to test all recently fixed bugs (swapped weights, bad description size and wrong last weight position)

… a remainder

…rite

…eeless literals

…iteration if required

… length

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

Co-authored-by: Wojciech Sipak <wsipak@antmicro.com> Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

magancarz · 2026-03-17T07:42:33Z

I've rebased changes on latest main with fixes from #3931 so the CI can pass.

magancarz · 2026-03-17T11:36:22Z

I've updated Bazel and pip lock files.

kbieganski · 2026-03-31T11:50:25Z

@proppy PTAL. We rebased earlier, but there are some new conflicts. We'd rather hold off on rebasing again for now until we get an initial round of feedback.

magancarz force-pushed the zstd_decoder_pr branch from dcab137 to 412ca7f Compare December 31, 2025 13:25

magancarz mentioned this pull request Dec 31, 2025

xls_benchmark_ir targets failing due to internal checks #3637

Closed

rw1nkler mentioned this pull request Jan 2, 2026

ZSTD decoder: cocotb utils and decoder testcases, verilog helper modules #3035

Closed

grebe requested a review from proppy January 2, 2026 22:31

hzeller mentioned this pull request Jan 22, 2026

zstd library dependency still needed ? #3708

Open

magancarz and others added 21 commits March 16, 2026 15:05

Revert "[Proc-scoped channels] Update zstd to compile with proc-scope…

f05ad59

…d channels; update flags, channel names and top names." This reverts commit 77dcc81.

modules: zstd: specify larger die dimensions

1109482

passes: check if a node is tracked before getting info

fb1d7f3

zstd: skip place and route targets with long tag

f06edf2

zstd: add long tag to long-running synth targets

31d5700

Add a separate test case for each test file

e6c2155

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

Add "pytype binary, library" hint

6389004

xls/modules/zstd: expose fifo verilog module

f3fc674

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/cocotb: Add ZSTD frame generator library

971e35c

This reverts commit 04ad379225b706ddf492d440c673e77348d7a409.

dependency_support: Add zstandard python library

d4046d2

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add verilog simulation of the ZstdDecoder

cab9f6a

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Use frame generator in tests

05d26f5

Co-authored-by: Robert Winkler <rwinkler@antmicro.com> Signed-off-by: Robert Winkler <rwinkler@antmicro.com> Signed-off-by: Krzysztof Obłonczek <koblonczek@antmicro.com>

modules/zstd/zstd_dec: Remove stream-based output interface

9ad44b0

Adjust cocotb test to: * Removal of Repacketizer proc * Removal of stream-based output channels from * SequenceExecutor * ZstdDecoder Internal-tag: [#67272] Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/memory/mem_writer: Reduce the amount of random test cases

44d929e

Caused by timeouts after rebase - looks like regression in the performance of codegen Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/memory: Move verilog sources to rtl subdirectory

19ce56a

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

xls/modules/zstd/external: Add docstring and license for third party …

3711dac

…sources Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Adjust cocotb test to removal of RESET CSR

8c78325

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

Style changes

8d5f00f

Improve formatting, wording and fix lint issues Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com> Signed-off-by: Wojciech Sipak <wsipak@antmicro.com>

magancarz force-pushed the zstd_decoder_pr branch from 412ca7f to 0d9c76b Compare March 17, 2026 07:35

sgizler and others added 24 commits March 17, 2026 08:40

modules: zstd: huffman weights dec: fix bad insertion position of las…

7d2060e

…t weight Previous version did not take into account weight swapping.

modules: zstd: fse table decoder: fix handling last read bit if it is…

9254c84

… a remainder

modules: zstd: rtl: ram_1r1w: fix handling of simultaneous read and w…

e892efa

…rite

modules: zstd: huffman ctrl: omit building new codes when decoding tr…

1bcd8b8

…eeless literals

modules: zstd: cocotb: print correct packet values less often

a0a95aa

modules: zstd: huffman weights dec: remove duplicate call

a898a33

modules: zstd: huffman weights dec: add more test cases for raw weights

0c69a3a

modules: zstd: huffman weights dec: send last raw weight in the next …

ece07dd

…iteration if required

modules: zstd: performance: add generating a throughput benchmark

c1d3a2f

modules: zstd: sequence executor: fix clearing current history buffer…

b4bf4bb

… length

modules: zstd: remove outdated/commented code

4731bbb

Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

modules: zstd: update README

3b97215

Co-authored-by: Wojciech Sipak <wsipak@antmicro.com> Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

modules: zstd: fix errors after rebase

bac2442

modules: zstd: cocotb: update deps to python 3.12

fed2208

dependencies: update python lock file

8805414

examples: ram: make 2RW ram proc public

87a630b

modules: zstd: sequence executor: rewrite seq exec architecture

71c5bcc

modules: zstd: fix parametrizing AXI bus width

e294e1b

modules: zstd: set worst case throughput to 5

f362e07

bazel: update bazel lock

64c8afe

modules: zstd: fix errors after rebase

d80f77a

modules: zstd: add manual tag to collect benchmark reports target

a21d6b6

magancarz force-pushed the zstd_decoder_pr branch from 0d9c76b to a21d6b6 Compare March 17, 2026 07:40

magancarz added 2 commits March 17, 2026 11:27

bazel: update lock file

c9e6ab6

python: update pip requirements lock

ce13d8d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZSTD decoder: cleanups, new sequence executor and fixes for subcomponents#3636

ZSTD decoder: cleanups, new sequence executor and fixes for subcomponents#3636
magancarz wants to merge 94 commits intogoogle:mainfrom
antmicro:zstd_decoder_pr

magancarz commented Dec 31, 2025

Uh oh!

google-cla Bot commented Dec 31, 2025

Uh oh!

proppy commented Mar 3, 2026 •

edited

Loading

Uh oh!

magancarz commented Mar 17, 2026 •

edited

Loading

Uh oh!

magancarz commented Mar 17, 2026

Uh oh!

kbieganski commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

magancarz commented Dec 31, 2025

Uh oh!

google-cla Bot commented Dec 31, 2025

Uh oh!

proppy commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

magancarz commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

magancarz commented Mar 17, 2026

Uh oh!

kbieganski commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

proppy commented Mar 3, 2026 •

edited

Loading

magancarz commented Mar 17, 2026 •

edited

Loading