[refactor] Serving into proper modules by SunMarc · Pull Request #44796 · huggingface/transformers

SunMarc · 2026-03-17T13:04:06Z

What does this PR do?

This PR refactors transformers serve so that it is not in a single file. We split it into multiple files with clear responsabilities. There were 2,293 lines initially in the serve.py file.

  ┌────────────────────────────┬───────┬──────────────────────────────────────────────┐
  │            File            │ Lines │                   Purpose                    │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/chat_completion.py │ 411   │ /v1/chat/completions handler                 │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/response.py        │ 565   │ /v1/responses handler (with queue draining)  │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/utils.py           │ 941   │ Shared types, streamers, generation managers │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/model_manager.py   │ 449   │ Model loading/caching                        │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/server.py          │ 127   │ FastAPI app + routes                         │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/transcription.py   │ 181   │ Audio transcription handler                  │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ serving/__init__.py        │ 17    │ Package init                                 │
  ├────────────────────────────┼───────┼──────────────────────────────────────────────┤
  │ Total new                  │ 2,691 │                                              │
  └────────────────────────────┴───────┴──────────────────────────────────────────────┘

I've added and fix some features that there missing or broken from the orignal serve. It was easier to fix those in this refactor as it helped me find the right design.

transcriptions fixed
CB streaming + non streaming tested with concurrency (up to 20k request at the same time - non streaming works nicely, streaming have a 10-20% gap compared to it)
generate and CB parity
response and chat completion parity
generate + compile
typing + docstring as much as possible cc @tarekziade
adding lots of tests, especially integration tests (1600 + 700 -)

For CB to work correctly, we need to first merge this PR: #45063 -> merged !

I'll add benchmarks in a follow-up as this PR is already big enough !

HuggingFaceDocBuilderDev · 2026-03-17T13:13:39Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LysandreJik

did a first pass!

LysandreJik · 2026-03-25T07:50:10Z

+
+        messages = body["messages"]
+
+        # HACK: tiny-agents sends requests ending with assistant message — skip


Maybe worth elaborating this comment a bit more in the future

yeah i kept it as it was there in the orignal serve.py but i didn't do any testing. Not sure if it is worth keeping

LysandreJik · 2026-03-25T08:07:21Z

why do we need an additional file for this btw? IMO we should focus on the existing test file, with the addition of other tests there. I would still make sure all existing tests in the existing module pass.

same as serve.py. I kept the original file to make sure that all the tests are covered as I iterate through the different features. I will update the name at the end of the refactor !

github-actions · 2026-03-31T16:53:14Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	a4a429cd	workflow commit (merge commit)
PR	ff02cd79	branch commit (from PR)
main	bc576731	base commit (on `main`)

Model CI Report

❌ 3 new failed tests from this PR 😭

cli:
tests/cli/test_serve.py::TestTranscription::test_transcription_openai_client (✅ ⟹ ❌)
tests/cli/test_serve.py::TestTranscription::test_transcription_returns_text (✅ ⟹ ❌)
tests/cli/test_serve.py::TestTranscription::test_transcription_streaming (✅ ⟹ ❌)

SunMarc · 2026-03-31T17:01:35Z

run-slow: cli

github-actions · 2026-03-31T17:03:01Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["cli"]
quantizations: []

github-actions · 2026-03-31T17:06:10Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	80567983	workflow commit (merge commit)
PR	ffe4c64e	branch commit (from PR)
main	bc576731	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

SunMarc · 2026-03-31T17:08:14Z

run-slow: cli

github-actions · 2026-03-31T17:09:22Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["cli"]
quantizations: []

github-actions · 2026-03-31T17:14:03Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	90ca80e2	workflow commit (merge commit)
PR	06a78815	branch commit (from PR)
main	bc576731	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

* new serve file * app * model_manager done * update serve * style * poc done * renaming * fix * new tests * update metrics and processor * hardcode n_batch for now * add response api + compile * more tests * add it for now but we will move it * remove cache impl * add back load_model * fix naming * add transcription * tool calls better ! * vlm support for both response and chat endpoints * update bench * fix vl test * first iteration of cb * cb tests * typing + review * update test * better benchmark * better stream * update bench * fix * serve refactored * merge * update * fix * style * simpler * style * update warmup * remove llamacpp integration for now * styke * styke * style again * remove annoattion * review ! * style * much cleaner * renamed * remove bench for now * batch output * style * type * better tests * update test * queue draining * some logs * readd nathan feature + some minor fixes * fix * guard transcription * better now * fix * adding lock to see if this helps * remove locks * lock again * update bench and remove lock for now

SunMarc added 7 commits March 16, 2026 17:57

new serve file

d8e7c45

app

f238867

model_manager done

be0291d

update serve

e84d82e

style

fb77305

poc done

d869d62

Merge remote-tracking branch 'origin/main' into refactor-serving

5aadd1a

SunMarc added 13 commits March 17, 2026 13:37

renaming

bd734e8

fix

69d3264

new tests

f5afd6c

update metrics and processor

fedad8e

hardcode n_batch for now

9b904b1

add response api + compile

0084b91

more tests

1d5d1cb

add it for now but we will move it

3d64a8c

Merge remote-tracking branch 'origin/main' into refactor-serving

74b3593

remove cache impl

552603c

add back load_model

3643ece

fix naming

12c0f55

add transcription

d4ffdf4

tarekziade mentioned this pull request Mar 23, 2026

refactor: improved the cli server module code organization #44875

Draft

tool calls better !

68cd5bc

SunMarc marked this pull request as ready for review March 24, 2026 09:22

SunMarc added 3 commits March 24, 2026 10:13

vlm support for both response and chat endpoints

6da3f3c

update bench

a92ebe2

fix vl test

76a5c83

LysandreJik reviewed Mar 25, 2026

View reviewed changes

SunMarc added 2 commits March 26, 2026 17:11

first iteration of cb

31e59c3

cb tests

962d039

SunMarc added 2 commits March 31, 2026 16:58

guard transcription

307498e

better now

ffe4c64

SunMarc requested a review from NathanHB March 31, 2026 17:01

fix

06a7881

adding lock to see if this helps

052cbc7

SunMarc force-pushed the refactor-serving branch from 24fb171 to 052cbc7 Compare April 1, 2026 10:26

SunMarc added 3 commits April 1, 2026 11:42

remove locks

6799727

lock again

3a07c86

update bench and remove lock for now

7a7abf2

SunMarc added this pull request to the merge queue Apr 1, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 1, 2026

SunMarc added this pull request to the merge queue Apr 1, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 1, 2026

Merge branch 'main' into refactor-serving

bbd5cb0

SunMarc enabled auto-merge April 1, 2026 15:32

SunMarc added this pull request to the merge queue Apr 1, 2026

Merged via the queue into main with commit 38593c2 Apr 1, 2026
30 checks passed

SunMarc deleted the refactor-serving branch April 1, 2026 16:03

stevhliu mentioned this pull request Apr 1, 2026

[docs] transformers serve #45174

Merged


		messages = body["messages"]

		# HACK: tiny-agents sends requests ending with assistant message — skip

Conversation

SunMarc commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Mar 17, 2026

Uh oh!

LysandreJik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LysandreJik Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

SunMarc Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LysandreJik Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

SunMarc Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 31, 2026

CI Results

Commit Info

Model CI Report

Uh oh!

SunMarc commented Mar 31, 2026

Uh oh!

github-actions Bot commented Mar 31, 2026

Uh oh!

github-actions Bot commented Mar 31, 2026

CI Results

Commit Info

Uh oh!

SunMarc commented Mar 31, 2026

Uh oh!

github-actions Bot commented Mar 31, 2026

Uh oh!

github-actions Bot commented Mar 31, 2026

CI Results

Commit Info

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SunMarc commented Mar 17, 2026 •

edited

Loading