llama : remove beam search by ggerganov · Pull Request #7736 · ggml-org/llama.cpp

ggerganov · 2024-06-04T11:32:57Z

The beam-search functionality should be implemented in user-code, utilizing the llama_kv_cache_ API. The existing implementation is likely not used at all, so I think it is ok to skip the deprecation phase

github-actions · 2024-06-04T13:22:12Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 550 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8474.72ms p(95)=20396.97ms fails=, finish reason: stop=498 truncated=52
Prompt processing (pp): avg=93.34tk/s p(95)=398.57tk/s
Token generation (tg): avg=45.29tk/s p(95)=46.06tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/remove-beam-search commit=fb1e7b08e8cfe84ecf5b9a3225b8dc84ab436fd2

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 457.05, 457.05, 457.05, 457.05, 457.05, 692.93, 692.93, 692.93, 692.93, 692.93, 686.31, 686.31, 686.31, 686.31, 686.31, 712.45, 712.45, 712.45, 712.45, 712.45, 775.99, 775.99, 775.99, 775.99, 775.99, 772.18, 772.18, 772.18, 772.18, 772.18, 792.06, 792.06, 792.06, 792.06, 792.06, 806.49, 806.49, 806.49, 806.49, 806.49, 822.94, 822.94, 822.94, 822.94, 822.94, 823.46, 823.46, 823.46, 823.46, 823.46, 827.84, 827.84, 827.84, 827.84, 827.84, 813.09, 813.09, 813.09, 813.09, 813.09, 824.2, 824.2, 824.2, 824.2, 824.2, 833.16, 833.16, 833.16, 833.16, 833.16, 845.88, 845.88, 845.88, 845.88, 845.88, 847.93, 847.93, 847.93, 847.93, 847.93, 843.64, 843.64, 843.64, 843.64, 843.64, 840.61, 840.61, 840.61, 840.61, 840.61, 819.69, 819.69, 819.69, 819.69, 819.69, 825.31, 825.31, 825.31, 825.31, 825.31, 831.55, 831.55, 831.55, 831.55, 831.55, 831.88, 831.88, 831.88, 831.88, 831.88, 833.12, 833.12, 833.12, 833.12, 833.12, 829.18, 829.18, 829.18, 829.18, 829.18, 831.63, 831.63, 831.63, 831.63, 831.63, 833.67, 833.67, 833.67, 833.67, 833.67, 845.88, 845.88, 845.88, 845.88, 845.88, 846.57, 846.57, 846.57, 846.57, 846.57, 846.77, 846.77, 846.77, 846.77, 846.77, 850.25, 850.25, 850.25, 850.25, 850.25, 851.82, 851.82, 851.82, 851.82, 851.82, 848.37, 848.37, 848.37, 848.37, 848.37, 853.36, 853.36, 853.36, 853.36, 853.36, 860.22, 860.22, 860.22, 860.22, 860.22, 865.08, 865.08, 865.08, 865.08, 865.08, 870.75, 870.75, 870.75, 870.75, 870.75, 865.62, 865.62, 865.62, 865.62, 865.62, 863.91, 863.91, 863.91, 863.91, 863.91, 865.04, 865.04, 865.04, 865.04, 865.04, 868.05, 868.05, 868.05, 868.05, 868.05, 872.74, 872.74, 872.74, 872.74, 872.74, 873.23, 873.23, 873.23, 873.23, 873.23, 872.42, 872.42, 872.42, 872.42, 872.42, 870.4, 870.4, 870.4, 870.4, 870.4, 869.8, 869.8, 869.8, 869.8, 869.8, 872.22, 872.22, 872.22, 872.22, 872.22, 873.13, 873.13, 873.13, 873.13, 873.13, 871.5, 871.5, 871.5, 871.5, 871.5, 871.36, 871.36, 871.36, 871.36, 871.36, 870.03, 870.03, 870.03, 870.03, 870.03, 871.77, 871.77, 871.77, 871.77, 871.77, 869.45, 869.45, 869.45, 869.45, 869.45, 867.63, 867.63, 867.63, 867.63, 867.63, 867.13, 867.13, 867.13, 867.13, 867.13, 868.02, 868.02, 868.02, 868.02, 868.02, 867.57, 867.57, 867.57, 867.57, 867.57, 868.71, 868.71, 868.71, 868.71, 868.71, 870.05, 870.05, 870.05, 870.05, 870.05, 869.33, 869.33, 869.33, 869.33, 869.33, 871.75, 871.75, 871.75, 871.75, 871.75, 872.08, 872.08]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 43.47, 31.04, 31.04, 31.04, 31.04, 31.04, 30.28, 30.28, 30.28, 30.28, 30.28, 31.95, 31.95, 31.95, 31.95, 31.95, 33.14, 33.14, 33.14, 33.14, 33.14, 33.95, 33.95, 33.95, 33.95, 33.95, 34.86, 34.86, 34.86, 34.86, 34.86, 35.23, 35.23, 35.23, 35.23, 35.23, 35.22, 35.22, 35.22, 35.22, 35.22, 35.12, 35.12, 35.12, 35.12, 35.12, 35.06, 35.06, 35.06, 35.06, 35.06, 34.38, 34.38, 34.38, 34.38, 34.38, 34.26, 34.26, 34.26, 34.26, 34.26, 32.2, 32.2, 32.2, 32.2, 32.2, 30.14, 30.14, 30.14, 30.14, 30.14, 29.78, 29.78, 29.78, 29.78, 29.78, 29.94, 29.94, 29.94, 29.94, 29.94, 30.27, 30.27, 30.27, 30.27, 30.27, 30.35, 30.35, 30.35, 30.35, 30.35, 30.51, 30.51, 30.51, 30.51, 30.51, 30.46, 30.46, 30.46, 30.46, 30.46, 30.69, 30.69, 30.69, 30.69, 30.69, 30.64, 30.64, 30.64, 30.64, 30.64, 30.93, 30.93, 30.93, 30.93, 30.93, 31.13, 31.13, 31.13, 31.13, 31.13, 30.96, 30.96, 30.96, 30.96, 30.96, 31.13, 31.13, 31.13, 31.13, 31.13, 31.34, 31.34, 31.34, 31.34, 31.34, 31.52, 31.52, 31.52, 31.52, 31.52, 31.51, 31.51, 31.51, 31.51, 31.51, 31.61, 31.61, 31.61, 31.61, 31.61, 31.72, 31.72, 31.72, 31.72, 31.72, 31.77, 31.77, 31.77, 31.77, 31.77, 31.64, 31.64, 31.64, 31.64, 31.64, 31.16, 31.16, 31.16, 31.16, 31.16, 30.89, 30.89, 30.89, 30.89, 30.89, 30.83, 30.83, 30.83, 30.83, 30.83, 30.92, 30.92, 30.92, 30.92, 30.92, 31.06, 31.06, 31.06, 31.06, 31.06, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 31.24, 30.92, 30.92, 30.92, 30.92, 30.92, 30.34, 30.34, 30.34, 30.34, 30.34, 29.97, 29.97, 29.97, 29.97, 29.97, 29.44, 29.44, 29.44, 29.44, 29.44, 29.43, 29.43, 29.43, 29.43, 29.43, 29.37, 29.37, 29.37, 29.37, 29.37, 29.29, 29.29, 29.29, 29.29, 29.29, 29.3, 29.3, 29.3, 29.3, 29.3, 29.37, 29.37, 29.37, 29.37, 29.37, 29.35, 29.35, 29.35, 29.35, 29.35, 29.22, 29.22, 29.22, 29.22, 29.22, 29.25, 29.25, 29.25, 29.25, 29.25, 29.27, 29.27, 29.27, 29.27, 29.27, 29.41, 29.41, 29.41, 29.41, 29.41, 29.52, 29.52, 29.52, 29.52, 29.52, 29.64, 29.64, 29.64, 29.64, 29.64, 29.73, 29.73, 29.73, 29.73, 29.73, 29.74, 29.74, 29.74, 29.74, 29.74, 29.75, 29.75]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.12, 0.12, 0.12, 0.12, 0.12, 0.38, 0.38, 0.38, 0.38, 0.38, 0.15, 0.15, 0.15, 0.15, 0.15, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.11, 0.11, 0.11, 0.11, 0.11, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.29, 0.29, 0.29, 0.29, 0.29, 0.34, 0.34, 0.34, 0.34, 0.34, 0.46, 0.46, 0.46, 0.46, 0.46, 0.43, 0.43, 0.43, 0.43, 0.43, 0.35, 0.35, 0.35, 0.35, 0.35, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.22, 0.22, 0.22, 0.22, 0.22, 0.23, 0.23, 0.23, 0.23, 0.23, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.21, 0.21, 0.21, 0.21, 0.21, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.32, 0.32, 0.32, 0.32, 0.32, 0.39, 0.39, 0.39, 0.39, 0.39, 0.24, 0.24, 0.24, 0.24, 0.24, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 0.2, 0.2, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.51, 0.44, 0.44, 0.44, 0.44, 0.44, 0.34, 0.34, 0.34, 0.34, 0.34, 0.21, 0.21, 0.21, 0.21, 0.21, 0.28, 0.28, 0.28, 0.28, 0.28, 0.25, 0.25, 0.25, 0.25, 0.25, 0.19, 0.19, 0.19, 0.19, 0.19, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.32, 0.32, 0.32, 0.32, 0.32, 0.11, 0.11, 0.11, 0.11, 0.11, 0.25, 0.25, 0.25, 0.25, 0.25, 0.13, 0.13, 0.13, 0.13, 0.13, 0.19, 0.19, 0.19, 0.19, 0.19, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.15, 0.15, 0.15, 0.15, 0.15, 0.22, 0.22, 0.22, 0.22, 0.22, 0.29, 0.29]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717506705 --> 1717507327
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0]

llama : remove beam search

fb1e7b0

github-actions Bot added the examples label Jun 4, 2024

martindevans added a commit to martindevans/LLamaSharp that referenced this pull request Jun 4, 2024

Removed beam search. Upstream is removing it: ggml-org/llama.cpp#7736

283358c

martindevans mentioned this pull request Jun 4, 2024

Remove Beam Search SciSharp/LLamaSharp#783

Merged

ggerganov merged commit 0cd6bd3 into master Jun 4, 2024

ggerganov deleted the gg/remove-beam-search branch June 4, 2024 18:23

AdolfVonKleist mentioned this pull request Jun 7, 2024

The beam search and llama_token_get_type related functions have abetlen/llama-cpp-python#1514

Closed

thxCode mentioned this pull request Jun 18, 2024

chore: clean useless beam search param #7985

Merged

4 tasks

BrickBee mentioned this pull request Mar 21, 2025

Feature Request: New sampling method that boosts reasoning performance - looks too good? #12479

Closed

4 tasks

WilliamTambellini mentioned this pull request May 19, 2025

llama : combined beam search + grammar sampling strategy #2923

Open

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

llama : remove beam search (ggml-org#7736)

3c618a1

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

llama : remove beam search (ggml-org#7736)

515db58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

llama : remove beam search#7736

llama : remove beam search#7736
ggerganov merged 1 commit intomasterfrom
gg/remove-beam-search

ggerganov commented Jun 4, 2024

Uh oh!

github-actions Bot commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ggerganov commented Jun 4, 2024

Uh oh!

github-actions Bot commented Jun 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant