server : return error on too large embedding input by ggerganov · Pull Request #7389 · ggml-org/llama.cpp

ggerganov · 2024-05-19T16:03:00Z

The input for embeddings must fit inside the physical batch size. If it does not, we now return an error:

$ ▶ curl -X POST "http://localhost:8080/embedding" --data '{"content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam rhoncus mauris eget magna semper, ut varius arcu eleifend. Vestibulum quis justo eget ex pretium sollicitudin. Nam euismod orci vulputate erat sagittis, sed pulvinar ante varius. Proin in dui non eros sodales tempus. Proin et mi scelerisque tellus eleifend auctor. Sed sagittis erat sapien, in porttitor augue bibendum nec. Nam ut mi accumsan lorem volutpat tempus. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. In ac nulla tempor, pharetra felis id, venenatis tortor. Donec felis turpis, egestas non ligula at, eleifend fringilla est. Fusce elit mi, fermentum a sapien eleifend, rutrum scelerisque eros. Sed et vestibulum orci. Quisque ut magna vel nibh accumsan dictum eget eu urna. Duis rhoncus, lacus in imperdiet tincidunt, turpis turpis vestibulum ante, at mollis nisi massa et purus. Phasellus sed ante eros. Aenean consequat nisi non massa eleifend finibus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce ultrices libero id metus consequat semper. Nam venenatis, est quis interdum commodo, nisl ex placerat diam, sed fringilla ex nisi sed sem. Pellentesque luctus orci id tellus dictum tristique. Integer molestie varius risus quis maximus. In id feugiat nulla, at scelerisque massa. Nulla neque diam, consequat ac orci laoreet, venenatis pharetra enim.Aenean rhoncus dapibus augue ac volutpat. Nullam laoreet, lorem quis fermentum scelerisque,"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1613  100   120  100  1493  32795   398k --:--:-- --:--:-- --:--:--  525k
{
  "error": {
    "code": 500,
    "message": "input is too large to process. increase the physical batch size",
    "type": "server_error"
  }
}

JohannesGaessler

Definitely an improvement over master.

github-actions · 2024-05-19T22:16:46Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 533 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8754.75ms p(95)=20897.88ms fails=, finish reason: stop=477 truncated=56
Prompt processing (pp): avg=99.93tk/s p(95)=440.6tk/s
Token generation (tg): avg=51.45tk/s p(95)=48.05tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/server-embd-err commit=85137247b2de79aeede8cf0a8dbd8be9ddc0bafb

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 304.83, 304.83, 304.83, 304.83, 304.83, 730.64, 730.64, 730.64, 730.64, 730.64, 728.45, 728.45, 728.45, 728.45, 728.45, 788.91, 788.91, 788.91, 788.91, 788.91, 848.64, 848.64, 848.64, 848.64, 848.64, 840.83, 840.83, 840.83, 840.83, 840.83, 842.84, 842.84, 842.84, 842.84, 842.84, 843.04, 843.04, 843.04, 843.04, 843.04, 856.74, 856.74, 856.74, 856.74, 856.74, 856.72, 856.72, 856.72, 856.72, 856.72, 874.51, 874.51, 874.51, 874.51, 874.51, 857.16, 857.16, 857.16, 857.16, 857.16, 851.66, 851.66, 851.66, 851.66, 851.66, 874.92, 874.92, 874.92, 874.92, 874.92, 882.05, 882.05, 882.05, 882.05, 882.05, 888.67, 888.67, 888.67, 888.67, 888.67, 885.46, 885.46, 885.46, 885.46, 885.46, 891.95, 891.95, 891.95, 891.95, 891.95, 899.53, 899.53, 899.53, 899.53, 899.53, 899.66, 899.66, 899.66, 899.66, 899.66, 903.84, 903.84, 903.84, 903.84, 903.84, 905.83, 905.83, 905.83, 905.83, 905.83, 903.78, 903.78, 903.78, 903.78, 903.78, 845.62, 845.62, 845.62, 845.62, 845.62, 847.34, 847.34, 847.34, 847.34, 847.34, 846.87, 846.87, 846.87, 846.87, 846.87, 840.47, 840.47, 840.47, 840.47, 840.47, 835.94, 835.94, 835.94, 835.94, 835.94, 836.87, 836.87, 836.87, 836.87, 836.87, 837.41, 837.41, 837.41, 837.41, 837.41, 841.62, 841.62, 841.62, 841.62, 841.62, 843.04, 843.04, 843.04, 843.04, 843.04, 840.4, 840.4, 840.4, 840.4, 840.4, 844.78, 844.78, 844.78, 844.78, 844.78, 857.29, 857.29, 857.29, 857.29, 857.29, 862.24, 862.24, 862.24, 862.24, 862.24, 862.47, 862.47, 862.47, 862.47, 862.47, 856.11, 856.11, 856.11, 856.11, 856.11, 854.2, 854.2, 854.2, 854.2, 854.2, 853.46, 853.46, 853.46, 853.46, 853.46, 857.91, 857.91, 857.91, 857.91, 857.91, 857.74, 857.74, 857.74, 857.74, 857.74, 851.36, 851.36, 851.36, 851.36, 851.36, 820.63, 820.63, 820.63, 820.63, 820.63, 821.75, 821.75, 821.75, 821.75, 821.75, 821.1, 821.1, 821.1, 821.1, 821.1, 818.13, 818.13, 818.13, 818.13, 818.13, 823.1, 823.1, 823.1, 823.1, 823.1, 825.71, 825.71, 825.71, 825.71, 825.71, 825.13, 825.13, 825.13, 825.13, 825.13, 824.86, 824.86, 824.86, 824.86, 824.86, 828.22, 828.22, 828.22, 828.22, 828.22, 826.37, 826.37, 826.37, 826.37, 826.37, 833.99, 833.99, 833.99, 833.99, 833.99, 832.5, 832.5, 832.5, 832.5, 832.5, 837.94, 837.94, 837.94, 837.94, 837.94, 841.42, 841.42, 841.42, 841.42, 841.42, 841.15, 841.15, 841.15, 841.15, 841.15, 842.28, 842.28, 842.28, 842.28, 842.28, 842.32, 842.32, 842.32, 842.32, 842.32, 843.18]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 41.67, 41.67, 41.67, 41.67, 41.67, 25.86, 25.86, 25.86, 25.86, 25.86, 27.65, 27.65, 27.65, 27.65, 27.65, 32.44, 32.44, 32.44, 32.44, 32.44, 31.54, 31.54, 31.54, 31.54, 31.54, 31.93, 31.93, 31.93, 31.93, 31.93, 33.11, 33.11, 33.11, 33.11, 33.11, 33.61, 33.61, 33.61, 33.61, 33.61, 34.12, 34.12, 34.12, 34.12, 34.12, 34.11, 34.11, 34.11, 34.11, 34.11, 33.51, 33.51, 33.51, 33.51, 33.51, 33.5, 33.5, 33.5, 33.5, 33.5, 33.35, 33.35, 33.35, 33.35, 33.35, 33.17, 33.17, 33.17, 33.17, 33.17, 30.75, 30.75, 30.75, 30.75, 30.75, 30.35, 30.35, 30.35, 30.35, 30.35, 30.44, 30.44, 30.44, 30.44, 30.44, 30.7, 30.7, 30.7, 30.7, 30.7, 30.1, 30.1, 30.1, 30.1, 30.1, 29.96, 29.96, 29.96, 29.96, 29.96, 30.06, 30.06, 30.06, 30.06, 30.06, 30.09, 30.09, 30.09, 30.09, 30.09, 30.04, 30.04, 30.04, 30.04, 30.04, 30.02, 30.02, 30.02, 30.02, 30.02, 30.05, 30.05, 30.05, 30.05, 30.05, 30.17, 30.17, 30.17, 30.17, 30.17, 30.39, 30.39, 30.39, 30.39, 30.39, 30.22, 30.22, 30.22, 30.22, 30.22, 30.3, 30.3, 30.3, 30.3, 30.3, 30.48, 30.48, 30.48, 30.48, 30.48, 30.64, 30.64, 30.64, 30.64, 30.64, 30.71, 30.71, 30.71, 30.71, 30.71, 30.8, 30.8, 30.8, 30.8, 30.8, 30.95, 30.95, 30.95, 30.95, 30.95, 31.02, 31.02, 31.02, 31.02, 31.02, 30.85, 30.85, 30.85, 30.85, 30.85, 30.59, 30.59, 30.59, 30.59, 30.59, 30.29, 30.29, 30.29, 30.29, 30.29, 30.39, 30.39, 30.39, 30.39, 30.39, 30.52, 30.52, 30.52, 30.52, 30.52, 30.7, 30.7, 30.7, 30.7, 30.7, 30.72, 30.72, 30.72, 30.72, 30.72, 30.92, 30.92, 30.92, 30.92, 30.92, 30.89, 30.89, 30.89, 30.89, 30.89, 30.65, 30.65, 30.65, 30.65, 30.65, 30.15, 30.15, 30.15, 30.15, 30.15, 29.05, 29.05, 29.05, 29.05, 29.05, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 28.95, 28.95, 28.95, 28.95, 28.95, 28.91, 28.91, 28.91, 28.91, 28.91, 28.9, 28.9, 28.9, 28.9, 28.9, 29.03, 29.03, 29.03, 29.03, 29.03, 29.04, 29.04, 29.04, 29.04, 29.04, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.94, 28.94, 28.94, 28.94, 28.94, 29.09, 29.09, 29.09, 29.09, 29.09, 29.23, 29.23, 29.23, 29.23, 29.23, 29.25, 29.25, 29.25, 29.25, 29.25, 29.36]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.11, 0.11, 0.11, 0.11, 0.11, 0.39, 0.39, 0.39, 0.39, 0.39, 0.09, 0.09, 0.09, 0.09, 0.09, 0.13, 0.13, 0.13, 0.13, 0.13, 0.24, 0.24, 0.24, 0.24, 0.24, 0.22, 0.22, 0.22, 0.22, 0.22, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.32, 0.32, 0.32, 0.32, 0.32, 0.31, 0.31, 0.31, 0.31, 0.31, 0.25, 0.25, 0.25, 0.25, 0.25, 0.37, 0.37, 0.37, 0.37, 0.37, 0.24, 0.24, 0.24, 0.24, 0.24, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.27, 0.27, 0.27, 0.27, 0.27, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.24, 0.24, 0.24, 0.24, 0.24, 0.21, 0.21, 0.21, 0.21, 0.21, 0.29, 0.29, 0.29, 0.29, 0.29, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.34, 0.34, 0.34, 0.34, 0.34, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.17, 0.17, 0.17, 0.17, 0.17, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.29, 0.29, 0.29, 0.29, 0.29, 0.19, 0.19, 0.19, 0.19, 0.19, 0.24, 0.24, 0.24, 0.24, 0.24, 0.23, 0.23, 0.23, 0.23, 0.23, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.31, 0.31, 0.31, 0.31, 0.31, 0.52, 0.52, 0.52, 0.52, 0.52, 0.53, 0.53, 0.53, 0.53, 0.53, 0.51, 0.51, 0.51, 0.51, 0.51, 0.23, 0.23, 0.23, 0.23, 0.23, 0.16, 0.16, 0.16, 0.16, 0.16, 0.24, 0.24, 0.24, 0.24, 0.24, 0.25, 0.25, 0.25, 0.25, 0.25, 0.12, 0.12, 0.12, 0.12, 0.12, 0.19, 0.19, 0.19, 0.19, 0.19, 0.12, 0.12, 0.12, 0.12, 0.12, 0.36, 0.36, 0.36, 0.36, 0.36, 0.19, 0.19, 0.19, 0.19, 0.19, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0]

server : return error on too large embedding input

8513724

ggerganov mentioned this pull request May 19, 2024

/embeddings endpoint sometimes does not return embedding #7277

Closed

github-actions Bot added examples server labels May 19, 2024

JohannesGaessler approved these changes May 19, 2024

View reviewed changes

mofosyne added refactoring Refactoring Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix labels May 20, 2024

ggerganov merged commit e932094 into master May 20, 2024

ggerganov deleted the gg/server-embd-err branch May 20, 2024 05:56

wsxiaoys mentioned this pull request May 20, 2024

server /embedding api doesn't handle cases when physical batch size < prompt length. #7422

Closed

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026

server : return error on too large embedding input (ggml-org#7389)

e82d4b8

phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026

server : return error on too large embedding input (ggml-org#7389)

9cc3a7c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server : return error on too large embedding input#7389

server : return error on too large embedding input#7389
ggerganov merged 1 commit intomasterfrom
gg/server-embd-err

ggerganov commented May 19, 2024

Uh oh!

JohannesGaessler left a comment

Uh oh!

github-actions Bot commented May 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ggerganov commented May 19, 2024

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants