Ignore tokens if their IDs in added token list are below the vocab's base size#3585

seungduk-yanolja wants to merge 5 commits intoggml-org:masterggml-org/llama.cpp:masterfrom

seungduk-yanolja:masterseungduk-yanolja/llama.cpp:masterCopy head branch name to clipboard