ATS failure when using the URI signing plugin

[Scumtron]

I have configured the URI Signing Plugin:
If JWT token is invalid - returns error 403
If JWT token is valid - returns empty response (ATS crash and restart)

Tested on ATS versions: 10.0.0, 9.1.2, 8.1.4

remap.config

map https://cache.tld/ http://src.tld/ @plugin=uri_signing.so @pparam=uri_signing.config

uri_signing.config

{ "CDN": { "iss": "CDN", "cdnistt": true, "cdniets": 30, "renewal_kid": "1", "keys": [ { "alg": "HS256", "kid": "0", "kty": "oct", "k": "1ee531596e8d" }, { "alg": "HS256", "kid": "1", "kty": "oct", "k": "1ee531596e9B" } ] } }

Crash log

May 29 15:56:55 edge traffic_server[14904]: NOTE: --- traffic_server Starting ---
May 29 15:56:55 edge traffic_server[14904]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 9.1.2 - (build # 0.cb7eda60d.el8 on May 28 2022 at 14:28:00)
May 29 15:56:55 edge traffic_server[14904]: NOTE: RLIMIT_NOFILE(7):cur(943718),max(943718)
May 29 15:56:55 edge traffic_manager[14904]: traffic_server: received signal 11 (Segmentation fault)
May 29 15:56:55 edge traffic_manager[14904]: traffic_server - STACK TRACE:
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x92)[0x4d8d82]
May 29 15:56:55 edge traffic_manager[14904]: /lib64/libpthread.so.0(+0x12ce0)[0x7f397c696ce0]
May 29 15:56:55 edge traffic_manager[14904]: /lib64/libc.so.6(+0xccc95)[0x7f397b858c95]
May 29 15:56:55 edge traffic_manager[14904]: /lib64/libc.so.6(hsearch_r+0x2b)[0x7f397b8b72fb]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/var/trafficserver/ba8b022b-38a5-4bc4-893a-693d6abb02f2/opt/trafficserver/libexec/trafficserver/uri_signing.so(+0x4b00)[0x7f397800cb00]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/var/trafficserver/ba8b022b-38a5-4bc4-893a-693d6abb02f2/opt/trafficserver/libexec/trafficserver/uri_signing.so(+0x7b6f)[0x7f397800fb6f]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/var/trafficserver/ba8b022b-38a5-4bc4-893a-693d6abb02f2/opt/trafficserver/libexec/trafficserver/uri_signing.so(TSRemapDoRemap+0x718)[0x7f397800c558]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN15RemapPluginInfo7doRemapEPvP13tsapi_httptxnP22_tm_remap_request_info+0x7d)[0x5d4e9d]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN12RemapPlugins10run_pluginEP15RemapPluginInst+0x75)[0x5ebbc5]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN12RemapPlugins16run_single_remapEv+0x53)[0x5ebc83]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN14RemapProcessor13perform_remapEP12ContinuationPN12HttpTransact5StateE+0x138)[0x5ddac8]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM16do_remap_requestEb+0x121)[0x5518d1]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x62a)[0x5712ca]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x43)[0x557a43]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x92)[0x56b332]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x74d)[0x56558d]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x173)[0x570e13]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x43)[0x557a43]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x92)[0x56b332]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x74d)[0x56558d]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x16b)[0x568fbb]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(TSHttpTxnReenable+0x191)[0x500da1]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/libexec/trafficserver/astats_over_http.so(+0x351c)[0x7f397d95451c]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x93)[0x4e9603]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZNK7APIHook6invokeEiPv+0xf1)[0x4ea631]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x1dc)[0x56501c]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x173)[0x570e13]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x43)[0x557a43]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x460)[0x55d5b0]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x100)[0x567de0]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xf5)[0x56b395]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x74d)[0x56558d]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_add_to_listEiPv+0x248)[0x568328]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP16ProxyTransactionP14IOBufferReader+0x5d6)[0x56a696]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN16ProxyTransaction15new_transactionEb+0x1aa)[0x72041a]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN18Http1ClientSession7releaseEP16ProxyTransaction+0x188)[0x5372a8]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN12ProxySession17handle_api_returnEi+0x8e)[0x71ed8e]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN12ProxySession14do_api_calloutE12TSHttpHookID+0x130)[0x71f580]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN18Http1ClientSession14new_connectionEP14NetVConnectionP9MIOBufferP14IOBufferReader+0x3e1)[0x537821]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN17HttpSessionAccept6acceptEP14NetVConnectionP9MIOBufferP14IOBufferReader+0x295)[0x530015]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN23ProtocolProbeTrampoline17ioCompletionEventEiPv+0x701)[0x71d7a1]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server[0x773f73]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server[0x779bd9]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x109)[0x767cb9]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x235)[0x768115]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN7EThread15execute_regularEv+0x8a9)[0x7b95f9]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x19e)[0x7b9cfe]
May 29 15:56:55 edge traffic_manager[14904]: /opt/trafficserver/bin/traffic_server[0x7b7992]
May 29 15:56:55 edge traffic_manager[14904]: /lib64/libpthread.so.0(+0x81cf)[0x7f397c68c1cf]
May 29 15:56:55 edge traffic_manager[14904]: /lib64/libc.so.6(clone+0x43)[0x7f397b7c5d83]
May 29 15:56:55 edge systemd[1]: Started Process Core Dump (PID 14924/UID 0).
May 29 15:56:55 edge systemd-coredump[14925]: Process 14904 ([TS_MAIN]) of user 176 dumped core.#12#012Stack trace of thread 14906:#12#0 0x00007f397b858c95 __strlen_avx2 (libc.so.6)#12#1 0x00007f397b8b72fb hsearch_r (libc.so.6)#12#2 0x00007f397800cb00 n/a (/opt/trafficserver/var/trafficserver/ba8b022b-38a5-4bc4-893a-693d6abb02f2/opt/trafficserver/libexec/trafficserver/uri_signing.so (deleted))
May 29 15:56:56 edge systemd[1]: systemd-coredump@16-14924-0.service: Succeeded.

[brbzull0]

@Scumtron can you enable debug and add the following tag to it: uri_signing . For the records, unit test on this plugin is passing locally in my dev box(our ci seems to have this disabled).
The crash seems to come from the validate_jws I guess, would help to see how far and the values he got. Also the crash logger should be dumping some information that may also help, share them if you have it.
Regards.

PS: Tried this on master.

[Scumtron]

@brbzull0, hello!

Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.526] [ET_NET 10] DIAG: (uri_signing) [uri_signing.c: 190] TSRemapDoRemap(): Processing request for https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4.
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.526] [ET_NET 10] DIAG: (uri_signing) [parse.c: 56] get_jws_from_uri(): Parsing JWS from query string: https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.526] [ET_NET 10] DIAG: (uri_signing) [parse.c: 124] get_jws_from_uri(): Unable to locate signing key in uri: https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.526] [ET_NET 10] DIAG: (uri_signing) [uri_signing.c: 358] TSRemapDoRemap(): Invalid JWT for https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.526] [ET_NET 10] DIAG: (uri_signing) [uri_signing.c: 360] TSRemapDoRemap(): Spent 74085 ns uri_signing verification of https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4.
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [uri_signing.c: 190] TSRemapDoRemap(): Processing request for https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4?URISigningPackage=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NTQyNDA3NDR9.7TuRb5EQFIQXxZuoHjKGG9AgFZutLjvO-6RyAouovQE.
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 56] get_jws_from_uri(): Parsing JWS from query string: https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4?URISigningPackage=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NTQyNDA3NDR9.7TuRb5EQFIQXxZuoHjKGG9AgFZutLjvO-6RyAouovQE
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 90] get_jws_from_uri(): Decoding JWS: URISigningPackage
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 96] get_jws_from_uri(): Parsed JWS: URISigningPackage ( 0x7fe49005bf30)
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 119] get_jws_from_uri(): Stripped URI: https://cache-75a482ce.u6.eu.replaced.net/data/0/293/293.mp4
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 162] validate_jws(): Validating JWS for 0x7fe49005bf30
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 171] validate_jws(): Spent 6007 ns getting jws plaintext
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 175] validate_jws(): Spent 9087 ns parsing jwt
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 189] validate_jws(): Spent 5908 ns initial validation of jwt
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 192] validate_jws(): Spent 5107 ns getting header of jws
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 199] validate_jws(): Spent 5462 ns getting kid of jws header
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: [Jun 3 10:13:04.674] [ET_NET 0] DIAG: (uri_signing) [parse.c: 213] validate_jws(): Searching all keys for issuer (null) for 0x7fe49005bf30
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: traffic_server: received signal 11 (Segmentation fault)
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: traffic_server - STACK TRACE:
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x92)[0x4a83b2]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /lib64/libpthread.so.0(+0x12ce0)[0x7fe49e0dece0]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /lib64/libc.so.6(+0xccc95)[0x7fe49d2a0c95]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /lib64/libc.so.6(hsearch_r+0x2b)[0x7fe49d2ff2fb]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/libexec/trafficserver/uri_signing.so(+0x4bb0)[0x7fe49c00fbb0]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/libexec/trafficserver/uri_signing.so(+0x7adf)[0x7fe49c012adf]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/libexec/trafficserver/uri_signing.so(TSRemapDoRemap+0x8da)[0x7fe49c00f71a]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN12RemapPlugins10run_pluginEP17remap_plugin_info+0x8d)[0x57ac6d]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN12RemapPlugins16run_single_remapEv+0x50)[0x57ad40]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN14RemapProcessor13perform_remapEP12ContinuationPN12HttpTransact5StateE+0x140)[0x574bf0]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM16do_remap_requestEb+0x116)[0x500986]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x827)[0x51f457]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x40)[0x508d20]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x92)[0x514182]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x39f)[0x51725f]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x2fb)[0x51ef2b]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x40)[0x508d20]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x92)[0x514182]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x39f)[0x51725f]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x16b)[0x519d8b]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(TSHttpTxnReenable+0x216)[0x4cb116]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/libexec/trafficserver/remap_stats.so(+0x1954)[0x7fe49f846954]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x73)[0x4b3303]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x475)[0x517335]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x16b)[0x519d8b]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(TSHttpTxnReenable+0x216)[0x4cb116]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/libexec/trafficserver/astats_over_http.so(+0x2afb)[0x7fe47e4c9afb]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x73)[0x4b3303]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x475)[0x517335]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x2fb)[0x51ef2b]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x40)[0x508d20]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x86e)[0x50b46e]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xf8)[0x519ae8]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x12a)[0x51421a]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x39f)[0x51725f]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM17state_add_to_listEiPv+0x250)[0x51e090]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP22ProxyClientTransactionP14IOBufferReader+0x5c4)[0x51e734]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN22ProxyClientTransaction15new_transactionEv+0x113)[0x6b56f3]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN18Http1ClientSession16state_keep_aliveEiPv+0x1a0)[0x4f27c0]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN18UnixNetVConnection19readSignalAndUpdateEi+0x47)[0x712e77]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x1b6d)[0x6ddaad]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN10NetHandler18process_ready_listEv+0x106)[0x6fefe6]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x285)[0x6ff435]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server(_ZN7EThread15execute_regularEv+0x72e)[0x7442ce]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /opt/trafficserver/bin/traffic_server[0x7427da]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /lib64/libpthread.so.0(+0x81cf)[0x7fe49e0d41cf]
Jun 3 10:13:04 cache-75a482ce traffic_manager[20483]: /lib64/libc.so.6(clone+0x43)[0x7fe49d20dd83]
Jun 3 10:13:04 cache-75a482ce systemd[1]: Started Process Core Dump (PID 20523/UID 0).
Jun 3 10:13:04 cache-75a482ce systemd-coredump[20524]: Removed old coredump core.[TS_MAIN].176.106f9a47877f41dd8c60fea550f681fd.18847.1654238594000000.lz4.
Jun 3 10:13:07 cache-75a482ce systemd-coredump[20524]: Process 20483 ([TS_MAIN]) of user 176 dumped core.#12#012Stack trace of thread 20485:#12#0 0x00007fe49d2a0c95 __strlen_avx2 (libc.so.6)#12#1 0x00007fe49d2ff2fb hsearch_r (libc.so.6)#12#2 0x00007fe49c00fbb0 find_keys (uri_signing.so)#12#3 0x00007fe49c012adf validate_jws (uri_signing.so)#12#4 0x00007fe49c00f71a TSRemapDoRemap (uri_signing.so)#12#5 0x000000000057ac6d _ZN12RemapPlugins10run_pluginEP17remap_plugin_info (traffic_server)#12#6 0x000000000057ad40 _ZN12RemapPlugins16run_single_remapEv (traffic_server)#12#7 0x0000000000574bf0 _ZN14RemapProcessor13perform_remapEP12ContinuationPN12HttpTransact5StateE (traffic_server)#12#8 0x0000000000500986 _ZN6HttpSM16do_remap_requestEb (traffic_server)#12#9 0x000000000051f457 _ZN6HttpSM14set_next_stateEv (traffic_server)#12#10 0x0000000000508d20 _ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE (traffic_server)#12#11 0x0000000000514182 _ZN6HttpSM17handle_api_returnEv (traffic_server)#12#12 0x000000000051725f _ZN6HttpSM17state_api_calloutEiPv (traffic_server)#12#13 0x000000000051ef2b _ZN6HttpSM14set_next_stateEv (traffic_server)#12#14 0x0000000000508d20 _ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE (traffic_server)#12#15 0x0000000000514182 _ZN6HttpSM17handle_api_returnEv (traffic_server)#12#16 0x000000000051725f _ZN6HttpSM17state_api_calloutEiPv (traffic_server)#12#17 0x0000000000519d8b _ZN6HttpSM18state_api_callbackEiPv (traffic_server)#12#18 0x00000000004cb116 TSHttpTxnReenable (traffic_server)#12#19 0x00007fe49f846954 handle_read_req_hdr (/opt/trafficserver/libexec/trafficserver/remap_stats.so)#12#20 0x00000000004b3303 _ZN15INKContInternal12handle_eventEiPv (traffic_server)#12#21 0x0000000000517335 _ZN6HttpSM17state_api_calloutEiPv (traffic_server)#12#22 0x0000000000519d8b _ZN6HttpSM18state_api_callbackEiPv (traffic_server)#12#23 0x00000000004cb116 TSHttpTxnReenable (traffic_server)#12#24 0x00007fe47e4c9afb astats_origin (astats_over_http.so)#12#25 0x00000000004b3303 _ZN15INKContInternal12handle_eventEiPv (traffic_server)#12#26 0x0000000000517335 _ZN6HttpSM17state_api_calloutEiPv (traffic_server)#12#27 0x000000000051ef2b _ZN6HttpSM14set_next_stateEv (traffic_server)#12#28 0x0000000000508d20 _ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE (traffic_server)#12#29 0x000000000050b46e _ZN6HttpSM32state_read_client_request_headerEiPv (traffic_server)#12#30 0x0000000000519ae8 _ZN6HttpSM12main_handlerEiPv (traffic_server)#12#31 0x000000000051421a _ZN6HttpSM17handle_api_returnEv (traffic_server)#12#32 0x000000000051725f _ZN6HttpSM17state_api_calloutEiPv (traffic_server)#12#33 0x000000000051e090 _ZN6HttpSM17state_add_to_listEiPv (traffic_server)#12#34 0x000000000051e734 _ZN6HttpSM21attach_client_sessionEP22ProxyClientTransactionP14IOBufferReader (traffic_server)#12#35 0x00000000006b56f3 _ZN22ProxyClientTransaction15new_transactionEv (traffic_server)#12#36 0x00000000004f27c0 _ZN18Http1ClientSession16state_keep_aliveEiPv (traffic_server)#12#37 0x0000000000712e77 _ZN18UnixNetVConnection19readSignalAndUpdateEi (traffic_server)#12#38 0x00000000006ddaad _ZN17SSLNetVConnection11net_read_ioEP10NetHandlerP7EThread (traffic_server)#12#39 0x00000000006fefe6 _ZN10NetHandler18process_ready_listEv (traffic_server)#12#40 0x00000000006ff435 _ZN10NetHandler15waitForActivityEl (traffic_server)#12#41 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#42 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#43 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#44 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20492:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20497:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00007fe49f8c15e1 _ZN11EventNotify4waitEv (/opt/trafficserver/lib/libtscore.so.8.1.4)#12#2 0x000000000059e15b _ZN3Log19preproc_thread_mainEPv (traffic_server)#12#3 0x00000000005a342d _ZN26LoggingPreprocContinuation9mainEventEiPv (traffic_server)#12#4 0x0000000000744402 _ZN7EThread7executeEv (traffic_server)#12#5 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#6 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#7 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20489:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20490:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20496:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20488:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20486:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20487:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20498:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00007fe49f8c15e1 _ZN11EventNotify4waitEv (/opt/trafficserver/lib/libtscore.so.8.1.4)#12#2 0x000000000059e5f1 _ZN3Log17flush_thread_mainEPv (traffic_server)#12#3 0x00000000005a344d _ZN24LoggingFlushContinuation9mainEventEiPv (traffic_server)#12#4 0x0000000000744402 _ZN7EThread7executeEv (traffic_server)#12#5 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#6 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#7 0x00007fe49d20dd83 __clone (libc.so.6)#12#012Stack trace of thread 20491:#12#0 0x00007fe49d303a27 epoll_wait (libc.so.6)#12#1 0x00000000006ff27a _ZN10NetHandler15waitForActivityEl (traffic_server)#12#2 0x00000000007442ce _ZN7EThread15execute_regularEv (traffic_server)#12#3 0x00000000007427da _ZL21spawn_thread_internalPv (traffic_server)#12#4 0x00007fe49e0d41cf start_thread (libpthread.so.0)#12#5 0x00007fe49d20dd83 __clone (libc
Jun 3 10:13:07 cache-75a482ce systemd[1]: systemd-coredump@55-20523-0.service: Succeeded.

[brbzull0]

@Scumtron thanks for the details.

It seems that the payload in your token is not right:

{
   "header":{
      "typ":"JWT",
      "alg":"HS256"
   },
   "payload":{
      "exp":1654240744
   }
}

If you check the log you'll notice:

Searching all keys for issuer (null) for 0x7fe49005bf30

Issuer is null(and not present in the payload up above), iss key should hold the issuer in this case.
As per the plugin's documentation

iss: Must be present. The issuer is used to locate the key for verification.

Please have a look at the token and how it's being generated.

Regards,
Damian.

[bryancall]

@dsouza93 is going to take a look at this issue.

[brbzull0]

I did spend some time on this last week. As noted up above, found out that the iss is not being validated(it should) as part of the payload which leads an invalid use of this pointer further down the line(so the crash). @dsouza93 as I've mentioned in the chat I have a small pr to fix this iss validation. Let me know if you also have a pr for this or want me to push this up and you can take it from there.
Thanks.

[brbzull0]

@Scumtron have you try this fix?

[Scumtron]

@brbzull0 Hello. No, i just added the iss to the payload.

[brbzull0]

ok then, the crash is gone. Can we close this issue or you still need some help around this? @Scumtron
Thanks.