Improve management of asmtransformers model's handling of tokenizer

[akaIDIOT]

Runtime shows warnings around the way the tokenizer and pre-processor code is hooked. There's room for improvement there, can we do this in a way that won't generate warnings and / or gets managed in the way it's meant to be done?