Is it ok to directly modify `model.cfg.n_ctx`?

[notoookay]

Hi, I have a question about model configuration. I want to use the model with a longer context. As most of the models' default context window is set to 2048, is it okay to change to a larger value below the original model context window, like 4096? Will it affect something like performance?
Btw, I saw a similar issue #491, it may help clarify my question.

[notoookay]

It doesn't work. Will there be an argument to change the n_ctx when loading the pre-trained model?

[tuomaso]

+1 to this. Would be very useful to have an argument in HookedTransformer.from_pretrained() that allows overriding n_ctx from config since config is artificially capping many models context windows lower than they actually are, e.g. Llama-3.2 models have 128k context window according to hf but config caps them at 2048. The only workaround I've found so far that works is to edit the config itself on a local copy of transformer_lens.loading_from_pretrained.py but surely there has to be a better solution.

[jlarson4]

@notoookay HookedTransformer & TransformerBridge now have a n_ctx parameter that will allow you to make this modification.

Note, a warning fires when overriding above the model's documented default, verify memory/model behavior at extended lengths
Direct post-load mutation of cfg.n_ctx is still not safe (won't resize positional embedding buffers) – always use the from_pretrained argument.