Backbone kwargs in config

[amyeroberts]

What does this PR do?

This enables configuring the backbones through the config directly e.g. passing in out_indices to the backbone. This enables configuring a model's backbone when it's loaded from a pretrained checkpoint. At the moment, this is only possible when loading from a backbone_config.

Example:

model = MaskFormer.from_pretrained(
    "facebook/maskformer-swin-base-ade",
    backbone="facebook/maskformer-swin-large-ade",
    backbone_kwargs={"out_indices": (-2, -1)}
)

This is necessary to replace th timm code currently there for models like DETR e.g. here, which is often hard coded.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Could you explain the motivations behind this? IMO would should push users to use a config for the backbone, as we do for the quantization_config for example.
I feel like `backbone = AutoBackbone.from_config(config.backbone_config, arg1 =x, arg2=y) could be used in the potential refactor?

Would help if you have a linked PR that has the expected behaviour to see how this is gonna be called 🤗

[amyeroberts]

@ArthurZucker Sure!

We want to remove the hard-coded conditional logic inside of models like DETR and be able to fully configure the backbone's behaviour. The use case is: image I want to create a new model to train. Instead of the default architecture of DETR, I want to use a different timm backbone, and I want the backbone to return different feature maps from the default.

At the moment this isn't possible because:

• use_timm_backbone behaviour is hard coded, so we can't load different timm backbones easily e.g. here i.e. I can't pass timm specific arguments to configure the backbone e.g. output_stride.
• I can't configure the backbone by passing in out_indices to set the feature maps.

I completely agree that it would be better to have this all as one argument! For context: when the backbones were first added, there were four arguments:

• backbone which specifies a checkpoint e.g. "facebook/detr-resnet-50"
• use_timm_backbone - whether or not to load the backbone from timm.
• use_pretrained_backbone which was ill-defined, but controlled the behaviour JUST for the timm backbones
• backbone_config a model config which defines a backbone e.g. ResNet.

The backbone_config is just a model config, and used to load backbone models from config:

backbone = AutoBackbone.from_config(backbone_config)

and backbone is a checkpoint used to load from pretrained:

backbone = AutoBackbone.from_pretrained(backbone)

So backbone and backbone_config are mutually exclusive. And backbone_config is of type PretrainedConfig so can only configure transformers models.

Adding backbone_kwargs isn't ideal, but I think it's the simplest solution. The alternative is creating a new backbone config which contains everything. Because the old values are used in 100s of configs, we wouldn't be able to immediately deprecate, so it'd be a case of having code which ingested old config arguments.

If you want, I'm happy to code something up and we can decide which is best from the two :)

For more context, here's an example of the final step for removing timm in the modeling files: https://github.com/amyeroberts/transformers/pull/114/files#diff-bdc82cb85f491576a99a341adabaf42260eac9cd797d70d0a2c564b0d4ee2930

You can see how all we need is a single call backbone = load_backbone(config) in a modules init.

[ArthurZucker]

Alright, looks good to me then!