Processor's attribute is mis-detected for child class from #41633

[MilkClouds]

System Info

transformers: v5.0.0.rc1

Who can help?

@yonigozlan @Cyrilvallez

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

From #41633, transformers removed manually defined attributes and replaced this into auto-detected from argument of __init__ using inspect.

transformers/src/transformers/processing_utils.py

Lines 1405 to 1409 in 05c0e1d

	@classmethod
	def get_attributes(cls):
	args_in_init = inspect.signature(cls.__init__).parameters.keys()
	attributes = []
	for sub_processor_type in args_in_init:

But it does not deal with the case for inherited class, e.g: there's no name present on signature of class B's init.

In [5]: import inspect

In [6]: class A:
   ...:     def __init__(self, *args, name: str, **kwargs):
   ...:         self.name = name
   ...:

In [7]: class B(A):
   ...:     def __init__(self, *args, name2: str, **kwargs):
   ...:         self.name2=name2
   ...:         super().__init__(*args, **kwargs)

In [12]: inspect.signature(B.__init__).parameters.keys()
Out[12]: odict_keys(['self', 'args', 'name2', 'kwargs'])

Although above is toy example, I'm using inherited class as Processor in my own research code. Following is tiny snippet from my own code. My own code broken and does not save attributes properly when save_pretrained is called after #41633.

class ActionTokenizerMixin:
    def __init__(self, *args, action_tokenizer: Optional[BaseActionTokenizer] = None, **kwargs):
        super().__init__(*args, **kwargs)
        self._action_tokenizer = action_tokenizer
        self._is_action_tokenizer_prepare_called = False

class AutoProcessorWithActionMixin(ActionTokenizerMixin, AutoProcessor):
    pass

Expected behavior

See above.

[MilkClouds]

And IMO, I believe goal of "auto-detecting everything" is neither realistic nor useful. Manual specification has it's own value, especially for Processor which is not strictly defined and has wide variety. Also detecting attributes from __init__ signature seems not that reliable, possibly leading to many loopholes.

[zucchini-nlp]

cc @yonigozlan , maybe we can allow to set explicit attributes as before and fallback to auto-detect if not set. It will give freedom to overwrite and define their own preprocessing classes for users

[yonigozlan]

Thanks for raising the issues @MilkClouds . Agreed with @zucchini-nlp that we can first check if sub_processor_class attributes are manually defines before trying to auto-detect everything. I'll open a PR for that today.