Preprocessing - categorical_data_preprocessor

[JanBenisek]

• We need to add option model_type which can be [classification, regression] to the class:

  cobra/cobra/preprocessing/categorical_data_processor.py

  Line 61 in 6ef6723

  def __init__(self, regroup: bool=True, regroup_name: str="Other",

• Think about implementation. I'd say keep this as it is, but add a paremeter model_type.

  cobra/cobra/preprocessing/categorical_data_processor.py

  Line 216 in 6ef6723

  pval = (CategoricalDataProcessor

• We need to add statistical test to regroup categories given continuous target. I'd choose non-parametric version of t-test mann whitney or kruskal walis. But I am open to suggestions. There'd be two methods and _compute_p_value would choose one or the other based on input parameter.

  cobra/cobra/preprocessing/categorical_data_processor.py

  Line 399 in 6ef6723

  def _compute_p_value(X: pd.Series, y: pd.Series, category: str,

• Document and write/modify unit tests

[MatthiasRoels]

As an alternative, you can use a more pragmatic approach and do the regrouping based on group size (regroup below a threshold size). You should benchmark this first to see if you don't loose a lot of model performance (I guess that is not the case). This has the added value that the number of categories will typically smaller than is currently the case, which is good for interpretability too.

[JanBenisek]

Agree, but I think the priority is to add functionality for linear regression, without changing current behavior.