Add support to compute metrics on list of channel-first data

[Nic-Ma]

Is your feature request related to a problem? Please describe.
After inverting transforms, it generates a list of channel-first data, we need to add support to compute metrics for them.
Now the metrics handlers can only support a batch data.

[Nic-Ma]

Hi @wyli and @rijobro ,

I will take look at this ticket ASAP to implement for v0.5 release.

Thanks.

[Nic-Ma]

Hi @vfdev-5 ,

May I ask a question: do ignite metrics support a list of [channel, H, W, [D]] shape data? As a batch of data may have different shape after inverting transforms, we flatten the batch into a list of tensor then need to compute metrics in this case.

My proposal is to enhance this line to support a list of data and add batch dim for the list item:
https://github.com/Project-MONAI/MONAI/blob/master/monai/handlers/iteration_metric.py#L77

Thanks.

[vfdev-5]

Hi @Nic-Ma ,

If I understand correctly, y_preds = [p1, p2, ..., pN] with pK being a float tensors but all not of the same shape such that we can't use tensor.stack(y_preds, dim=0) to make a single tensor of shape (N, C, H, W, ...).

Currently, ignite metrics work on tensor input only and assumes the batch dimension to be the first one.

My proposal is to enhance this line to support a list of data and add batch dim for the list item:

Yes, I think this could be doable:

        y_pred, y = output
        if isinstance(y_pred, Sequence) and isinstance(y, Sequence):
            score_list = []
            for y_pred_, y_ in zip(y_pred, y):
                score_list.append(self.metric_fn(y_pred_[None, ...], y_[None, ...]))
            score = torch.tensor(score_list).item()
        else:
            score = self.metric_fn(y_pred, y)

I think this is something could be also done from ignite's side as well... Let me open an issue for that in ignite and see what others are thinking about.

Thanks

[Nic-Ma]

Hi @vfdev-5 ,

Thanks for your double-confirm and suggestions.
This could be very important because we may need to restore prediction and label shape to align with the original input label shape, which may be different for every item in a batch, so can't stack to a batch anymore.

Another question is that: currently, MONAI post transforms execute on the engine.state.output dict, the post transforms can't get some useful information in engine.state.batch, for example, the meta_dict of images or some transform context we put in the engine.state.batch. As we discussed in another ignite ticket before, I think we need an end-to-end dict, which is something like "combine engine.state.batch and engine.state.output", maybe it's better to use only 1 dict: engine.state.data instead, what do you think?
CC @wyli and @rijobro

Thanks.

[vfdev-5]

Another question is that: currently, MONAI post transforms execute on the engine.state.output dict, the post transforms can't get some useful information in engine.state.batch, for example, the meta_dict of images or some transform context we put in the engine.state.batch. As we discussed in another ignite ticket before, I think we need an end-to-end dict, which is something like "combine engine.state.batch and engine.state.output", maybe it's better to use only 1 dict: engine.state.data instead, what do you think?

I think both solutions are possible. Personally, i would include necessary info from state.batch to state.output as for example target (or label) can be propagated from batch to output:

def training_step(engine, batch):
    x, y = batch["x"], batch["y"]
    # ...

    return {
        "y_pred": y_pred,
        "y": y,
        # ...
        "meta_data": batch["meta_data"],
    }

Maybe, a small remark on data lifespan for state.data which by default is not bounded. We are also currently discussing about reducing the lifespan of state.batch and state.output between GET_BATCH_COMPLETED and ITERATION_COMPLETED (see pytorch/ignite#1919).

[Nic-Ma]

Hi @vfdev-5 ,

Sounds interesting, maybe we should copy engine.state.batch to engine.state.output as:

engine.state.output = copy(engine.state.batch)
engine.state.output.update({Keys.IMAGE: inputs, Keys.LABEL: targets})

Or directly use the same dict from batch to output:

engine.state.output = engine.state.batch

At this line:
https://github.com/Project-MONAI/MONAI/blob/master/monai/engines/trainer.py#L160

What do you think?

Thanks.

[vfdev-5]

I agree that under certain assumptions we can build an application where state.batch and state.output could point to the same data structure (e.g. a dictionary):

state.output = state.batch = {
    Keys.IMAGE: inputs,
    Keys.LABEL: targets,
    Keys.META_DATA: inputs_meta,
    Keys.PREDICTION: y_preds,
    ...
}

On the other hand, usually, handlers receive engine as argument and thus they can access engine.state and fetch both batch and output...
For example, here is a visualization handler:

https://github.com/pytorch/ignite/blob/13a42b90589112e67bcc6e1780e7c826a5a34199/examples/references/segmentation/pascal_voc2012/vis.py#L100-L129

where we need input images, masks and predictions and thus handler should fetch the data from different parts of the state.

Anyway, using a single structure I think can be a convention that could a bit simplify handlers development from the user's side.

[Nic-Ma]

Hi @vfdev-5 ,

Thanks for your discussion.
Do you guys have any future plan to involve a state.data as end-to-end data flow?
We may consider to enhance this conception if keep receiving feature requests from Clara or MONAI workflow users.

Thanks.

[vfdev-5]

@Nic-Ma sounds good !
I think this could be an interesting addition, we can discuss it in more details here : pytorch/ignite#1923