Discrepancies in FWT and BWT Metric Formulas in Paper and Code

[ZhixinHan]

In your paper, you mention the evaluation metric Forward Transfer (FWT). However, the formula provided in your paper appears to differ from the one in the original paper you cited (Lopez-Paz and Ranzato, 2017). Specifically, your formula is:

Whereas the original paper presents the formula as:

It seems that these two formulas are not equivalent. Could you provide clarification on this discrepancy and explain why your formula differs from the original?

References: David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30.

Additionally, I noticed another discrepancy regarding the Backward Transfer (BWT) metric in your paper. The formula given in the paper is:

In this formula, the denominator is T-1. However, in your code (score.py), BWT is computed as follows:

Bwt=sum([scores_array[-1][i] - scores_array[i][i] for i in range(task_num)])/task_num

Here, the denominator is T. Could you please explain this difference between the formula in the paper and the implementation in the code?

I greatly appreciate your paper. Your work has been incredibly valuable to my research, and I look forward to your clarification. Thanks in advance for your help!

[circle-hit]

In your paper, you mention the evaluation metric Forward Transfer (FWT). However, the formula provided in your paper appears to differ from the one in the original paper you cited (Lopez-Paz and Ranzato, 2017). Specifically, your formula is: Whereas the original paper presents the formula as: It seems that these two formulas are not equivalent. Could you provide clarification on this discrepancy and explain why your formula differs from the original?

References: David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30.

Additionally, I noticed another discrepancy regarding the Backward Transfer (BWT) metric in your paper. The formula given in the paper is: In this formula, the denominator is T-1. However, in your code (score.py), BWT is computed as follows:

Bwt=sum([scores_array[-1][i] - scores_array[i][i] for i in range(task_num)])/task_num

Here, the denominator is T. Could you please explain this difference between the formula in the paper and the implementation in the code?

I greatly appreciate your paper. Your work has been incredibly valuable to my research, and I look forward to your clarification. Thanks in advance for your help!

Thank you for your attention.

For FWT, in fact, this formula is directly referred from [1]. And in the last page of [1], the authors explain why this formula is different from that in [2]. We think this may be a typo in our work and we feel so sorry for this confusion.

[1] Ke Z, Liu B. Continual learning of natural language processing tasks: A survey[J]. arXiv preprint arXiv:2211.12701, 2022.
[2] Lopez-Paz D, Ranzato M A. Gradient episodic memory for continual learning[J]. Advances in neural information processing systems, 2017, 30.

For BWT, if i = T, the term (scores_array[-1][T] - scores_array[T][T] equals to 0), which is computationally equivalent when the denominator is T-1. We appreciate your suggestion to change the denominator is T-1, but actually this will not affect the results.

[ZhixinHan]

Thank you for your clarification! Your explanation regarding FWT has resolved my doubts very well!

However, as for BWT, as you mentioned, For BWT, if i = T, the term (scores_array[-1][T] - scores_array[T][T] equals to 0), the numerator in the formula Bwt=sum([scores_array[-1][i] - scores_array[i][i] for i in range(task_num)])/task_num remains the same. But for the denominator, using task_num or task_num-1 yields different results. For example:

If T = 3:
According to the formula in the paper: BWT=((a31-a11)+(a32-a22))/2;
According to the code: BWT=((a31-a11)+(a32-a22)+(a33-a33))/3

Nevertheless, whether the denominator is task_num or task_num-1 does not affect the correctness of your conclusion.

Clearly, your code takes into account the scenario when T = 1. What I would like to know is whether the implementation of this metric in your code was optimized using an approximation approach. Did you reference code from earlier methods when implementing this metric? Have other similar papers also adopted the same approach in their code when calculating BWT?

Your response is very important to me as it will help me fairly compare the experimental results from your paper with those from other studies.

Thank you again for your help!

[circle-hit]

Thank you for your clarification! Your explanation regarding FWT has resolved my doubts very well!

However, as for BWT, as you mentioned, For BWT, if i = T, the term (scores_array[-1][T] - scores_array[T][T] equals to 0), the numerator in the formula Bwt=sum([scores_array[-1][i] - scores_array[i][i] for i in range(task_num)])/task_num remains the same. But for the denominator, using task_num or task_num-1 yields different results. For example:

If T = 3:
According to the formula in the paper: BWT=((a31-a11)+(a32-a22))/2;
According to the code: BWT=((a31-a11)+(a32-a22)+(a33-a33))/3

Nevertheless, whether the denominator is task_num or task_num-1 does not affect the correctness of your conclusion.

Clearly, your code takes into account the scenario when T = 1. What I would like to know is whether the implementation of this metric in your code was optimized using an approximation approach. Did you reference code from earlier methods when implementing this metric? Have other similar papers also adopted the same approach in their code when calculating BWT?

Your response is very important to me as it will help me fairly compare the experimental results from your paper with those from other studies.

Thank you again for your help!

In fact, we did not refer code from earlier methods for the implementation of BWT because few works reported this metric in their paper. As you mentioned above, I agree with you that it is okay to fairly compare the experimental results of different methods under the same settings.

Thanks again for your attention and discussion.

[ZhixinHan]

Thank you for your response.