HDDS-11268. Add --table mode for OM/SCM Roles CLI #7016

slfan1989 · 2024-08-01T14:26:28Z

What changes were proposed in this pull request?

When viewing roles of OM/SCM via CLI, a lengthy string is typically displayed, which lacks user-friendly formatting. We aim to improve its presentation. I have added a table option to allow displaying this data in a tabular format.

OM

om-current/bin/ozone admin om roles

om32 : FOLLOWER (bigdata-ozone-online32)
om30 : FOLLOWER (bigdata-ozone-online30)
om31 : LEADER (bigdata-ozone-online31)

om-current/bin/ozone admin om roles --table

+------------------------------------------------------+
|                 Ozone Manager Roles                  |
+---------------------------------+---------+----------+
|            Host Name            | Node ID |   Role   |
+---------------------------------+---------+----------+
| bigdata-ozone-online32          |  om32   | FOLLOWER |
| bigdata-ozone-online30          |  om30   | FOLLOWER |
| bigdata-ozone-online31          |  om31   |  LEADER  |
+---------------------------------+---------+----------+

SCM

scm-current/bin/ozone admin scm roles

bigdata-ozone-online31:9894:FOLLOWER:61b1c8e5-da40-4567-8a17-96a0234ba14e:100.3.197.98
bigdata-ozone-online32:9894:LEADER:e428ca07-b2a3-4756-bf9b-a4abb033c7d1:100.3.192.89
bigdata-ozone-online30:9894:FOLLOWER:41f90734-b3ee-4284-ad96-40a286654952:100.3.196.51

scm-current/bin/ozone admin scm roles --table

+---------------------------------------------------------------------------------------------------------------+
|                                        Storage Container Manager Roles                                        |
+---------------------------------+------------+----------+--------------------------------------+--------------+
|            Host Name            | Ratis Port | Role     |                 Node ID              | Host Address |
+---------------------------------+------------+----------+--------------------------------------+--------------+
| bigdata-ozone-online31          |    9894    | FOLLOWER | 61b1c8e5-da40-4567-8a17-96a0234ba14e | 100.3.197.98 |
| bigdata-ozone-online32          |    9894    |  LEADER  | e428ca07-b2a3-4756-bf9b-a4abb033c7d1 | 100.3.192.89 |
| bigdata-ozone-online30          |    9894    | FOLLOWER | 41f90734-b3ee-4284-ad96-40a286654952 | 100.3.196.51 |
+---------------------------------+------------+----------+--------------------------------------+--------------+

What is the link to the Apache JIRA

JIRA: HDDS-11268. [CLI] Improve CLI Display OM/SCM Roles.

How was this patch tested?

To validate this in a production environment.

errose28 · 2024-08-02T17:00:02Z

Thanks @slfan1989 the table looks nice. In the description the Role and Node ID headers are swapped. I wonder if it would be better to make this format the "verbose" output and trigger it with a --verbose flag, which is more standard across CLI tools. I don't have a strong opinion on this though.

slfan1989 · 2024-08-04T14:28:05Z

Thanks @slfan1989 the table looks nice. In the description the Role and Node ID headers are swapped. I wonder if it would be better to make this format the "verbose" output and trigger it with a --verbose flag, which is more standard across CLI tools. I don't have a strong opinion on this though.

Thank you for your attention to this pr! I have tried to output verbose information using --verbose, but encountered some issues. ozone admin om does not yet support recognizing the verbose option.

Usage: ozone admin om [-hV] [--verbose] [-conf=<configurationPath>]
                      [-D=<String=String>]... [COMMAND]
Ozone Manager specific admin operations
      -conf=<configurationPath>

  -D, --set=<String=String>

  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
      --verbose   More verbose output. Show the stack trace of the errors.

[hadoop@bigdata-ozone-online32 ~/om-current/bin]$ ./ozone admin om --verbose
Incomplete command

After completing this pr, I can submit a new pr to add support for the verbose option in ozone admin om/scm, including support for tabular-style output.

adoroszlai · 2024-08-05T11:30:27Z

Thanks @slfan1989 for the patch.

Please check acceptance test failure:

List om roles as TABLE with OM service ID passed                      | FAIL |
No keyword with name 'Assert Leader Present in TABLE' found. Did you try using keyword 'Assert Leader Present' and forgot to use enough whitespace between keyword and arguments?
--
List om roles as TABLE without OM service ID passed                   | FAIL |
No keyword with name 'Assert Leader Present in TABLE' found. Did you try using keyword 'Assert Leader Present' and forgot to use enough whitespace between keyword and arguments?
--
Om-Roles :: Smoke test for listing om roles.                          | FAIL |
6 tests, 4 passed, 2 failed

https://github.com/apache/ozone/actions/runs/10236952464/job/28319907708?pr=7016#step:6:9

slfan1989 · 2024-08-06T01:57:40Z

@errose28 @adoroszlai Can you help me review this pr again? Thank you very much!

ChenSammi · 2024-08-08T06:04:17Z

hadoop-ozone/dist/src/main/smoketest/omha/om-roles.robot

                                    Should Contain                  ${output_without_id_passed}      no Ozone Manager service ID specified
+
+List om roles as TABLE with OM service ID passed
+    ${output_without_id_passed} =   Execute                         ozone admin om roles --table


@slfan1989 , this should be "output_with_id_passed". BTW, the first command "ozone admin om roles --table" doesn't have service ID set, why it's under this "with OM service ID passed" category?

Thank you for helping review the code. The issue was due to my oversight, and I have updated the unit tests.

@ChenSammi Can you help review this pr again? thank you very much!

whbing

Thanks @slfan1989 for working on this. Overall looks good, left a few comments.

hadoop-ozone/dist/src/main/smoketest/omha/om-roles.robot

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/utils/FormattingCLIUtils.java

whbing · 2024-08-27T06:47:47Z

Seems some related ci failures, such as org.apache.hadoop.ipc.RemoteException(com.google.protobuf.UninitializedMessageException): Message missing required fields: scmHaEnabled.

@slfan1989 From my point, maybe we don't need to make so many changes. How about simplifying the codes similar to the —json ? cc @adoroszlai @ChenSammi

ChenSammi · 2024-08-27T08:52:20Z

@slfan1989 , thanks for improving the patch. For this scmHaEnabled, can you explain a little bit about its purpose?

slfan1989 · 2024-08-28T15:31:42Z

...-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisRolesSubcommand.java

+
+      // Determine which header to use based on whether Ratis is enabled or not.
+      if (isRatisEnabled) {
+        formattingCLIUtils.addHeaders(RATIS_SCM_ROLES_HEADER);


@whbing @ChenSammi

@slfan1989 , thanks for improving the patch. For this scmHaEnabled, can you explain a little bit about its purpose?

Thank you very much for reviewing this PR! I’d like to explain why I added the scmRatisEnable:

When using --table to output SCM information, we need to first determine the length of the table headers.

For non-HA SCMs, we will display it in the following format:

+-----------------------------------------------+ | Storage Container Manager Roles | +---------------------------------+-------------+ | Host Name | Port | +---------------------------------+-------------+

For HA SCMs, we will display it in the following format:

+---------------------------------------------------------------------------------------------------------------+ | Storage Container Manager Roles | +---------------------------------+------------+----------+--------------------------------------+--------------+ | Host Name | Ratis Port | Role | Node ID | Host Address | +---------------------------------+------------+----------+--------------------------------------+--------------+

I need a clear method to determine if SCM has Ratis enabled.

I am considering two approaches:

Approach 1: Use the ratisRole length of the returned data for judgment.
Approach 2: Add a return attribute that explicitly indicates whether Ratis is being used.

From my perspective, Approach 2 is better, as the code looks clearer.

slfan1989 · 2024-08-28T15:34:40Z

Seems some related ci failures, such as org.apache.hadoop.ipc.RemoteException(com.google.protobuf.UninitializedMessageException): Message missing required fields: scmHaEnabled.

@slfan1989 From my point, maybe we don't need to make so many changes. How about simplifying the codes similar to the —json ? cc @adoroszlai @ChenSammi

@whbing Thank you for your question! I believe using a clear method to indicate whether SCM is using Ratis would be better. While a JSON output approach is also feasible, determining whether Ratis is used based on the size of the length might not be very reliable.

ChenSammi · 2024-09-04T06:44:35Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java

+   * while `false` indicates that it is in STANDALONE mode.
+   * @throws IOException  an I/O exception of some sort has occurred.
+   */
+  boolean isSCMRatisEnable() throws IOException;


isSCMRatisEnable -> isScmRatisEnable

Thank you for reviewing the code! I will improve this part of the code.

ChenSammi · 2024-09-04T06:54:44Z

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java

    ScmInfo.Builder builder = new ScmInfo.Builder()
        .setClusterId(resp.getClusterId())
        .setScmId(resp.getScmId())
+        .setScmRatisEnabled(resp.getScmRatisEnabled())


Here to be backward compatible, we should first check whether the response has "scmRatisEnabled" field. If it doesn't have, which means the SCM server is an old version server, then we need to set this ScmRatisEnabled field in other way, for example, the size of RolesList.

Here to be backward compatible, we should first check whether the response has "scmRatisEnabled" field.

Thank you for the reminder! It's indeed very necessary. I will check if ScmRatisEnabled exists.

If it doesn't have, which means the SCM server is an old version server, then we need to set this ScmRatisEnabled field in other way, for example, the size of RolesList.

We use the size of RolesList to determine if Ratis is enabled. If RolesList.size > 1, we consider Ratis enabled. If RolesList.size <= 1, we consider Ratis not enabled ？

There is one case that SCM ratis is ended, but only one SCM instance in the raft group. In this case, we can check whether the role string has LEADER or FOLLOWER key word, if neither key words exist, then ratis is not enabled.

Thank you for your suggestion! I will continue to improve the code.

@ChenSammi Could you review this PR again? Thank you very much!

slfan1989 · 2024-09-12T05:55:34Z

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java

        .setRatisPeerRoles(resp.getPeerRolesList());

-    return builder.build();
+    // By default, we assume that SCM Ratis is not enabled.


We have considered whether the ScmRatisEnabled field exists in the proto. If the field is not present, we determine whether Ratis is enabled based on whether peerRolesList includes leader or follower.

@ChenSammi Kindly ping, Can you review this PR again? Thank you very much!

whbing

Thanks @slfan1989 for update. Overall looks good. Some comments left.

whbing · 2024-09-19T08:13:13Z

hadoop-ozone/dist/src/main/smoketest/omha/om-roles.robot

                                     Should Not Be Equal            ${leader}       ${EMPTY}
+Assert Leader Present in TABLE
+     [Arguments]                     ${output}
+     Should Match Regexp             ${output}                     \|\s+om\d+\s+\|\s+LEADER\s+\|


om host is a user-defined string and does not necessarily start with om, right ?

NIT: string don't look aligned.

I verified this RE, and it seems that any format will pass. So, this RE looks like it needs to be changed.
You can install robot in python and execute it locally:
om-roles-test.robot :

*** Settings *** Documentation Smoke test for listing om roles. Library OperatingSystem Test Timeout 5 minutes *** Keywords *** Assert Leader Present in TABLE [Arguments] ${output} Should Match Regexp ${output} \|\s+om\d+\s+\|\s+LEADER\s+\| *** Test Cases *** List om roles as TABLE without OM service ID passed ${result}= Set Variable | host32 | om32 | OTHER Should Be Equal ${result} | host32 | om32 | OTHER Assert Leader Present in TABLE ${result}

run: robot om-roles-test.robot

============================================================================== Om-Roles-Test :: Smoke test for listing om roles. ============================================================================== List om roles as TABLE without OM service ID passed | PASS | ------------------------------------------------------------------------------ Om-Roles-Test :: Smoke test for listing om roles. | PASS | 1 test, 1 passed, 0 failed ==============================================================================

om host is a user-defined string and does not necessarily start with om, right ?

This regex matches strings containing "om" followed by digits, specifically formatted with "LEADER" surrounded by pipe delimiters.

If we replace it with other characters (like "scm"), it should not match.

NIT: string don't look aligned.

This should be fixed, and I will modify the format issue.

I verified this RE, and it seems that any format will pass. So, this RE looks like it needs to be changed.
You can install robot in python and execute it locally:

Sorry, I don't think there's an issue with this regex.

Test1 regex test

The regex validation indicates that, based on the provided example, it cannot match the regex.

I tested the regex functionality in VSCode, and everything is working fine. I'm not sure if Robot has issues recognizing the regular expression, so I changed it to ensure that it works properly with Robot.

============================================================================== Om-Roles ============================================================================== List om roles as TABLE without OM service ID passed | FAIL | '| host32 | om32 | OTHER' does not match '\|.*LEADER.*' ------------------------------------------------------------------------------ Om-Roles | FAIL | 1 test, 0 passed, 1 failed ==============================================================================

Test Passed.

Assert Leader Present in TABLE [Arguments] ${output} Should Match Regexp ${output} \\|.*LEADER.* *** Test Cases *** List om roles as TABLE without OM service ID passed ${result}= Set Variable | host32 | om32 | LEADER Log ${result} Assert Leader Present in TABLE ${result}

============================================================================== Om-Roles ============================================================================== List om roles as TABLE without OM service ID passed | PASS | ------------------------------------------------------------------------------ Om-Roles | PASS | 1 test, 1 passed, 0 failed ==============================================================================

hadoop-ozone/dist/src/main/smoketest/admincli/scmrole.robot

slfan1989 · 2024-09-23T05:04:08Z

@whbing @ChenSammi Could you please help review this PR again? Thank you very much!

whbing · 2024-09-24T12:03:31Z

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java

+   * This is used to check if 'leader' or 'follower' exists,
+   * in order to confirm whether we have enabled Ratis.
+   */
+  private final List<String> scmRatisRolesToCheck = Arrays.asList("leader", "follower");


The variable scmRatisRolesToCheck is only referenced in one place, and I'm not sure if defining it as a global variable is a good idea. Additionally, from my understanding, using static might be better if it needs to be used. @ChenSammi Please help give some suggestions.

@errose28 @adoroszlai May I take up some of your time to ask for some advice? Thank you very much!

@whbing How should I modify this? Should I change the variable to uppercase and make it static?

slfan1989 · 2024-09-30T01:48:34Z

@whbing @ChenSammi This PR is a small feature, but I believe it is valuable. We have already deployed it internally for 2 months, and I use this feature whenever switching SCM or OM. It should also make the information display more user-friendly for other users. If there are no major issues, I hope we can merge this PR into the master branch.

cc: @errose28 @adoroszlai

slfan1989 · 2024-10-03T08:28:21Z

@adoroszlai @errose28 @whbing @ChenSammi

I'm sorry to disturb everyone, but I really need this pr to assist with some subsequent work, such as #7266 (HDDS-11463. Track and display failed DataNode storage locations in SCM). I think we should provide a way to present this in a table format. Currently, I'm not sure what issues this PR has.

adoroszlai

Thanks @slfan1989 for the patch. I have some minor code comments, otherwise looks good.

adoroszlai · 2024-10-03T09:37:39Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmInfo.java

  private final String clusterId;
  private final String scmId;
  private final List<String> peerRoles;
+  private final Boolean scmRatisEnabled;


Is this intentionally Boolean instead of boolean? Since its value is set in the constructor from a boolean parameter, it doesn't seem to be ever set to null.

Thank you very much for helping review the code! I will improve this part of the code.

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java

...-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisRolesSubcommand.java

adoroszlai · 2024-10-03T09:44:51Z

...-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisRolesSubcommand.java

+        }
+
+        // In STANDALONE mode, output only the Host Name and Port information.
+        if (roleItems.length == 2) {


Should we check that roleItems.length matches the value expected according to isRatisEnabled?

Your revision looks great. We can simplify it by removing the array size check, since we'll standardize it to formattingCLIUtils.addLine(roleItems);.

…/scm/GetScmRatisRolesSubcommand.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

slfan1989 · 2024-10-04T06:42:36Z

@adoroszlai Could you help review this PR again? Thank you very much!

adoroszlai

Thanks @slfan1989 for updating the patch. Two previous minor items still apply, but no need to push another commit just for these.

@ChenSammi @whbing do you have any other comments?

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java

…/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

slfan1989 · 2024-10-04T13:53:08Z

@adoroszlai Thank you very much for reviewing the code. Friendly ping. @whbing @ChenSammi

whbing · 2024-10-04T14:39:42Z

Sorry for the late reply due to the holiday. I'll have a look ASAP.

whbing · 2024-10-04T23:20:33Z

Thanks for @slfan1989 for the update, +1.

slfan1989 · 2024-10-05T07:43:15Z

@adoroszlai Thank you very much for reviewing this PR!

Regarding whbing, he/she has agreed to this PR.

Regarding ChenSammi , he/she may have been quite busy recently and hasn’t responded to this PR in the past month. If he/she has any feedback later, I’m willing to make improvements to the code at any time.

Can we merge this PR into the master branch? so that I can continue working on HDDS-11463(#7266) as discussed with Ethan.

adoroszlai · 2024-10-05T08:17:27Z

Thanks @slfan1989 for the patch, @ChenSammi, @errose28, @whbing for the review.

slfan1989 · 2024-10-05T08:49:13Z

@adoroszlai Thank you again for your help with this PR!

slfan1989 force-pushed the HDDS-11268 branch from 5a1bb1d to cbb7e1a Compare August 1, 2024 23:32

ChenSammi reviewed Aug 8, 2024

View reviewed changes

slfan1989 force-pushed the HDDS-11268 branch from 028675d to 0b4b02d Compare August 20, 2024 01:12

slfan1989 requested a review from ChenSammi August 21, 2024 07:04

whbing reviewed Aug 22, 2024

View reviewed changes

hadoop-ozone/dist/src/main/smoketest/omha/om-roles.robot Outdated Show resolved Hide resolved

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/utils/FormattingCLIUtils.java Show resolved Hide resolved

slfan1989 force-pushed the HDDS-11268 branch from 5bd12bc to c56053e Compare August 25, 2024 02:24

slfan1989 commented Aug 28, 2024

View reviewed changes

slfan1989 force-pushed the HDDS-11268 branch 2 times, most recently from 4845692 to e7b81f3 Compare September 1, 2024 02:51

slfan1989 requested a review from whbing September 1, 2024 05:08

ChenSammi reviewed Sep 4, 2024

View reviewed changes

slfan1989 added 2 commits September 11, 2024 21:33

HDDS-11268. [CLI] Improve CLI Display OM/SCM Roles.

5d51969

HDDS-11268. Improve Code.

b5e7a3a

slfan1989 force-pushed the HDDS-11268 branch from 9ae14fa to b5e7a3a Compare September 11, 2024 13:55

slfan1989 added 2 commits September 11, 2024 21:57

HDDS-11268. Fix CheckStyle.

b37d601

HDDS-11268. Fix CheckStyle.

c6ce277

slfan1989 commented Sep 12, 2024

View reviewed changes

slfan1989 requested a review from ChenSammi September 18, 2024 11:58

whbing reviewed Sep 19, 2024

View reviewed changes

HDDS-11268. Fix CheckStyle.

92a11d0

slfan1989 requested a review from whbing September 23, 2024 05:00

whbing reviewed Sep 24, 2024

View reviewed changes

Merge branch 'apache:master' into HDDS-11268

f12f7f6

slfan1989 requested a review from whbing October 3, 2024 08:29

adoroszlai reviewed Oct 3, 2024

View reviewed changes

slfan1989 and others added 3 commits October 3, 2024 20:32

Update hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin…

e69f1d6

…/scm/GetScmRatisRolesSubcommand.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin…

1b0fbc2

…/scm/GetScmRatisRolesSubcommand.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

HDDS-11268. [CLI] Improve CLI Display OM/SCM Roles.

d427166

slfan1989 requested a review from adoroszlai October 4, 2024 06:42

adoroszlai reviewed Oct 4, 2024

View reviewed changes

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Outdated Show resolved Hide resolved

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Outdated Show resolved Hide resolved

adoroszlai changed the title ~~HDDS-11268. [CLI] Improve CLI Display OM/SCM Roles.~~ HDDS-11268. Add --table mode for OM/SCM Roles CLI Oct 4, 2024

slfan1989 and others added 2 commits October 4, 2024 16:04

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm…

e8b99ad

…/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

Update hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm…

1360309

…/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>

whbing approved these changes Oct 4, 2024

View reviewed changes

adoroszlai merged commit eb26677 into apache:master Oct 5, 2024

HDDS-11268. Add --table mode for OM/SCM Roles CLI #7016

HDDS-11268. Add --table mode for OM/SCM Roles CLI #7016

Uh oh!

Conversation

slfan1989 commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

errose28 commented Aug 2, 2024

Uh oh!

slfan1989 commented Aug 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai commented Aug 5, 2024

Uh oh!

slfan1989 commented Aug 6, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whbing left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

whbing commented Aug 27, 2024

Uh oh!

ChenSammi commented Aug 27, 2024

Uh oh!

slfan1989 Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whbing left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slfan1989 Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

slfan1989 commented Sep 23, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Sep 30, 2024

slfan1989 commented Aug 1, 2024 •

edited

Loading

slfan1989 commented Aug 4, 2024 •

edited

Loading

slfan1989 Aug 28, 2024 •

edited

Loading

slfan1989 commented Aug 28, 2024 •

edited

Loading

ChenSammi Sep 4, 2024 •

edited

Loading

slfan1989 Sep 20, 2024 •

edited

Loading

whbing commented Oct 4, 2024 •

edited

Loading