HDDS-10414. Some acceptance tests fail with Docker Compose V2 by ArafatKhan2198 · Pull Request #6269 · apache/ozone

ArafatKhan2198 · 2024-02-24T13:27:35Z

What changes were proposed in this pull request?

Recon Acceptance test is failing for the robot test :-

hadoop-ozone/dist/src/main/smoketest/recon/recon-api.robot

The discrepancy arises because, in local Docker setups, DataNodes are named using hyphens, such as datanode-1, whereas in the upstream (CI) environment, DataNodes utilize underscores, like datanode_1. This variation in naming conventions leads to inconsistencies across different environments.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10414

How was this patch tested?

The acceptance test passed locally :-

------------------------------------------------------------------------------
Check if Recon picks up DN heartbeats                                 | PASS |
------------------------------------------------------------------------------
Check if Recon Web UI is up                                           | PASS |
------------------------------------------------------------------------------
Check web UI access                                                   | PASS |
------------------------------------------------------------------------------
Check admin only api access                                           | PASS |
------------------------------------------------------------------------------
Check unhealthy, (admin) api access                                   | PASS |
------------------------------------------------------------------------------
Check normal api access                                               | PASS |
------------------------------------------------------------------------------

And in the forked CI :- https://github.com/ArafatKhan2198/ozone/actions/runs/8030499503

… DN heartbeats: failed.

ArafatKhan2198 · 2024-02-25T11:06:53Z

@dombizita @devmadhuu Could you please take a look.

adoroszlai · 2024-02-25T11:17:30Z

@ArafatKhan2198 Please include information about the Docker (and Docker Compose) version where the test is failing.

ArafatKhan2198 · 2024-02-26T11:23:30Z

So my local docker versions are the following :-

(base) mohammad.khan:ozone/ (HDDS-10324*) $ docker --version                                    
Docker version 24.0.5, build ced0996

(base) mohammad.khan:ozone/ (HDDS-10324*) $ docker-compose --version                            
Docker Compose version v2.20.2-desktop.1

adoroszlai · 2024-02-27T17:47:34Z

@ArafatKhan2198 please check whether hadoop-ozone/dist/src/main/smoketest/topology/cli.robot is also failing with the same Docker version, it has similar host names in assertion:

hadoop-ozone/dist/src/main/smoketest/topology/cli.robot
29:                        Should contain   ${output}         10.5.0.7(ozone-topology_datanode_4_1.ozone-topology_net)    IN_SERVICE    /rack2
33:                        Should contain   ${output}         10.5.0.7(ozone-topology_datanode_4_1.ozone-topology_net) IN_SERVICE
36:                        Should contain   ${output}         10.5.0.7(ozone-topology_datanode_4_1.ozone-topology_net)    IN_SERVICE    /rack2
39:                        Should contain   ${output}         10.5.0.7(ozone-topology_datanode_4_1.ozone-topology_net)    IN_SERVICE    /rack2

ArafatKhan2198 · 2024-03-11T09:12:36Z

Hi @adoroszlai can you please take a look
The forked CI ran green :- https://github.com/ArafatKhan2198/ozone/actions/runs/8229346145

The hadoop-ozone/dist/src/main/smoketest/topology/cli.robot ran locally as well

(base) mohammad.khan:ozone/ (HDDS-10414*) $ ../test-single.sh om topology/cli.robot                                                                                                                                                                                                                     [13:21:04]
==============================================================================
Cli :: Smoketest ozone cluster startup                                        
==============================================================================
Run printTopology                                                     | PASS |
------------------------------------------------------------------------------
Run printTopology -o                                                  | PASS |
------------------------------------------------------------------------------
Run printTopology --operational-state IN_SERVICE                      | PASS |
------------------------------------------------------------------------------
Run printTopology --node-state HEALTHY                                | PASS |
------------------------------------------------------------------------------
Cli :: Smoketest ozone cluster startup                                | PASS |
4 tests, 4 passed, 0 failed
==============================================================================

adoroszlai

Thanks @ArafatKhan2198 for the patch.

adoroszlai · 2024-03-11T17:01:11Z

hadoop-ozone/dist/src/main/smoketest/topology/cli.robot

+                        Should Contain   ${output}         State = HEALTHY
+                        Should Contain          ${output}         IN_SERVICE
+                        Should Match Regexp     ${output}         .*datanode[-_]\\d+.*IN_SERVICE.*


State = HEALTHY was not in the original expected content, why is it added?

Why is check for rack2 removed?

Each test case begins by checking for the State = HEALTHY output, ensuring that the overall state of the system or the datanodes is as expected.
All the commands start with this line.

sh-4.2$ ozone admin printTopology State = HEALTHY 172.19.0.8(ozone-datanode-4.ozone_default):HTTP=9882,CLIENT_RPC=19864,REPLICATION=9886,RATIS=9858,RATIS_ADMIN=9857,RATIS_SERVER=9856,RATIS_DATASTREAM=9855,STANDALONE=9859 IN_SERVICE /default-rack 172.19.0.10(ozone-datanode-1.ozone_default):HTTP=9882,CLIENT_RPC=19864,REPLICATION=9886,RATIS=9858,RATIS_ADMIN=9857,RATIS_SERVER=9856,RATIS_DATASTREAM=9855,STANDALONE=9859 IN_SERVICE /default-rack 172.19.0.7(ozone-datanode-5.ozone_default):HTTP=9882,CLIENT_RPC=19864,REPLICATION=9886,RATIS=9858,RATIS_ADMIN=9857,RATIS_SERVER=9856,RATIS_DATASTREAM=9855,STANDALONE=9859 IN_SERVICE /default-rack

And I have added the check for rack again

I think it would be better to remove it since we want this test to work regardless of the health of DN's

adoroszlai · 2024-03-11T17:02:02Z

hadoop-ozone/dist/src/main/smoketest/topology/cli.robot

+                        Should Contain   ${output}         State = HEALTHY
+                        Should Contain   ${output}         IN_SERVICE
+                        Should Match Regexp   ${output}         .*ozone.*datanode[-_]\\d+.*IN_SERVICE.*


Let's avoid duplicating assertions. Please extract a Keyword and use it in all test cases.

adoroszlai · 2024-03-12T10:45:03Z

Thanks @ArafatKhan2198 for updating the patch, latest patch looks good. Let's have someone with Docker Compose V2 also check it.

Tejaskriya · 2024-03-12T13:29:47Z

Tested this patch locally in docker, LGTM
Docker version on my system: Docker Compose version v2.18.1

$ ../test-single.sh om topology/cli.robot                                    
==============================================================================
Cli :: Smoketest ozone cluster startup                                        
==============================================================================
Run printTopology                                                     | PASS |
------------------------------------------------------------------------------
Run printTopology -o                                                  | PASS |
------------------------------------------------------------------------------
Run printTopology --operational-state IN_SERVICE                      | PASS |
------------------------------------------------------------------------------
Run printTopology --node-state HEALTHY                                | PASS |
------------------------------------------------------------------------------
Cli :: Smoketest ozone cluster startup                                | PASS |
4 tests, 4 passed, 0 failed
==============================================================================


$ ../test-single.sh om recon/recon-api.robot  
==============================================================================
Recon-Api :: Smoke test to start cluster with docker-compose environments.    
==============================================================================
Check if Recon picks up OM data                                       | PASS |
------------------------------------------------------------------------------
Check if Recon picks up DN heartbeats                                 | PASS |
------------------------------------------------------------------------------
Check if Recon Web UI is up                                           | FAIL |
'<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403 Forbidden</h2>
<table>
<tr><th>URI:</th><td>/</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>Forbidden</td></tr>
<tr><th>SERVLET:</th><td>org.eclipse.jetty.servlet.DefaultServlet-5cfed0ba</td></tr>
</table>

</body>
</html>' does not contain 'Ozone Recon'
------------------------------------------------------------------------------
Check web UI access                                                   | FAIL |
'403' does not contain '200'
------------------------------------------------------------------------------
Check admin only api access                                           | PASS |
------------------------------------------------------------------------------
Check unhealthy, (admin) api access                                   | PASS |
------------------------------------------------------------------------------
Check normal api access                                               | PASS |
------------------------------------------------------------------------------
Recon-Api :: Smoke test to start cluster with docker-compose envir... | FAIL |
7 tests, 5 passed, 2 failed
==============================================================================

(I have some issues with recon UI working in my system so I think the other test failures are related to that)

devmadhuu

Thanks @ArafatKhan2198 for working on the fix. Changes LGTM +1 and tested on following docker and docker-compose versions:
docker-compose --version
Docker Compose version v2.18.1
docker --version
Docker version 24.0.2, build cb74dfc

../test-single.sh om topology/cli.robot 
==============================================================================
Cli :: Smoketest ozone cluster startup                                        
==============================================================================
Run printTopology                                                     | PASS |
------------------------------------------------------------------------------
Run printTopology -o                                                  | PASS |
------------------------------------------------------------------------------
Run printTopology --operational-state IN_SERVICE                      | PASS |
------------------------------------------------------------------------------
Run printTopology --node-state HEALTHY                                | PASS |
------------------------------------------------------------------------------
Cli :: Smoketest ozone cluster startup                                | PASS |
4 tests, 4 passed, 0 failed
==============================================================================

../test-single.sh om recon/recon-api.robot
==============================================================================
Recon-Api :: Smoke test to start cluster with docker-compose environments.    
==============================================================================
Check if Recon picks up OM data                                       | PASS |
------------------------------------------------------------------------------
Check if Recon picks up DN heartbeats                                 | PASS |
------------------------------------------------------------------------------
Check if Recon Web UI is up                                           | PASS |
------------------------------------------------------------------------------
Check web UI access                                                   | PASS |
------------------------------------------------------------------------------
Check admin only api access                                           | PASS |
------------------------------------------------------------------------------
Check unhealthy, (admin) api access                                   | PASS |
------------------------------------------------------------------------------
Check normal api access                                               | PASS |
------------------------------------------------------------------------------
Recon-Api :: Smoke test to start cluster with docker-compose envir... | PASS |
7 tests, 7 passed, 0 failed
==============================================================================

adoroszlai · 2024-03-12T16:05:27Z

Thanks @ArafatKhan2198 for the patch, @devmadhuu, @Tejaskriya for the review.

…#6269) (cherry picked from commit 91af26a)

…#6269)

ArafatKhan2198 added 2 commits February 23, 2024 12:47

HDDS-10414. ozone acceptance: Recon.Recon-Api.Check if Recon picks up…

bda2ee6

… DN heartbeats: failed.

Using both the format datanode_1 and datanode-1

33ae10c

adoroszlai added recon test labels Feb 26, 2024

adoroszlai changed the title ~~HDDS-10414. ozone acceptance: Recon.Recon-Api.Check if Recon picks up DN heartbeats: failed.~~ HDDS-10414. Some acceptance tests fail with Docker Compose V2 Feb 29, 2024

adoroszlai removed the recon label Feb 29, 2024

Using [-_] for cli.robot

32fa37a

ArafatKhan2198 marked this pull request as draft February 29, 2024 11:27

ArafatKhan2198 added 3 commits March 11, 2024 11:20

Removed changes for cli.robot

df15c6b

Original changes

c232223

Fix for the failing test cli.robot in docker V2

6b303b2

ArafatKhan2198 marked this pull request as ready for review March 11, 2024 09:12

adoroszlai marked this pull request as draft March 11, 2024 13:41

adoroszlai marked this pull request as ready for review March 11, 2024 13:41

adoroszlai reviewed Mar 11, 2024

View reviewed changes

ArafatKhan2198 added 2 commits March 12, 2024 00:03

Made review comment changes

bf3a778

Removed the healthy DN assertion

45499f9

adoroszlai requested a review from devmadhuu March 12, 2024 10:43

devmadhuu approved these changes Mar 12, 2024

View reviewed changes

Tejaskriya approved these changes Mar 12, 2024

View reviewed changes

adoroszlai merged commit 91af26a into apache:master Mar 12, 2024

adoroszlai pushed a commit to adoroszlai/ozone that referenced this pull request Jul 12, 2024

HDDS-10414. Some acceptance tests fail with Docker Compose V2 (apache…

625a9ca

…#6269) (cherry picked from commit 91af26a)

adoroszlai pushed a commit to adoroszlai/ozone that referenced this pull request Jul 12, 2024

HDDS-10414. Some acceptance tests fail with Docker Compose V2 (apache…

a035b65

…#6269) (cherry picked from commit 91af26a)

devabhishekpal pushed a commit to devabhishekpal/ozone that referenced this pull request Jul 12, 2024

HDDS-10414. Some acceptance tests fail with Docker Compose V2 (apache…

360729d

…#6269)

devabhishekpal mentioned this pull request Jul 12, 2024

[DO NOT MERGE] Backport some fixes from master to ozone-1.4 #6929

Merged

Conversation

ArafatKhan2198 commented Feb 24, 2024

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

ArafatKhan2198 commented Feb 25, 2024

Uh oh!

adoroszlai commented Feb 25, 2024

Uh oh!

ArafatKhan2198 commented Feb 26, 2024

Uh oh!

adoroszlai commented Feb 27, 2024

Uh oh!

ArafatKhan2198 commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

ArafatKhan2198 Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

ArafatKhan2198 Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

adoroszlai Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

ArafatKhan2198 Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Mar 12, 2024

Uh oh!

Tejaskriya commented Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devmadhuu left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Mar 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ArafatKhan2198 commented Mar 11, 2024 •

edited

Loading

Tejaskriya commented Mar 12, 2024 •

edited

Loading