test framework compatible with m1 arm64 by zhyyu · Pull Request #322 · apache/skywalking-java

zhyyu · 2022-09-28T08:48:58Z

If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
Update the CHANGES log.

wu-sheng · 2022-09-28T09:02:15Z

2022-09-28T08:59:21.1450158Z [INFO] Copying files to /home/runner/work/skywalking-java/skywalking-java/test/plugin/containers/jvm-container/target/docker/skywalking/agent-test-jvm/1.0.0/build/maven
2022-09-28T08:59:21.1703577Z [INFO] Building tar: /home/runner/work/skywalking-java/skywalking-java/test/plugin/containers/jvm-container/target/docker/skywalking/agent-test-jvm/1.0.0/tmp/docker-build.tar
2022-09-28T08:59:21.2850662Z [INFO] DOCKER> [skywalking/agent-test-jvm:1.0.0]: Created docker-build.tar in 143 milliseconds
2022-09-28T08:59:25.3836083Z [ERROR] DOCKER> Unable to build image [skywalking/agent-test-jvm:1.0.0] : "The command '/bin/sh -c apt-get update -y' returned a non-zero code: 127"  ["The command '/bin/sh -c apt-get update -y' returned a non-zero code: 127" ]

Build image failure from the CI process.

zhyyu · 2022-09-28T09:31:13Z

2022-09-28T08:59:21.1450158Z [INFO] Copying files to /home/runner/work/skywalking-java/skywalking-java/test/plugin/containers/jvm-container/target/docker/skywalking/agent-test-jvm/1.0.0/build/maven
2022-09-28T08:59:21.1703577Z [INFO] Building tar: /home/runner/work/skywalking-java/skywalking-java/test/plugin/containers/jvm-container/target/docker/skywalking/agent-test-jvm/1.0.0/tmp/docker-build.tar
2022-09-28T08:59:21.2850662Z [INFO] DOCKER> [skywalking/agent-test-jvm:1.0.0]: Created docker-build.tar in 143 milliseconds
2022-09-28T08:59:25.3836083Z [ERROR] DOCKER> Unable to build image [skywalking/agent-test-jvm:1.0.0] : "The command '/bin/sh -c apt-get update -y' returned a non-zero code: 127"  ["The command '/bin/sh -c apt-get update -y' returned a non-zero code: 127" ]

Build image failure from the CI process.

  [INFO] DOCKER> Pulled adoptopenjdk/openjdk8:alpine in 6 seconds 
  Error:  DOCKER> Unable to build image [skywalking/agent-test-jvm:1.0.0] : "The command '/bin/sh -c apt-get update -y' returned a non-zero code: 127"  ["The command '/bin/sh -c apt-get update -y' returned a non-zero code: 127" ]
  [INFO] ------------------------------------------------------------------------
  [INFO] Reactor Summary for SkyWalking Plugins Tests 1.0.0:

It's still adoptopenjdk/openjdk8:alpine, I will change .github CI and retry.

…emurin:8-jdk

zhyyu · 2022-09-30T10:16:31Z

m1 arm64 have to excute

socat TCP-LISTEN:2375,range=127.0.0.1/32,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock
export DOCKER_HOST=tcp://127.0.0.1:2375

before run

bash ./test/plugin/run.sh -f ${scenario_name}

wu-sheng · 2022-09-30T10:48:41Z

You could use shell to check and run more.

wu-sheng · 2022-10-09T01:51:43Z

@zhyyu Would you continue this?

zhyyu · 2022-10-11T02:16:30Z

@zhyyu Would you continue this?

Yes, I will continue. These days my work is busy, I will go on when I am free.

zhyyu · 2022-10-11T09:57:58Z

The m1 user have to install socat.

wu-sheng · 2022-10-11T10:38:50Z

You could add through CI, and update docs to help on local use.

kezhenxu94 · 2022-10-11T13:48:27Z

m1 arm64 have to excute

socat TCP-LISTEN:2375,range=127.0.0.1/32,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock
export DOCKER_HOST=tcp://127.0.0.1:2375

before run

bash ./test/plugin/run.sh -f ${scenario_name}

Can you explain or paste some related links about why the socat command is needed?

zhyyu · 2022-10-12T02:17:25Z

m1 arm64 have to excute
socat TCP-LISTEN:2375,range=127.0.0.1/32,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock
export DOCKER_HOST=tcp://127.0.0.1:2375
before run
bash ./test/plugin/run.sh -f ${scenario_name}
Can you explain or paste some related links about why the socat command is needed?

https://stackoverflow.com/questions/71300031/docker-image-build-failed-on-mac-m1-chip , the second answer

https://juejin.cn/post/7094919855643557896

https://nxt.engineering/blog/kafka-docker-image/

https://dmp.fabric8.io/ the 3. Global configuration section, DOCKER_HOST description

…se m1

kezhenxu94

Don't have a m1 machine to verify, just some code level review.

kezhenxu94 · 2022-10-13T08:59:16Z

 parse_commandline "$@"

+if [[ $(uname) == 'Darwin' ]] && [[ $(uname -m) == 'arm64' ]]; then
+  nohup socat TCP-LISTEN:2375,range=127.0.0.1/32,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock > /dev/null 2>&1 &


Can you add a trap to capture the exit signal and kill this process so when the tests finished this can be released? Otherwise this process is still running even when the tests finished.

From the document, https://github.com/fabric8io/docker-maven-plugin/blob/master/doc/changelog.md

The M1 support has already been added from 0.38.1 and I've run this patch without socat, everything is fine.

From the document, https://github.com/fabric8io/docker-maven-plugin/blob/master/doc/changelog.md

The M1 support has already been added from 0.38.1 and I've run this patch without socat, everything is fine.

What version of mac m1 docker desktop are you using?

I use the newest version of m1 docker desktop, still error if not use socat

[INFO] [INFO] --- docker-maven-plugin:0.40.2:build (build) @ tomcat-container --- ./test/plugin/run.sh: line 186: 50210 Killed: 9 ${mvnw} --batch-mode -f ${home}/pom.xml -DskipTests -Dbase_image_java=${base_image_java} -Dbase_image_tomcat=${base_image_tomcat} -Dcontainer_image_version=${container_image_version} clean package

Can you add a trap to capture the exit signal and kill this process so when the tests finished this can be released? Otherwise this process is still running even when the tests finished.

I have removed socat nohup, when the run.sh exit, socat exit.

I've confirmed with another laptop (13' M2 2022). Previously I've checked on my Mac Studio with M1 Ultra. Still the same results.

$ bash ./test/plugin/run.sh -f spring-4.3.x-scenario

I'm using the default configuration with Docker Desktop 4.13.1 (90346),

I guess your jvm may not be built for arm64. Anyway, I don't think socat is necessary.

@lujiajing1126 Could you submit another PR to bump up the version?
Then I could do another round check locally on my M1 Max.

@lujiajing1126 Could you submit another PR to bump up the version? Then I could do another round check locally on my M1 Max.

Sure

wu-sheng · 2022-10-13T09:03:57Z

I will do local verification on my M1 Max once everything is ready for CI and @kezhenxu94 's review.

kezhenxu94 · 2022-10-13T09:08:17Z

m1 arm64 have to excute
socat TCP-LISTEN:2375,range=127.0.0.1/32,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock
export DOCKER_HOST=tcp://127.0.0.1:2375
before run
bash ./test/plugin/run.sh -f ${scenario_name}
Can you explain or paste some related links about why the socat command is needed?
https://stackoverflow.com/questions/71300031/docker-image-build-failed-on-mac-m1-chip , the second answer

https://juejin.cn/post/7094919855643557896

https://nxt.engineering/blog/kafka-docker-image/

https://dmp.fabric8.io/ the 3. Global configuration section, DOCKER_HOST description

In the link https://stackoverflow.com/a/71301494/5200161, the other answer suggests switching to fabric8-maven-plugin, dockerfile-maven-plugin from Spotify is inactive for a long time so I think for any reason we should switch to fabric8.

kezhenxu94 · 2022-10-13T09:09:04Z

The changes in this PR is a workaround and not a long-term solution in my opinion.

kezhenxu94 · 2022-10-15T01:49:16Z

Or, we can even just use exec maven plugin to execute the Docker command, this has the best compatibility. But we need to extract the existing maven configurations into a Dockerfile. Like https://github.com/apache/dolphinscheduler/blob/cd5fb12e46026f224ed24ad52a6e8711ff212cfe/pom.xml#L436-L466

wu-sheng · 2022-11-03T00:09:00Z

@zhyyu What is the status of this PR?

lujiajing1126 · 2022-11-03T01:47:08Z

 parse_commandline "$@"

+if [[ $(uname) == 'Darwin' ]] && [[ $(uname -m) == 'arm64' ]]; then
+  nohup socat TCP-LISTEN:2375,range=127.0.0.1/32,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock > /dev/null 2>&1 &


From the document, https://github.com/fabric8io/docker-maven-plugin/blob/master/doc/changelog.md

The M1 support has already been added from 0.38.1 and I've run this patch without socat, everything is fine.

zhyyu · 2022-11-04T02:23:45Z

On my local m1 mac, If I don't use socat, this error happen.

[INFO] 
[INFO] --- docker-maven-plugin:0.40.2:build (build) @ tomcat-container ---
./test/plugin/run.sh: line 186: 50210 Killed: 9               ${mvnw} --batch-mode -f ${home}/pom.xml -DskipTests -Dbase_image_java=${base_image_java} -Dbase_image_tomcat=${base_image_tomcat} -Dcontainer_image_version=${container_image_version} clean package

…th-m1 # Conflicts: # test/plugin/agent-test-tools/pom.xml

wu-sheng · 2022-11-08T01:52:50Z

#379 works.

compatible with m1 arm64

b03c77a

zhyyu changed the title ~~compatible with m1 arm64~~ test framework compatible with m1 arm64 Sep 28, 2022

wu-sheng reviewed Sep 28, 2022

View reviewed changes

Comment thread test/plugin/run.sh Outdated

wu-sheng added enhancement New feature or request CI Setting labels Sep 28, 2022

wu-sheng requested review from dmsolr, kezhenxu94 and wu-sheng September 28, 2022 08:54

wu-sheng added this to the 8.13.0 milestone Sep 28, 2022

yuzhongyu added 2 commits September 29, 2022 14:49

change base_image_java from adoptopenjdk/openjdk8:alpine to eclipse-t…

0079c1b

…emurin:8-jdk

change eclipse-temurin:17-alpine to eclipse-temurin:17-jdk

8e3481d

auto check m1 and export DOCKER_HOST

afdd066

kezhenxu94 reviewed Oct 11, 2022

View reviewed changes

Comment thread test/plugin/containers/jvm-container/pom.xml

update Plugin-test.md, add socat into Environment Requirements when u…

b204b33

…se m1

kezhenxu94 reviewed Oct 13, 2022

View reviewed changes

lujiajing1126 requested changes Nov 3, 2022

View reviewed changes

yuzhongyu added 2 commits November 4, 2022 10:26

Merge remote-tracking branch 'origin/main' into feature-compatible-wi…

597782d

…th-m1 # Conflicts: # test/plugin/agent-test-tools/pom.xml

remove socat nohup, when the run.sh exit, socat exit

35e6e60

wu-sheng removed this from the 8.13.0 milestone Nov 6, 2022

lujiajing1126 mentioned this pull request Nov 7, 2022

Basic testing support on arm64/v8 platforms #379

Merged

4 tasks

wu-sheng closed this Nov 8, 2022

wu-sheng added this to the 8.14.0 milestone Nov 8, 2022

Conversation

zhyyu commented Sep 28, 2022

Uh oh!

Uh oh!

wu-sheng commented Sep 28, 2022

Uh oh!

zhyyu commented Sep 28, 2022

Uh oh!

zhyyu commented Sep 30, 2022

Uh oh!

wu-sheng commented Sep 30, 2022

Uh oh!

wu-sheng commented Oct 9, 2022

Uh oh!

zhyyu commented Oct 11, 2022

Uh oh!

zhyyu commented Oct 11, 2022

Uh oh!

wu-sheng commented Oct 11, 2022

Uh oh!

Uh oh!

kezhenxu94 commented Oct 11, 2022

Uh oh!

zhyyu commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kezhenxu94 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wu-sheng commented Oct 13, 2022

Uh oh!

kezhenxu94 commented Oct 13, 2022

Uh oh!

kezhenxu94 commented Oct 13, 2022

Uh oh!

kezhenxu94 commented Oct 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wu-sheng commented Nov 3, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhyyu commented Nov 4, 2022

Uh oh!

wu-sheng commented Nov 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhyyu commented Oct 12, 2022 •

edited

Loading

kezhenxu94 commented Oct 15, 2022 •

edited

Loading