HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK… #3904

stoty · 2021-12-01T12:54:51Z

…eyValue()

Change-Id: I33099354f40b04b0f4f79f5cbd32445fe9d0d818

…eyValue() Change-Id: I33099354f40b04b0f4f79f5cbd32445fe9d0d818

Apache-HBase · 2021-12-01T13:27:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 53s	master passed
+1 💚	compile	0m 28s	master passed
+1 💚	shadedjars	8m 56s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 40s	the patch passed
+1 💚	compile	0m 28s	the patch passed
+1 💚	javac	0m 28s	the patch passed
+1 💚	shadedjars	8m 16s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 7s	hbase-common in the patch passed.
		32m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3904
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 32db26357eef 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d1762f5`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/testReport/
Max. process+thread count	300 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-01T13:29:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 42s	master passed
+1 💚	compile	0m 28s	master passed
+1 💚	shadedjars	9m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 30s	the patch passed
+1 💚	compile	0m 26s	the patch passed
+1 💚	javac	0m 26s	the patch passed
+1 💚	shadedjars	9m 44s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 0s	hbase-common in the patch passed.
		33m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3904
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a76b2da62ea1 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d1762f5`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/testReport/
Max. process+thread count	360 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-01T13:36:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 10s	master passed
+1 💚	compile	0m 52s	master passed
+1 💚	checkstyle	0m 27s	master passed
+1 💚	spotbugs	0m 48s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 47s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	checkstyle	0m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	19m 23s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 56s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		40m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3904
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 5186b05cc77b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d1762f5`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2021-12-04T14:58:24Z

Checked the code in KeyValueUtils, there is a method getSerializedSize

  public static int getSerializedSize(Cell cell, boolean withTags) {
    if (withTags) {
      return cell.getSerializedSize();
    }
    if (cell instanceof ExtendedCell) {
      return ((ExtendedCell) cell).getSerializedSize(withTags);
    }
    return length(cell.getRowLength(), cell.getFamilyLength(), cell.getQualifierLength(),
      cell.getValueLength(), cell.getTagsLength(), withTags);
  }

It seems that if withTags is true, we will always call cell.getSerializedSize directly. I guess this is why we use getSerializedSize there.

So let's add some comments to say that, the byte array is not used for serializaing, as it will copy all the fields such as column family, qualifier, etc, so it is not suitable to call getSerializedSize here, we should calculate it with all the fields.

Thanks.

stoty · 2021-12-06T07:07:31Z

Thanks for the review @Apache9 , I have added the comment.

Apache9 · 2021-12-06T07:52:45Z

Thanks for the review @Apache9 , I have added the comment.

Forgot to push it? Seems the PR remains unchanged...

Change-Id: Ibf9419d391560a209404bb7dcc6b583502d12bda

stoty · 2021-12-06T08:02:29Z

@Apache9 Yes, I pushed to the wrong repo.
Please check now.

Apache-HBase · 2021-12-06T08:34:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 56s	master passed
+1 💚	compile	0m 26s	master passed
+1 💚	shadedjars	9m 13s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 28s	the patch passed
+1 💚	compile	0m 27s	the patch passed
+1 💚	javac	0m 27s	the patch passed
+1 💚	shadedjars	8m 21s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 5s	hbase-common in the patch passed.
		32m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3904
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux d8c31e4bdb1b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dcd622d`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/testReport/
Max. process+thread count	297 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-06T08:34:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 1s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 31s	master passed
+1 💚	compile	0m 23s	master passed
+1 💚	shadedjars	9m 3s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 16s	the patch passed
+1 💚	compile	0m 25s	the patch passed
+1 💚	javac	0m 25s	the patch passed
+1 💚	shadedjars	9m 9s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 1s	hbase-common in the patch passed.
		32m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3904
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux fbe740a80a17 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dcd622d`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/testReport/
Max. process+thread count	228 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-12-06T09:03:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	8m 57s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 1s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	5m 36s	master passed
+1 💚	compile	0m 55s	master passed
+1 💚	checkstyle	0m 31s	master passed
+1 💚	spotbugs	0m 56s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 7s	the patch passed
+1 💚	compile	0m 51s	the patch passed
+1 💚	javac	0m 51s	the patch passed
+1 💚	checkstyle	0m 29s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	26m 35s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	1m 1s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		61m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3904
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 587f6a4aefbe 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `dcd622d`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-common U: hbase-common
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3904/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…eyValue() (#3904) Signed-off-by: Andrew Purtell <apurtell@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

…eyValue() (apache#3904)

…eyValue() (apache#3904) Signed-off-by: Andrew Purtell <apurtell@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK…

13897a0

…eyValue() Change-Id: I33099354f40b04b0f4f79f5cbd32445fe9d0d818

stoty marked this pull request as draft December 1, 2021 12:55

apurtell approved these changes Dec 1, 2021

View reviewed changes

stoty marked this pull request as ready for review December 2, 2021 10:21

added comment as requested by Duo

d754dce

Change-Id: Ibf9419d391560a209404bb7dcc6b583502d12bda

Apache9 approved these changes Dec 6, 2021

View reviewed changes

Apache9 merged commit d9315fa into apache:master Dec 6, 2021

Apache9 pushed a commit that referenced this pull request Dec 6, 2021

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK…

e64b089

…eyValue() (#3904) Signed-off-by: Andrew Purtell <apurtell@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Dec 6, 2021

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK…

a5a8cd8

…eyValue() (#3904) Signed-off-by: Andrew Purtell <apurtell@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

guangxuCheng pushed a commit to guangxuCheng/hbase that referenced this pull request Mar 28, 2022

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK…

77dc0c0

…eyValue() (apache#3904)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK… #3904

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK… #3904

Uh oh!

stoty commented Dec 1, 2021

Uh oh!

Apache-HBase commented Dec 1, 2021

Uh oh!

Apache-HBase commented Dec 1, 2021

Uh oh!

Apache-HBase commented Dec 1, 2021

Uh oh!

Apache9 commented Dec 4, 2021

Uh oh!

stoty commented Dec 6, 2021

Uh oh!

Apache9 commented Dec 6, 2021

Uh oh!

stoty commented Dec 6, 2021

Uh oh!

Apache-HBase commented Dec 6, 2021

Uh oh!

Apache-HBase commented Dec 6, 2021

Uh oh!

Apache-HBase commented Dec 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK… #3904

HBASE-26527 ArrayIndexOutOfBoundsException in KeyValueUtil.copyToNewK… #3904

Uh oh!

Conversation

stoty commented Dec 1, 2021

Uh oh!

Apache-HBase commented Dec 1, 2021

Uh oh!

Apache-HBase commented Dec 1, 2021

Uh oh!

Apache-HBase commented Dec 1, 2021

Uh oh!

Apache9 commented Dec 4, 2021

Uh oh!

stoty commented Dec 6, 2021

Uh oh!

Apache9 commented Dec 6, 2021

Uh oh!

stoty commented Dec 6, 2021

Uh oh!

Apache-HBase commented Dec 6, 2021

Uh oh!

Apache-HBase commented Dec 6, 2021

Uh oh!

Apache-HBase commented Dec 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants