[GLUTEN-10546][FLINK] Support all flink operators for nexmark by shuai-xu · Pull Request #10548 · apache/gluten

shuai-xu · 2025-08-27T03:56:12Z

What changes are proposed in this pull request?

This pr contains:

support all the operators use for nexmark test, e.g. GroupAgg, Rank, TopN
nexmark q18, q19, q20, q21 run pass, while the result is not equal to flink yet.
support checkpoint related interfaces.

How was this patch tested?

This pr is tested by manually run nexmark tests.

github-actions · 2025-08-27T03:56:30Z

#10546

philo-he

Some minor comments. Thanks.

...va/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecGroupWindowAggregate.java

philo-he · 2025-08-29T06:54:44Z

gluten-flink/planner/src/main/java/org/apache/gluten/rexnode/WindowUtils.java

    int rowtimeIndex = -1;
    int windowType = -1;
    WindowSpec windowSpec = windowing.getWindow();
+    System.out.println("WindowSpec " + windowSpec);


philo-he · 2025-08-29T07:17:33Z

...in/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecWindowAggregate.java

+    final ZoneId shiftTimeZone =
+        TimeWindowUtil.getShiftTimeZone(
+            windowing.getTimeAttributeType(), TableConfigUtils.getLocalTimeZone(config));
+    final WindowAssigner windowAssigner = createWindowAssigner(windowing, shiftTimeZone);


shiftTimeZone and windowAssigner are not used after the modification. Maybe, we can remove it.

shiftTimeZone is used, so keep it.

...va/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecGroupWindowAggregate.java

ryyyyyy1 · 2025-08-30T04:01:31Z

try to run but it seems to fail due to memory leak error:
E20250830 11:49:14.568943 3949495 MemoryManager.cc:264] [Velox4J MemoryManager DTOR] Memory leak found on Velox memory pool: Memory Pool[Decoding Memory Pool LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 128.00MB used 512B available 1023.50KB reservation [used 512B, reserved 1.00MB, min 0B] counters [allocs 4, frees 0, reserves 0, releases 0, collisions 0])>. Please make sure your code released all opened resources already. E20250830 11:49:14.569058 3949495 MemoryManager.cc:264] [Velox4J MemoryManager DTOR] Memory leak found on Velox memory pool: Memory Pool[root AGGREGATE root[root] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 128.00MB used 512B available 0B reservation [used 0B, reserved 1.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>. Please make sure your code released all opened resources already. E20250830 11:49:14.569092 3949495 MemoryPool.cpp:461] [MEM] Memory leak (Used memory): Memory Pool[Decoding Memory Pool LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 128.00MB used 512B available 1023.50KB reservation [used 512B, reserved 1.00MB, min 0B] counters [allocs 4, frees 0, reserves 0, releases 0, collisions 0])> E20250830 11:49:14.569126 3949495 Exceptions.h:66] Line: velox4j/src/main/cpp/main/velox4j/memory/MemoryManager.cc:92, Function:removePool, Expression: pool->reservedBytes() == 0 (1048576 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError' what(): Exception: VeloxRuntimeError Error Source: RUNTIME Error Code: INVALID_STATE Reason: (1048576 vs. 0) Retriable: False Expression: pool->reservedBytes() == 0 Function: removePool File: velox4j/src/main/cpp/main/velox4j/memory/MemoryManager.cc Line: 92 Stack trace: 0 _ZN8facebook5velox7process10StackTraceC1Ei 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_ 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_ 3 _ZN7velox4j12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev 6 _ZN7velox4j13MemoryManager11tryDestructEv 7 _ZN7velox4j13MemoryManagerD2Ev 8 _ZNSt10_HashtableIjSt4pairIKjSt10shared_ptrIvEESaIS4_ENSt8__detail10_Select1stESt8equal_toIjESt4hashIjENS6_18_Mod_range_hashingENS6_20_Default_ranged_hashENS6_20_Prime_rehash_policyENS6_17_Hashtable_traitsILb0ELb0ELb1EEEE8_M_eraseEmPNS6_15_Hash_node_baseEPNS6_10_Hash_nodeIS4_Lb0EEE 9 _ZN7velox4j11ResourceMapISt10shared_ptrIvEE5eraseEj 10 _ZN7velox4j11ObjectStore15releaseInternalEj 11 _ZN7velox4j12_GLOBAL__N_116releaseCppObjectEP7JNIEnv_P8_jobjectl 12 0x0000ffff80769cfc
any idea to solve?

shuai-xu · 2025-09-04T01:37:48Z

try to run but it seems to fail due to memory leak error: E20250830 11:49:14.568943 3949495 MemoryManager.cc:264] [Velox4J MemoryManager DTOR] Memory leak found on Velox memory pool: Memory Pool[Decoding Memory Pool LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 128.00MB used 512B available 1023.50KB reservation [used 512B, reserved 1.00MB, min 0B] counters [allocs 4, frees 0, reserves 0, releases 0, collisions 0])>. Please make sure your code released all opened resources already. E20250830 11:49:14.569058 3949495 MemoryManager.cc:264] [Velox4J MemoryManager DTOR] Memory leak found on Velox memory pool: Memory Pool[root AGGREGATE root[root] parent[null] MALLOC track-usage thread-safe]<unlimited max capacity capacity 128.00MB used 512B available 0B reservation [used 0B, reserved 1.00MB, min 0B] counters [allocs 0, frees 0, reserves 0, releases 0, collisions 0])>. Please make sure your code released all opened resources already. E20250830 11:49:14.569092 3949495 MemoryPool.cpp:461] [MEM] Memory leak (Used memory): Memory Pool[Decoding Memory Pool LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 128.00MB used 512B available 1023.50KB reservation [used 512B, reserved 1.00MB, min 0B] counters [allocs 4, frees 0, reserves 0, releases 0, collisions 0])> E20250830 11:49:14.569126 3949495 Exceptions.h:66] Line: velox4j/src/main/cpp/main/velox4j/memory/MemoryManager.cc:92, Function:removePool, Expression: pool->reservedBytes() == 0 (1048576 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError' what(): Exception: VeloxRuntimeError Error Source: RUNTIME Error Code: INVALID_STATE Reason: (1048576 vs. 0) Retriable: False Expression: pool->reservedBytes() == 0 Function: removePool File: velox4j/src/main/cpp/main/velox4j/memory/MemoryManager.cc Line: 92 Stack trace: 0 _ZN8facebook5velox7process10StackTraceC1Ei 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_ 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_ 3 _ZN7velox4j12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev 6 _ZN7velox4j13MemoryManager11tryDestructEv 7 _ZN7velox4j13MemoryManagerD2Ev 8 _ZNSt10_HashtableIjSt4pairIKjSt10shared_ptrIvEESaIS4_ENSt8__detail10_Select1stESt8equal_toIjESt4hashIjENS6_18_Mod_range_hashingENS6_20_Default_ranged_hashENS6_20_Prime_rehash_policyENS6_17_Hashtable_traitsILb0ELb0ELb1EEEE8_M_eraseEmPNS6_15_Hash_node_baseEPNS6_10_Hash_nodeIS4_Lb0EEE 9 _ZN7velox4j11ResourceMapISt10shared_ptrIvEE5eraseEj 10 _ZN7velox4j11ObjectStore15releaseInternalEj 11 _ZN7velox4j12_GLOBAL__N_116releaseCppObjectEP7JNIEnv_P8_jobjectl 12 0x0000ffff80769cfc any idea to solve?

It seems this is not the root reason, the job may fail by some other reason, you can check the log.

philo-he

Looks good. Thanks.

shuai-xu requested a review from philo-he August 27, 2025 03:56

github-actions bot added INFRA FLINK labels Aug 27, 2025

shuai-xu mentioned this pull request Aug 27, 2025

[FLINK] add tests for nexmark q18-21 #10550

Closed

philo-he reviewed Aug 29, 2025

View reviewed changes

shuai-xu added 11 commits September 4, 2025 10:15

[GLUTEN-9939][FLINK] support nexmark q3

4a049ee

[GLUTEN-10002] support nexmark q4

c160832

support to q9

bdac73a

remove useless header

aebb22c

support state

952fe85

nexmark pass

5396243

[Gluten-10317][FLINK] Support state related operation

d1b0e49

rollback useless change

520642b

refine window util

8fe270b

[GLUTEN-10546][FLINK] Support all flink operators for nexmark

e3e3411

remove println

e200314

shuai-xu force-pushed the q9 branch from 7f74ae7 to e200314 Compare September 4, 2025 02:20

rollback useless changes

9e2cf65

philo-he approved these changes Sep 4, 2025

View reviewed changes

shuai-xu merged commit be9f4f6 into apache:main Sep 4, 2025
4 checks passed

shuai-xu deleted the q9 branch September 5, 2025 08:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-10546][FLINK] Support all flink operators for nexmark#10548

[GLUTEN-10546][FLINK] Support all flink operators for nexmark#10548
shuai-xu merged 12 commits intoapache:mainfrom
bigo-sg:q9

shuai-xu commented Aug 27, 2025

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

philo-he left a comment

Uh oh!

Uh oh!

philo-he Aug 29, 2025

Uh oh!

philo-he Aug 29, 2025

Uh oh!

shuai-xu Sep 4, 2025

Uh oh!

Uh oh!

ryyyyyy1 commented Aug 30, 2025

Uh oh!

shuai-xu commented Sep 4, 2025

Uh oh!

philo-he left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shuai-xu commented Aug 27, 2025

What changes are proposed in this pull request?

How was this patch tested?

Uh oh!

github-actions bot commented Aug 27, 2025

Uh oh!

philo-he left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

philo-he Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

philo-he Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

shuai-xu Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryyyyyy1 commented Aug 30, 2025

Uh oh!

shuai-xu commented Sep 4, 2025

Uh oh!

philo-he left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants