Skip to content

fallback queryNumber() if there are uncommitted records#2060

Merged
imbajin merged 3 commits intomasterfrom
fallback-queryNumber-with-uncommitted
Dec 29, 2022
Merged

fallback queryNumber() if there are uncommitted records#2060
imbajin merged 3 commits intomasterfrom
fallback-queryNumber-with-uncommitted

Conversation

@javeme
Copy link
Copy Markdown
Contributor

@javeme javeme commented Dec 22, 2022

fixed sub-issue 1 of #2058

@javeme javeme mentioned this pull request Dec 22, 2022
1 task
@imbajin
Copy link
Copy Markdown
Member

imbajin commented Dec 23, 2022

image

@imbajin imbajin added ci-cd Build or deploy tests Add or improve test cases labels Dec 23, 2022
@imbajin imbajin added this to the 1.0.0 milestone Dec 23, 2022
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 26, 2022

Codecov Report

Merging #2060 (973e6b0) into master (6c3170c) will decrease coverage by 0.01%.
The diff coverage is 75.86%.

@@             Coverage Diff              @@
##             master    #2060      +/-   ##
============================================
- Coverage     68.36%   68.35%   -0.02%     
- Complexity      977      979       +2     
============================================
  Files           481      481              
  Lines         39807    39821      +14     
  Branches       5592     5598       +6     
============================================
+ Hits          27216    27221       +5     
- Misses         9952     9957       +5     
- Partials       2639     2643       +4     
Impacted Files Coverage Δ
.../apache/hugegraph/backend/tx/GraphTransaction.java 79.87% <74.07%> (-0.27%) ⬇️
...java/org/apache/hugegraph/backend/query/Query.java 86.56% <100.00%> (+0.10%) ⬆️
...hugegraph/backend/store/raft/RaftBackendStore.java 73.11% <0.00%> (-2.16%) ⬇️
.../hugegraph/backend/store/rocksdb/RocksDBTable.java 67.11% <0.00%> (-2.02%) ⬇️
...hugegraph/backend/store/raft/rpc/RpcForwarder.java 69.49% <0.00%> (-1.70%) ⬇️
...ugegraph/backend/store/raft/StoreStateMachine.java 64.86% <0.00%> (-0.91%) ⬇️
...a/org/apache/hugegraph/util/collection/IntSet.java 73.72% <0.00%> (-0.85%) ⬇️
...g/apache/hugegraph/backend/store/BackendTable.java 85.11% <0.00%> (+0.59%) ⬆️
...n/java/org/apache/hugegraph/schema/IndexLabel.java 77.27% <0.00%> (+1.13%) ⬆️
...e/hugegraph/backend/tx/SchemaIndexTransaction.java 79.54% <0.00%> (+6.81%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@javeme
Copy link
Copy Markdown
Contributor Author

javeme commented Dec 26, 2022

cassandra can't run on mac m1:

bin/cassandraapache-cassandra-3.11.14 [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:bin/../logs/gc.log instead.
intx ThreadPriorityPolicy=42 is outside the allowed range [ 0 ... 1 ]
Improperly specified VM option 'ThreadPriorityPolicy=42'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
bin/cassandra            
                                                                                                                                                                      
The stack size specified is too small, Specify at least 640k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
➜  apache-cassandra-3.11.14 vim conf/jvm.options
ERROR [main] 2022-12-26 23:40:03,898 NativeLibraryDarwin.java:64 - Failed to link the C library against JNA. Native methods will be unavailable.
java.lang.UnsatisfiedLinkError: /private/var/folders/3q/9d9hmtzd6c397wsb1fchsvyh0000gn/T/jna-101015171/jna6875563382134887427.tmp: dlopen(/private/var/folders/3q/9d9hmtzd6c397wsb1fchsvyh0000gn/T/jna-101015171/jna6875563382134887427.tmp, 0x0001): tried: '/private/var/folders/3q/9d9hmtzd6c397wsb1fchsvyh0000gn/T/jna-101015171/jna6875563382134887427.tmp' (fat file, but missing compatible architecture (have (i386,x86_64), need (arm64e)))
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
	at java.lang.Runtime.load0(Runtime.java:783)
	at java.lang.System.load(System.java:1100)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826)
	at com.sun.jna.Native.<clinit>(Native.java:140)
	at com.sun.jna.NativeLibrary.<clinit>(NativeLibrary.java:84)
	at org.apache.cassandra.utils.NativeLibraryDarwin.<clinit>(NativeLibraryDarwin.java:55)
	at org.apache.cassandra.utils.NativeLibrary.<clinit>(NativeLibrary.java:90)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:212)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786)
WARN  [main] 2022-12-26 23:40:03,899 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN  [main] 2022-12-26 23:40:03,899 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR [main] 2022-12-26 23:40:03,899 CassandraDaemon.java:803 - The native library could not be initialized properly.

@imbajin
Copy link
Copy Markdown
Member

imbajin commented Dec 27, 2022

some simple ways to handle this problem:

  1. replace a JNA jar in 3.x - refer how-to-start-cassandra-on-a-m1-macbook
  2. use 4.x directly (seems no compatible problem)
  3. use 3.x with apple transfer compile

@imbajin
Copy link
Copy Markdown
Member

imbajin commented Dec 27, 2022

cassandra can't run on mac m1:

bin/cassandraapache-cassandra-3.11.14 [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:bin/../logs/gc.log instead.
intx ThreadPriorityPolicy=42 is outside the allowed range [ 0 ... 1 ]
Improperly specified VM option 'ThreadPriorityPolicy=42'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
bin/cassandra            
                                                                                                                                                                      
The stack size specified is too small, Specify at least 640k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
➜  apache-cassandra-3.11.14 vim conf/jvm.options
ERROR [main] 2022-12-26 23:40:03,898 NativeLibraryDarwin.java:64 - Failed to link the C library against JNA. Native methods will be unavailable.
java.lang.UnsatisfiedLinkError: /private/var/folders/3q/9d9hmtzd6c397wsb1fchsvyh0000gn/T/jna-101015171/jna6875563382134887427.tmp: dlopen(/private/var/folders/3q/9d9hmtzd6c397wsb1fchsvyh0000gn/T/jna-101015171/jna6875563382134887427.tmp, 0x0001): tried: '/private/var/folders/3q/9d9hmtzd6c397wsb1fchsvyh0000gn/T/jna-101015171/jna6875563382134887427.tmp' (fat file, but missing compatible architecture (have (i386,x86_64), need (arm64e)))
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
	at java.lang.Runtime.load0(Runtime.java:783)
	at java.lang.System.load(System.java:1100)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826)
	at com.sun.jna.Native.<clinit>(Native.java:140)
	at com.sun.jna.NativeLibrary.<clinit>(NativeLibrary.java:84)
	at org.apache.cassandra.utils.NativeLibraryDarwin.<clinit>(NativeLibraryDarwin.java:55)
	at org.apache.cassandra.utils.NativeLibrary.<clinit>(NativeLibrary.java:90)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:212)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786)
WARN  [main] 2022-12-26 23:40:03,899 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN  [main] 2022-12-26 23:40:03,899 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR [main] 2022-12-26 23:40:03,899 CassandraDaemon.java:803 - The native library could not be initialized properly.

some simple ways to handle this problem:

  1. replace a JNA jar in 3.x - refer how-to-start-cassandra-on-a-m1-macbook
  2. use 4.x directly (seems no compatible problem)
  3. use 3.x with apple transfer compile

@zyxxoo
Copy link
Copy Markdown
Contributor

zyxxoo commented Dec 28, 2022

这里 select count(*) 的结果,在解析的时候 count 被视为column,但是我们在 hugeKey 中没有定义

row2Entry:429, CassandraTable (org.apache.hugegraph.backend.store.cassandra)
lambda$results2Entries$5:418, CassandraTable (org.apache.hugegraph.backend.store.cassandra)
apply:-1, 646504864 (org.apache.hugegraph.backend.store.cassandra.CassandraTable$$Lambda$192)
fetch:115, CassandraEntryIterator (org.apache.hugegraph.backend.store.cassandra)
hasNext:62, BackendEntryIterator (org.apache.hugegraph.backend.store)
fetch:86, ExtendableIterator (org.apache.hugegraph.iterator)
hasNext:40, WrappedIterator (org.apache.hugegraph.iterator)
lambda$flatMap$0:226, QueryResults (org.apache.hugegraph.backend.query)
apply:-1, 253709867 (org.apache.hugegraph.backend.query.QueryResults$$Lambda$249)
fetch:62, FlatMapperIterator (org.apache.hugegraph.iterator)
hasNext:40, WrappedIterator (org.apache.hugegraph.iterator)
fetch:42, MapperIterator (org.apache.hugegraph.iterator)
hasNext:40, WrappedIterator (org.apache.hugegraph.iterator)
keepInputOrderIfNeeded:103, QueryResults (org.apache.hugegraph.backend.query)
queryVerticesFromBackend:821, GraphTransaction (org.apache.hugegraph.backend.tx)
queryVerticesFromBackend:243, CachedGraphTransaction (org.apache.hugegraph.backend.cache)
queryVertices:801, GraphTransaction (org.apache.hugegraph.backend.tx)
lambda$queryNumber$1:583, GraphTransaction (org.apache.hugegraph.backend.tx)
apply:-1, 984832924 (org.apache.hugegraph.backend.tx.GraphTransaction$$Lambda$266)
iterator:161, QueryList$OptimizedQuery (org.apache.hugegraph.backend.page)
apply:-1, 1055453694 (org.apache.hugegraph.backend.page.QueryList$$Lambda$248)
lambda$flatMap$0:225, QueryResults (org.apache.hugegraph.backend.query)
apply:-1, 253709867 (org.apache.hugegraph.backend.query.QueryResults$$Lambda$249)
fetch:62, FlatMapperIterator (org.apache.hugegraph.iterator)
hasNext:40, WrappedIterator (org.apache.hugegraph.iterator)
reduce:93, Aggregate$AggregateFunc (org.apache.hugegraph.backend.query)
reduce:50, Aggregate (org.apache.hugegraph.backend.query)
queryNumber:593, GraphTransaction (org.apache.hugegraph.backend.tx)
queryNumber:703, StandardHugeGraph (org.apache.hugegraph)
verticesCount:91, HugeGraphStep (org.apache.hugegraph.traversal.optimize)
count:80, HugeGraphStep (org.apache.hugegraph.traversal.optimize)
processNextStart:77, HugeCountStep (org.apache.hugegraph.traversal.optimize)
next:135, AbstractStep (org.apache.tinkerpop.gremlin.process.traversal.step.util)
next:40, AbstractStep (org.apache.tinkerpop.gremlin.process.traversal.step.util)
next:230, DefaultTraversal (org.apache.tinkerpop.gremlin.process.traversal.util)
testQueryCountWithOptimizeByIndex:5793, VertexCoreTest (org.apache.hugegraph.core)
invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:62, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
invoke:566, Method (java.lang.reflect)
runReflectiveCall:50, FrameworkMethod$1 (org.junit.runners.model)
run:12, ReflectiveCallable (org.junit.internal.runners.model)
invokeExplosively:47, FrameworkMethod (org.junit.runners.model)
evaluate:17, InvokeMethod (org.junit.internal.runners.statements)
evaluate:26, RunBefores (org.junit.internal.runners.statements)
evaluate:27, RunAfters (org.junit.internal.runners.statements)
runLeaf:325, ParentRunner (org.junit.runners)
runChild:78, BlockJUnit4ClassRunner (org.junit.runners)
runChild:57, BlockJUnit4ClassRunner (org.junit.runners)
run:290, ParentRunner$3 (org.junit.runners)
schedule:71, ParentRunner$1 (org.junit.runners)
runChildren:288, ParentRunner (org.junit.runners)
access$000:58, ParentRunner (org.junit.runners)
evaluate:268, ParentRunner$2 (org.junit.runners)
evaluate:26, RunBefores (org.junit.internal.runners.statements)
evaluate:27, RunAfters (org.junit.internal.runners.statements)
run:363, ParentRunner (org.junit.runners)
runChild:128, Suite (org.junit.runners)
runChild:27, Suite (org.junit.runners)
run:290, ParentRunner$3 (org.junit.runners)
schedule:71, ParentRunner$1 (org.junit.runners)
runChildren:288, ParentRunner (org.junit.runners)
access$000:58, ParentRunner (org.junit.runners)
evaluate:268, ParentRunner$2 (org.junit.runners)
run:363, ParentRunner (org.junit.runners)
run:137, JUnitCore (org.junit.runner)
startRunnerWithArgs:69, JUnit4IdeaTestRunner (com.intellij.junit4)
execute:38, IdeaTestRunner$Repeater$1 (com.intellij.rt.junit)
repeat:11, TestsRepeater (com.intellij.rt.execution.junit)
startRunnerWithArgs:35, IdeaTestRunner$Repeater (com.intellij.rt.junit)
prepareStreamsAndStart:235, JUnitStarter (com.intellij.rt.junit)
main:54, JUnitStarter (com.intellij.rt.junit)

@zyxxoo
Copy link
Copy Markdown
Contributor

zyxxoo commented Dec 28, 2022

以前是走 queryNumber:

lambda$queryNumber$1:121, CassandraTable (org.apache.hugegraph.backend.store.cassandra)
apply:-1, 417160915 (org.apache.hugegraph.backend.store.cassandra.CassandraTable$$Lambda$266)
query:152, CassandraTable (org.apache.hugegraph.backend.store.cassandra)
queryNumber:117, CassandraTable (org.apache.hugegraph.backend.store.cassandra)
queryNumber:333, CassandraStore (org.apache.hugegraph.backend.store.cassandra)
queryNumber:131, AbstractTransaction (org.apache.hugegraph.backend.tx)
lambda$queryNumber$1:559, GraphTransaction (org.apache.hugegraph.backend.tx)
apply:-1, 265556987 (org.apache.hugegraph.backend.tx.GraphTransaction$$Lambda$269)
iterator:161, QueryList$OptimizedQuery (org.apache.hugegraph.backend.page)
apply:-1, 1962610818 (org.apache.hugegraph.backend.page.QueryList$$Lambda$247)
lambda$flatMap$0:225, QueryResults (org.apache.hugegraph.backend.query)
apply:-1, 1695173607 (org.apache.hugegraph.backend.query.QueryResults$$Lambda$248)
fetch:62, FlatMapperIterator (org.apache.hugegraph.iterator)
hasNext:40, WrappedIterator (org.apache.hugegraph.iterator)
reduce:93, Aggregate$AggregateFunc (org.apache.hugegraph.backend.query)
reduce:50, Aggregate (org.apache.hugegraph.backend.query)
queryNumber:583, GraphTransaction (org.apache.hugegraph.backend.tx)
queryNumber:703, StandardHugeGraph (org.apache.hugegraph)
verticesCount:91, HugeGraphStep (org.apache.hugegraph.traversal.optimize)
count:80, HugeGraphStep (org.apache.hugegraph.traversal.optimize)
processNextStart:77, HugeCountStep (org.apache.hugegraph.traversal.optimize)
next:135, AbstractStep (org.apache.tinkerpop.gremlin.process.traversal.step.util)
next:40, AbstractStep (org.apache.tinkerpop.gremlin.process.traversal.step.util)
next:230, DefaultTraversal (org.apache.tinkerpop.gremlin.process.traversal.util)
testQueryCountWithOptimizeByIndex:5783, VertexCoreTest (org.apache.hugegraph.core)
invoke0:-1, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:62, NativeMethodAccessorImpl (jdk.internal.reflect)
invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
invoke:566, Method (java.lang.reflect)
runReflectiveCall:50, FrameworkMethod$1 (org.junit.runners.model)
run:12, ReflectiveCallable (org.junit.internal.runners.model)
invokeExplosively:47, FrameworkMethod (org.junit.runners.model)
evaluate:17, InvokeMethod (org.junit.internal.runners.statements)
evaluate:26, RunBefores (org.junit.internal.runners.statements)
evaluate:27, RunAfters (org.junit.internal.runners.statements)
runLeaf:325, ParentRunner (org.junit.runners)
runChild:78, BlockJUnit4ClassRunner (org.junit.runners)
runChild:57, BlockJUnit4ClassRunner (org.junit.runners)
run:290, ParentRunner$3 (org.junit.runners)
schedule:71, ParentRunner$1 (org.junit.runners)
runChildren:288, ParentRunner (org.junit.runners)
access$000:58, ParentRunner (org.junit.runners)
evaluate:268, ParentRunner$2 (org.junit.runners)
evaluate:26, RunBefores (org.junit.internal.runners.statements)
evaluate:27, RunAfters (org.junit.internal.runners.statements)
run:363, ParentRunner (org.junit.runners)
runChild:128, Suite (org.junit.runners)
runChild:27, Suite (org.junit.runners)
run:290, ParentRunner$3 (org.junit.runners)
schedule:71, ParentRunner$1 (org.junit.runners)
runChildren:288, ParentRunner (org.junit.runners)
access$000:58, ParentRunner (org.junit.runners)
evaluate:268, ParentRunner$2 (org.junit.runners)
run:363, ParentRunner (org.junit.runners)
run:137, JUnitCore (org.junit.runner)
startRunnerWithArgs:69, JUnit4IdeaTestRunner (com.intellij.junit4)
execute:38, IdeaTestRunner$Repeater$1 (com.intellij.rt.junit)
repeat:11, TestsRepeater (com.intellij.rt.execution.junit)
startRunnerWithArgs:35, IdeaTestRunner$Repeater (com.intellij.rt.junit)
prepareStreamsAndStart:235, JUnitStarter (com.intellij.rt.junit)
main:54, JUnitStarter (com.intellij.rt.junit)

@javeme javeme force-pushed the fallback-queryNumber-with-uncommitted branch from e769b85 to a022486 Compare December 28, 2022 13:29
Change-Id: Icefb8089c9ef5231a9bf4b1e5e6e359935feda51
Change-Id: I4a08d6ee3cdc4c914e02d27396bda488c424f136
@javeme javeme force-pushed the fallback-queryNumber-with-uncommitted branch from a022486 to 973e6b0 Compare December 28, 2022 13:33
@imbajin imbajin merged commit eb7a65c into master Dec 29, 2022
@imbajin imbajin deleted the fallback-queryNumber-with-uncommitted branch December 29, 2022 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd Build or deploy tests Add or improve test cases

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants