Use join instead of views #8321

vishesh92 · 2023-12-06T16:45:22Z

Description

To be merged after #8410 to ensure this doesn't break anything.

This PR contains changes similar to #8012 which is to directly query the tables and make required joins only when there is a requirement to filter on some other field. This will help reduce the query times and thus reduce load on the database as well.

Resource	Count	Before	After	Difference
volumes	748	50	47	3
networks	191	140	131	9
events	7515	61	31	30
accounts	52	100	94	6
domains	51	35	32	3
hosts	101	306	163	143
storage pools	100	41	40	1
service offerings	603	53	41	12
disk offerings	705	48	37	11
templates	603	155	109	46

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2023-12-06T16:54:44Z

Codecov Report

Attention: Patch coverage is 68.29522% with 305 lines in your changes are missing coverage. Please review.

Project coverage is 30.85%. Comparing base (0208e09) to head (61f1382).
Report is 5 commits behind head on 4.19.

Files	Patch %	Lines
...ain/java/com/cloud/api/query/QueryManagerImpl.java	70.45%	138 Missing and 26 partials ⚠️
...c/main/java/com/cloud/utils/db/GenericDaoBase.java	70.66%	16 Missing and 6 partials ⚠️
...b/src/main/java/com/cloud/utils/db/SearchBase.java	47.36%	9 Missing and 11 partials ⚠️
...va/com/cloud/api/query/dao/AccountJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
...m/cloud/api/query/dao/DiskOfferingJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
...ava/com/cloud/api/query/dao/DomainJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
...loud/api/query/dao/ServiceOfferingJoinDaoImpl.java	53.12%	11 Missing and 4 partials ⚠️
.../src/main/java/com/cloud/utils/db/JoinBuilder.java	51.72%	14 Missing ⚠️
...ava/org/apache/cloudstack/acl/RoleManagerImpl.java	56.25%	4 Missing and 3 partials ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	92.98%	2 Missing and 2 partials ⚠️
... and 8 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8321      +/-   ##
============================================
- Coverage     30.91%   30.85%   -0.07%     
+ Complexity    34249    34183      -66     
============================================
  Files          5354     5355       +1     
  Lines        376094   376619     +525     
  Branches      54696    54809     +113     
============================================
- Hits         116258   116194      -64     
- Misses       244542   245131     +589     
  Partials      15294    15294

Flag	Coverage Δ
simulator-marvin-tests	`24.64% <68.19%> (-0.09%)`	⬇️
uitests	`4.39% <ø> (ø)`
unit-tests	`16.56% <5.19%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-12-10T06:51:42Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java

github-actions · 2023-12-14T08:00:18Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

engine/schema/src/main/java/com/cloud/service/ServiceOfferingVO.java

vishesh92 · 2024-01-03T13:48:12Z

@blueorangutan package

blueorangutan · 2024-01-03T13:50:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

vishesh92 · 2024-01-04T06:47:45Z

@blueorangutan package

blueorangutan · 2024-01-04T06:48:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-04T08:01:23Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8196

harikrishna-patnala · 2024-01-09T09:18:48Z

@blueorangutan test matrix

blueorangutan · 2024-01-09T09:20:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-01-09T22:03:38Z

[SF] Trillian test result (tid-8765)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44377 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t8765-xenserver-71.zip
Smoke tests completed. 119 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_04_change_offering_small	`Error`	100.44	test_service_offerings.py
test_06_disk_offering_strictness_false	`Error`	148.42	test_service_offerings.py
test_change_service_offering_for_vm_with_snapshots	`Error`	98.27	test_vm_snapshots.py
test_01_create_vm_snapshots	`Error`	90.24	test_vm_snapshots.py
test_02_revert_vm_snapshots	`Failure`	90.12	test_vm_snapshots.py
test_03_delete_vm_snapshots	`Failure`	0.01	test_vm_snapshots.py

blueorangutan · 2024-01-09T23:25:18Z

[SF] Trillian test result (tid-8766)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49277 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t8766-vmware-67u3.zip
Smoke tests completed. 115 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_root_disk	`Error`	0.07	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:setup	`Error`	213.28	test_snapshots.py
test_01_snapshot_usage	`Error`	0.07	test_usage.py
test_02_balanced_drs_algorithm	`Failure`	130.02	test_cluster_drs.py
ContextSuite context=TestListIdsParams>:setup	`Error`	0.00	test_list_ids_parameter.py
test_04_change_offering_small	`Error`	97.46	test_service_offerings.py
test_06_disk_offering_strictness_false	`Error`	150.06	test_service_offerings.py
test_change_service_offering_for_vm_with_snapshots	`Error`	144.69	test_vm_snapshots.py
test_01_create_vm_snapshots	`Error`	123.73	test_vm_snapshots.py
test_02_revert_vm_snapshots	`Failure`	90.22	test_vm_snapshots.py
test_03_delete_vm_snapshots	`Failure`	0.02	test_vm_snapshots.py

vishesh92 · 2024-01-10T08:58:36Z

@blueorangutan package

blueorangutan · 2024-01-10T09:04:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

vishesh92 · 2024-01-11T05:18:12Z

@blueorangutan test matrix

blueorangutan · 2024-01-11T05:20:03Z

@vishesh92 a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-01-11T18:04:58Z

[SF] Trillian test result (tid-8788)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44397 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t8788-xenserver-71.zip
Smoke tests completed. 118 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	341.81	test_safe_shutdown.py
test_04_change_offering_small	`Error`	100.42	test_service_offerings.py
test_06_disk_offering_strictness_false	`Error`	158.00	test_service_offerings.py
test_change_service_offering_for_vm_with_snapshots	`Error`	211.21	test_vm_snapshots.py

vishesh92 · 2024-02-07T07:30:03Z

@blueorangutan package

blueorangutan · 2024-02-07T07:32:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

github-actions · 2024-02-13T09:32:10Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…iews

weizhouapache · 2024-03-13T12:22:36Z

@blueorangutan package

weizhouapache

code lgtm

yadvr · 2024-03-14T07:36:10Z

@blueorangutan package

blueorangutan · 2024-03-14T07:38:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-14T08:39:01Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8932

vishesh92 · 2024-03-14T09:56:00Z

@blueorangutan package

blueorangutan · 2024-03-14T09:58:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-14T10:54:17Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8935

yadvr · 2024-03-14T12:21:38Z

@blueorangutan test matrix

blueorangutan · 2024-03-14T12:22:03Z

@rohityadavcloud a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-03-14T23:54:49Z

[SF] Trillian test result (tid-9482)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40129 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t9482-xenserver-71.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-03-15T01:32:59Z

[SF] Trillian test result (tid-9483)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 45959 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t9483-vmware-67u3.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-03-15T02:03:38Z

[SF] Trillian test result (tid-9484)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47737 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8321-t9484-kvm-centos7.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.69	test_safe_shutdown.py

vladimirpetrov · 2024-03-18T15:28:46Z

Tested on a simulator hypervisor, with a separate MS (4 CPUs, 8 GB RAM) and DB VM (8 CPUs, 16 GB RAM), using Apache Benchmark tool (ab), measuring mean response time from 100 executions per list operation.

Number of resources:
Hosts: 1001
Storage pools: 100
Volumes: 25110
Events: 155118
Instances: 8370

Results, mean time in milliseconds

list hosts

	Page=20	Page=50	Page=100
Unpatched	304	676	1052
Patched	259	647	1246
Difference	-15%	-4%	18%

list storage pools

	Page=20	Page=50	Page=100
Unpatched	204	487	972
Patched	193	381	843
Difference	-5%	-22%	-13%

list volumes

	Page=20	Page=50	Page=100
Unpatched	663	878	1205
Patched	389	558	961
Difference	-41%	-36%	-20%

list volume metrics

	Page=20	Page=50	Page=100
Unpatched	699	1057	1329
Patched	374	732	1299
Difference	-46%	-31%	-2%

list events

	Page=20	Page=50	Page=100
Unpatched	2609	2750	2748
Patched	396	522	457
Difference	-85%	-81%	-83%

list instance metrics

	Page=20	Page=50	Page=100
Unpatched	914	1804	3273
Patched	898	1650	3255
Difference	-2%	-9%	-1%

vladimirpetrov

LGTM based on performance measurements with and without the patch

weizhouapache · 2024-03-18T17:07:48Z

Merging based on approvals and CI/manual test results
Thanks @vishesh92 @vladimirpetrov

boring-cyborg bot added component:database component:storage labels Dec 6, 2023

vishesh92 force-pushed the use-join-instead-of-views branch from f56421f to 91eef99 Compare December 6, 2023 17:03

github-actions bot added the status:has-conflicts label Dec 10, 2023

vishesh92 force-pushed the use-join-instead-of-views branch from ee2388b to 66bb65c Compare December 11, 2023 05:43

github-actions bot removed the status:has-conflicts label Dec 11, 2023

boring-cyborg bot added the component:api label Dec 13, 2023

DaanHoogland reviewed Dec 13, 2023

View reviewed changes

server/src/main/java/com/cloud/api/query/QueryManagerImpl.java Outdated Show resolved Hide resolved

github-actions bot added the status:has-conflicts label Dec 14, 2023

github-actions bot removed the status:has-conflicts label Dec 15, 2023

vishesh92 marked this pull request as ready for review December 15, 2023 09:51

vishesh92 commented Dec 15, 2023

View reviewed changes

engine/schema/src/main/java/com/cloud/service/ServiceOfferingVO.java Outdated Show resolved Hide resolved

boring-cyborg bot added the component:networking label Dec 15, 2023

github-actions bot added the status:has-conflicts label Feb 13, 2024

Merge remote-tracking branch 'origin/4.19' into use-join-instead-of-v…

5a6bd99

…iews

github-actions bot removed the status:has-conflicts label Feb 13, 2024

vishesh92 mentioned this pull request Feb 21, 2024

list by isEncrypted #8643

Merged

13 tasks

weizhouapache approved these changes Mar 13, 2024

View reviewed changes

yadvr approved these changes Mar 14, 2024

View reviewed changes

Merge branch '4.19' into use-join-instead-of-views

61f1382

DaanHoogland added the status:needs-testing label Mar 15, 2024

vladimirpetrov approved these changes Mar 18, 2024

View reviewed changes

weizhouapache merged commit 0043540 into apache:4.19 Mar 18, 2024

vishesh92 deleted the use-join-instead-of-views branch March 19, 2024 07:36

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Mar 21, 2024

Use join instead of views (apache#8321)

7d442b7

gpordeus mentioned this pull request Jun 18, 2024

[UI] Add option to select owner of volume when creating it #9265

Merged

8 tasks

Use join instead of views #8321

Use join instead of views #8321

Uh oh!

Conversation

vishesh92 commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Dec 10, 2023

Uh oh!

Uh oh!

github-actions bot commented Dec 14, 2023

Uh oh!

Uh oh!

vishesh92 commented Jan 3, 2024

Uh oh!

blueorangutan commented Jan 3, 2024

Uh oh!

vishesh92 commented Jan 4, 2024

Uh oh!

blueorangutan commented Jan 4, 2024

Uh oh!

blueorangutan commented Jan 4, 2024

Uh oh!

harikrishna-patnala commented Jan 9, 2024

Uh oh!

blueorangutan commented Jan 9, 2024

Uh oh!

blueorangutan commented Jan 9, 2024

Uh oh!

blueorangutan commented Jan 9, 2024

Uh oh!

vishesh92 commented Jan 10, 2024

Uh oh!

blueorangutan commented Jan 10, 2024

Uh oh!

vishesh92 commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

blueorangutan commented Jan 11, 2024

Uh oh!

vishesh92 commented Feb 7, 2024

Uh oh!

blueorangutan commented Feb 7, 2024

Uh oh!

github-actions bot commented Feb 13, 2024

Uh oh!

weizhouapache commented Mar 13, 2024

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Mar 14, 2024

Uh oh!

blueorangutan commented Mar 14, 2024

Uh oh!

blueorangutan commented Mar 14, 2024

Uh oh!

vishesh92 commented Mar 14, 2024

Uh oh!

blueorangutan commented Mar 14, 2024

Uh oh!

blueorangutan commented Mar 14, 2024

Uh oh!

yadvr commented Mar 14, 2024

Uh oh!

blueorangutan commented Mar 14, 2024

Uh oh!

vishesh92 commented Dec 6, 2023 •

edited

Loading

codecov bot commented Dec 6, 2023 •

edited

Loading