Skip to content

Conversation

@rtibbles
Copy link
Member

Summary

  • I had noticed this a while ago, and I think even mentioned it on slack, but never actioned it - the sync session instance ids are not properly deduplicated in our distinct query, so we end up making repeated requests to network clients even when we know they're not available
  • Fixes this by using list/set to deduplicate the instance_ids.

Reviewer guidance

Without this change, I see this every time I run my devserver:

DEBUG    2025-08-11 12:46:04,644 Dispatching on_connect hooks for network location 9328c91cb0c34664a631892452d19ea5
DEBUG    2025-08-11 12:46:04,740 Processing content requests
DEBUG    2025-08-11 12:46:04,759 Attempting to import missing metadata before content import
DEBUG    2025-08-11 12:46:04,778 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,780 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,791 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,793 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,797 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,800 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,801 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,804 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,806 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,807 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,810 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,811 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,813 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,815 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,817 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,819 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,820 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,821 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,823 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,824 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,914 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
INFO     2025-08-11 12:46:04,919 No acceptable peer device for importing content metadata for 62 nodes
DEBUG    2025-08-11 12:46:04,920 Starting automated import of content
INFO     2025-08-11 12:46:04,946 Processing content import request for node 80a9be1687bc58f5a0f51f674d612d3c
DEBUG    2025-08-11 12:46:04,968 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,971 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,982 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,985 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,988 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,990 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,991 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:04,993 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,039 Dispatching on_disconnect hooks for network location 9328c91cb0c34664a631892452d19ea5
DEBUG    2025-08-11 12:46:05,051 Dispatching on_disconnect hooks for network location e7c2973b9da99351f5e03dcf2179be41
DEBUG    2025-08-11 12:46:05,181 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
WARNING  2025-08-11 12:46:05,183 Unable to import 80a9be1687bc58f5a0f51f674d612d3c from peers
DEBUG    2025-08-11 12:46:05,222 Checking connection status for network location 9328c91cb0c34664a631892452d19ea5
INFO     2025-08-11 12:46:05,222 Attempting connection to: https://kolibri-demo.learningequality.org/
DEBUG    2025-08-11 12:46:05,227 Starting new HTTPS connection (1): kolibri-demo.learningequality.org:443
DEBUG    2025-08-11 12:46:05,541 Received ADD event for Zeroconf service: e7c2973b9da99351f5e03dcf2179be41.Kolibri._sub._http._tcp.local.
DEBUG    2025-08-11 12:46:05,752 https://kolibri-demo.learningequality.org:443 "GET /api/public/info/?v=3 HTTP/1.1" 200 None
INFO     2025-08-11 12:46:05,753 Success! We connected to: https://kolibri-demo.learningequality.org/api/public/info/?v=3
DEBUG    2025-08-11 12:46:05,763 Dispatching on_connect hooks for network location 9328c91cb0c34664a631892452d19ea5
DEBUG    2025-08-11 12:46:05,864 Processing content requests
DEBUG    2025-08-11 12:46:05,881 Attempting to import missing metadata before content import
DEBUG    2025-08-11 12:46:05,916 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,922 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
INFO     2025-08-11 12:46:05,930 Kolibri instance 'e7c2973b9da99351f5e03dcf2179be41' joined zeroconf network; device info: {'application': 'kolibri', 'kolibri_version': '0.19.0a0.dev0+git.169.g19b2f287', 'instance_id': 'e7c2973b9da99351f5e03dcf2179be41', 'device_name': 'Alex dsafasdf', 'operating_system': 'Linux', 'subset_of_users_device': False, 'min_content_schema_version': '5'}
DEBUG    2025-08-11 12:46:05,964 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,970 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,977 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,980 Received UPDATE event for Zeroconf service: e7c2973b9da99351f5e03dcf2179be41.Kolibri._sub._http._tcp.local.
DEBUG    2025-08-11 12:46:05,983 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,988 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:05,997 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,004 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,013 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,025 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,029 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,033 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,038 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,049 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,052 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,123 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,135 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,141 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,145 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,254 Creating `DynamicNetworkLocation` for instance e7c2973b9da99351f5e03dcf2179be41
DEBUG    2025-08-11 12:46:06,414 Enqueuing connection check for network location e7c2973b9da99351f5e03dcf2179be41
DEBUG    2025-08-11 12:46:06,563 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
INFO     2025-08-11 12:46:06,568 No acceptable peer device for importing content metadata for 62 nodes
DEBUG    2025-08-11 12:46:06,568 Starting automated import of content
INFO     2025-08-11 12:46:06,600 Processing content import request for node 80a9be1687bc58f5a0f51f674d612d3c
DEBUG    2025-08-11 12:46:06,672 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,676 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,713 Checking connection status for network location e7c2973b9da99351f5e03dcf2179be41
INFO     2025-08-11 12:46:06,714 Attempting connection to: http://172.27.216.170:8000/
DEBUG    2025-08-11 12:46:06,718 Starting new HTTP connection (1): 172.27.216.170:8000
DEBUG    2025-08-11 12:46:06,733 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,739 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,745 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,749 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,752 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,756 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,760 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,766 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,771 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,776 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,779 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,784 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,788 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,792 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,797 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,804 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,808 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,813 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:06,942 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
WARNING  2025-08-11 12:46:06,946 Unable to import 80a9be1687bc58f5a0f51f674d612d3c from peers
INFO     2025-08-11 12:46:07,042 172.27.216.170 - - "GET /api/public/info/" 200 0 "" "Kolibri/0.19.0a0.dev0+git.169.g19b2f287 python-requests/2.27.1"
DEBUG    2025-08-11 12:46:07,123 http://172.27.216.170:8000 "GET /api/public/info/?v=3 HTTP/1.1" 200 247
INFO     2025-08-11 12:46:07,124 Success! We connected to: http://172.27.216.170:8000/api/public/info/?v=3
DEBUG    2025-08-11 12:46:07,129 Dispatching on_connect hooks for network location e7c2973b9da99351f5e03dcf2179be41
DEBUG    2025-08-11 12:46:07,331 Processing content requests
DEBUG    2025-08-11 12:46:07,353 Attempting to import missing metadata before content import
DEBUG    2025-08-11 12:46:07,379 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,380 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,390 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,391 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,395 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,396 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,398 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,399 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,401 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,403 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,405 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,407 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,409 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,410 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,412 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,414 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,416 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,417 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,419 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,420 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,511 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
INFO     2025-08-11 12:46:07,515 No acceptable peer device for importing content metadata for 62 nodes
DEBUG    2025-08-11 12:46:07,515 Starting automated import of content
INFO     2025-08-11 12:46:07,534 Processing content import request for node 80a9be1687bc58f5a0f51f674d612d3c
DEBUG    2025-08-11 12:46:07,545 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,547 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,555 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,556 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,558 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,559 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,561 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,562 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,564 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,565 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,567 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,568 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,570 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,571 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,573 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,574 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,576 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,578 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,579 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,581 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:46:07,670 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
WARNING  2025-08-11 12:46:07,671 Unable to import 80a9be1687bc58f5a0f51f674d612d3c from peers

After this change, I see this:

DEBUG    2025-08-11 12:47:02,068 Processing content requests
DEBUG    2025-08-11 12:47:02,081 Attempting to import missing metadata before content import
DEBUG    2025-08-11 12:47:02,097 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
DEBUG    2025-08-11 12:47:02,108 Starting new HTTPS connection (1): telemetry.learningequality.org:443
INFO     2025-08-11 12:47:02,109 No acceptable peer device for importing content metadata for 62 nodes
DEBUG    2025-08-11 12:47:02,109 Starting automated import of content
INFO     2025-08-11 12:47:02,135 Processing content import request for node 80a9be1687bc58f5a0f51f674d612d3c
DEBUG    2025-08-11 12:47:02,145 Peer 2a824768-819a-a2be-c5ce-cbc06a31ec1e is not available
WARNING  2025-08-11 12:47:02,147 Unable to import 80a9be1687bc58f5a0f51f674d612d3c from peers

@github-actions github-actions bot added DEV: backend Python, databases, networking, filesystem... SIZE: small labels Aug 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 11, 2025

@rtibbles
Copy link
Member Author

Just need to fix assertions in one test that assumed a specific order that is not retained after set/list operation.

@rtibbles rtibbles force-pushed the unique_instance_ids branch from cd31bf2 to f66c57b Compare August 12, 2025 19:48
@rtibbles
Copy link
Member Author

Updated

Copy link
Member

@LianaHarris360 LianaHarris360 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes make sense to me and should be good to merge.

@LianaHarris360 LianaHarris360 merged commit a2410b1 into learningequality:release-v0.18.x Aug 26, 2025
63 of 64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DEV: backend Python, databases, networking, filesystem... SIZE: small

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants