Track regualrity through distance index#4857
Merged
Conversation
…re start-start or end-end connected
Member
Author
|
Right now this passes all the unit tests for the distance index: I'm not sure it really covers all the edge cases for regular snarls, like turnarounds in the middle of chains. |
…he middle of chains
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Member
Author
|
Looks like I'm introducing a bunch of test failures in the |
cache_payloads was single-threaded despite the -t flag; with 164M nodes on an HPRC graph it hung for hours. Two fixes: 1. Pass `true` to for_each_handle to enable OpenMP parallelism; guard the non-thread-safe writes (oversized_zipcodes vector and node_id_to_payload map) with named omp critical sections. 2. Call distance_index->preload(true) immediately before cache_payloads in build_minimizer_index. find_frequent_kmers runs for ~3300 s before this point and evicts the mmap'd index pages, causing a page fault on every snarl-tree lookup in fill_in_zipcode_from_pos. Reloading here ensures the index is warm when the parallel loop starts. Also add a depth guard (abort at >10000) in fill_in_zipcode_from_pos to catch any future infinite loops in the snarl tree traversal. Also use distance_index.get_snarl_child_count() (O(1) record read) instead of for_each_child iteration in get_regular/irregular_snarl_code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parallel payload caching
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This should hopefully help @electricEpilith with the minimizer indexing from hub labeling distance indexes.
Instead of traversing the net graph/hub labeling distance data when trying to decide if a snarl is regular while creating the zip codes during minimizer indexing, we now track the (strict, so in internal reversals) notion of regularity as we build up the distance index.
This includes a distance index version bump.
I'm PRing this against the
hublabelbranch.