Skip to content

Conversation

@sirtimid
Copy link
Contributor

@sirtimid sirtimid commented Feb 12, 2025

Closes: #329

  • Add handlers for syscall delivery messages related to distributed garbage collection:
    • dropImports: Vat tells kernel it no longer needs certain imported references
    • retireImports: Vat acknowledges retirement of imports and promises not to use them again
    • retireExports: Vat tells kernel it's retiring certain exported objects
    • abandonExports: Vat tells kernel it's abandoning exported objects (even if they are live)
  • Add incrementRefCount and decrementRefCount functions to manage reference counts for kernel objects and promises
  • Extend deleteClistEntry to handle reachability flags and reference counting
  • impement both "reachable" and "recognizable" refcounts on all kernel objects
  • update these refcounts during clist additions/removals
  • track the refcounts that touch zero, add them to a new maybeFreeKrefs ephemeral set
  • process that set at end-of-delivery, to see what GC actions should be scheduled
  • add those actions to a new durable GC Actions set (in the kernelDB)
  • scan the pending actions before each new delivery, to prioritize GC work
  • deliver the highest-priority non-negated GC Action to the appropriate vat
  • restrucure ported GC code for better structure and organization
  • update all unit tests as necessary, add new ones

@sirtimid sirtimid changed the title feat(kernel): Support liveslots distributed garbage collection feat(kernel): [WIP] Support liveslots distributed garbage collection Feb 14, 2025
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch from 894b13b to 536c336 Compare February 25, 2025 22:41
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch from 536c336 to 9a651d1 Compare March 11, 2025 19:26
Comment on lines 58 to 65
import { makeBaseStore } from './base-store.ts';
import { makeCListStore } from './clist-store.ts';
import { makeGCStore } from './gc-store.ts';
import { makeIdStore } from './id-store.ts';
import { makeObjectStore } from './object-store.ts';
import { makePromiseStore } from './promise-store.ts';
import { makeQueueStore } from './queue-store.ts';
import { makeRefCountStore } from './refcount-store.ts';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not completely wild about this refactor of kernel-store. To be clear: I think breaking it apart into more manageably sized pieces is well motivated, given that it suffered a bit too much from having fallen into the big-ball-of-mud pattern (which it acquired by way of its SwingSet heritage, which in turn got that way because it just growed). Moreover, I think the partitioning of functionality here seems mostly sensible.

However, given that its job is simply wrapping various bits of higher-level, kernel-specific functionality around a single underlying (and fairly stupid) key/value store, I'm not comfortable characterizing the various sub-groupings of function as "stores" in their own right (though doing that arguably might make sense in a world where we implement more of those bits of functionality directly in the database with their own individualized tables, schemata, and prepared statements, and maybe that's an idea we should talk about -- a refactoring like this might make sense as a precursor to moving things in that direction). While calling them "stores" might actually just be a naming quibble, reifying each of them as its own object (with accompanying factory function) feels like a pretty heavy weight approach, especially given the fairly complicated graph of dependencies between them -- I'm betting that getting the order-of-initialization logic right for them all was a bit of work. Given those cross dependencies, I'm concerned that the whole thing may be fairly brittle under maintenance. On top of this, the big-ball-of-mud API is still retained in the end because that's how remains exposed to the rest of the kernel

In short, my intuition is that if you're going to refactor the kernel store, you should go all the way (perhaps exposing the "kernel store" as an explicit bag of sub-functionality groups), otherwise I'm not sure it pays for itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! That’s actually great to hear because the main reason I kept the factory-based approach was to avoid deviating too much from the existing pattern. I was already aware that it introduces order-of-initialization challenges and I had considered refining it further to align more closely with the overall code style. I held back from making major changes in this pass but now that it's in this state I think it’s clear that a more thorough refactor would be worthwhile.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FUDCo Check it out e00a14b

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This refactor is still going to collide with critical bug fixes in #448. I'd be much more comfortable if the kernel store refactor was separated into its own PR so that we can land the GC stuff now.

Copy link
Contributor Author

@sirtimid sirtimid Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I've reverted the refactor and will do it on a separate PR after #448 is merged. I will also add the unit tests on that PR because it will be a big hustle to adjust them for the one file now. Added a task to track it

@sirtimid sirtimid marked this pull request as ready for review March 11, 2025 22:04
@sirtimid sirtimid requested a review from a team as a code owner March 11, 2025 22:04
@sirtimid sirtimid changed the title feat(kernel): [WIP] Support liveslots distributed garbage collection feat(kernel): Support liveslots distributed garbage collection Mar 11, 2025
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch from 0c58cb2 to e00a14b Compare March 12, 2025 17:11
@sirtimid sirtimid requested a review from FUDCo March 12, 2025 18:01
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch 2 times, most recently from b27f593 to 11778ef Compare March 18, 2025 18:09
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch from b336429 to 381e30e Compare March 26, 2025 13:23
@sirtimid sirtimid enabled auto-merge (squash) March 26, 2025 17:50
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch from 189658d to 045188e Compare March 26, 2025 23:32
@sirtimid sirtimid force-pushed the sirtimid/distributed-garbage-collection branch from e4da823 to ca199c6 Compare March 27, 2025 16:33
Copy link
Contributor

@FUDCo FUDCo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Let's all plug our ears and wait for the kaboom!

@sirtimid sirtimid merged commit 2f6e0d3 into main Mar 27, 2025
18 checks passed
@sirtimid sirtimid deleted the sirtimid/distributed-garbage-collection branch March 27, 2025 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement kernel-side support for liveslots distributed garbage collection

3 participants