Requests for Comments

[thygrrr]

Dear Maintainer, if you have a little bit of time, I'd love to learn more about the internals. :) I'm generally familiar with ECS (maintaining one myself), but not quite familiar with Bevy.

The code looks well structured but has few comments explaining the ECS API/Architecture and its goals. Understandable, for sure, given it's so fresh. 💙

What's a "Fetch"?

This appears to be a core concept of pollus's ECS, but it's hard for me to glimpse from the code what it really represents.

  var t0 = ((IFetch<T0>)t0Fetch.Fetch).DoFetch(world, this);

It seems to be some sort of Archetype access registry?

What's bool Commands.needsSort / Command Buffer "Priority"?

What's being sorted (the command buffers or commands?) The sorting seems to operate by some kind of priority based sorting, what are these priorities?

What are System Parameters?

e.g. in SystemBuilder Create<TSystemParam> and in

public Param<T0> DoFetch(World world, ISystem system)
{
    var t0 = ((IFetch<T0>)t0Fetch.Fetch).DoFetch(world, system);
    return new Param<T0> { Param0 = t0 };
}

Why are Entities stored in a bespoke Pollus.Collections.NativeArray<Entity>?

I can understand why it's useful for certain components that need to get passed on to native functions (I - perhaps incorrectly - tend to prefer Memory<T> over Span<byte> for that), but the Entity?

How are the native arrays pinned / protected against modification, e.g. when Entities move out of a chunk?

Why are components stored in NativeArray<byte>?

Same question, almost, but more along the lines "does that mean managed components are impossible?"

Thank you very much in advance if you find the time, if not, no hard feelings if you close the issue.

[Refsa]

Sure, I can try and answer these questions. A lot of the system and query code is source generated, so it can be a bit messy to follow.

Whats a fetch:

Systems in this ECS arent directly related to querying the entity-component part but instead allows for creating systems one level above that. The World contains Resources and Archetypes which the systems can pull data from. Resources is an open type container that allows the user to put anything into it, working like sort of a DI container.

Fetch is how the system pulls data out of the World and into the systems. Two main types of fetch are setup to pull data out of world, ResourceFetch and QueryFetch. If I add a resource to the world world.Resources.Add(new Time()) it then allows a system to access that resource like so (Time time) => {}. A query is also requested as a parameter to the system like so (Time time, Query<Pos> qPos

As long as a type implementing IFetch<T> is registered in the Fetch class it can be used as a parameter for a System. Examples of this is AssetsFetch<T> or Param<T>, where they can pull data out of other resources or combine multiple resources. Each System also has their own Resources container, which can for example be used for LocalFetch<T> where per system data can be stored using just a delegate for the system.

The Fetch.Register method also allows specifying dependencies, which for now is mostly unused, but can allow for better automatic scheduling of systems if they are allowed to run concurrently in the future.

This setup is pretty similar to how Bevy does it.

Command buffer priority

Commands are sorted based on the command type, and there is no sorting between commands of the same type. Priority is controlled by the command itself so SpawnEntityCommand<TBuilder> defines a priority of 100 and DespawnEntityCommand a priority of 0 as an example. It's pretty loosely defined right now but the main goal is just to ensure that adds are done before edits.

use of NativeArray

I've had some different ideas around this and right now there is no purpose for it. I had some ideas around keeping all chunks in contiguous memory and allowing more efficient moves. Having the ability to easily couple with queries implemented in native libraries was also another reason for it. It's most likely better to handle the chunks with managed memory the way it works now.

Each ArchetypeChunk owns its own memory and that memory is never moved out of it, just copied to other chunks. Any interaction with the internals is optimally done via public methods or Spans to ensure that it doesn't leak. There is also an obvious need to ensure all chunks are properly disposed to avoid unmanaged memory leaks.

By pinned here do you mean for the GC? if so the memory is allocated via the NativeMemory class which allocates it in the unmanaged memory block so it doesnt need to be pinned as it's never moved after allocation.

Managed components

This hasn't been a priority as the goal was to ensure that blittable data can be queried and iterated fast. It's also something I'd like to avoid doing as it forces me to write queries over simple data. The way managed data is stored on components is via an asset Handle that is a pointer to data in the AssetServer that resides in the Resources container on the World. The Handle and AssetServer concept is also something that was borrowed from Bevy.

Hope this answered your questions. Feel free to ask more, and provide insights from learnings you've made creating Fennecs.

[thygrrr]

Awesome, thank you for the detailed infos. :)

I like how fennecs turned out, it was always intended as a drop-in ECS to use with other engines and works like a charm. But it's also got its rough edeges and many limitations, so I want to flesh it out more. (also, my fingers are itching to try dabbling with a pure C# engine)

Some maybe relevant learnings & thoughts:

---

MethodImplOptions.AggressiveInlining hardly ever helps

So far RyuJIT has always inlined everything fine, and AggressiveInlining won't magically make it inline otherwise. (there's various inlining profiler/analyzers that can tell you why a function wasn't inlined)

---

MethodImplOptions.AggressiveOptimization usually hurts

It turns off Tiered PGO, which is responsible for the majority of performance gains I see in systems and streams ('queries'). I recommend avoiding it, it's a misnomer from the past.

---

64-bit Entities carry a surprising memory bandwidth cost

Especially when enumerating them to a runner's ("system's") code. It's so bad that for instance in a system that despawns Entities when they go out of bounds, the Entity parameter passed to the system code makes up 40% of the total memory bandwidth used when looking at Vector3 and 50% when looking at single Vector2 components.

Therefore, I'm refactoring living Entity IDs to be a 32 bit ref struct or local index only; which carries an overhead of just 25% when working on a single Vector3 component / 33% for Vector2. A metadata table already stores generation and other flags, so can get baked into a non-ref struct 64 bit ID only when someone wants to store the Entity itself outside the ECS.

Another approach could be an Entity-Chunk-accessor parameter. Sadly, passing in a ref is always 64 bits; but if the ref doesn't change or only increments, PGO and escape analysis etc. can lead to the value never being overwritten. (it's just left on the stack during each iteration).
In practice, I found this a little fiddly, but with chunks there could be a huge gain (e.g. just increment a 16 bit accessor, and have one 16 bit memory increment and 0 memory copies per entity per iteration...?)

Also potentially frees up an entire cache prefetch lane. 🤔

---

When to discard Archetypes is a hard decision for both users and the library

Discarding an empty archetype and freeing its memory is a difficult decision (all queries need to be modified, and not even one tick later, new entities might want to migrate into that archetype). My current Storage in Archetypes is a big contiguous array; I want to switch that to chunks because then empty archetypes lose most of their memory requirements.

I tried some sort of "if the archetype is still empty after X world catchups, GC it", or having the user call World.GC() [I hate that pattern so much] or World.Forget(Archetype) [so we're exposing Archetypes to users now?]. I was happy with neither.

---

Managed Components are awesome (to coexist with unmanaged ones)

They are just inutitive and useful; e.g. keeping a pointer to mesh data, image references, etc. in a component is just too good. It also enables intuitive "shared" components. Managed components don't negatively affect the performance of unmanaged ones. With chunked storage even migrating (many) entites from one archetype to another doesn't even suffer the managed reference copy overhead.

---

Span<T> and Memory<T> seem to have a hidden overhead per thread

Not sure what it is, but my ECS struggles with multi-threading or context switches. I don't know if the same is true for native memory.

Using just vector intrinsics, single threaded fennecs performance can compete with the fastest ECS out there, including some native ones, including multi-threaded ones.

I guess memory locality and fast processing outweights spending hours on trying to come up with a good thread scheduler. (FriFloEngine.ECS has a great scheduler, but for instance it oversaturates my CPU, so it sure is fast, but at what cost)

My hope is for async2 which could then also make jobs awaitable without the currently somewhat brutal overheads.

---

I miss Command buffers, even if I hate them

I only have one deferred operations queue per world, and all it has done was force me to add lock {} statements all over the place and obscure some behaviour. Structural changes done on threads cause significant congestion. (so at the very least, future me wants a command buffer per thread).

I do like the automatic semantics that structural changes are applied when the runner's ("system's") scope ends, so I will probably keep that and allow users to pass in an optional command buffer.

---

Sparse sets might have a place...

There are a few use cases where a sparse set type "component lookup" outperforms, e.g. when you have many structural changes or wide (many components) or too many (fragmented) archetypes, sparse sets probably vastly outperform moving entities between chunks. (benchmark pending)

I hear Bevy and flecs have some sort of hybrid storage ("do not fragment"), but filtering entities for presence of a component stored in a sparse set, or even just enumerating archetype components alongside sparse set components would likely tank performance.

OTOH, sparse set storage allows to trivially remove a component from a all entities without even iterating Archetypes. Component data is also globally contiguous, so Blit / Clear operations are also very fast.

Generally, I prefer archetypes, though, as sparse sets don't do well when you want to work on 2 or more components at once.

I don't think there's a good middle ground (I'd love some kind of ID-ordered component storage...), yet an optional sparse set storage type is on my to do list now, right after void storage. That is for tags, currently my tags eat 1 byte because that's a builtin limitation of C#: no data type can be size 0.

---

Cheers :)

[Refsa]

That was an interesting read, thanks for the information. This will surely come in handy whenever I do another pass over the ECS.

Didn't know about the MethodImplOptions.AggressiveOptimization, but it might explain some issues I've related to GDV when using delegates or static methods in the query iteration. Start of .NET 8 I think they added some optimizations to avoid vtable calls for methods and delegates, so the use of structs as a container for iteration callbacks came from some optimizations I had to do in projects before. I've had some hit and miss with getting it to work properly even after .NET 8 so this might be something I need to try out.

Chunks is something that works very well for allowing fast and cheap resize of archetype storage. Another benefit is that it could be easier to ensure alignment so that cache and memory lookups are more efficient, but that might be negligible when compared to one contiguous array.

I havent looked at any multi-threading support yet as I want to ensure WASM support. It will be interesting to see if there is any benefit to unmanaged memory there, but the runtime is getting so good at optimizing so who knows.

With your use of vector intrinsics and SIMD related optimizations, how ergonomic do you find that in use? Is there a lot of manual work to select exactly what data you want to iterate with. Say you have a Transform with a Position, Scale and Rotation and a Rigidbody with Velocity. How would an iteration adding the Velocity to Position look like when using Fennecs?