[BUG] index out of bounds in et_storage

[cocoon]

Just started to try this lib today and after receiving some entries it fails here:

thread '<unnamed>' panicked at .cargo\registry\src\index.crates.io-1949cf8c6b5b557f\entrace_core-0.1.2\src\mmap\et_storage.rs:73:36:
index out of bounds: the len is 70 but the index is 70

entrace/entrace_core/src/mmap/et_storage.rs

Line 73 in b1c9d0d

child_lists[entry.parent as usize].children.push(len);

[algorithmiker]

Hello!
Thanks a lot for using entrace! I think you're the first user. This means a lot.

Can you reproduce this bug consistently?
I have an example generator program in this repo.
I tried to reproduce this via cargo run --release -- --log-mode disk-et spammer for example, but wasn't able to.

[cocoon]

Hi, yes I was searching something like TraceSpy for windows that works with Rust traces and found entrace.

I directly used it in one of my new programs that refused to start as windows service and so needed a way to see traces and I did not want to log always to disk.

It always happened after some traces for me really quick, but the program is producing traces constantly.

I already replaced entrace with Venator (https://github.com/kmdreko/venator) as this was working stable and uses just a layer and has one big benefit, I can just have my program running and afterwards start Venator and start collecting traces plug and play. The only thing there I struggle sometimes with the time range feature, that it does not display entries.

Before that I tried to somehow add a reconnect feature to entrace but failed to do it in short time.

Another project that was similar is "tracing-live", but I could not compile the web dist:
https://github.com/ycysdf/tracing-live

I might have a backup of the old code with entrace, and maybe try to get something to reproduce it again ...

I had one additional big problem with the entrace UI, when I started to collect in the UI and just closed my program, it totally killed windows sometimes so that it was totally unresponsive and I had to shutdown the VM.

[algorithmiker]

Thanks for mentioning these other tracing projects, I wasn't aware of them when I wrote this.

Before that I tried to somehow add a reconnect feature to entrace but failed to do it in short time.

I didn't need this myself, but maybe I'll implement it. Thanks for the suggestion.

I might have a backup of the old code with entrace, and maybe try to get something to reproduce it again ...

That would be really helpful if you find it.

I had one additional big problem with the entrace UI, when I started to collect in the UI and just closed my program, it totally killed windows sometimes so that it was totally unresponsive and I had to shutdown the VM.

Hmmm. I haven't tested entrace on Windows at all, because I don't use Windows. I'll check.

---

As for the bug itself: Are you emitting spans in multiple threads?

my current hypothesis is that in tree_layer::on_new_span, the TreeLayer submits the span to EtStorage via .on_new_span, but .on_new_span does some small extra work before submitting it to the worker thread.
In this part maybe if you have multiple threads writing child spans to the same parent span, maybe it could cause a data race if the threads are preempted at a wrong point, the child entry could arrives first instead of the parent.

The difference between Windows and Linux scheduling might also explain why I can't reproduce this.

[cocoon]

Are you emitting spans in multiple threads

yes that was the same I was thinking, as yes I have multiple threads running.

update:
and I can reproduce it.
See attached project.
Simply leave it running for ~20-30 seconds and the error appeared for me.

etrace-test.zip

[algorithmiker]

Hey! I haven't been to make this crash on my machine; and when I exit the program the written trace works fine.

I will try on Windows later, as it sounds like a windows specific issue. I'm pretty sure it's an OS-specific thing now.

[algorithmiker]

I was able to reproduce your issue on Windows, and I think I also fixed it

Here's the reproducer I used:

entrace_fixed_reproducer.zip

This reproduces the panic with 0.1.2 but not with #8. Your reproducer is also fixed.

I will make a release soon.