New kitty image backend

[itsjunetime]

Alright, this one is beautiful (once I finish all the final pieces). It replaces, when using a kitty-supporting terminal, the ratatui-image image renderer with a new one I built called kittage. This supports a ton more than ratatui-image, including:

1. Sending images via shared memory objects, which is significantly faster than sending all the image data over stdout
2. Sending images without the alpha channel, which is basically always unused data with pdfs.
3. Deleting images - this isn't immediately useful, but will allow us to fix [consistent] Some pages not rendering #61 once we hook it all up (i.e. delete all non-displayed images once we get an error back from the terminal that it couldn't display an image due to lack of memory)
4. Zooming/panning/all that jazz - because we have much more control over how the image is placed with kittage, this will allow us to zoom and pan and stretch and whatever very easily (albiet only on kitty for now) - this will finally allow us to fix really small text #7.
5. I guess this would allow us to close Add support for Warp #65 somewhat more easily since warp supports kitty

Things still to do:

• Add support for tmux to kittage. Hopefully this will be easier as I understand the protocol a lot better and we have a lot more control over how codes are sent
• Hook up deletion-upon-memory-exhaustion - fairly easy, I just haven't done it yet
• Make it work with windows. As of right now, the shared memory object implementation I'm using only works with unix 'cause that's the only machine I have, but I'd like to get it working windows as well. If all else fails, though, I can just not use shms on windows and just do normal direct data transfer instead.

So this isn't quite ready for prime time yet, but I'm very happy with it so far. I'm hoping to run some benchmarks soon to see if we can get numbers on the performance improvement. And who knows, if it goes well here maybe I'll try to get it into ratatui-image or yazi as their backends for displaying kitty images.

[itsjunetime]

Ok, here are some (not-fully-representative) numbers from the current benchmarks (none of which show regressions over main):

render_full/benches/adobe_example.pdf
                        time:   [43.464 ms 44.251 ms 45.089 ms]
                        change: [−5.1472% −1.9929% +1.3250%] (p = 0.23 > 0.05)
                        No change in performance detected.
Found 1 outliers among 40 measurements (2.50%)
  1 (2.50%) high mild

render_full/benches/example_dictionary.pdf
                        time:   [816.48 ms 818.97 ms 821.54 ms]
                        change: [−21.433% −21.136% −20.793%] (p = 0.00 < 0.05)
                        Performance has improved.

render_full/benches/geotopo.pdf
                        time:   [374.26 ms 375.44 ms 376.75 ms]
                        change: [−10.392% −9.9231% −9.4319%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 40 measurements (7.50%)
  3 (7.50%) high mild

render_first_page/benches/adobe_example.pdf
                        time:   [19.610 ms 20.093 ms 20.525 ms]
                        change: [−5.4113% −2.4964% +0.2469%] (p = 0.10 > 0.05)
                        No change in performance detected.
Found 3 outliers among 40 measurements (7.50%)
  3 (7.50%) low mild

render_first_page/benches/example_dictionary.pdf
                        time:   [10.311 ms 10.350 ms 10.386 ms]
                        change: [−7.8038% −6.7169% −5.5129%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 40 measurements (7.50%)
  3 (7.50%) low mild

render_first_page/benches/geotopo.pdf
                        time:   [36.364 ms 37.461 ms 38.558 ms]
                        change: [−12.943% −9.6350% −5.8420%] (p = 0.00 < 0.05)
                        Performance has improved.

only_converting/benches/adobe_example.pdf
                        time:   [1.1816 ms 1.1901 ms 1.2000 ms]
                        change: [−56.662% −54.011% −51.035%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 40 measurements (2.50%)
  1 (2.50%) high severe

only_converting/benches/example_dictionary.pdf
                        time:   [110.77 ms 115.42 ms 120.05 ms]
                        change: [−46.149% −43.495% −41.123%] (p = 0.00 < 0.05)
                        Performance has improved.

only_converting/benches/geotopo.pdf
                        time:   [29.162 ms 30.732 ms 32.345 ms]
                        change: [−47.263% −43.894% −40.468%] (p = 0.00 < 0.05)
                        Performance has improved.

The only_converting benchmarks basically track the Vec<u8> to ratatui_image::Protocol conversion, which is the thing that this PR is replacing, so we can see that this PR will make that approximately 2x as fast.

I also think these numbers don't fully represent the improvements, just by virtue of how much faster tdf clearly is when I'm using it now. I can just hold down the right arrow, even in heavy PDFs, and it pages through without stuttering at all (whereas previously it would be taking ~400ms to load each page). I'll try to get some whole-system benchmarks that can accurately measure just paging through an entire pdf, thus also considering how long the terminal takes to render.

[itsjunetime]

Also TODO: fix the help page

[itsjunetime]

Also, we should make sure this still works correctly when using tdf remotely. I don't know if there's any sure-fire way to detect when the tty is remote (since you could be using mosh, ssh, or any other protocol to do so) but $SSH_TTY should cover most cases, and maybe we can add a flag to force tdf to act like it's remote and be conservative with using shms and such.

[kovidgoyal]

The kitty graphics protocol itself allows you detect if you share a filesystem/shared memory witht heterminal emulator. Use that and you will be robust against all such situations which include remote systems, containers, etc.

[itsjunetime]

The kitty graphics protocol itself allows you detect if you share a filesystem/shared memory witht heterminal emulator

As far as I can tell, the method of detecting shared fs/memory doesn't actually distinguish between an shm just not existing vs. not sharing fs/memory at all. In my tests, running echo -en "\e_Gi=3948,a=q,t=s,s=1,v=1;$(echo -n "this_is_a_file" | base64)\e\\" results in the same reported error code on both a remote computer vs a local one.

This obviously isn't too bad - if we mistakenly think that we're on a remote computer when we're actually local, everything will still work fine, albeit a bit slower. I just want to make sure I'm not missing some part of the protocol that allows me to actually distinguish between a non-existent shm and a remote computer.

[kovidgoyal]

On Mon, Jun 09, 2025 at 04:04:58PM -0700, June wrote: itsjunetime left a comment (itsjunetime/tdf#74) > The kitty graphics protocol itself allows you detect if you share a filesystem/shared memory witht heterminal emulator As far as I can tell, the method of detecting shared fs/memory doesn't actually distinguish between an shm just not existing vs. not sharing fs/memory at all. In my tests, running `echo -en "\e_Gi=3948,a=q,t=s,s=1,v=1;$(echo -n "this_is_a_file" | base64)\e\\"` results in the same reported error code on both a remote computer vs a local one.

You create the shm, a simple dummy four byte one for instance and then transmit its name. Same for files.

[itsjunetime]

Alright, I'm happy enough to merge this and push another release.