stmt and stmt_html output are too low level

[abadams]

For example, they show already-compiled PTX assembly for cuda kernels instead of stmt ir, because those have already been offloaded. As more and more of codegen creeps into lowering, this problem will get worse. We need to identify a point in lowering at which the stmt should be preserved for stmt and stmt_html output. I propose just after the custom passes and before hexagon kernel offload.

See #7507

[antonysigma]

Uploading the screenshots to illustrate the stmt output, and why they are too low level since Halide 14.0 (?)

[antonysigma]

For that to happen we'd have to emit the .stmt file before the offload gpu pass, e.g. in Lower.cpp around line 441. I guess we'd have to stash that earlier stmt in the Module somewhere for use in the .stmt outputs.

@maaz139 Other than line 441 of lower.cpp, is there anywhere I can contribute as a Halide beginner? I suppose I can add an optional argument intent=stmt | codegen for a start?

Module lower(const std::vector<Function> &output_funcs, ..., Halide::my_intent_t = Stmt_only);

[mcourteaux]

I'd like to chime in, and state that I ❤️ the PTX code in the Stmt HTML to get a good understanding of what's happening in a single CUDA thread. I examined these for months when working on something for work. Much like how one wants to check the generated assembly for CPU code, you'd want to validate what got generated for the GPU code.

In my opinion, the compilation pipeline should be respected for what it is: a pipeline. The code can be useful to inspect at different stages. Statement IR can be useful, PTX code can be useful. Just like we are able to get a stmt file showing IR and an s file showing the CPU assembly, I believe we should be able to see an output PTX output file. I'd imagine .s and .ptx. Or perhaps .x86-64.s and .ptx.s? Maybe there are existing conventions here already that I'm unaware of.

Alternatively, we output the PTX code in the assembly file, but that would cause us to lose the colors, and in general is hard to navigate, as the s file is less verbose and intuitive.

Considering all the discussions regarding the VizIR HTML, the problems it has, and the fact that now PTX code is likely to be another variable, I believe a possible solution could be to have more generator emit options: now there is only stmt and stmt_html. I think the following would be useful:

• stmt (plain text stmt)
• stmt_html (base mode for html stmt)
• stmt_html_vizir (option for stmt_html which will include the vizir)
• stmt_html_asm (option for the stmt_html which will include the asm)
• stmt_html_ptx (option for the stmt_html which will lower the ir to ptx and include that)

In the above, stmt_html_xxxx options are meaningless without the base mode.

Additionally, to generalize the assembly emit option:

• assembly (which will be the CPU instruction set of the Halide target)
• assembly_ptx

If this sounds reasonable, I could give this a try to implement this. What I like about this, is that it will be backward-compatible. Feedback on this idea?

[antonysigma]

Attaching a side-by-side comparison. This is to illustrate the difference among: Halide IR, PTX IR, and assembly outputs.

[mcourteaux]

To make the discussion complete, I'll copy paste my thoughts I posted on the other PR here:

---

I have been thinking -- like most of you -- quite a bit about this. I believe it just makes sense to indeed pick a few point in the lowering process, stash the IR away into the Module for later, and emit multiple files. Each file representing one point in the lowering process.

I'm absolutely not sure what to do with the PTX code and the assembly tab. It is assembly, and therefore could go there, but it's not the assembly of the pipeline. It really is a buffer containing assembly for the GPU. Maybe... Perhaps... It is more sensible to have a "Buffers" tab?

Overall, I believe the HTML-way of doing things is getting convoluted, and I'm thinking of an approach where we just dump a very information-rich IR tree to some file format and have a custom DearImGui-based tool be the visualizer? I don't know what the recently-merged "experimental serializer" is exactly capable of, but maybe piggybacking on that to dump flatbuffers of the IR at a few points of the lowering process can be a nice trick here to build such a tool. @TH3CHARLie Can the serializer you built be used for dumping an IR tree?

[steven-johnson]

Can the serializer you built be used for dumping an IR tree?

Yes, that's pretty much the only thing it can be used for :-)

[antonysigma]

Moving the discussions from the draft PR here. Feel free to clarify if I mis-quoted you.

maaz139 : From Given that we now have an assembly tab when using the html output, perhaps we can instead simply add a jump [from the gpu_block block in the IR tab] to the corresponding [decoded PTX IRs in the rodata (?) buffer] line in assembly.

@mcourteaux pointed out the PTX code dump in the cuda_gpu_source_kernels indeed is an extension of the LLVM IR. I took a minute to read the LLVM documentation, and yea, indeed it is called the internal representation (IR) of Nvidia's GPU code. Thanks @mcourteaux .

@maaz139 I am not sure if the "assembly tab" will visualize the PTX code, even if we add a "jump" button. If I understand the docs correctly, the PTX IR is further "lowered" by LLVM into Nvidia bytecode, then encoded/obfuscated into a buffer entry in the assembly. So, the jump button may be useless here.

maaz139 : I think printing any IR other than what is present after all lowering steps may be confusing or misleading (if thats is the only output). Perhaps the answer is allowing users to view the IR after each lowering stage or more realistically at a few key checkpoints during the lowering pipeline?

Understood. I retracted my draft PR. It does serve my most immediate needs. That is, to reason about the gpu_block and gpu_thread sizes, and to flush out redundant compute by inspecting the 3GL style Halide IR.

Agreed it can be misleading if (other) users' goal, unlike mine, is to cross-reference the lowered IR and the assembly output.

[antonysigma]

mcourteaux : [The presence of buffer cuda_gpu_source_kernels print out in the IR tab] boils down to the other comment of @maaz139. Because that [PTX code] really is the lowered IR.

Perhaps we use the term "assembly tab/code/dump" too broadly, creating confusions. There's a more formal term available: 2GL and 3GL. When I said the PTX as an IR, looks too low level, I mean the PTX dump looks too much like the 2GL. I was expecting a 3GL-style textual representation of the Halide IR back in Halide 10.0.

@maaz139 , could you please drive the discussion on the work scope? That is, how low, 2GL or 3GL, should Halide IR should be printed as stmt ?

In the meantime, I will pay the effort to learn the PTX IR syntax.

(An off-topic opinion: I personally call the Halide language the 4GL, and languages like the the CVXGEN the 5GL.)

[mcourteaux]

@mcourteaux pointed out the PTX code dump in the cuda_gpu_source_kernels indeed is an extension of the LLVM IR. I took a minute to read the LLVM documentation, and yea, indeed it is called the internal representation (IR) of Nvidia's GPU code. Thanks @mcourteaux .

@antonysigma To clear this up: what I meant there is that the IR printed to the HTML file is actually the IR that gets compiled as CPU code. The gpu_kernel_sources is actually already "fully compiled" in the sense that it just gets handed over to the CUDA driver at runtime. The IR surrounding this raw byte-buffer, is the code that actually will pass it to the CUDA driver and perform the kernel launch.

I'm not sure about the LLVM IR PTX you describe... As far as I understood, the buffer contains PTX code, which is NVIDIA specific, and not really anything related to LLVM.

---

Yes, that's pretty much the only thing it can be used for :-)

@steven-johnson Seems tho, that at this point, the interface only supports dumping a Pipeline. I guess adding an interface that will take a LoweredFunc or Stmt is an easy addition.

[steven-johnson]

Yeah -- it's still experimental, and thus the precise use cases that are sensible to allow (or disallow) are still a bit fluid (and likely will be for a version, at least). Feel free to offer a PR that allows de/serializing fragments.

[TH3CHARLie]

Stmt should be easy, not sure about LoweredFunc though, the serializer was created only to serialize front-end IRs.

[maaz139]

@maaz139 , could you please drive the discussion on the work scope? That is, how low, 2GL or 3GL, should Halide IR should be printed as stmt ?

Thats a great question. In my personal opinion, the statement files are meant to look like 3GL code. The whole point of printing the Stmt file is to look at code that is not 2GL -- at least in my use cases. I can't really comment on why folks decided to print low-level PTX code inside the stmt files, I was not involved with the code back then but I can imagine there was a demand for it.

Generally speaking, I agree that different users typically seek out different details. Generating different variations of the stmt_html files is one way to disentangle different "views" into the Halide program and have consistent expectations with each view. I am not sure if there should be dedicated generator flags for each, perhaps we can generate all the of versions whenever a user selects stmt_html as a target. This approach is especially appealing if users can still jump across views without losing where they are in the program.

Overall, I believe the HTML-way of doing things is getting convoluted, and I'm thinking of an approach where we just dump a very information-rich IR tree to some file format and have a custom DearImGui-based tool be the visualizer? I don't know what the recently-merged "experimental serializer" is exactly capable of, but maybe piggybacking on that to dump flatbuffers of the IR at a few points of the lowering process can be a nice trick here to build such a tool. @TH3CHARLie Can the serializer you built be used for dumping an IR tree?

That seems like a fair bit of work but I concur that the HTML based method is not very scalable. I'm happy to contribute on the de-serialization of missing lower-level IR constructs.