[Proof of concept] Generate spec from the compiler

[wilzbach]

This is a proof of concept following from #1683 and the idea is to generate the spec based on the ddoc comments.
This PR isn't perfect yet, but I would wait with further tweaking until we decide that this is the direction in which we want to go.

The respective PR at dmd is dlang/dmd#6877

[CyberShadow]

More makefile rules D:

[CyberShadow]

I am opposed to this solely because it further complicates the build process. It needs to have a strong justification for the additional complication.

[wilzbach]

I am opposed to this solely because it further complicates the build process. It needs to have a strong justification for the additional complication.

Well, the idea is to move to a stage where the spec can't get outdated anymore, because it's generated from the library frontend.

CC @andralex

[andralex]

cc @RazvanN7 this is the kind of stuff we need to enable with the parser as a library.

This looks like a bit of a steep upfront cost. @MartinNowak what do you think?

[CyberShadow]

Do I understand correctly that this adds a dependency on a 3rd-party library we don't control for the single instance of {{ grammarTraitsList }}? If so, a std.array.replace call would be more pragmatic.

[wilzbach]

There are three variables defined and used in the Mustache context (1 in grammar.d and 2 in traits.d, so replace would look a bit uglier, but yes it would definitely work as well.

Btw mustache-d is one of the 20 libs we test on Jenkins.

So I am happy to change this if we decide to go this road.

[CyberShadow]

Ah yes, the traits.dd ones were not loaded so didn't show up in Ctrl+F.

Still I think it would be good to avoid adding a third template engine to the dlang.org build process (on top of DDoc and Diet). In fact, it would be more idiomatic to generate a .ddoc file and use a DDoc macro - but that's all pending the larger decision on whether we want to do this.

[wilzbach]

cc @RazvanN7 this is the kind of stuff we need to enable with the parser as a library.

@RazvanN7 yes the libdparse API is pretty sweet. In case you aren't familiar a couple of examples:

• Dscanner analysis plugins
• assert_rewrite_magic
• dlang/tools
  • tests_extractor
  • has_public_example
• using Dscanner to auto-upgrade Phobos

[wilzbach]

@andralex @WalterBright we still depend on the high-level decision whether we want to go into this direction.

[wilzbach]

@andralex @WalterBright we still depend on the high-level decision whether we want to go into this direction.

Seeing that the updating the spec manually is tedious work that often gets forgotten, it would be great if we can have a general go ahead on automatically generating the spec from the compiler sources.

I'm pretty sure that once we have a green light on this direction that @CyberShadow and I can work out the technical details ;-)

[andralex]

I do think automation in general is great. Automation of documentation is obviously good, too. However, keeping the solution commensurate with the problem and advantageous (in spite of added complexity) compared to simpler alternatives is a difficult balancing act. This PR does not seem like a slam dunk in terms of bang for the buck. All of a sudden we have libdparse and mustache and new scripts to run - just to keep in sync a few names that are seldom updated.

I was thinking of easing into things before getting too sophisticated. For example, some of the compiler documentation pages can be written - with plan old ddo(c|x) - to take part to the specification pages. At least, the spec could point to something like "Click here for the feature documentation in the reference implementation".

[wilzbach]

just to keep in sync a few names that are seldom updated.

Hehe it still happens to get out of sync regularly, e.g. #1683

And where is isDeprecated?

dlang/dmd#7178

Generating the documentation from the compiler will prevent this from happening again and again.

This PR does not seem like a slam dunk in terms of bang for the buck. All of a sudden we have libdparse and mustache and new scripts to run

We just added a DDoc wrapper in #2060
This allow us to define "dynamic" Ddoc macros without any Makefile complexities and of course to drop the mustache dependency. I could look into using the DaaL (DMD as a library) instead of libdparse.
It essentially be like defining LIST_TRAITS in D code - no new scripts required.

For example, some of the compiler documentation pages can be written - with plan old ddo(c|x) - to take part to the specification pages. At least, the spec could point to something like "Click here for the feature documentation in the reference implementation".

Hmm, this doesn't help the specification. I doubt that anyone is actively looking at the compiler documentation when he wants to know something about a language.

[braddr]

A thought, without having dived through the links here or the pull itself: as an alternate approach, detecting if parts of the doc are missing. That could be used as a build failure trigger. The docs might be essentially empty or crappy, but let the humans decide that part during the review.