An idea of using Cirru syntax as an alternative of text format

[tiye]

...I'm just thinking from the syntax side, but not being experienced in compiler. I will just discuss on the syntax side.

I made a video to demonstrate the main ideas behind Cirru:
https://www.youtube.com/watch?v=1ShYO_A8g-0 (14mins)

Short version:

• Cirru is trying to build a syntax tree that we can edit directly
• that syntax tree has a text syntax, with parser and formatter
• that text syntax is based on indentations
• I've been thinking about it for several years and have been using it for months

For Cirru code like this:

defn add2 (x y) (+ x y)

it is equivalent to a piece of data in JSON:

["defn", "add2", ["x", "y"], ["+", "x", "y"]]

once you get JSON you are free to make an DOM editor or simply render to HTML/CSS:

and also it can be turned back to Cirru code as you want.

Check these links if you need details.
http://cirru.org/
https://github.com/Cirru/
https://twitter.com/cirrulang
https://github.com/Cirru/cirru-wasm-ast (mostly based on @indutny 's work)
https://github.com/Cirru/cirru-wasm-cli

From my side I got some experience using Cirru:

Pros

• Preciously we are programming by manipulating text files, we all say it's about AST, which is a tree, but we need the help of text syntax to do it. Some languages pushed syntax even further and it becomes hard for newcomers to pick up. Cirru is giving more possibilities. I mean we can manipulate the syntax tree directly now.
• Cirru syntax is simple. It's based on indentations and reduced the usage of parentheses a lot.
• Cirru syntax is flexible. When recovered from binary files, it can be turned into serveral styles of text form, such as "smart folding look like human wrote it", "expression unfolded for the purpose of add break points", "tree view for a better user interface", etc.
• I've been generating my JavaScript code and ClojureScript code in Cirru for many months, the syntax is quite handy now.

Cons

• People hate to learn one more syntax, what's worse, it's different from C, Java or Python.
• String syntax in Cirru can be confusing according to my experience introducing Cirru to others. Because Cirru was originally designed as a tree editor and people does not quote marks in a tree editor.
• After Cirru code is parsed into a tree, we need another step to turn that tree into an AST. I'm not quite clear about how compilers will handle this. So I'm not sure it's good or bad.
• I think we don't have very much experience programming with a tree rather than a text file. Our toolchains are mostly based on text files.

Even if we don't want the graphical part of CIrru, we can still try its text syntax. It's basic Lisp in indentations(with string syntax a bit strange):
https://gist.github.com/jiyinyiyong/0fc58e2ed7c641973d9b

module
  export :even $even
  export "odd" $odd

  func $even (param $n i32) (result i32)
    if (i32.eq (get_local $n) (i32.const 0))
      i32.const 1
      call $odd (i32.sub (get_local $n) (i32.const 1))

  func $odd (param $n i32) (result i32)
    store_global $scratch (get_local $n)
    if (i32.eq (get_local $n) (i32.const 0))
      i32.const 0
      call $even (i32.sub (get_local $n) (i32.const 1))

  global $scratch i32

I'm a big fan of Lisp(mainly Clojure), but dislike the parentheses in Lisp. So this is my idea to fix it and that in WebAssembly.

[qwertie]

You have some competition.... from me.

I made LES (LESv2), a language inspired by LISP but syntactically based on Javascript and C#. Its relationship to JSON is somewhat different - LES is a superset of JSON, rather than being convertible to JSON (one could define such a reverse mapping, of course, I just haven't done so, and if JSON code is treated as LES code and then converted with any reasonable mapping back to JSON, it would probably have to expand in size). Edit: LES is probably a more sophisticated language than Wasm needs, so I was going to propose a subset of LES to the Wasm CG, I just haven't got around to defining it yet. And I was considering other refinements.

Hmm... given that, in Cirru, an ident becomes "ident" in JSON, what about a string like "ident" in Cirru? Are strings and identifiers exactly equivalent? And what about the dollar sign in the above code? The documentation says $ as a function to fold code, see Haskell, but I suppose that's not what's happening here.

[tiye]

I'm looking into the "Cirru problems" first,

For ident, my video covered it. Yes I realise this problem as soon as I began to use it. There are already examples on solving this. It's like 'something-quoted in Scheme, and :a-keyword in Clojure. My solution is to prefix that with one character, for JavaScript "str" is written in :str, and in Clojure I use |str.

For $, in Haskell it's an infix function which evaluates x before f is calling in:

($) :: (a -> b) -> (a -> b)
f $ x = f x

Yeah, it's internally different in Cirru since Cirru Parser is doing syntax tree transformation at the end of parsing. I want to say the meaning is learnt from Haskell. f $ a b is evaluated like f (a b).

Back to LES... personally I don't like writing brackets, but I have to say LES is far more mature than Cirru is and Cirru is hardly a competitor as an idea at this moment(well, I suppose Scheme syntax is by now the best). But I'm still glad Scheme is not our only option to write WebAssembly :D

By the way even people don't like Cirru, I can still compile Cirru to WebAssembly by some ways..

• CirruScript -> JavaScript -> wasm
• Cirru -> ClojureScript -> wasm
• Cirru -> wasm AST -> wasm

[qwertie]

How old is Cirru, by the way? It seems to me that if the CG decided they just wanted an s-expression parser, they could go with Sweet Expressions, which appear to be 2.5 years 3.5 years old. I heard about sweet-expressions (a.k.a. Readable Lisp S-expressions) a couple of years ago, and I don't even use LISP. So.... I wonder, did you know about sweet expressions already when you started Cirru? Personally I think infix operators are a key to readability; sweet exprs have them but I don't see them in Cirru. Heck, if it weren't for the fact that everybody still writes LISP in s-exprs instead of sweet-exprs, I'm sure I would use a language in the LISP family by now (the real problem is that I can't learn LISP with sweet exprs: no one is teaching it that way.)

LES is far more mature than Cirru

Uhh, did you really mean to say that? Cirru is older than LES LESv2 (LESv1 is 2.7 years old), it has its own domain name, and it has parsers written in a whole bunch of languages. LES is great if we want a Javascript-style syntax, but Cirru (and sweet expressions) appear to be mature.

About the $, I was referring to the code you wrote earlier like (param $n i32) - this means (param (n 132))?

Edit added corrections after you replied

[tiye]

Cirru was started in the middle 2012 but only as a tree editor, and later got a text syntax by the end of the year, according to my git commits and tweets at weibo. It was only a toy before I can actually compile it to JavaScript in 2015. I saw Sweet before, but I don't know how to use it or where to find its interpreter. So, I did it before Sweet.

The way I came to Cirru, I was making an AST editor after learning http://norvig.com/lispy2.html but failed to make it handy back that time with raw JavaScript. There was not React at that time, I built my first version of tree editor in canvas and it's hardly an editor I can use. So I realised I need a text syntax which looks very similar to Cirru Editor.

Clojure is OK for me, Polish notation, so prefix syntax, it feels good to me. Ever since I built my own "lispy" interpreter I found it very nice, which JavaScript syntax bothered me a lot. I mean I think Lisp is still readable comparing to C-style syntax since the later one has much more syntax rules.

Yes, I think LES is more mature in the part of WebAssembly. Well, you got a spec. Cirru does not have a spec, it only contains some special functions to help transforming code to JavaScript or Clojure, also some WebAssembly. So it's more like a code generator at this moment. However, I would say Cirru is more advanced as "an AST tree exploring project". :)

[qwertie]

Edit corrected my earlier post.

LES has a spec, but it's not complete, and if the CG considers LES as the Wasm text format, I will pretty much change it according to the whims of the "important" members like @lukewagner and @jfbastien. Heck, I could even stop calling it "my" idea - if that's the price :)

The most important thing is that Wasm should choose a parser that isn't solely dedicated to Wasm itself - like s-expressions, it should be a syntax that lends itself to many different use cases. Why? Look, none of the "friendly" languages for representing syntax trees - sweet expressions, LES, Cirru - are popular. People don't use them because they aren't popular, and they aren't popular because people don't use them. Chicken and egg. I just talked to a prominent member of the C# community; he said he wouldn't use the Enhanced C# language I created, not because of anything wrong with it, but simply because "hardly anyone else knows" it.

That means WebAssembly CG has a huge opportunity that they shouldn't squander. Whatever they pick automatically becomes popular. I hope that they will choose a variation of LES, because Javacript/C syntax is by far the most popular syntactic style, and as far as I know, LES is the only language that parses a language resembling Javacript et al into a simple data structure.

Fun fact: originally LES had a Python-style mode where it was indentation-sensitive. I took that out in favor of just C/JS-style, because I think people will warm up to that more than the Python style. I don't really care one way or the other - I quite like Python's indentation-as-structure, but the important thing is to bring those LISP ideas of "syntax independent of semantics" and "code is data" over to the mainstream.

JavaScript syntax bothered me a lot. I mean I think Lisp is still readable comparing to C-style syntax since the later one has much more syntax rules.

I have difficulty understanding this perspective. I mean, I've been coding for over 20 years, and in that time I have spent far, far more time learning about library APIs than syntax. More time learning semantics than syntax. Etc. C/C++ syntax is definitely confusing in some ways, but Javascript? It's a lot easier. Also, infix operators are present in natural language ("five plus four"), the order of operations is taught before high school ("1 + 2 × 3 < 9" vs "< (+ 1 (× 2 3)) 9"), and some other parts of the syntax are popular in mathematics as well as programming languages (e.g. (tuple, of, things) and [list, of, things]).

[tiye]

That's far more than I have thought. Popularity matters a lot in language choosing I think. WebAssembly is still new as a compiling target so choosing a popular syntax may benefit from existing toolchains. S-expression is nice since it's simple to built from scratch and people already explored it enough to bypass the potential problems. And in my point, Cirru is just a simple version of S-expression and better designed for the future, which is enabling graphical tool in programming.

So in short, Cirru is not the best choice at this moment. But if people want 1) a cleaner version of S-expression(by using indentations), 2) build graphical tools for WebAssembly, then Cirru might be a good start to the future plans.

I've been learning programming for about 4 years, maybe 5, only. And from the day I began to learn, CoffeeScript is becoming popular and lots of ideas are making to choose an efficient way of writing code, like making less mistakes, bringing less confusions. That's why I'm exploring a new one trying to get more by using a graphical way. JavaScript syntax, in my view is tedious to write and modify. I can bear using it in large projects that enhance stability and team work, but not in all my projects that I want to code fast.

[shaunlebron]

For anyone wanting quick context on this discussion: WebAssembly will be a binary format but there are some cases in which a Text Format is also desired. An official text format hasn't been decided, but S-expressions are currently used for the sake of simplicity in the spec. Hence, the discussion here on what the official text format should be by looking at alternative syntaxes explicitly designed to represent ASTs.

A way to evaluate

I want to offer that choosing a syntax may be an optimization problem of the following things:

simplicity	number of syntax rules
flexibility	ability and ease of adding rules
familiarity*	popularity of syntax
readability	a subjective function of simplicity, familiarity, and preference
writability	how simple it is for a human to produce it w/ or w/o tools

* I think familiarity is set by community defaults, for which WebAssembly has carte blanche, as @qwertie said.

Evaluating: S-expressions

• simplicity - obviously good
• flexibility - obviously good
• familiarity - obviously bad
• readability
  • suffers from 'too much information' for newcomers
  • users learn that indentation style is equivalent to Python
  • infix is missing
• writability
  • balancing parens is difficult without plugins
  • we are working on Parinfer to allow indentation to influence structure and vice versa

Evaluating: Cirru and LES

I like what I've seen from both, but I can't fully evaluate these for lack of experience, so I will leave it to their authors...

My thoughts

I would place less weight on familiarity and more weight on simplicity and
flexibility because that will contribute the most to future-proofing how we
express the AST of the web, which will probably be around for a while! ;)

@qwertie wasm does not have many of the operators of LES and LES seems to be described as having a fixed set of operators which seems necessary for the code-is-data use case. e.g. Wasm does not have +, rather it has i32.add i64.add etc. Languages and data do not in general map to a fixed set of operators so a fixed set of precedence and parsing rules seems to have limited utility. How could this be addressed?

[qwertie]

@JSStats LES actually has an unlimited number of operators (sequences of punctuation marks), with precedence chosen by a set of rules based on operators from several languages, especially C/C++/Java/C#/Javascript. There is also a backquote pseudo-operator, which lets you treat any identifier as an operator. For instance, the s-expression

(i32.add (get_local N) (i32.const 1))

could be directly transliterated to LES in prefix notation...

@i32.add(get_local(N), @i32.const(1));

or with "superexpressions"

@i32.add(get_local N, @i32.const 1);

or with a i32.add and a superexpression...

(get_local N) `i32.add` (@i32.const 1);

or even (worst of all) a backquote fest:

`get_local` N `i32.add` `i32.const` 1;

These are all equivalent, due to the equivalence of operators and "calls()". They are also ugly and not recommended; I'm just showing the different ways LES allows you to express the same thing. (Note that the @ in @int32.add is needed because the identifier i32.add contains a dot character and we probably don't want . to be considered a binary operator.)

Wasm does not have +, rather it has i32.add i64.add

Quite right. However, I do think it would make a lot of sense to include operators like + in the text format, requiring the "assembler" to infer whether it's i32.add or i64.add. There are other things we can readily do to make the text format more friendly. Let's consider this adder:

(func $add (param $x i32) (param $y i32) (result i32) 
    (i32.add (get_local $x) (get_local $y)))

(Incidentally, one extra @ sign makes this into valid LES code.)

It could look something like this in LES:

func add(x: i32, y: i32) -> i32 { x + y };

That's if we decide users can access locals unqualified (or we could use #x or $x or whatever); similarly we could use simply 123 for i32.const 123 and 123L for i64.const 123. . The structure of this expression (expressed as an LES-compatible s-expression) is

(func (@-> (add (@: x i32) (@: y i32)) (i32)) (@`{}` (@+ #x #y)));

Languages and data do not in general map to a fixed set of operators so a fixed set of precedence and parsing rules seems to have limited utility.

The fixed set of rules is designed to reflect the existing consensus of many different programming languages. I think the existing rule set would serve Wasm quite well - but it's not a finalized spec, and if the CG wants changes, then changes there will be. Even better would be if eyeballs outside the Wasm community deliberated on it, but I'm not sure how to make that happen.

@qwertie None of those seem 'familiar' or an improvement over s-exp to me, but it's a subjective matter. Avoiding explicit get_local operators seems an obvious thing to improve the readability. I suggest waiting to see the latest thinking on a post-order encoding because with a one pass validator the type derivation will be bottom up and I can't see any advantage to having i32.add and i64.add etc rather than just add and determining the type from the argument types? This seems to work fine for CIL. If the types could be removed from many of the operators then they would map to familiar operators.

[qwertie]

@JSStats The 'add' example doesn't seem familiar? It looks like Rust and Swift. Since the text format isn't used directly on the web, I was assuming a one-pass compiler wouldn't be necessary for it (and certainly one could do multiple passes over individual functions), so some conveniences can be added, so + could be used for adding even if there is no single opcode for it. Heck, even if one really wanted to optimize the parser, that feature is still possible (given forward declarations for all functions).

@qwertie Sorry if I missed it but nothing in you comment immediately above looked like familiar JS style? Do you have a wasm printer that emits wasm using the format you suggest to give an example? 'Rust and Swift' were not something i would consider 'familiar' (subjective).

I think it would help if people could develop some alternatives to add to the discussion. If removing the type from operators such as add allows them to map to familiar infix and precedence syntax then make the case for this change and I may well support this as no reason against doing this comes to mind. Show people the substantial difference it could make to the text format.

It's an interesting idea of LES to have a fixed set of familiar operators with familiar infix and precedence rules for them, and with all other operators handled by generic paths so as to support the code-is-data use case. It might create a compelling reason to 'overload' some operators so that they match familiar patterns. This same property would probably help text formatters in general, and it affects the binary encoding too unless formatters are expected to translate.

Personally I hope the browsers can support an extension model for viewing the source so that there can be some personalization in this area and it seems possible for the text format to evolve somewhat independent of the binary encoding.

[qwertie]

On the LES page, you'll LES code that looks like Javascript, but since Wasm is statically typed it couldn't/shouldn't look quite like JS, hence the Rust/Swift style. I haven't made a LES-to-Wasm (or vice versa) converter yet, and I've been puzzling over which language I should write a parser in... JS? C++? OCaml? Currently it's just C# which presumably doesn't interest folks here.