More control over Go lang (and other lang) tagging.

[rocky]

I am writing a debugger for Go and came across this project via Dmitri Shuralyov and his use of this which uses go/scanner.

One improvement I'd like to see is more fined-grained control over what gets tagged. Rather than reinvent stuff when not necessary, I consulted Pygments to see what it has.

This is from pygments/formatters/terminal.py;

    Whitespace:         ('lightgray',   'darkgray'),
    Comment:            ('lightgray',   'darkgray'),
    Comment.Preproc:    ('teal',        'turquoise'),
    Keyword:            ('darkblue',    'blue'),
    Keyword.Type:       ('teal',        'turquoise'),
    Operator.Word:      ('purple',      'fuchsia'),
    Name.Builtin:       ('teal',        'turquoise'),
    Name.Function:      ('darkgreen',   'green'),
    Name.Namespace:     ('_teal_',      '_turquoise_'),
    Name.Class:         ('_darkgreen_', '_green_'),
    Name.Exception:     ('teal',        'turquoise'),
    Name.Decorator:     ('darkgray',    'lightgray'),
    Name.Variable:      ('darkred',     'red'),
    Name.Constant:      ('darkred',     'red'),
    Name.Attribute:     ('teal',        'turquoise'),
    Name.Tag:           ('blue',        'blue'),
    String:             ('brown',       'brown'),
    Number:             ('darkblue',    'blue'),

There are some things above that aren't relevant to Go (but you might want to add them for other languages), and there are some things that should be added above for go like Rune, possibly Case label and so on. So here is a list below:

• Whitespace
• Comment
• Keyword
• Type name
• Built in
• Function
• Variable
• Exported name
• Number
• Rune

Kind is a uint8 so there is plenty of space to add constants to this.

If you would like me to create a pull request for this, let me know.

Thanks.

[dmitshur]

For reference, the current list of Kind values is:

const (
    Whitespace Kind = iota
    String
    Keyword
    Comment
    Type
    Literal
    Punctuation
    Plaintext
    Tag
    HTMLTag
    HTMLAttrName
    HTMLAttrValue
    Decimal
)

Which ones do you think are missing?

I think they were based on google-code-prettify CSS classes.

[rocky]

See #8

Pygments uses "Number" instead of "Decimal" and I think Pygments is right here. Because numbers in hexadecimal would be formatted the same way and even though 010 could be decimal, it is in fact octal.

If you look at the list in above-cited issue, or the above list you'll see things like Rune, built-in function name, and exported name (capitalized variable).

For the record, I think it wrong to try to reuse one of the above names (like HTMLTag) to mean something else (like Rune) when there is plenty of room for more constants.

[dmitshur]

Ok, I agree that Number sounds a little better that Decimal. What about the others?

Can you make your proposal a little more formal and concrete by showing suggested new values for Kind? For example, something like this:

 const (
     Whitespace Kind = iota
     String
     Keyword
     Comment
     Type
     Literal
     Punctuation
     Plaintext
     Tag
-    HTMLTag
-    HTMLAttrName
-    HTMLAttrValue
-    Decimal
+    yourSuggestionHere?
+    ???
+    ???
+    Number
)

I think changing those consts can be done without breaking anything (it might require updating some dependent packages), so if what you're proposing is a worthwhile improvement, it might be possible, if @sqs agrees. We've made a change to make those const names more idiomatic in the past, see #6.

[sqs]

I'm fine changing them.

[rocky]

Below is a stab at this. But first are comments about the list.

Although I think the 13 constants that are cited above for Kind is a bit too small, I don't think it important to obsess about this to try to get it right to the last detail. That said, it's probably worth putting a little effort to get it roughly right as I've tried below.

I think it important to try to include examples of what was intended as I've done below. There are just so many things that could mean more than one thing. For example, is "KeywordType" the literal "type" or is it a type name. Are things like C's "static", "extern" Keyword types? Dunno. Below is my guess; please double check with pygments which is where I got this from. (And I chose that simply because it seems to be parallel to this effort and has had to deal with this already).

KeywordType was originally Keyword.Type and I think in pygments the intent is that this a specific kind of Keyword. So, possibly NameType is what should be below. And possibly there are languages where separating the keyword "type" would make visual sense. Again, Dunno. You all discuss and decide.

You'll see I included an alias at the end of the list. As far as I can tell, a Go Rune is exactly the same as a C literal char. (And I think Literal was not intended to be the same as a C literal or Go Rune, but I don't know). In those cases like this where at least to me it is clear that Rune in Go means literal char in C, I do think it helpful to use the same number, because the styling or appearance of a C literal char is probably the same as that of a Go Rune.

I believe when one has an assignment like this, iota is going to still increment. So that's why I put them at the end of the list.

Lastly, I am not against Decimal, HTMLTag, HTMLAttrName, HTMLAttrValue. I just don't know what they are, or how exactly they'd be used.

const (
    Whitespace  Kind = iota

    Comment
    CommentPreproc // e.g. #ifdef, #else in C/C++ ?
    Keyword        // e.g. "if", "for" in many languages
    KeywordType    // the specific keyword "type" ? 
    Label          // C/C++ or Perl labels, e.g: loop:
    Literal
    LiteralChar       // 'a' or '\n' in C
    LiteralSymbol   // :foo in Ruby, 'foo in Lisp
    NameBuiltin   // e.g. len() in Go or Python
    NameClass     // Class name in Java, C++, Ruby, Python...
    NameDecorator // @classmethod in Python?
    NameException // e.g. Ruby/Python exception name, e.g. IOError
    NameFunction  // user-defined function or method, e.g gcd in "def gcd(a,b)" in Ruby, Python
    NameConstant  // e.g Ruby or Perl constant, The Pi in Ruby:  Pi=3, or Perl: use constant Pi => 3 
    NameType    // e.g. int_t in `typedef int int_t` in C or `type int_t  int` in Go
    NameTypeBuiltin // e.g. int in C, C++, Go
    NameVariable  // user-defined program variable
    NameNamespace // package *name* in Go, Perl; module name in Python
    Number        // e.g 99, 0xff, 010, possibly 1.0 or 1.0i depending on lexer
    OperatorWord  // +, *, <= in most languages
    Plaintext
    Punctuation
    String

    // From pygments/formatters/terminal.py not totally sure what there are
    NameTag
    GenericDeleted
    GenericInserted
    GenericHeading
    GenericSubheading
    GenericError
    Error
    // Aliases. Need to be at the end
    LiteralRune = LiteralChar // Go 'a' or '\n'
)

[rocky]

Hmm. Maybe NameTag above is the same thing as LiteralSymbol.

And maybe Label should be NameLabel.

[dmitshur]

Hi @rocky,

Thanks for the proposal. After thinking about it, I'm beginning to think that what you're suggesting falls outside of scope of this library, and may be better suited as a separate standalone library.

My thinking is that this package is for highlighting code; in most practical use cases I've only seen a couple styles. I mean, it's really hard to actually have more than 20 visual styles that are still distinguishable one from another.

Go code as displayed on golang.org website actually only has 2 styles, green text for comments and black text for all other code.

Most other websites that display code that I'm aware of, including github.com, code.google.com, bitbucket.com, sourcegraph.com, gotools.org, have at most 5-10 different styles.

It doesn't seem to me that most users of this package would want to care about such detailed distinctions; it's more of keyword vs literal vs type, etc. So it would be computationally wasteful to get that extra information, and harder to use the API. If you really want such high detail level, perhaps you should be using the go/scanner and go/token packages directly?

I'd be curious to hear @sqs thoughts on this.

[rocky]

After thinking about it, I'm beginning to think that what you're suggesting falls outside of scope of this library, and may be better suited as a separate standalone library.

I don't mind using another library or modifying this one, but I'm lost here. What is the scope of this library?

The README seems to suggest that JavaScript, Java, Ruby, Python, Go, and C are intended to be used.

My thinking is that this package is for highlighting code; in most practical use cases I've only seen a couple styles. I mean, it's really hard to actually have more than 20 visual styles that are still distinguishable one from another.

Ok. But what does that have to do with say, allowing those few styles decide that they'd like to distinguish Runes from other things in Go or distinguish decorator symbols in Python from other kinds of variables?

It's not about how many code styles are used, just what is made available to the styles. Whether or not this package uses the distinctions defined in syntaxhighlight.Kind, not having them defined in this package precludes someone else writing a package and using this as a base. That's why I brought up the issue.

It doesn't seem to me that most users of this package would want to care about such detailed distinctions;

You may be right. I'm going by largely what pygments decided on. You asked for a list, so you got one. You want a smaller list? Okay, change it.

it's more of keyword vs literal vs type, etc. So it would be computationally wasteful to get that extra information, and harder to use the API.

I don't see it as computationally wasteful. I bet if you benchmarked the difference, you'd find the time and space negligible. As Knuth says, premature optimization is the root of evil. That said, if you really do find a big time/space difference I'd be very interested.

The scanner already breaks things out into detailed tokens, so it is just a matter how the case statement that looks at the token classes combines these into Kind. Of course, if there is a map that has 20 entries instead of 10, or a case statement that has 20 clauses instead of 10, yes that's more space, but probably not much for time. The extra time to index a table of 20 things instead of 10, or select them is negligible.

If you really want such high detail level, perhaps you should be using the go/scanner package directly?

Well, that's what shurcooL/go/highlight_go was using in distinction to this package. The only reason I'm not using that is because of the bug mentioned in that project, where white space attributes come from the token that follows the white space. (And by the way, fixing that is really tangential to this discussion).

I never said I wanted such a high level of detail; again, you asked for a list. But a little more detail would be nice. How about if we start out with making a distinction between a C literal/Go Rune and a character string?

[dmitshur]

I don't mind using another library or modifying this one, but I'm lost here. What is the scope of this library?

The README seems to suggest that JavaScript, Java, Ruby, Python, Go, and C are intended to be used.

That is correct. This package is for general syntax highlighting of code, including many languages:

// Package syntaxhighlight provides syntax highlighting for code. It currently uses a
// language-independent lexer and performs decently on JavaScript, Java, Ruby, Python,
// Go, and C.

And not just Go.

You may be right. I'm going by largely what pygments decided on. You asked for a list, so you got one. You want a smaller list? Okay, change it.

My goal is not to try to be as thorough and detailed as possible, but rather as simple as possible while offering just enough functionality to be useful in the contexts it will be used.

How about if we start out with making a distinction between a C literal/Go Rune and a character string?

I think that is reasonable. You mean differentiating "some string" vs 'a', right?

[rocky]

You mean differentiating "some string" vs 'a', right?

Yep.

[dmitshur]

My highlight_go package already does that:

"some string" -> syntaxhighlight.String
'a' -> syntaxhighlight.Decimal

It currently returns syntaxhighlight.Decimal for character literals, as well as numbers. But it's different from string literals.

[owenthereal]

👍 to following Pygment's token names since https://github.com/jingweno/ccat can benefit from reusing the same theme from Pygment 😸