Modifying AST and generating source code

[jmthomas]

Forgive my ignorance but I'm trying to use Prism to modify Ruby source (add nodes) and then regenerate the source to be interpreted. I'm using Prism.parse(text) and then implementing a Prism::Visitor to potentially modify the nodes as they are visited. I assume this is possible but I haven't really seen any examples of it. Finally I would like to 'unparse' the AST back into source code.

The Python equivalent is indeed unparse which is where I'm taking many of the ideas. They also have a NodeTransformer which is allowed to modify the nodes in the visit pattern. I don't see equivalents in Prism but perhaps you're always allowed to modify the nodes ... this is Ruby afterall.

Thanks for any help pointing me in the right direction!

[jmthomas]

If it can't be done to generate the source, I don't require the generated source code as long as I can run the modified AST result. Something like eval the modified AST. Not sure if that is possible.

[jmthomas]

@kddnewton Any insights on this?
@Earlopain Any ideas?

[Earlopain]

Not quite sure why you pinged me on this 😅 I guess because I worked on the parser translator here, and RuboCop would need something like this as well in the future. But right now I haven't invested too much time into thinking about how RuboCop would make use of prism without a middleman. No ideas from me, sorry

[jmthomas]

@Earlopain thanks ... I was just looking for advice for someone knowledgeable about Prism to weigh in on whether this was even possible right now.

[mbj]

@jmthomas Evenutally unparser will support prism see:

• Requesting new release #3553
• Add prism parser support mbj/unparser#387
• Using prism translation instead of parser/current at 3.4 mbj/unparser#385

This is done via the translation layer to the parser AST, but I plan to drop to native as soon I do not have to support 3.2 anymore.

[rafaelfranca]

Closing this since it is more a question, but the answer is yes, you can use a visitor to modify the AST, or even build a new AST. And they use something unparser to generate the ruby code again.

[Earlopain]

unparser depends on the parser gem which I think makes it rather undesirable in combination with prism. I haven't checked it out yet but somewhat doubt that using the native prism ast would even work with unparser. This is not such a big deal for folks that currently use one of the translation layers since they can just continue to use what they already use (be that unparser, Parser::TreeRewrite or something else). But when you use prism bare it is a bit of a bother.

Previously kddnewton expressed desire to add such an api to prism itself here: whitequark/parser#1046 (comment)

[eregon]

Reading that thread, whitequark/parser#1046 (comment) seems also relevant.

I think unparser should/could work, by parsing, mutating the Prism AST, applying the translation to parser AST, use unparser. Obviously not the most efficient though.

[jmthomas]

I think unparser is probably what I'm looking for. Looks like they are waiting for a new release of Prism to incorporate a few fixes before they support Ruby 3.4 and beyond.

[Earlopain]

I think prism should make this possible, supporting the most basic operations wouldn't be too complicated (I guess). I image it to be used like this:

class RewriteVisitor < Prism::Visitor
  def initialize(rewriter)
    @rewriter = rewriter
  end

  def visit_def_node(node)
    @rewriter.replace(node.name_loc, "#{node.name}_renamed")
    super
  end
end

parse_result = Prism.parse(DATA.read)
rewriter = Prism::Rewriter.new(parse_result)
visitor = RewriteVisitor.new(rewriter)
visitor.visit(parse_result.value)

puts rewriter.source
# def foo_renamed
#   def bar_renamed
#   end
# end

__END__
def foo
  def bar
  end
end

I have something naive that accomplishes this, currently it's very far from ready. The API will obviously be very inspired by the parser gem. What do you think @kddnewton?