feat: format with generator

[H4ad]

This PR adds a new method to be able to consume the format edits using generators, which can help reduce the amount of memory needed to process a heavier json.

With the current implementation, using the benchmark of #81, we had a slowdown while using sync result:

End: 1264.72ms
RSS: 693.6484375MB

But, if we consider how this API can be used, for example to replace the current implementation of format on vscode-json-language-service, we have good results:

Before:

End of Sync: 1832.79ms
RSS: 937.5625MB

text-format-with-map.js

const heavyJson = require('fs').readFileSync('./rrdom-benchmark-1.json', 'utf8');

const lib = require('./lib/umd/main');

class Range {
  construtor(start, end) {
    this.start = start;
    this.end = end;
  }
}

class TextEdit {
  construtor(range, content) {
    this.range = range;
    this.content = content;
  }
}

const startTime = performance.now();
const edits = lib.format(heavyJson, undefined, {
  tabSize: 2,
  insertFinalNewline: true,
  insertSpaces: true,
});
const mappedEdits = edits.map(e => {
  return new TextEdit(
    new Range(e.offset, e.offset + e.length),
    e.content,
  );
});
console.log(`End of Sync: ${(performance.now() - startTime).toFixed(2)}ms`);
console.log(`RSS: ${process.memoryUsage.rss() / 1024 / 1024}MB`);

After:

End of Generator: 1661.84ms
RSS: 471.7890625MB

test-format-with-map-generator.js

const heavyJson = require('fs').readFileSync('./rrdom-benchmark-1.json', 'utf8');

const lib = require('./lib/umd/main');

class Range {
  construtor(start, end) {
    this.start = start;
    this.end = end;
  }
}

class TextEdit {
  construtor(range, content) {
    this.range = range;
    this.content = content;
  }
}

const startTime = performance.now();
const mappedEdits = [];

for (const e of lib.formatGenerator(heavyJson, undefined, {
  tabSize: 2,
  insertFinalNewline: true,
  insertSpaces: true,
})) {
  mappedEdits.push(new TextEdit(
    new Range(e.offset, e.offset + e.length),
    e.content,
  ));
}

console.log(`End of Generator: ${(performance.now() - startTime).toFixed(2)}ms`);
console.log(`RSS: ${process.memoryUsage.rss() / 1024 / 1024}MB`);

This new approach will be faster and also will consume less memory 🚀

[H4ad]

Hey @aeschli, did you have time to take a look on this one?

[H4ad]

@aeschli Sorry for pinging again, but can you take a look on this one?

[aeschli]

Using the generator is an interesting idea. It's not something we in VS Code could use right away given the way the rest of our APIs are designed.
So I'm a bit hesitant about adding a new API (and maintaining it) unless there's considerable demand for it.

[H4ad]

I think this feature can help reduce memory usage but it also introduces generators by default even when we don't want them, which leads to slower formatting since generators are slower.

I hope in the future vscode will support a way to send partial updates to the UI, if it happens, then we can use a feature like this with the full power.

[aeschli]

At this moment there's just too much missing to make the generator API useful.

• LSP doesn't support streaming for format
• VS Code API expecting a TextEdit array, not a stream
• Applying a TextEdit array to a document is not trivial. In a TextEdit[], all TextEdit ranges refer to the original document, so once you have applied an edit you have to transform the range of the next TextEdit.

I don't see us getting there any time soon. I'd rather not maintain an API that we don't use. If you depend on it, I think it is fait to copy the code and maintain it on your side
Closing the PR