[Transforms] Minimal authoring support for down-level generator functions

[rbuckton]

Our current implementation of down-level support for generator functions in the transforms branch only supports generator functions created as part of the async/await transformation. This pull request relaxes the restrictions we added to the checker to allow developers to author generator functions that can be used in down-level host environments such as ES5/ES3. As a result, developers are able to use powerful features of generator functions in ES5/ES3 for scenarios such as coroutines in the same way TypeScript leverages this support for async functions.

The following features of ES6 generators are supported down-level:

• Supports generator functions or methods, returning an Iterator<T> (but not an IterableIterator<T>).
• Supports the yield expression.
• Supports the yield* expression when its value is an Iterator<T>.

The following features of ES6 generators are not supported down-level:

• Does not support using the result of a generator function or method directly in for..of, spread, or destructuring, as it is not an array.
• Does not support passing an array (or any other non-Iterator<T>) to the yield* expression.

The following features differ between ES6 and down-level generators when type-checking:

• The return type of a generator function or method is inferred as Iterator<T> rather than IterableIterator<T>.

Here are some examples of the above features and restrictions:

// Supported
function * f() { // ok, implicitly typed as Iterator<number>
  yield 0; // ok
}
function * g() { // ok, implicitly typed as Iterator<numer>
  yield* f(); // ok, f() is an Iterator<number>
  yield 1; // ok
}
const iterator = g(); // ok, returns Iterator<number>
iterator.next(); // ok, returns IteratorResult<number>
iterator.throw(e); // ok, throws exception at last yield
iterator.return(); // ok, returns IteratorResult<number>

// Not supported
const [a] = f(); // not ok, destructuring of Iterator is not supported in ES5/3.
console.log(...f()); // not ok, spread of Iterator is not supported in ES5/3.
for (const x of f()); // not ok, for..of over Iterator is not supported in ES5/3.

function* f() {
  yield* []; // not ok, Array is not an Iterator.
}

The reason we've chosen to only support iterators for yield* is to allow for generator composition.

Please note that it is possible to partially work around the above restrictions in your code by implementing your own conversion functions, for example:

function toArray<T>(iterator: Iterator<T>): T[] {
  const array: T[] = [];
  for (let result = iterator.next(); !iterator.done; result = iterator.next()) {
    array.push(result.value);
  }
  return array;
}
function toIterator<T>(array: T[]): Iterator<T> {
  let i = 0;
  return {
    next(): IteratorResult<T> {
      if (i < array.length) {
        return { value: array[i++], done: false };
      }
      return { value: undefined, done: true };
    }
  };
}

const [a] = toArray(f()); // ok, destructuring of Array is supported in ES5/3.
console.log(...toArray(f())); // ok, spread of Array is supported in ES5/3.
for (const x of toArray(f())); // ok, for..of over Array is supported in ES5/3.

function* f() {
  yield* toIterator([1]); // ok, yield* over Iterator is supported in ES5/3.
}

We do not perform the above operations by default as they do not have the same runtime semantics as ES6 (i.e. toArray eagerly consumes the iterator), and we cannot guarantee the availability of a global Symbol.iterator down-level.

[rbuckton]

I discussed this with @mhegazy prior to leaving for vacation, with the exception of the support for yield*. I feel that adding yield* support purely for iterators and not arrays makes generator functions more compositional. As a result, you can lazily delegate operations to sub-generators using yield* following the majority of ES6 runtime semantics (other than the Symbol.iterator lookup).

If we chose to have yield* only operate on array then we lose the ability to delegate operations such as return(), throw(), or next(value) to nested generators as we are forced to evaluate eagerly using something like toArray above.

@DanielRosenwasser, @ahejlsberg, @mhegazy, et. al. - Any questions or concerns on the PR or my rationale regarding yield*?

[weswigham]

Instead of suggesting using a standard function, why not suggest using a rest parameter, ie ...args: any[]?

[rbuckton]

The same argument could be made for arrow functions, which have pretty much the same error message. I'd rather not overcomplicate the error message here.

[rbuckton]

Closing this PR based on design meeting feedback.