How can we export the statistics about the dataset?

[buffxz]

One typical use case about transformation is that, we compute the all the statistics summary about the dataset (e.g. the quantile value for the continuous features), then at training time, we can bucketize our continuous features using those quantile values.

But seems like the statistics object is not implemented.

How can we use the statistics?

[buffxz]

Seems the QuantilesSpec is not implemented:

def _impl_for_analyzer(spec, temp_assets_dir):
  if isinstance(spec, analyzers.NumericCombineSpec):
    return _NumericCombineAnalyzerImpl(spec)
  elif isinstance(spec, analyzers.UniquesSpec):
    return _UniquesAnalyzerImpl(spec, temp_assets_dir)
  else:
    raise NotImplementedError(spec.__class__)

[KesterTong]

At the moment, tf.Transform typically generates a statistic for a tensor, then uses it to transform a tensor. E.g. string_to_int generates a vocab and uses it to transform a string to an int.

For quantiles, the problem is that the QuantilesSpec is not yet implemented. When it is, you will be able to use tf.Transform to both compute, and apply, quantiles.

[katsiapis]

We plan to release Quantiles support as part of Tensorflow Transform 0.4 (which should happen not too long after Tensorflow 1.4 is released).

@KesterTong or @elmer-garduno, could we apply the 0.4 Milestone to this issue for tracking purposes?

[katsiapis]

tf.transform 0.4 has been released (and it includes a quantiles analyzer and a bucketize transform). With the bucketize transform you can both compute and apply the quantile buckets, as @KesterTong mentioned above.

As such, closing this issue. Feel free to reopen in case you run into any problems.