Serialisation of static environments is slow for large functorised programs

[melsman]

Serialisation of static environments during static interpretation is slow for large functorised programs (e.g., MLton). When environments are serialised, which they are in particular when functor closures are serialised during static interpretation, it seems that quite some time is spend on discovering sharing (with parts of previously serialised environments).

See #97 .

[melsman]

Some issues fixed on commit fc16a08. We now avoid problematic hash collisions when serialising elaboration environments by transforming StatObject types, type schemes, type functions, and realisations into simpler non-unifiable objects prior to serialisation. The compilation of MLton on my Macbook Pro (2016) model has now dropped from 44min to 28min (when parallel compilation is used).

There are still some room for improvements, however. The Pickle structure does not allow us to express that parent picklers should not attempt to create sharing. Because the transformations are passed to the Pickle.convert0 function, children are repeatedly transformed (the functiontype_to_pty in StatObject.sml is called many times) whenever a parent needs to compute a hash, which happens to obtain sharing. The Pickle structure allows us to overwrite the hash and equality functions, which is one way of circumventing the problem, I guess. This solution, however, will create unequal-sized hash-table buckets, which we definitely want to avoid...

[melsman]

I have now improved serialisation further (commit 4b74e80) and arranged, semiautomatically, that sharing is not attempted for the pickle conversions (i.e., type_to_pty as discussed above). The Pickle library now supports a noshare combinator:

val noshare : 'a pu -> 'a pu

The combinator disables attempts to share data involving the values of type 'a (as that would trigger calls to eq and hash functions in the child pickler). This inheritance work for all combinators except dataGen, data2Gen, and data3Gen. Special combinators dataGenNoShare and data2GenNoShare are now available. These combinators explicitly disable sharing in serialised data.

It turns out, however, that there is a tradeoff between sharing and non-sharing. A good mix is to enable sharing at the environment level. To avoid that the type_to_pty functions and friends are called during hashing and equality tests, tailor-made hashing and equality functions are created in StatObject.sml (commit f4c27d0).

Compiling MLton is now down at 19m42s on my Macbook Pro (2016). There might still be possibilities for improvements. For instance, some hash collisions seem to appear during serialisation of RegionStatEnv values...

[melsman]

I'm closing this issue now. To further significantly improve compile times, we can perhaps support a compilation mode where groups of source files are compiled in sequence without serialising to disc. Mlb-files already have the bas...end grouping construct - perhaps an mlb-annotation can be used to specify that intermediate elaboration (and compilation) bases should not be serialised and that compilation of such a group should be implemented in a single in-memory process.