structural hashing truncates numbers to their integer part

[pedramamani]

The new structural hashing coerces numbers (64-bit floats) into 32-bit integers in db-ivm/src/hashing/murmur.ts thus hashing 2.0 and 2.5 the same. In queries with distinct(), this prevents propagation of updates that change only the decimal part of a number.

Minimal repro: stackblitz.com/edit/tsdm

[pedramamani]

GPT-5 recommends this instead.

const buf = new ArrayBuffer(8)
const view = new DataView(buf)
view.setFloat64(0, chunk, true) // little-endian
for (let i = 0; i < 8; i++) this._writeByte(view.getUint8(i))

[kevin-dp]

Hi @pedramamani, thank you for finding this bug, its root cause, and proposing a solution! I opened PR #525 that modifies the unit tests to reproduce the bug and fixes it. I also opened an issue tc39/proposal-composites#22 on the composites proposal which is the implementation of structural hashing that we based ours on.

[pedramamani]

Thanks! Having read your conversation with acutmore, I suppose the bigger issue is we falsely assume same hash implies same values.

[kevin-dp]

Thanks! Having read your conversation with acutmore, I suppose the bigger issue is we falsely assume same hash implies same values.

Indeed. I raised this with @samwillis yesterday. We're going to look into this soon. We think we may want to swap out the 32 bit murmurhash by a 64 bit hashing implementation. That should significantly reduce the chances of a collision especially since ts/db collections aren't expected to contain tens or hundres of millions of rows.