Issues with hashing/saving QuadPrecDType

[dlakaplan]

Within PINT we currently need to be able to hash individual QuadPrecision values (for data integrity checks) and save arrays to npz files for rapid access. I don't think either of those currently work:

import numpy as np
from numpy_quaddtype import QuadPrecDType, QuadPrecision

hash(QuadPrecision(np.float32(2)))

fails (while hash(np.float128(2)) works using the native numpy type).

And:

d = np.array([1, 2, 3], dtype=QuadPrecDType)
np.savez('test.npz',d=d)

fails while the same for d=np.array([1,2,3],dtype=np.float128) works.

My hope is that these are both relatively easy fixes?

[ngoldbaum]

My hope is that these are both relatively easy fixes?

Hashing should be easy to fix.
Unfortunately saving to npz files is going to need changes in NumPy: numpy/numpy#25374 (comment).

Most of the discussion there is around StringDType, which raises issues that a quad dtype doesn't have because it's not variable-width. In principle it should be possible to get fixed-width user dtypes working in the npy file format. There might be issues though because the npy format has backwards compatibility concerns it needs to deal with and old parsers shouldn't break when loading an npy file written using a new writer.

[dlakaplan]

OK, thanks. If saving files will be a long-term issue, we might have to see how best to resolve that.

[SwayamInSync]

I'll investigate on this about what can be incompatible, but a work-around can be a helper that cast quad->string (fix-width) and then store and while loading do the reverse (not sure, but @ngoldbaum we might can implement this internally as well, if strings are compatible with npz)
This way precision can be maintained, same goes for having tp_hash as I was just checking and CPython hash only does the double values

[dlakaplan]

So actually, I misspoke slightly earlier. Rather than saving with np.savez, we pickle explicitly, but I think that has the same underlying code.

[ngoldbaum]

Oh that's good, pickle is actually easier to fix. StringDType supports pickle. @SwayamInSync let me know if you want a pointer.

[dlakaplan]

Interesting. It looked to me that it was trying to fall back on pickle earlier:

In [5]: np.savez('test.npz',d=d)
/Users/kaplan/miniforge3/envs/nptest/lib/python3.14/site-packages/numpy/lib/_format_impl.py:393: UserWarning: Custom dtypes are saved as python objects using the pickle protocol. Loading this file requires allow_pickle=True to be set.
  d['descr'] = dtype_to_descr(array.dtype)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[5], line 1
----> 1 np.savez('test.npz',d=d)

File ~/miniforge3/envs/nptest/lib/python3.14/site-packages/numpy/lib/_npyio_impl.py:673, in savez(file, allow_pickle, *args, **kwds)
    580 @array_function_dispatch(_savez_dispatcher)
    581 def savez(file, *args, allow_pickle=True, **kwds):
    582     """Save several arrays into a single file in uncompressed ``.npz`` format.
    583 
    584     Provide arrays as keyword arguments to store them under the
   (...)    671 
    672     """
--> 673     _savez(file, args, kwds, False, allow_pickle=allow_pickle)

File ~/miniforge3/envs/nptest/lib/python3.14/site-packages/numpy/lib/_npyio_impl.py:786, in _savez(file, args, kwds, compress, allow_pickle, pickle_kwargs)
    784         # always force zip64, gh-10776
    785         with zipf.open(fname, 'w', force_zip64=True) as fid:
--> 786             format.write_array(fid, val,
    787                                allow_pickle=allow_pickle,
    788                                pickle_kwargs=pickle_kwargs)
    789 finally:
    790     zipf.close()

File ~/miniforge3/envs/nptest/lib/python3.14/site-packages/numpy/lib/_format_impl.py:762, in write_array(fp, array, version, allow_pickle, pickle_kwargs)
    760     if pickle_kwargs is None:
    761         pickle_kwargs = {}
--> 762     pickle.dump(array, fp, protocol=4, **pickle_kwargs)
    763 elif array.flags.f_contiguous and not array.flags.c_contiguous:
    764     if isfileobj(fp):

RuntimeError: Custom dtypes cannot use the default pickle implementation for NumPy dtypes. Add a custom pickle implementation to the DType to avoid this error
when serializing tuple item 2
when serializing numpy.ndarray state
when serializing numpy.ndarray object

but maybe that wasn't intentional. In any case, if you can get a variant of this working that would be great.

[SwayamInSync]

Oh that's good, pickle is actually easier to fix. StringDType supports pickle. @SwayamInSync let me know if you want a pointer.

Yup that would be great :) I can direct take the inspiration

[ngoldbaum]

I initially added pickle support in https://github.com/numpy/numpy-user-dtypes/pull/33/files. One load-bearing piece is to add NPY_LIST_PICKLE to the dtype flags: numpy/numpy-user-dtypes@457fc76

You can see the latest version of the implementation in numpy by looking at stringdtype__reduce__: https://github.com/numpy/numpy/blob/dd80e20bc4b7f78ba92177c87d11c236bb7d194d/numpy/_core/src/multiarray/stringdtype/dtype.c#L725-L757

[dlakaplan]

Hi, @SwayamInSync Actually, I was looking for updates on this issue (not the one I mentioned - apologies). Are there updates here I could test?

[SwayamInSync]

Yeah so here the pickling/unpickling of arrays support is added.
Hashing of scalar is todo, will wrap that as well very soon