SparseDataFrame.to_coo does not convert the default fill value when is not 0

[gm-spacagna]

DataFrame.sparse.to_coo() should raise when the fill_value is not zero, since scipy.sparse only supports filling with 0.

In [21]: df = pd.DataFrame({"A": pd.SparseArray([1, 1, 1, 2], fill_value=1)})

In [22]: df.sparse.to_coo().todense()
Out[22]:
matrix([[0],
        [0],
        [0],
        [2]])

That's incorrect. It should instead raise a ValueError with a nice message.

[TomAugspurger]

Can you shorten the example to remove the unnecessary lines?

Can you show the actual output and the expected output?

Finally, can you try on 0.24.0rc1? I don't recall if these were fixed or not, but sparse got an overhaul.

pip install --pre pandas or conda install -c conda-forge/label/rc pandas in a new env.

[gm-spacagna]

Can you shorten the example to remove the unnecessary lines?

Can you show the actual output and the expected output?

Finally, can you try on 0.24.0rc1? I don't recall if these were fixed or not, but sparse got an overhaul.

pip install --pre pandas or conda install -c conda-forge/label/rc pandas in a new env.

I have updated to 0.24.0rc1 and added the output of each block. The problem persists.

[gm-spacagna]

I guess the right behaviour of fillna on a sparse dataframe should be sparse_df_filled =sparse_df.to_dense().fillna(-1).to_sparse() or we should not support a fill_value different from zero, just like scipy.

[jorisvandenbossche]

To avoid surprises, I would raise an informative error when the fill value is not zero, pointing people towards doing changing the fill value before converting to scipy (since scipy simply does not support any other fill value).

Ideally we do this before 0.25, and only for the DataFrame.sparse.to_coo version, since that is still new.