Python 3 problems with versions > 1.1.6

[kwilcox]

Appending to a file using a context manager seems to close the Dataset (or it gets garbaged collected) before completing, usually resulting in a segfault.

This may have to do with #251.

I've used 1.1.6 for a long awhile now (it "just works" for me) and I was trying to upgrade things to 1.2.1 (so I can use the now built-in get_variables_by_attributes).

Here is a simple script to demonstrate. File to go along with script (looks in the same directory for it). Everything works fine with Python 2.7, but fails with Python 3.4 when the netcdf4-python version is above 1.1.6. I didn't test 1.1.7 because that release never made it to conda.

[jswhit]

Confirmed. Don't understand why the context manager or the garbage collector should behave differently in 3.4 though.

[jswhit]

Don't believe it has anything to do with the context manager or garbage collection. Even if I modify your script to just open the file and try to modify the attributes, it segfaults with python 3.4 (but not 2.7).

import netCDF4
nc = netCDF4.Dataset('axiom_cruise_profiles.nc', 'a')
for vname in nc.variables.keys():
    v = nc.variables[vname]
    if hasattr(v,'units'):
        v.units = 'foobar'
nc.close()

[jswhit]

Probably related to pull request #389. Even though the attributes are stored as ascii text in the original file, they are written as strings in python 3, and the C lib does not like overwriting a text attribute with a vlen string.

[jswhit]

Note that replacing

v.units = 'foobar'

with

v.units = numpy.array('foobar',dtype='S6')

fixes the segfault on python 3.4, confirming the problem is associated with trying to write the attribute as a string and not text.

[jswhit]

@shoyer, do you have any thoughts on how to handle this?

[shoyer]

Well, the C lib should probably not be crashing when you ask it to set an attribute to a new type :). I assume this also happens if you try to do other types of attribute type conversion?

One sensible approach would be to delete and recreate attributes if they exist but have a different type before attempting to set them.

[jswhit]

The C lib appears able to handle conversions of numeric types, just not text to strings.

from netCDF4 import Dataset
import numpy as np
nc = Dataset('test_att_convert.nc','w')
#nc.foo = np.array('bar','S')
nc.foo = np.array([1,2,3],np.int8)
nc.close()
nc = Dataset('test_att_convert.nc','a')
#nc.foo = np.array('bar','U')             <--- causes segfault
nc.foo = np.array([1,2,3],np.float64)
nc.close()

[jswhit]

Unidata/netcdf-c#149

[jswhit]

workaround suggested by @shoyer in pull request #486

[jswhit]

I've committed a workaround for this in master, and @WardF has fixed the problem in netcdf-c master.

Thanks for your quick action @WardF !