Round trip converts text attributes to strings

[jjhelmus]

When netcdf4-python is used to read a "text" attribute from a file and write that attribute to a second file the resulting attribute will be a string where-as the original attribute was "text".

For example:

import netCDF4

# create a reference netCDF file with a text (non-unicode) attribute
ds1 = netCDF4.Dataset('test1.nc', 'w')
ds1.setncattr('foo', 'bar')
ds1.close()

# copy the attribute to a second file, result is a string attribute
ds1 = netCDF4.Dataset('test1.nc', 'r')
ds2 = netCDF4.Dataset('test2.nc', 'w')
ds2.setncattr('foo', ds1.getncattr('foo'))
ds1.close()
ds2.close()

ncdumps of the two files show that the text attribute was changed to a string attribute.

 ncdump test1.nc; ncdump test2.nc
netcdf test1 {

// global attributes:
        :foo = "bar" ;
}
netcdf test2 {

// global attributes:
        string :foo = "bar" ;
}

From what I can tell this is a result of #389 which addressed #388. Unicode Python variables are written as string attributes (using nc_put_att_string) but reading a "text" or string attribute returns unicode. One possible fix would be to have "text" attribute be returned as str (Python 2) or bytes (Python 3).

[jjhelmus]

I think the issue is that both NC_CHAR and NC_STRING attribute types are decoded to unicode on lines 1101-1121 of _netCDF4.pyx. I'm not sure what would happen if NC_CHAR attributes were not decoded on line 1113 but it might be an option.

[jswhit]

I don't think there is any easy way to preserve the round-trip, especially with python 3, without adding significant complication and confusion. Under what conditions would this actually be a problem?

[jjhelmus]

The issues seems to be that other netcdf software does not support the "string" type qualifier on attributes. In the case I ran into was that one of our users was having issues reading a file in Matlab after the "string" attribute modification were made (ARM-DOE/pyart#487). A similar issue in xarray(pydata/xarray#681) has a bit more discussion and an alternative to try encoding all strings to ascii prior to writing the file.

@shoyer or @jhamman might be able to provide more details on on this topic.

[jswhit]

The trick suggested in pydata/xarray#681 could be used in your original example, i.e:

import netCDF4
def maybe_encode(string, encoding='ascii'):
    try:
        return string.encode(encoding)
    except UnicodeEncodeError:
        return string
# create a reference netCDF file with a text (non-unicode) attribute
ds1 = netCDF4.Dataset('test1.nc', 'w')
ds1.setncattr('foo', 'bar')
ds1.close()
# copy the attribute to a second file, result is a string attribute
ds1 = netCDF4.Dataset('test1.nc', 'r')
ds2 = netCDF4.Dataset('test2.nc', 'w')
ds2.setncattr('foo', maybe_encode(ds1.getncattr('foo')))
ds1.close()
ds2.close()

[jhamman]

@jswhit - we we're going to take the maybe_encode route in xarray? Would you consider putting it in netcdf4 instead? I don't have strong opinions but as far as I'm concerned, it only needs to go in one library.

[jswhit]

If we do that, then all attributes that can be encoded as ascii will be returned as ascii text. I'm not sure this is a good idea, but I'd like to hear opinions.

[jswhit]

I spoke too soon - the solution is really pretty trivial (pull request #530). If the attribute requested is of type NC_CHAR, the result is returned as an ascii (not unicode) string.

[jswhit]

Note: to ensure that an attribute gets written as ascii (NC_CHAR), even with python3, use

ds1.setncattr('foo', b'bar')

[jswhit]

This example now works as expected in python 2 and 3:

import netCDF4
# create a reference netCDF file with a text (non-unicode) attribute
ds1 = netCDF4.Dataset('test1.nc', 'w')
ds1.setncattr('foo', b'bar')
ds1.close()
# copy the attribute to a second file, result is an ascii attribute
ds1 = netCDF4.Dataset('test1.nc', 'r')
ds2 = netCDF4.Dataset('test2.nc', 'w')
ds2.setncattr('foo', ds1.getncattr('foo'))
ds1.close()
ds2.close()

A test is still needed for pull request #530.

[jswhit]

This fix could break code that assumes that ascii (NC_CHAR) attributes are always returned as strings. In python3, NC_CHAR attributes will now be returned as bytes. This in fact broke some utility functions in netcdf4-python.

[jjhelmus]

This fix could break code that assumes that attributes are always returned as strings. In python3, NC_CHAR attributes will now be returned as bytes. This in fact broke some utility functions in netcdf4-python.

Agreed. I think it Python 2 returning str objects when the attribute is stored as a NC_CHAR and unicode when stored as NC_STRING makes sense. In Python 3 this gets a bit more difficult as both ascii and other unicode encoding should likely be returned as str as there is no unicode. Returning ascii as bytes in Python 3 will cause some issues since bytes behavior different than str (for examples "hello" + b" world" raises an TypeError.

Perhaps another option is to have an toggle similar to set_auto_mask which controls how unicode is written in attributes. Maybe set_char_attr_on and set_char_attr_off? When on unicode objects will create NC_CHAR attributes.

[jjhelmus]

Just tested one of the file with a NC_STRING type global attribute with Matlab 2011b. I can confirm that at least this version of Matlab cannot read these attributes but can read NC_CHAR type attributes.

>> ncdisp('test.nc')
Source:
           /home/jhelmus/test.nc
Format:
           netcdf4
Global Attributes:
           foo1 = 'bar'
           foo2 = UNSUPPORTED DATATYPE

>> ncid = netcdf.open('test.nc', 'NOWRITE');
>> netcdf.getAtt(ncid,netcdf.getConstant('NC_GLOBAL'), 'foo2')
Error using netcdf.getAtt (line 64)
Attribute datatype 12 is not recognized.

>> netcdf.getAtt(ncid,netcdf.getConstant('NC_GLOBAL'), 'foo1')

ans =

bar

[jswhit]

Returning NC_CHAR attributes as strings with python 3 will not fix your round-trip issue though. I'm leaning towards keeping NC_CHAR attributes as bytes in python 3. We could add an 'encoding' arg to getncattr that would convert to ascii on python 3.

[DennisHeimbigner]

This may be just a noise comment, but remember how this is handled in C.
If the type of the attribute is char, then an array of characters is returned (which in c looks
like a non-null terminated string). This is true for netcdf-3 and netcdf-4.
If the type is string (netcdf-4 only) then a vector of strings is returned.
Technically, the charset in both cases is UTF-8, but for chars, this reduces to ascii
(the 8 bit only subset of UTF-8). Of course people routinely cheat and include non-ascii characters
when they are really storing a string as an array of characters. They will often add the null termination
character as well.
So, out of curiosity, why does python not follow this convention also?