Create a conventional netcdf using to_netcdf()

[Marston]

Hi,

I'm having trouble understanding how to write a netcdf4 file with 2 groups (two sets with different dimensions) using the xarray.to_netcdf() in Python.
The simple example given in the Xarray doc is far too simple and I cannot see how to exploit the Xarray Dataset/DataArray creation to create a netcdf that looks like/similar this:

netcdf my_netcdf4 {

group: gaw {
  types:
    ubyte enum features_3d {Temperature = 0, Potential_temperature = 1,
        Pseudo_adiabatic_potential_temperature = 2,
        Dew_point_temperature = 3, Specific_humidity = 4,
        Relative_humidity = 5, Humidity_mixing_ratio = 6, Wind_direction = 7,
        Wind_speed = 8, Geopotential_height = 9, Geometric_Height = 10,
        Density = 11, Absolute_vorticity = 12, Relative_vorticity = 13,
        Relative_divergence = 14} ;

    ubyte enum features_2d {Temperature_2m = 0, Temperature_tropopause = 1,
        Latent_heat_flux = 2, Sensible_heat_flux = 3,
        Surface_Skin_Temperature = 4, Precipitable_water_ea = 5,
        Absolute_Humidity = 6, Maximum_absolute_humidity = 7,
        Horizontal_moisture_convergence = 8, Potential_evaporation_rate = 9,
        Vertical_speed_shear = 10, Pressure_at_the_tropopause = 11,
        Pressure_reduced_to_MSL = 12, Geopotential_height_fzglvl = 13,
        Geopotential_height_tro = 14} ;

  dimensions:
        time = UNLIMITED ; // (1 currently)
        lat = 721 ;
        lon = 1440 ;
        plevels = 30 ;
        features3d = 15 ;
        features2d = 15 ;

  variables:
        double time(time) ;
                time:units = "seconds since 1990-01-01 00:00:00" ;
                time:long_name = "time" ;
                time:axis = "T" ;
                time:calendar = "Standard" ;
        
        float lat(lat) ;
                lat:units = "degrees_north" ;
                lat:long_name = "latitude" ;
                lat:axis = "Y" ;
        
        float lon(lon) ;
                lon:units = "degrees_east" ;
                lon:long_name = "longitude" ;
                lon:axis = "X" ;

        int plevels(plevels) ;
        int features3d(features3d) ;
        int features2d(features2d) ;
        float variables3d(time, features3d, plevels, lat, lon) ;
        float variables2d(time, features2d, lat, lon) ;

  } // group gaw
}

I would also not like to create very large 5D and 4 D datasets and then dump it to a file but rather write one 2d array at a time and for 3d variables, one level at a time, so the chunking in this case would be a 2d (lat, lon) variable size. The reason for this is because I have only 5GB to work with. In the past, I've done this in python by first creating the netcdf container and then writing to it as described above, but this is slow and I would like to leverage Xarray in this and perhaps utilize its Dask interface.

[TomNicholas]

I'm having trouble understanding how to write a netcdf4 file with 2 groups (two sets with different dimensions) using the xarray.to_netcdf() in Python.

You should be able to do this using the group and mode="a" arguments to to_netcdf(), like this

import xarray as xr

ds1 = xr.Dataset({"a": 0})

ds2 = xr.Dataset({"b": 1})

ds1.to_netcdf("file.nc", group="A")
ds2.to_netcdf("file.nc", group="B", mode="a")

$ ncdump file.nc 
netcdf file {

group: A {
  variables:
  	int64 a ;
  data:

   a = 0 ;
  } // group A

group: B {
  variables:
  	int64 b ;
  data:

   b = 1 ;
  } // group B
}

Another way would be to use xarray-datatree.

The reason for this is because I have only 5GB to work with.

Do you mean you only have 5GB of RAM? Or 5GB of data? Dask is intended to help if it's the former.

In the past, I've done this in python by first creating the netcdf container and then writing to it as described above, but this is slow and I would like to leverage Xarray in this and perhaps utilize its Dask interface.

Consider using Zarr. It is similar to netCDF, also supports groups, but is chunked natively and will help you avoid various performance bottlenecks when you later use dask on your data.

[Marston]

Thank you. I'll look into these options.

[dcherian]

Great answer! We should add this info to the docs.

[Marston]

My platform where I do my processing only allows each user 5GB of RAM. This makes it hard to keep large datasets in memory.

[TomNicholas]

We should add this info to the docs.

I've opened #7338 to improve the documentation on reading and writing multiple groups to the same file.

My platform where I do my processing only allows each user 5GB of RAM.

Dask may be able to help you with this. I still recommend using Zarr. If you have other issues please open more discussions / issues!

[Marston]

One more thing I would like to ask:

I was not able to get xarray to write conventional variable attributes as seen in the dimensions:

    double time(time) ;
            time:units = "seconds since 1990-01-01 00:00:00" ;
            time:long_name = "time" ;
            time:axis = "T" ;
            time:calendar = "Standard" ;
    
    float lat(lat) ;
            lat:units = "degrees_north" ;
            lat:long_name = "latitude" ;
            lat:axis = "Y" ;
    
    float lon(lon) ;
            lon:units = "degrees_east" ;
            lon:long_name = "longitude" ;
            lon:axis = "X" ; 

How do I setup a time variable that is not a pandas string, for example?

[TomNicholas]

If you want to save the attributes of a variable as something other than a string you just need to make sure that the .attrs value is set to the type you want. e.g.

In [1]: group1['a'].attrs["x"] = 10

In [2]: group1.to_netcdf("file.nc", group="A")

In [3]: opened1 = xr.open_dataset("file.nc", group="A")

In [4]: opened1['a'].attrs["x"]
Out[4]: 10

In [5]: type(opened1['a'].attrs["x"])
Out[5]: numpy.int64

Is that what you were asking about?

[Marston]

Thank you