Define structures for event data

[dlebauer]

• Define JSON structure. reference management-events.schema.json
• provide example events.json
• Explain casting from geoTIFF to JSON
• Define format for agronomic events PecanProject/sipnet#20
• Explain casting from JSON to sipnet events file
• provide example events.tiff
• Get feedback from team

[dlebauer]

Example of minimal JSON events

Information what can be inferred from satellite data.

{
    "management": {
        "events": [
            {
                "mgmt_operations_event": "tillage",
                "date": "2022-03-01"
            },
            {
                "mgmt_operations_event": "tillage",
                "date": "2022-04-05"
            },
            {
                "mgmt_operations_event": "planting",
                "date": "2022-04-15"
            },
            {
                "mgmt_operations_event": "harvest",
                "date": "2022-10-25",
                "harvest_residue_placement": "left in field"
            },
            {
                "mgmt_operations_event": "planting",
                "date": "2022-11-15",
            }
        ]        
    }
}

This example is not compliant with the management-events.schema.json because it is missing required fields.

Example of events JSON with additional information gap-filled

This is information that will be required by SIPNET, and probably(?) also by the statistical downscaling algorithm. Thus most of this should be in the raster data layers.

{
    "management": {
        "events": [
            {
                "mgmt_operations_event": "tillage",
                "date": "2022-03-01",
                "tillage_implement": "subsoiler",
                "tillage_operations_depth": "20"
            },
            {
                "mgmt_operations_event": "tillage",
                "date": "2022-04-05",
                "tillage_implement": "disk, tandem",
                "tillage_operations_depth": "10"
            },
            {
                "mgmt_operations_event": "planting",
                "date": "2022-04-15",
                "planting_list": [
                    {
                        "planted_crop": "wheat",
                    }
                ]
            },
            {
                "mgmt_operations_event": "fertilizer",
                "date": "2022-03-20",
                "fertilizer_total_amount": 200,
                "fertilizer_material": "urea"
                "C_in_applied_fertilizer": 40,
                "N_in_applied_fertilizer": 93
            },
            {
                "mgmt_operations_event": "fertilizer",
                "date": "2022-07-15",
                "fertilizer_type": "organic material",
                "fertilizer_total_amount": 200,
                "organic_material": "compost"
                "C_in_applied_fertilizer": 40
                "N_in_applied_fertilizer": 5
            },
            {
                "mgmt_operations_event": "harvest",
                "date": "2022-10-25",
                "harvest_list": [
                    {
                        "harvest_crop": "wheat",
                        "harvest_operat_component": "grain"
                    }
                ],
                "harvest_residue_placement": "left in field",
                "harvest_yield_harvest_dw_total": 3000
            },
            {
                "mgmt_operations_event": "planting",
                "date": "2022-11-15",
                "planting_list": [
                    {
                        "planted_crop": "alfalfa",
                    }
                ]
            }
        ]
    }
}

Notes:

• it is not clear that the hierarchy provided by the <event>_list array is necessary for the CCMMF project. Retained to be consistent w/ management-events.json schema (though adding these w/o a list may also be compliant)
• gap filling will require a lot of assumptions and look-up tables to get from 'event occurred on date' to event attributes
  • for example, for fertilizer start with material and amount, then can look up C, N, NO3, NH4, Urea organic-n components
• can define mowing as a type of harvest as opposed to separate event type as in management-events.json schema

[dlebauer]

Raster File type

I first looked into using netcdf to support different event types, each with optional attributes, but @mdietze suggested geotiff for simplicity and familiarity, and this makes sense.

Using one file per event type per event_type per date, named something like <event_type>_YYYYMMDD.tif would be simpler to deal with.

Proposed GeoTIFF format:

• Band 1: event_present, 0 no_event, 1 event_presence
• Bands 2..n, additional attributes (tillage_depth, implement, fertilizer type, amount, etc)
  • each attribute could be a separate file
• Can use TIFF Tags, Optional tags for metadata

• One file per event type will be more modular, and easier to associate an event_type with specific attributes like tillage_depth or harvest_operat_component.

For satellite derived data:

• Keep gridded data to a minimum - starting with event type (boolean) and date (or year).
  • GeoTiff isn't designed for hierarchical data, but additional attributes can be added as layers or separate files
• Is there any information other than event type and date that we expect to be able to sense from satellite?
• Retain all observations - both event_present and no_event. This creates more data points, but knowing the last not detected would constrain the date the event occurred. Also, boolean is small and easily compressed. Alternatively in addition could create an additional attribute days_since_last_tillage_not_present

For inputs to statistical downscaling models:

• may likely want more information in gridded data layers (fert amount, tillage intensity, etc)

[dlebauer]

Example conversion from netcdf --> JSON

Update: prob. going with GeoTIFF

This is rough, but the goal of writing this out is to evaluate how hard the conversion will be.

Design objective is keeping the "gridded layer --> JSON --> SIPNET" use case simple.

# for event_type in c("tillage", ...){

nc <- nc_open(paste0(event_type, ".nc"))


date <- ncvar_get(nc, "time") |> 
   mutate(date = f(time) # convert from days since to YYYYMMDD
event <- ncvar_get(nc, "tillage_present")

latitude <- ncvar_get(nc, "latitude")
longitude <- ncvar_get(nc, "longitude")

event_date <- date[event] # now we are loosing not event present
previous_event_not_present_date <- date[f(event)] # some logic to find closest previous 0
# Close the NetCDF file
nc_close(nc)


data_list = list()
for (date_i in event_date){
  append(data_list, 
    list(
      date = date_i,
      mgmt_operations_event = event_type
      earliest_event_date = date - previous_event_not_present_date
    )
  )
)
data_list <- list("management" = list("events" = data_list))

# Convert the list to JSON
json_data <- toJSON(data_list, pretty = TRUE)
writeLines(json_data, file = paste0(round(latitude, 3), round(longitude, 3), event_type, ".json"))

[infotroph]

byte tillage_present(time, latitude, longitude) ;

I would have expected tillage not present would just be defined by an absence of events for that pixel. Is it necessary to track explicit event absence?

GeoTIFF should be simpler to deal with, not least of which is using YYYYMMDD instead of 'days since 1700-01-01'.
One file per event type per event_type per date, name something like <event_type>_YYYYMMDD.tif.

I think of GeoTiff as well optimized for single numeric rasters, so it's not immediately obvious to me how well they'd work for events with multiple attributes of several types -- See e.g. the last line of https://www.earthdata.nasa.gov/about/esdis/esco/standards-practices/geotiff: "GeoTIFF is not suitable for storing complex multi-dimensional data structures nor for storing vector data with many attributes or topology information."

[dlebauer]

Update - project will vectors instead of rasters for SIPNET model runs

[dlebauer]

Reopening and assigning to @hdpriest-ui.

Summary

• we will be using vector data with one polygon / field, so a standard GeoTIFF may not be necessary - but would be nice to have specification that can be represented in both vectors and rasters.
• we will need to store a time series of management events in some sort of PEcAn standardized format.

Outcome:

• standard format(s) for event data specified and documented in PEcAn documentation