Insert plot level height and / or percent cover into BETYdb

[dlebauer]

Description

@ZongyangLi has generated extractors that summarize plot level phenotypes. The first two are height and percent cover.

These need to be inserted into the BETYdb API.

TODO

• Need to make sure that we have plots defined in BETYdb consistent with the geospatial bounding boxes that you used.
  • Season 2 bounding boxes based on gantry coordinates are here: http://terraref.ncsa.illinois.edu/bety/api/beta/sites?sitename=~Season+2&limit=none&key=
  • See Confirm that plot boundaries for sorghum crop 2 generated from field scanner coordinates are consistent with those based on RTK GPS #187 for comparison of two bounding boxes
• generate a 'PUT' request with one or more values per plot.
  • This can go in the current extractor or be added to a new extractor
  • See PlantCV extractor for an example of the JSON format
  • the technical documentation here: https://github.com/PecanProject/betydbdoc-dataentry/blob/master/trait_insertion_api.md
  • ask @gsrohde for help

[max-zilla]

@ZongyangLi here is an example of creating a CSV to push into BETYdb:
https://github.com/terraref/extractors-lemnatec-indoor/blob/master/plantcv/terra.plantcv.py

# BETYdb instance information for submitting output CSV (skipped if betyAPI is empty)
betyAPI = "https://terraref.ncsa.illinois.edu/bety/api/beta/traits.csv"
betyKey = "CREATE_USER_ACCOUNT_FOR_A_KEY"

def submitToBety(csvfile):
    global betyAPI, betyKey

    if betyAPI != "":
        sess = requests.Session()
        print(csvfile)
        print("%s?key=%s" % (betyAPI, betyKey))
        r = sess.post("%s?key=%s" % (betyAPI, betyKey),
                  data=file(csvfile, 'rb').read(),
                  headers={'Content-type': 'text/csv'})

        if r.status_code == 200 or r.status_code == 201:
            print("CSV successfully uploaded to BETYdb.")
        else:
            print("Error uploading CSV to BETYdb %s" % r.status_code)
            print(r.text)

The CSV file is generated here:
https://github.com/terraref/extractors-lemnatec-indoor/blob/master/plantcv/PlantcvClowderIndoorAnalysis.py

# fields are the CSV column headers
fields = ('entity', 'cultivar', 'treatment', 'local_datetime', 'sv_area', 'tv_area', 'hull_area',
              'solidity', 'height', 'perimeter', 'access_level', 'species', 'site',
              'citation_author', 'citation_year', 'citation_title', 'method')
# traits is the dictionary of values for CSV (this script only has 1 row per csv)
    traits = {'plant_barcode' : '',
              'genotype' : '',
              'treatment' : '',
              'imagedate' : '',
              'sv_area' : [],
              'tv_area' : '',
              'hull_area' : [],
              'solidity' : [],
              'height' : [],
              'perimeter' : [],
              'access_level': '2',
              'species': 'Sorghum bicolor',
              'site': 'Danforth Plant Science Center Bellweather Phenotyping Facility',
              'citation_author': '"Fahlgren, Noah"',
              'citation_year': '2016',
              'citation_title': 'Unpublished Data from Sorghum Test Runs',
              'method': 'PlantCV'}

def generate_average_csv(fname, fields, trait_list):
    """ Generate CSV called fname with fields and trait_list """
    csv = open(fname, 'w')
    csv.write(','.join(map(str, fields)) + '\n')
    csv.write(','.join(map(str, trait_list)) + '\n')
    csv.close()

    return fname

[ZongyangLi]

Start working on this. @dlebauer

• Make sure that we are using same plots definition
  1. For 'Percent Cover' traits, I believe I am using a same plot plan and field book that @tingli3 was using. For now I only generated some Season 1 data.
  2. For 'plot level height', although we have some of position parameters like 'scan position, scan direction, scan distance' and so on, we don't have enough parameter to locate plots. Ply files can be shown as below:
     
     
     The most important parameter I need is 'what is the original point represents for'. Before I get this parameter, I have to make a guess like 'the first plant in the ply represents for plot row 2'. So I don't think it's a good timing to add 'height distribution' into BETYdb. I generated some height distribution data using those ply files in Season 2 and sent it to @rmgarnett to compared with the hand measurements. I am looking forward to his feed back.
• generate a 'PUT' request with one or more values per plot.
  1. I think it should be a new extractor for 'percent cover' data into BETYdb, it is different from those geotiff creations.
  2. @max-zilla Thanks for these examples! I will try this.

[ZongyangLi]

@dlebauer I am writing to let you know my plan of this task, please let me know if I made any misunderstanding.

1. Get leaf area index value(percent cover) for each vis-image, that means each image have a value to submit.
2. Create a .csv file that records all the information about the trait(location, row#, col#, time and trait value).
3. use clowder extractor to sent this .csv file into BETYdb

[dlebauer]

@ZongyangLi yes, this is correct. However, instead of using the row and column, you will need to construct a sitename field.

• Currently they are constructed as MAC Field Scanner Field Plot <plotid>
  • For season 1 these will match the records here that do not include "Season 2" https://terraref.ncsa.illinois.edu/bety/api/beta/sites?sitename=~Field+Scanner&limit=none&key=
  • do you have the plot # or just the row # and col #?
• In addition to passing a single value like 'canopy_cover', you can optionally insert a sample size and statistic (like SD). We can define how this is computed in the method protocol.

[dlebauer]

@ZongyangLi regarding the position of the points in the ply files - I was under the impression that these were in the gantry xyz coordinate system. If this is not the case, please open a separate issue and assign to @solmazhajmohammadi.

[ZongyangLi]

@dlebauer I am not quite understand the link above?
{ "metadata": { "URI": "https://terraref.ncsa.illinois.edu/bety/api/beta/sites?sitename=%7EField+Scanner&limit=none&key=", "timestamp": "2016-10-28T11:29:24-05:00", "count": null }, "errors": "authenication failed" }
Is that means I should construct a sitename field in the .csv file that contains of 'timestamp, cont, errors'?
I use a plot plan and field book to get a plot # base on the gantry position.

[dlebauer]

@ZongyangLi you need a key. Try https://terraref.ncsa.illinois.edu/bety/api/beta/sites?sitename=~Field+Scanner&limit=none&key=9999999999999999999999999999999999999999

[dlebauer]

@ZongyangLi if you use the plot numbers in the field book and construct the sitename as MAC Field Scanner Field Plot <plotid> where ` is [1, 2, ...849] that will work.

[remotesensinglab]

Has anyone measured in-situ leaf area index, vegetation fraction so that can be used to validate/refine extraction algorithms?

[dlebauer]

@remotesensinglab there is radiation interception but I am not sure of LAI - @NewcombMaria would know

[dlebauer]

@remotesensinglab I think this issue just refers to % cover - it seems like the camera is the most accurate way to directly measure this, and most of any error would come from the identification of leaf vs soil. This isn't a direct measure of LAI. By 'vegetation fraction', do you mean 'canopy cover'?