Add function to Optionally get `site.info` if not present #3324

Sweetdevil144 · 2024-07-05T18:35:11Z

Description

This PR aims to refactor site.id and improve str_ns. The end goal is to make the the System independent of DB. Currently, I'm refactoring to create a siteID if not present already. I'll add test cases to check this util function too.
Work still in pending

Some comments from @mdietze :

Here's my slightly different interpretation :

Require the user to input a dataframe with lat/lon and optionally siteID; if siteID is not provided, >construct a unique identifier from lat/lon

replace str_ns with provided unique identifier

Provide a helper function that takes in a BETY connection and site IDs and returns lat, lon, and str_ns

Motivation and Context

May Fix a Subtask of #3307

Review Time Estimate

Immediately
Within one week
When possible

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
My name is in the list of CITATION.cff
I have updated the CHANGELOG.md.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

In do_conversion.R Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Sweetdevil144 · 2024-07-05T20:33:59Z

@meetagrawal09 can you cross check if corresponding changes in test.met.process are valid or not? :)

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

modules/data.atmosphere/tests/testthat/test.met.process.R

base/db/R/get.new.site.R

base/workflow/R/do_conversions.R

infotroph · 2024-07-08T16:41:32Z

I think you're taking the wrong approach to task 2 here: Yes, "if siteID is not provided, construct a unique identifier from lat/lon", but it needs to be constructed without using the DB. As we discussed in Slack, this could be as simple as something like id_str <- paste0(lat, "_", lon).

base/db/R/get.new.site.R

modules/data.atmosphere/R/met.process.R

modules/data.land/R/ic_process.R

modules/data.land/R/soil_process.R

… siteID-refactor

base/db/R/get.new.site.R

….site

Signed-off-by: Abhinav Pandey <abhinavpandey1230@gmail.com>

Sweetdevil144 · 2025-06-05T17:30:12Z

Fixing the CI checks and test errors

resolving https://github.com/PecanProject/pecan/pull/3324/files/3c537f6552023ceb415ccd139383aefc831f7bca#r1918394219

mdietze

What are the examples you are using to test your code as part of development? I ask because (1) there are no testthat tests defined for your new functions and (2) there's a good bit of code here that doesn't look like it would run at all. I want to make sure your not just writing code that "makes sense" and submitting it without verifying that the individual functions work as expected under all the different cases and that the overall workflow still works

mdietze · 2025-06-06T00:13:59Z

base/db/R/get.new.site.R

+##' @return a dataframe with new site information on lat, lon and time_zone
+##' @export
+##' @author Abhinav Pandey
+##' @importFrom digest digest


What does this do? First, as far as I can tell this function is never used here. Second, where it is used I don't see why it can't be called by namespace

mdietze · 2025-06-06T00:14:07Z

base/db/NAMESPACE

 export(var_names_all)
 export(workflow)
 export(workflows)
+importFrom(digest,digest)


Should not be needed

mdietze · 2025-06-06T00:16:20Z

base/db/R/get.new.site.R

+#' @export
+generate_site_id <- function(lat, lon) {
+  latlon_str <- paste0(lat, lon)
+  uid <- digest::digest(latlon_str, algo = "xxhash64")


Do we really need to add an entirely new package dependency to PEcAn just for this one call in one function? Is there an easier solution, or one that relies on existing packages?

Doesn't necessarily warrant using it, but that was my suggestion since I was using it in ccmmf workflows, and it was already a dependency b/c of the api.

pecan/apps/api/R/auth.R

Line 14 in 7faa280

dig <- digest::digest(

There's also rlang::hash

mdietze · 2025-06-06T00:25:47Z

base/db/R/get.new.site.R

+            ) {
+                site.id <- paste0(site$lat, "_", site$lon)
+                new.site <- data.frame(
+                    id = as.numeric(site.id),


the as.numeric here seems like it's going to always return NA given how site.id is constructed

mdietze · 2025-06-06T00:26:46Z

base/db/R/get.new.site.R

+                    lat = site$lat,
+                    lon = site$lon
+                )
+                str_ns <- paste0(new.site$lat, "_", new.site$lon)


this seems redundant with site.id

mdietze · 2025-06-06T00:29:29Z

base/db/R/get.new.site.R

+                PEcAn.logger::logger.info(paste0("Generated siteID using lat and lon:", site.id))
+                site.id <- generate_site_id(site$lat, site$lon)
+                new.site <- data.frame(
+                    id = as.numeric(site.id),


does this id need to be numeric? If so, why? Is the algorithm generating it guaranteed to produce numbers?

It won't be numeric if using digest as is.

mdietze · 2025-06-06T00:32:27Z

modules/data.atmosphere/R/met.process.R

-                         lat = latlon$lat, 
-                         lon = latlon$lon)
-  str_ns <- paste0(new.site$id %/% 1e+09, "-", new.site$id %% 1e+09)
+  site.info <- PEcAn.DB::get.new.site(site, con=con, latlon = latlon)


latlon undefined

mdietze · 2025-06-06T00:33:05Z

modules/data.land/R/ic_process.R

-                         lon = latlon$lon)
+  # Setup site database number, lat, lon and name and copy for format.vars if new input
+  # Then extract new.site and str_ns from site.info 
+  site.info <- PEcAn.DB::get.new.site(site, con=con, latlon = latlon)


latlon undefined

mdietze · 2025-06-06T00:34:10Z

modules/data.land/R/ic_process.R

 ##' @param input Taken from settings$run$inputs. This should include id, path, and source
 ##' @param dir settings$database$dbfiles
 ##' @param overwrite Default = FALSE. whether to force ic_process to proceed
+##' @param site Current site information


This line is defining a new argument to the function, but the argument has't been added to the function itself

mdietze · 2025-06-06T00:37:06Z

base/db/DESCRIPTION

    tidyverse,
-    withr
+    withr,
+    openssl


Package added as a dependency, but I can't find any places in your PR where it's actually used. Can this be removed?

modules/benchmark/DESCRIPTION

infotroph · 2025-06-08T06:00:11Z

base/db/R/get.new.site.R

+#' @export
+generate_site_id <- function(lat, lon) {
+  latlon_str <- paste0(lat, lon)
+  uid <- digest::digest(latlon_str, algo = "xxhash64")


There's also rlang::hash

infotroph · 2025-10-21T09:31:38Z

@Sweetdevil144 Thank you for the extensive work you put into this PR. It has become quite large compared to the simple idea it started from, and at the same time we've been gaining more experience running PEcAn with no database and learning how much we can simplify things by making more inputs truly required rather than trying to fill them in for you. Specifically, it's now clear that we almost always want to force the user to provide a site id rather than try to create it for them, and that in the cases we create an ID on the fly we should generally be thinking of it as a site name (i.e. something that lets a human figure out which site's in this folder) rather than guarantee its permanence or uniqueness as a global identifier.

I've taken the liberty of cherry-picking commits from this PR into a much smaller set of changes in #3656, which will close this one. I think the replacement PR captures the core ideas of this one (avoiding DB access when we know location already, allowing met download from locations with no preassigned siteid) with a much smaller footprint and better alignment with our current run setup approaches.

Sweetdevil144 · 2025-10-22T05:55:15Z

Let's close this Pr once #3656 is merged

Sweetdevil144 added 8 commits July 3, 2024 20:08

Accept site.data as function input

cb41971

In do_conversion.R Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Pass site.data via args

8b82f2b

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Read Command Line arguments

8c998fd

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Update str_ns declaration

7a5a848

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Update site$lon

74365fd

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Update if-else block for site.id

a0ae689

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Add helper function to get new.site

18dd086

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Roxygenise the documentations

7a69113

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

github-actions bot added Modules Base labels Jul 5, 2024

Sweetdevil144 added 3 commits July 6, 2024 00:15

Update Function Name

604634b

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Export dependency to DB rather than utils

0f154e5

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Update test case

fae37a8

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

github-actions bot added the Tests label Jul 5, 2024

Update 'exampes'

cb42033

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Sweetdevil144 added 10 commits July 6, 2024 16:27

Return null when site$id is not present

5ee450a

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Remove NULL calling

e37b93b

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Fixing 'no global binding err'

144ce9c

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

update global variable handling

54ce644

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Remove global binding changes

6d76455

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Update global variables

ca0b50e

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Ipdate all instances of latlon

8082194

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Roxygenise documents

65cdea7

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Usage Updates added

acb8507

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

Doc Updates added

59f451a

Signed-off-by: Abhinav Pandey <abhinav.pandey.met22@itbhu.ac.in>

meetagrawal09 reviewed Jul 8, 2024

View reviewed changes

modules/data.atmosphere/tests/testthat/test.met.process.R Outdated Show resolved Hide resolved

infotroph reviewed Jul 8, 2024

View reviewed changes

base/db/R/get.new.site.R Show resolved Hide resolved

infotroph reviewed Jul 8, 2024

View reviewed changes

base/workflow/R/do_conversions.R Outdated Show resolved Hide resolved

Sweetdevil144 and others added 5 commits October 7, 2024 16:17

Merge branch 'develop' into siteID-refactor

c411a6e

Merge branch 'develop' into siteID-refactor

0009c6e

Merge branch 'develop' into siteID-refactor

9039c23

Merge branch 'PecanProject:develop' into siteID-refactor

71e293c

Merge branch 'develop' into siteID-refactor

3c537f6

mdietze requested changes Jan 16, 2025

View reviewed changes

infotroph mentioned this pull request Feb 1, 2025

only abbreviate siteid if numeric and over a billion #3428

Merged

14 tasks

Sweetdevil144 added 2 commits February 15, 2025 21:00

Merge branch 'develop' of https://github.com/Sweetdevil144/pecan into…

f9e8b4a

… siteID-refactor

Merge branch 'PecanProject:develop' into siteID-refactor

2939e52

dlebauer reviewed Mar 6, 2025

View reviewed changes

base/db/R/get.new.site.R Outdated Show resolved Hide resolved

Sweetdevil144 added 6 commits March 12, 2025 10:01

Merge branch 'PecanProject:develop' into siteID-refactor

06b62db

Merge branch 'PecanProject:develop' into siteID-refactor

4f42372

Merge branch 'PecanProject:develop' into siteID-refactor

25f5548

Merge branch 'PecanProject:develop' into siteID-refactor

013c9eb

Merge branch 'PecanProject:develop' into siteID-refactor

a3faa1c

Export generate_new_site, refactor inefficiency while calling get.new…

7831dc7

….site

Sweetdevil144 requested a review from infotroph May 13, 2025 01:28

Sweetdevil144 and others added 4 commits May 13, 2025 09:39

fix R check NOTE

b599171

Update NAMESPACE

81f53d3

Signed-off-by: Abhinav Pandey <abhinavpandey1230@gmail.com>

Add DESCRIPTION

a68c928

Signed-off-by: Abhinav Pandey <abhinavpandey1230@gmail.com>

Merge branch 'develop' into siteID-refactor

bd58f69

remove latlon arg from get.new.site

8132ae6

resolving https://github.com/PecanProject/pecan/pull/3324/files/3c537f6552023ceb415ccd139383aefc831f7bca#r1918394219

mdietze requested changes Jun 6, 2025

View reviewed changes

infotroph reviewed Jun 8, 2025

View reviewed changes

infotroph mentioned this pull request Oct 21, 2025

Simpler str_ns handling #3656

Merged

14 tasks

infotroph closed this in #3656 Oct 22, 2025

Sweetdevil144 deleted the siteID-refactor branch October 22, 2025 13:26

Add function to Optionally get site.info if not present #3324

Add function to Optionally get site.info if not present #3324

Uh oh!

Conversation

Sweetdevil144 commented Jul 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Review Time Estimate

Types of changes

Checklist:

Uh oh!

Sweetdevil144 commented Jul 5, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

infotroph commented Jul 8, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sweetdevil144 commented Jun 5, 2025

Uh oh!

mdietze left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

infotroph commented Oct 21, 2025

Uh oh!

Sweetdevil144 commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add function to Optionally get `site.info` if not present #3324

Add function to Optionally get `site.info` if not present #3324

Sweetdevil144 commented Jul 5, 2024 •

edited

Loading