Test for wide range of parameter space in designers

[jaspercooper]

• not enough to have coverage = 100%, we also want to test how well designers work with broad range of
• using expand grid and do.call, test every parameter in the designs

Here is how I would construct a test for a designer where I thought every combination should work, for example. You could do another one for the combinations where it should break.

Note that I am not using expand.grid() directly on parameters here, but am using indices and pulling parameters according to those. Why? Because sometimes you will have to provide vectors and this approach is robust to that (though may need some debugging -- I think Map() call may need [[)

library(testthat)

test_that(
  desc = "simple_two_arm_designer does not break over a combination of lots of parameters", 
  code = {
    # Get default args
    arguments <- formals(simple_two_arm_designer)
    
    # Define param space for each argument
    arguments$N <- c(1,50,1000)
    arguments$prob <- c(0,.5,1)
    arguments$control_mean <- c(0,1)
    arguments$control_sd <- c(0,1)
    arguments$ate <- c(0,1)
    arguments$treatment_mean <- c(0,1)
    arguments$treatment_sd <- c(0,1)
    arguments$rho <- c(0,.5,1)
    
    # Get indices for expanding out parameter space
    argument_lengths <- sapply(arguments,length)
    indices <- lapply(argument_lengths, function(x) 1:x)
    # Get indices for every combination of parameters
    expand_indices <- do.call(expand.grid, indices)
    # Create lists of arguments by grabbing parameters that correspond to indices
    argument_list <- apply(expand_indices,1,function(indices) Map(`[`, arguments, indices))
    
    # Try to declare every design for every param combination using the designer
    # expect_error(code, NA) means you expect the code to run
    # This might be better as a loop that prints the argument_list so you know
    # where it is breaking
    expect_error(expand_design(simple_two_arm_designer, argument_list),NA)
  }
)

[nfultz]

See also the code from the shiny app that uses a grid search to cache diagnosis.