Implement 'fake' back-end fitting flow

[jonc125]

Alongside preparing the front-end aspects, we can actually implement fitting in the back-end fc-runner, similarly to Aidan's prototype. The back-end web service can have an extra API added that takes a 'fitting spec' and 'dataset' as well as model and protocol, and runs a single hardcoded fit, returning results. A temporary hack in the front-end can then recognise a 'magic' protocol and call this new API instead of the standard simulation.

This will require changes to at least 4 repos, not just this one. But I've stuck the issue here for convenience! @MichaelClerx you can create separate issues if desired.

WebLab:

• Recognise a protocol that has a .csv file in it, and call the new API rather than the standard experiment runner. Pass the protocol's URL as the dataset and fitting-spec parameters.
• Possibly the protocol should also contain a file with pints code to provide a realistic fitting spec?

fc-runner:

• Add the new API to chastewebservice.py.
  • It should download and unpack the dataset and fitting spec, since from the above these should be real URLs and at least the dataset should contain a .csv with data.
• Ensure it has suitable error handling logic as for the existing experiment runner (perhaps refactoring as needed)

weblab-fc:

• Implement the special-case fit in here? (Keeping with pycml etc for code generation.)

deployment (I can help with this bit @MichaelClerx - perhaps work on it together on Thurs?):

• Deploy the new fc-runner (i.e. the branch of the deployment repo that tests all this must deploy the corresponding fc-runner branch)
• Deploy the code that runs the fit (
• Deploy the hacky WebLab branch

Related to #19 (if needed).

[MichaelClerx]

Thanks @jonc125 . What does the current WL2 run on? Is it something we could swap out for the bitbucket code with Aidan's modifications?

[jonc125]

Forward simulations for the current WL2 are still using the C++/Python hybrid from WL1. But fitting experiments can easily use something different. The fc-runner just treats the simulator as an executable anyway.

[MichaelClerx]

Might be easiest to just stick Aidan's prototype in then, instead of redoing that in a branch of fc?

[jonc125]

The idea was to move towards what we'll eventually be using for fitting, rather than just Aidan's code. Does that make sense? Can start with Aidan's code and then replace it though if that's easier?

[MichaelClerx]

I don't know! If the above is basically doing this (https://bitbucket.org/joncooper/fcweb/branch/cardiac-fitting-pints) and then removing it again later, I'm not sure we should bother? Bit lost at the moment though :-)

[MichaelClerx]

OK, looking at the diff that looks a bit messy, I'll maybe start on a cleaner re-implement then, based on a version of fc that still has PyCML in it...

[jonc125]

Yeah, it's that kind of thing, but making the webservice call look like the eventual solution and refactoring fc-runner to support it in a nicer way, so it's easier to switch to the final version. And so we can properly deploy it for the workshop. Basically I was trying to figure out what steps we could take on the back-end for this to move us towards workshop-ready while the front-end isn't there yet :)

[MichaelClerx]

To select a different fc-runner branch, update Vagrantfile in the deployment repository to add an ansible.extra_vars, celery_git_branch

e.g.

ansible.extra_vars = {
    django...
    celery_git_branch: 'fitting',
}

where fitting is the name of the fc-runner branch

[MichaelClerx]

Recognise a protocol that has a .csv file in it, and call the new API rather than the standard experiment runner. Pass the protocol's URL as the dataset and fitting-spec parameters.

@jonc125 I've been looking around for a while now, but I have no idea how to do this. Looks like the javascript should be updated?

[jonc125]

Recognise a protocol that has a .csv file in it, and call the new API rather than the standard experiment runner. Pass the protocol's URL as the dataset and fitting-spec parameters.

@jonc125 I've been looking around for a while now, but I have no idea how to do this. Looks like the javascript should be updated?

Try the submit_experiment function in experiments/processing.py

[MichaelClerx]

@jonc125 I've discovered I can (heuristically) detect the CSV and extra TXT in fc-runner, without changes to the front-end. Is that acceptable for this prototype?

[jonc125]

You can start there for an initial commit, but then move towards doing that heuristic check in the front-end so we get closer to the final solution.

[MichaelClerx]

More questions for @jonc125 :-)

Have got to the point where I'm calling Aidan's cardiac.py, but it can't find its dependencies. Have updated the fc_virtualenv but not sure that's being used here?

The same script, tasks.py, calls CheckSyntax.py without issue, assuming that runs in a virtualenv too?

[MichaelClerx]

Oh! Testing on the VM it seems CheckSyntax.py can run without a virtualenv. So presumably no virtualenv is activated at this point...
Bit confused where the fc_virtualenv comes into play then