Stack release for 2017-07-19 meeting

[jar600]

A stack release is required for dissemination to participants in a meeting that starts 19 July. It is desirable that the release and its documentation by ready for internal review by 17 July.

We are assuming for this purpose that the user is providing a server (which could be AWS but that's not our worry).

• Decision needed: which configuration is to be documented or recommended i.e. how do we expect the stack to be accessed - via browser running on same machine as stack, via HTTP (perhaps over WAN), or via ssh tunnel?
• If access is over WAN, how is authentication to be handled?
• Decision needed: where should people send bug reports? Some email address, or create github issues - if the latter, which repository?
• What is the recommended hardware configuration i.e. how much RAM and how many cores have been used in successful deployments so far?
• Documentation page needs to be written - prerequisites, download, install, use. (JAR charged with drafting this, in progress)
• Where should the download/install/run documentation page reside (in this repo? what directory?) and what sort of markup should it use (html? what css?)?

[jar600]

Also, where should the download/install/run documentation page reside (in this repo? what directory?) and what sort of markup should it use (html? what css?)? (adding to checklist)

[jar600]

c3.8xlarge = 32 cores, 60Gb

[jar600]

Deployment options:

1. Recommended configuration: a workstation that has at least 32 cores, 60G RAM, and a display and keyboard.
2. Compromise: fewer cores and/or less RAM, leading to significant limitations in what data sets can be analyzed.
3. It is also possible to run in client/server mode, but that is "outside the scope of this document" (?? two ways to get secure the connection: https, requiring a certificate and some fussy initialization; and ssh, requiring an ssh client).

[axch]

• We have been recommending 32 cores and 60G RAM.
• We generally bundle a collection of tutorial material from the workshop-materials repository into the distribution (see scripts/content-package.sh therein)
• Two configurations are "currently implemented":
  • The Docker file docker/ubuntu1604-jupyter-full. This expects the base instance to be built from the Docker file docker/ubuntu1604-jupyter, and augments it with our tutorial material; see release-container.sh in this repo. This configuration was coded under the expectation that the user would connect locally or though an SSH tunnel, so implements no encryption or authentication.
  • The AWS setup backing stack.probcomp.net. This is defined in AWS terms in aws/stack.yaml. Spinning one up is automated in stack.sh (the idea is that there is one stack per user). The UNIX interior is defined by ami/setup.sh, the results of which are baked into the Amazon machine image ami-751b2c63, which is then tweaked at startup by the actions stack.sh performs on the instance. This configuration was coded to be accessed over the WAN; however, the only way we found to use an SSL certificate (that would cover many subdomains) was to rely on Amazon's Elastic Load Balancer functionality for https, so the configuration itself only deals with authenticating the user (by means of a Jupyter notebook password, whose installation on the machine is handled by stack.sh). The expectation when coding the AWS configuration was also that we would be administering the servers, though in principle it ought not to be too difficult to adapt to allow the user to run it against their own AWS account.
• The sources of the web site are in the probcomp-website repository. This is coded in the Middleman static site generator, which unfortunately is distributed as a Ruby gem. Installation and deployment instructions in the repo.
• The web site's primary residence is at http://probcomp.csail.mit.edu. Making that https is https://github.com/probcomp/probcomp-sysadmin/issues/16. In particular, the last page to provide instructions for how to download the probcomp stack is http://probcomp.csail.mit.edu/stack (user/pass oreilly/oreilly, because it was thought of as a one-off for that event).
• FWIW, Amazon has a tutorial on how to start a bare EC2 instance that listens to ssh connections, and log in to it. It may be feasible to write a tutorial for how to run the probcomp stack on a private EC2 instance (using a Docker container, presumably) and connect to it with an ssh tunnel, partly by referring to that document.

[jar600]

I think you've already done a good job of documenting options, and your comment here fills in some useful details. What we need for Wednesday is to decide what to tell participants at the PI meeting regarding release; this entails removing options (from the release documentation) so that the explanation is as simple as we can bear. For this audience we cannot / will not provide AWS instances, so docker it is. My first draft of release notes is here: https://github.com/probcomp/probcomp-stack/blob/20170713-jar-releasepage/web/release-page.html . The subtext is that they'll want to use an ssh tunnel, but for some reason we're not supposed to say that outright, thus the peculiar explanation that's currently there. I'm sure what I've written will require major revision.