docs: document deployment to Cloud Foundry

[hacksparrow]

Document deployment to Cloud Foundry. Addresses #1604.

Checklist

• npm test passes on your machine
• New tests added or existing tests modified to cover all changes
• Code conforms with the style guide
• API Documentation in code was updated
• Documentation in /docs/site was updated
• Affected artifact templates in packages/cli were updated
• Affected example projects in examples/* were updated

[dhmlau]

I understand this is working in progress, however, I think if we assume the app is running fine locally, we can omit the sections on running cloudant on docker. It's clearer on what this doc trying to accomplish.
I'm thinking the following flow (see if it makes to you):

• clone the todo example using lb4 example command
• create the cloudant service and todo database
• make the changes in LB4 app
  1. add cloud foundry artifacts
  2. vcapService
  3. datasource, etc.
• deployment to IBM Cloud

[dhmlau]

If putting this as the root layer, the title should be consistent with the rest of the titles.
Suggestion: Deploying to IBM Cloud (or something along that line)

After the title has been finalized, might want to change the file name as well. (typo in the file name as well)

[dhmlau]

make it consistent with the title after it's been finalized.

[dhmlau]

• might want to explain a bit that the link is derived from the name and domain in the manifest.yml file.

[dhmlau]

@hacksparrow , @raymondfeng mentioned some time ago that there's IBM Cloud command we can use instead of the cf command. Have you had a chance to take a look to see whether that's a better approach?

[hacksparrow]

@dhmlau I think that was not a separate tool, it was cf's own command, if I am not wrong. @raymondfeng was it something else?

I have updated the doc accordingly.

[bajtos]

I like the new code, it's so much simpler!

It would be great if somebody can run through the instructions to verify they are complete and work as described. I am not sure if I'll be able to find time to do that in the next few days.

[bajtos]

typo: Cloudan

[bajtos]

-s to save this connector as a regular (production) dependency.

With the proposed -S, the connector will be available only at dev-time.

[hacksparrow]

ooops!

[bajtos]

Use -s for --save for production.

[bajtos]

Let's simplify this block a little bit. How about the following?

if (!options) options = {};
if (!options.rest) options.rest = {};
options.rest.port = appEnv.port || options.rest.port;

Also: do we need to check appEnv.isLocal flag before applying appEnv.port?

[virkt25]

Here's my index.ts that I think we can leverage as I think it's simpler. Not the default value being set in the main function.

export async function main(options: ApplicationConfig = {}) {
  options.rest = options.rest || {};
  options.rest.port = appEnv.isLocal ? options.rest.port : appEnv.port;

  const app = new ShoppingApplication(options);

  if (!appEnv.isLocal) {
    userDataSource.url = appEnv.getServiceURL('t-shopping-cloudant');
    userDataSource.connector = 'cloudant';

    app.bind('datasources.config.user').to(userDataSource);
  }

  console.log('cloudant url: ', userDataSource.url);

  await app.boot();
  await app.start();

  const url = app.restServer.url;
  console.log(`Server is running at ${url}`);
  console.log(`Try ${url}/ping`);

  return app;
}

I am making two PRs to propose reading the port from env as a default and setting the options as a {} by default in hopes of simplifying this step.

[hacksparrow]

Writing to directly userDataSource will have side-effects for other files loading the datasource JSON file. I have uptimized, though.

Good proposals.

[bajtos]

I would personally structure the code slightly differently:

const appEnv = cfenv.getAppEnv();
if (!appEnv.isLocal) {
  // 'myCloudant' is the name of the provisioned Cloudant service
  datasourceDb.url = appEnv.getServiceURL('myCloudant');
}

What do you think?

[hacksparrow]

Agreed, since we don't need to refer the url anywhere else.

[bajtos]

Modifying the object returned by require('./datasources/db.datasource.json') is a working solution, but I find it rather hacky. When writing a code calling require on a JSON file, most people expect the require call to return the exact data as stored in the file, not a value that was modified by somebody.

A better solution is to create a new object and set the modified properties there.

const appEnv = cfenv.getAppEnv();
if (!appEnv.isLocal) {
  // 'myCloudant' is the name of the provisioned Cloudant service
  datasourceDb = Object.assign({}, datasourceDb, {
    url: appEnv.getServiceURL('myCloudant')
  });
  app.bind('datasources.config.db').to(datasourceDb);
}

[hacksparrow]

It was hacky indeed. Let's have that better implementation.

[bajtos]

I have a question regarding a host name and the application nam mylb4app. Shouldn't users pick a unique application name? I don't see how the same domain mylb4app.eu-gb.mybluemix.net can serve every LB4 user trying out cloud deployment by following this guide.

Am I missing something here?

I think the guide should be more clear about the fact that mylb4app is just a placeholder that people need to change to an actual unique application name.

[dhmlau]

I think there's one more glitches here. After going to that URL, it got redirect to another url that won't work. I have to remove the "8080" port from the redirected url and change to https. for example:
https://loopback.io/api-explorer/?url=https://loopback4-example-shopping.mybluemix.net/openapi.json
instead of the original:
https://loopback.io/api-explorer/?url=http://loopback4-example-shopping.mybluemix.net:8080/openapi.json

[hacksparrow]

Good find. This is a bug either in the rest package or loopback.io. Will investigate further and open an issue accordingly.

[hacksparrow]

@bajtos I made the changes you suggested. Thanks!

[virkt25]

Great guide!! Love how simple it is to read, follow and understand. Just have some questions / feedback.

[virkt25]

I think it's a safe assumption that if someone is reading this page they already have a working LB4 app and hence have Node.js installed and likely have the LB4 CLI installed as well.

[virkt25]

We don't need to specify -s as npm saves it by default in package.json. That said if you want it then flags should be placed before the connector name: npm i -s loopback-connector-cloudant.

[bajtos]

Maybe use a more explicit --save instead of -s, to make the command easier to understand for people that are not very familiar with npm.

[virkt25]

Same comment as above for -s

[virkt25]

I find it weird that we tell them to use the name myCloudant but we don't provision the service till the next step. I feel like these should be flipped.

[virkt25]

I'm not sure I agree with this part! Building locally and deploying (without the node_modules) folder seems weird. What if there is a new dependency released vs. the target your app was built with? Not to mention it seems like a pain to have to build the app yourself before deploying it when the build folder isn't stored in GitHub.

The way I personally see builds working is as follows:

• BASIC: Build the app on the cloud as you would locally.
• Intermediate: Set up a CI/CD pipeline to build / deploy the app, something like Travis / IBM Cloud DevOps offering. This would help limit downtime during a switch if not using blue-green deployments.

Building locally should not be a thing and the deploy should be with the dependencies it's built with IMO.

[hacksparrow]

Looks like building on IBM Cloud will require lb-tsc to be installed globally. Moving @loopback/build to dependencies does not make lb-tsc available to npm script like it was installed globally.

Any suggestions?

[virkt25]

Really? Because I don't have typescript installed globally and have never had issues with lb-tsc not working locally. @raymondfeng any ideas as to why this might happen? Only thing that comes to my mind is it might be related to Node 8?

[hacksparrow]

Maybe something to do with the environment on IBM Cloud. I logged in to the shell and checked, there's no node or npm.

[bajtos]

Building locally and deploying (without the node_modules) folder seems weird. What if there is a new dependency released vs. the target your app was built with?

Or what if the public npm registry goes down and our production box cannot install the dependencies?

IMO, if we deploy from a local computer (instead of using a build/deploy pipeline), then we should run the build (TypeScript compilation) locally AND also deploy node_modules dependencies as we have installed them locally.

At least this was our conclusion when we were discussing different deployment options back in 2014/2015. We created https://github.com/strongloop/strong-build to make that process easy to automate.

@rmg could you please chime in? What's your opinion on the deployment process for Node.js apps requiring a build step?

[virkt25]

I think it might be worthwhile to explain in a comment here as to what this is doing aka overriding / re-binding the config if on IBM Cloud from cfenv.

[dhmlau]

@hacksparrow , i was trying to follow your instruction and ran into a few issues. I think I've captured what I know so far, but couldn't get the app to work fully yet. I might have some more comments after getting everything running.

cc @virkt25 @jannyHou (thanks for ur help!)

[dhmlau]

i think you meant src\index.ts

[dhmlau]

Need to login first before running this command.
cf login -a <endpoint> -u <user> -o <org> -s <space>
cf login --sso

Actually cf create-app-manifest cannot be run if the app does not exist on IBM cloud already.

[dhmlau]

I got errors when somewhat copying and pasting your code snippet. The 2 errors:

1. In the line of app.bind('datasources.config.db').to(datasourceDb);, app hasn't been defined yet
2. const datasourceDb meaning it's not changeable but indeed we are assigning another value a few lines after.

For your reference, here is my index.ts:

import {ShoppingApplication} from './application';
import {ApplicationConfig} from '@loopback/core';

// IBM Cloud deployment
let datasourceDb = require('./datasources/user.datasource.json');
const cfenv = require('cfenv');
const appEnv = cfenv.getAppEnv();

export {ShoppingApplication};

export async function main(options?: ApplicationConfig) {
  // Set the port assigned for the app
  if (!options) options = {};
  if (!options.rest) options.rest = {};

  const app = new ShoppingApplication(options);
  if (appEnv.isLocal) options.rest.port = options.rest.port;
  else {
    datasourceDb = Object.assign({}, datasourceDb, {
      url: appEnv.getServiceURL('myCloudant'),
    });
    app.bind('datasources.config.db').to(datasourceDb);
    options.rest.port = appEnv.port;
  }

  await app.boot();
  await app.start();

  const url = app.restServer.url;
  console.log(`Server is running at ${url}`);
  console.log(`Try ${url}/ping`);

  return app;
}

[hacksparrow]

Fixed in the latest commit. Thanks.

[dhmlau]

I think there's a missing step of binding the service in the app (from the IBM Cloud console)

[hacksparrow]

It is mentioned under "Connecting the database to app".

[dhmlau]

I think there's one more glitches here. After going to that URL, it got redirect to another url that won't work. I have to remove the "8080" port from the redirected url and change to https. for example:
https://loopback.io/api-explorer/?url=https://loopback4-example-shopping.mybluemix.net/openapi.json
instead of the original:
https://loopback.io/api-explorer/?url=http://loopback4-example-shopping.mybluemix.net:8080/openapi.json

[virkt25]

So I tried to deploy loopback4-example-shopping to IBM Cloud as a regular app and this was my experience:

• Forked the repo so I can make changes

• Goal was to swap the database in production from mongodb to Cloudant since it offers a free tier

• Step 1 was to deploy as is without creating an app on IBM Cloud first. The goal was simple, push to Cloud as is, no changes. cf push t-shopping-app failed.

  • This created an app in IBM Cloud but failed to deploy. Missing buildpack. Specified sdk-for-nodejs as the buildpack via command-line (only needs to be done once) -- But I added it to my manifest.yml later as well.
  • Attempt to deploy again failed. Missing lb-tsc. IBM Cloud is a production environment and doesn't install devDependencies.
  • Moved @loopback/build to be a regular dependency. Got errors about tsc missing @types/* packages as they are a devDependency.

• I decided to build locally and deploy the app to make sure I can do that. Failed to do so since I hadn't installed before and doing so now was attempting to push node_modules folder which isn't allowed. I created a .cfignore and excluded node_modules.

  • Pushed and app failed as I forgot to set port from env in prod. Fixed that.
  • Push finally worked and I got the ping endpoint working.
  • The redirect to /swagger-ui does not work as expected. It sets port as :8080 in front of the domain name ... which causes it to not load. https://loopback.io/api-explorer/?url=https://t-shopping.mybluemix.net:8080/openapi.json -- Removing the :8080 makes it work fine.

Now to work out the database issues.

• Update code to swap out the database connection info and connector on a production environment
• Got annoyed by forgetting to re-build the app before a push. Got sidetracked to find a better solution and decided to use IBM Toolchains.

IBM Toolchain Setup

• It created it's own private Git repo. I already had GitHub setup -- so I had to re-configure some stuff and install the GitHub ToolChain thingy (Not sure if it's called an app / plugin / extension for the toolchain).
• IBM Toolchain for npm in the build stage offers Node 0.10.x as a default! This caused failures. The max Node version that is pre-installed is Node 6.7.x. Had to figure out how to install nvm and my own version of Node since LB4 requires Node 8 or higher. Came across: https://gist.github.com/DMeechan/2f6e962622c3387785bb2e44298a153c
  • The script didn't work out of the box as is. Here's my final Build Stage script I used.

#!/bin/bash
# Credit: http://gh-blog.mybluemix.net/blogs/cokeSchlumpf/rethink-it/posts/bluemix/node-buildpipeline.md?cm_sp=dw-bluemix-_-nospace-_-answers
# Paste the script below into IBM Cloud Delivery Pipeline Build process (shell script)
# Set your Node and NVM version:
export NODE_VERSION=8.11.4
export NVM_VERSION=0.33.11
export NVM_DIR=/home/pipeline/nvm

mkdir $NVM_DIR

npm config delete prefix \
  && curl https://raw.githubusercontent.com/creationix/nvm/v${NVM_VERSION}/install.sh | sh \
  && . $NVM_DIR/nvm.sh \
  && nvm install $NODE_VERSION \
  && nvm alias default $NODE_VERSION \
  && nvm use default \
  && node -v \
  && npm -v

export NVM_DIR="$HOME/nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"

npm install
npm run build

• Had to configure the rest of the toolchain (namely the Deploy step) to use the output of the previous build step to deploy.
• Toolchain works every time a commit is merged to master

Back to DataBase.

• Configured a Cloudant instance in the dashboard and bound it to the application. Set authentication as Legacy and IAM.
• Was getting 404 errors from trying to POST to /users.
• Figured it might be because I was missing a database. Created a user and users database. Didn't work.
• Explicitly set the database as user in user.datasource.json.
• Errors showed that the connector was trying to save data to a database named test despite the config saying user. Created a database named test in cloudant ... and everything just worked.

---

I'm not a fan of trying to remember to build locally before deploying. I know one can setup local commit hooks or something but I personally like the Toolchain since it updated the app automatically after a merge to master.

My GitHub Repo: https://github.com/virkt25/loopback4-example-shopping
The app link: https://t-shopping.mybluemix.net/

[raymondfeng]

@virkt25 Do you want to try the following?

• https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring
• Support for disabling "production" flag when running npm install cloudfoundry/nodejs-buildpack#16

[hacksparrow]

Thanks for the findings @virkt25.

For someone who just wants to see a basic LB4 app running on IBM Cloud, the overall experience seems overly complex. Thoughts?

[rmg]

@virkt25 to specify the node and npm versions that the buildpack should use you specify them in engines in package.json.

[virkt25]

to specify the node and npm versions that the buildpack should use you specify them in engines in package.json

This works for the actual deployed Application to IBM Cloud but IBM Toolchain doesn't seem to honour this. Below is the default build for npm in IBM Toolchain

[virkt25]

Do you want to try the following?

https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring
cloudfoundry/nodejs-buildpack#16

The issue is that the IBM Toolchain environment is very different from IBM Cloud where the app is deployed and hence the need for that script.

[dhmlau]

I think we can make this round of documentation to be just deploying a LB4 app to IBM Cloud without all the toolchain setup. Keep it minimal and easy to follow so that users know what are the steps involved. As a follow up, we can create an additional section on the toolchain. Even that, I'm thinking whether we can piggyback on some of the existing docs, meaning we only add the instructions specific or additional to LB app deployment.

Not sure if it helps in the discussion, with the help of @virkt25, I was able to deploy my own LB4 app to IBM Cloud. Similar to the starting point @virkt25 had above:

• clone the app
• make it to be cloudant connector, i.e. modify the datasource.json
• modify index.ts (https://github.com/virkt25/loopback4-example-shopping/blob/master/src/index.ts)'
• build locally using npm run build
  • not sure if i'm doing something wrong, i got errors when pushing if i keep the prestart in package.json
• Call cf login to login first
• push the app using cf push command
• Go to IBM Cloud, create the cloudant service and name it to match with what's used in index.ts
• In IBM Cloud, go to the app, create connection to bind the cloudant service with the app
• go to browser to open up API Explorer (with modified url that we've discussed above)

I think if we have the above steps (as an outline), it is good for the purpose of telling the users how to deploy to IBM Cloud.

Thoughts?

[hacksparrow]

PR updated with @virkt25 and @dhmlau inputs, except for the build process. Can we keep it simple (as-is in the latest commit), so our users can just follow the mechanism of the app running on IBM Cloud, and nothing more than that.

[virkt25]

We're almost there. Some nitpicks. I'm not sure if we need to talk about the local setup ... at least not in the early part of it.

Also ... why are we discussing cloudant integration / local setup? I find there to be a bit of a disconnect in the flow of the document. If we want to keep things simple then we can do a tutorial along the following lines:

1. Deploy the app as is. Point the limitation that data may be lost between restarts. Gets users comfortable with the idea of deploying.

2. Explain why we are adding Cloudant (persistance). Explain the local setup ... get the app working locally.

3. Show how to deploy with Cloudant on BlueMix.

And OUT OF SCOPE of this PR, but the next step in the deploy to IBM Cloud would be to show them the Pipeline setup so it auto-deploys when merging / pushing to master.

[virkt25]

nitpick: sentence should end in : instead of .

[virkt25]

nitpick: lets not add . at the end of the list items.

[virkt25]

I'm not sure how I feel about the local step -- it's not necessary for a deploy of the application.

[hacksparrow]

Removed.

[virkt25]

Here's my index.ts that I think we can leverage as I think it's simpler. Not the default value being set in the main function.

export async function main(options: ApplicationConfig = {}) {
  options.rest = options.rest || {};
  options.rest.port = appEnv.isLocal ? options.rest.port : appEnv.port;

  const app = new ShoppingApplication(options);

  if (!appEnv.isLocal) {
    userDataSource.url = appEnv.getServiceURL('t-shopping-cloudant');
    userDataSource.connector = 'cloudant';

    app.bind('datasources.config.user').to(userDataSource);
  }

  console.log('cloudant url: ', userDataSource.url);

  await app.boot();
  await app.start();

  const url = app.restServer.url;
  console.log(`Server is running at ${url}`);
  console.log(`Try ${url}/ping`);

  return app;
}

I am making two PRs to propose reading the port from env as a default and setting the options as a {} by default in hopes of simplifying this step.

[bajtos]

I quickly skimmed through the current version of the deployment guide and it looks good enough to me.

I agree the build process and/or deployment pipeline is out of scope of this pull request (and the related user story). Having said that, can we add a note to tell readers that a manual build is not the best approach, they should consider setting up a build pipeline and give them a link to an existing documentation for build pipelines? I think it's good to let them be aware of other solutions, even if we don't provide any additional LoopBack-specific guidelines (yet).

Also ... why are we discussing cloudant integration / local setup? I find there to be a bit of a disconnect in the flow of the document. If we want to keep things simple then we can do a tutorial along the following lines:

• Deploy the app as is. Point the limitation that data may be lost between restarts. Gets users comfortable with the idea of deploying.

• Explain why we are adding Cloudant (persistance). Explain the local setup ... get the app working locally.

• Show how to deploy with Cloudant on BlueMix.

I love this new outline!

On the other hand, this pull request has been opened for 9 days already, maybe it's best to land it in the current form and then rework it to the new outline in a follow-up pull request (or even a follow-up user story)?

[hacksparrow]

@virkt25 @dhmlau added these additional contextual info.

1. "This is a basic guide for deploying a LoopBack 4 (LB4) app to IBM Cloud. In the setup explained below, your app will use a local instance of Cloundant when running locally, and a provisioned Cloudant service when running on the cloud."

2. NOTE: Production deployment to IBM Cloud is a much bigger topic with many possible options, refer to "IBM Cloud Continuous Delivery: Build, deploy, and manage apps with toolchains" for the details.

3. We will be replacing the default memory-based connector of the app with a Cloudant connector, so data is persisted.

[dhmlau]

I'm good with landing this PR as-is too.

+1 on the new outline of removing the local setup. As I've mentioned before, it is a bit confusing to have the local set up / deploy instructions because we can assume the prereq is the app running locally already.