[Spike] Use Express / Koa for HTTP processing

[dhmlau]

See original text in #1038

To evaluate different approach:

• see how much effort to improve our HTTP pipeline (our homegrown) to deal with common HTTP features/policies
• pros and cons to leverage Express base without routing plus the common middleware, and Koa
• if we can cherrypick the enhancement from Express/Koa, then we can reuse the middleware and no need to maintain the modules.

Coming from discussion with @raymondfeng

[dhmlau]

assigning this to @raymondfeng as he has a PR #1082 for this

[b-admike]

From our spike result meeting today regarding #1082, here are some questions and tasks that we think follow from it:

• Refactor more common interfaces into @loopback/http-server package from @loopback/rest (like low-level http response writer, parser etc.). In other words, from Raymond's comment, [Continue to] Refactor the http layer constructs and logic out of the @loopback/rest module for better reusability and cleaner separations.
• Refactor testlab to two packages testlab and testlab-http (circular dependency will arise otherwise)
• Get rid of http request and response dependency in sequence actions (rely on context completely) (https://github.com/strongloop/loopback-next/blob/master/packages/rest/src/internal-types.ts)
• Which web framework will we use? find out what is best: express, koa? or do we keep/improve our own homegrown module?
  • List supported use-cases for http processing; what do we need? CORS, compression, static file serving etc.
  • Look into maturity, availability of middleware, adoption, etc
• How do we align the chosen framework's middleware pipeline with our sequence of actions?
• Create a common http client for outbound connections (common http tier to do pre/post processing)
  @strongloop/lb-next-dev @bajtos Thoughts?

[bajtos]

My thoughts:

Refactor more common interfaces into @loopback/http-server package from @loopback/rest (like low-level http response writer, parser etc.). In other words, from Raymond's comment, [Continue to] Refactor the http layer constructs and logic out of the @loopback/rest module for better reusability and cleaner separations.

This sounds reasonable 👍

Refactor testlab to two packages testlab and testlab-http (circular dependency will arise otherwise)

I need more details to be able to understand this point and provide feedback.

Ideally, we should improve our REST package to make it more compatible with supertest, so that testlab-http becomes obsolete and people can use supertest directly. I am not sure if that's possible though!

Get rid of http request and response dependency in sequence actions (rely on context completely) (https://github.com/strongloop/loopback-next/blob/master/packages/rest/src/internal-types.ts)

Hmm, this deserves deeper consideration. What are the reasoning for this change?

From what I remember, the current Sequence is intentionally designed to be independent of/decoupled from DI/IoC. Ideally, people should not be forced into our IoC to write a custom Sequence, but I am ok to revisit this idea.

The second use case to consider is the short-hand app.handler((req, res, sequence) => {/* custom sequence*/}). Do we want to keep it around? Will it be possible to implement this API when request and response objects are stored in context only?

Which web framework will we use? find out what is best: express, koa? or do we keep/improve our own homegrown module?

+1

How do we align the chosen framework's middleware pipeline with our sequence of actions?

I'd like to specifically see how we will approach error handling. When mounting a LB4 app on express/koa/etc., users should have the possibility to decide whether they want to handle errors inside LB4 or use the app-wide express/koa/etc.-level error handler for all endpoints.

Create a common http client for outbound connections (common http tier to do pre/post processing)

IMO, http client and outbound connections should be left out of scope of the work on hardening/improving our HTTP server (inbound connections) and moved to the story/epic covering integration with REST/SOAP services - see #1036.

[b-admike]

Thank you for your feedback @bajtos. I think the best person to explain further for the concerns you raised is @raymondfeng.

[raymondfeng]

Ideally, we should improve our REST package to make it more compatible with supertest, so that testlab-http becomes obsolete and people can use supertest directly. I am not sure if that's possible though!

In my PR, the HttpEndpoint already exposes url (bound) which can be used by supertest (request(endpoint.url)).

[raymondfeng]

Hmm, this deserves deeper consideration. What are the reasoning for this change?
From what I remember, the current Sequence is intentionally designed to be independent of/decoupled from DI/IoC. Ideally, people should not be forced into our IoC to write a custom Sequence, but I am ok to revisit this idea.
The second use case to consider is the short-hand app.handler((req, res, sequence) => {/* custom sequence*/}). Do we want to keep it around? Will it be possible to implement this API when request and response objects are stored in context only?

Once we introduce a web framework like Express, we should find the divide of responsibility between the native middleware and our sequence of actions. I think our actions should be mostly handle concerns beyond the transport, such as how to route REST API calls. Lower level http req/res handling should be hooked into the web framework as much as possible. I'm open to being practical.

DI or not for the action composition is another discussion out of scope of this issue.