Can't handle page with embedded images

[marcoskirsch]

The server quickly runs out of memory and crashes if a page with a few embedded images is requested. See example cars.html

  Requested URI: /cars.html
  Requested URI: /cars-mas.jpg
  Requested URI: /cars-ferrari.jpg
  Requested URI: /cars-lambo.jpg
  PANIC: unprotected error in call to Lua API (not enough memory)
  ������1�)m�1�)�!������B

[marcoskirsch]

Well, sometimes it works.

[devyte]

it's a race condition. I understand that the ESP accepts up to 5 concurrent connections, and in the case of a webpage with elements, a browser may decide to request those elements concurrently.
The httpserver mem footprint is rather large.

• Serving one element is unlikely to make the ESP crash (unless it's a lua script being run, and it does something stupid).
  -Serving a second element concurrently means a second httpserver context. Currently, if the element is small, it may work, or if it's a second script, it will most likely crash.
  -I don't think a 3rd element will work, because 3 httpserver contexts don't fit in memory.

I see 2 possible solutions:

1. Limit the number of concurrent connections that the ESP accepts. This could be done by having a configuration option in the net module inside the firmware, or by implementing a kind of lock in the socket listening function.
   This solution may be ok for nodemcu-httpserver, but it would not be viable for my nodemcu-platform, so I would prefer to avoid it.
2. Reduce the mem footprint of the httpserver significantly. I would prefer to this be pursued, it would make for a better implementation.

[marcoskirsch]

A little more informative if I add a few prints for debugging. You can see the memory go down as expected until crash:

  GET: /cars.html
  Begin sending:    http/cars.html
  node.heap():  27216
  Sent: 713 of 713
  Finished sending:     http/cars.html
  GET: /cars-ferrari.jpg
  Begin sending:    http/cars-ferrari.jpg
  node.heap():  25008
  GET: /cars-lambo.jpg
  Begin sending:    http/cars-lambo.jpg
  node.heap():  16240
  GET: /cars-mas.jpg
  Begin sending:    http/cars-mas.jpg
  node.heap():  5776
  GET: /cars-porsche.jpg
  Begin sending:    http/cars-porsche.jpg
  node.heap():  2648
  Error:    not enough memory
  GET: /cars-mas.jpg
  Begin sending:    http/cars-mas.jpg
  node.heap():  2536
  PANIC: unprotected error in call to Lua API (not enough memory)
  ���I�x�1���Ό)������!��

1. Limit the number of concurrent connections that the ESP accepts.

Sounds like this won't happen in the short term based on the discussion at nodemcu/nodemcu-firmware#1066

1. Reduce the mem footprint of the httpserver significantly.

This would be nice, but I doubt we can get that far. I'll check out the code and look for areas of improvement.

[devyte]

Ok, so a 3rd solution came up: buffer the incoming requests and process them serially. I have an implementation that allows this testcase to work, but it sometimes leaks memory, and I have no idea why yet.

[ATAMAH]

Right solution: load html page only with only <div> tags where must be images and load images serially by JS and insert into that divs.

Modern browsers always try to load page elements in many concurrent connections.

[marcoskirsch]

Latest code works with 3 images in cars.html... fourth image crashes the server.
Could be due to running out of memory, limit in concurrent connections, or something else. More debugging needed.

[ATAMAH]

Fixed in my latest pull request.

You can add loading progress bar. Google "xhr.onprogress".

[CharlesScoville]

There appears to be more than just this issue with concurrent connections and nodemcu-httpserver.

Currently, if you have an html page that needs to run some script only once, it won't work right. The page gets requested by the browser up to a dozen times concurrently, so whatever init() you have gets run that many times. This includes any lua, since any call comes from the html, which is, again, loaded all those times. If you want to have any kind of initialization for a page, it seems you have to shoehorn it into the init.lua script.

Any one know what the deal with this is? Know a workaround that's better.

I've tried to use a semaphore in JavaScript and lua. And I've tried using the closure technique, or the window.onload method. Nothing has worked yet.

... I half suspect this may be a browser issue. Maybe I'll try with my lappy just to see.

Side Note:
I'm actually fixing a GPIO bug in garage_door_opener, and this is being a problem. The GPIOs default to high when the system starts, they should default to gnd for garage_door_opener. I have code that does the trick for my ESP-01, but it only does GPIO0 and GPIO2, it's flaky for many more than two. (which I guess is fine for the demo case)

[marcoskirsch]

I'm actually fixing a GPIO bug in garage_door_opener, and this is being a problem. The GPIOs default to high when the system starts, they should default to gnd for garage_door_opener. I have code that does the trick for my ESP-01, but it only does GPIO0 and GPIO2, it's flaky for many more than two. (which I guess is fine for the demo case)

This depends on the how you have the hardware wired. I am using a Wemos D1 mini pro with their relay shield, and things work just fine. This is as "official" as it gets hardware-wise as far as I know, so I think it's fine to leave as is. It's an example after all. How I hated the ESP-01... you should give this Wemos thing a try.

[HHHartmann]

Another method would be to separate static content from dynamic content.
The construct using the coroutine is very convenient for dynamic content generated by a lua script but it also has a large footprint.
For just serving a file this is a huge overhead. Only the filename and the position has to be retained between sends.
I have an implementation for that but need to sort out other changes before creating a PR.

[Fractal147]

So I've had some ideas and done a variety of tests with cars.lua. I've got some findings:
Different browsers make different patterns of requests, and behave differently with full refreshes and normal refreshes. The most aggressive I have found so far is mobile firefox on android, which submits all 6 image GET requests at once. Desktop Chrome often spreads out the GET requests, doing one first, then all the rest after the first has finished, but is more aggressive with a full refresh (ctrl-shift-r).
server:listen won't give a callback beyond 5 sockets open, the 6th is either queued opaquely, or dropped. Sometimes the 6th will get triggered as soon as there's a connection freed up, but by that point it's usually out of memory!

Every incoming connection creates a new socket, with memory usage being considerable until the connection is closed. More memory is used once the httpserver processes the request, and then when it decides what to do with it - i.e. assigns a fileServeFunction, and again when a bufferedConnection is made. Peak memory usage is during a connection:send, for a full buffer, spiking at least 1.4 kB. This is usually the moment it generates an error.

I've had success with immediately processing a GET request to reduce it to just a URI, fileServefunction string, socket, and the args, then storing that in a table. This means the full headers are gc'd quickly. This can store all the image requests before serving them one-by-one - but there's still a httpserver instance per connection at several kB each.
This has got cars.lua?n=6 to load repeatedly on desktop chrome, with free heap starting at 27 kB.
Running memory usage can probably be reduced further by loading the httpserver-x modules with "require" and then removing them from the package.loaded table when finished with (httpserver-request will be my first candidate).

I'm in process of making these hacked together changes a bit more pushable - my project requires robust image handling, and more flexible delays, so it's going to happen!