'json' encoding stringifies data before being inserted into db

[ianwremmel]

Since JSON is the native encoding to IndexedDB, it shouldn't be necessary to stringify data before storing it. Is it AbstractLevelDOWN that does the conversion? Does it make sense to replace 'json' encoding with raw:true when the database is opened?

I'm using the following to make sure I can cleanly work with JSON in both node and browser environments, but it feels like a hack:

package.json:

  ...
  "browser": {
    "leveldown": "level-js",
  }
  ...

index.js:

var levelup = require('levelup');
var leveldown = require('leveldown');
var leveljs = require('level-js');

var options = {
  db: leveldown
};
if (leveljs !== leveldown) {
  options.valueEncoding = 'json';
}
var db = levelup(name, options);

[mcollina]

Do you have some numeric data that shows how much we are losing in term of speed for this unneeded conversion? Some form of serialization happen anyway, and the native one should be faster, but it might not be worth the change here and the added complexity.

@dominictarr your work on encodings can help here?

[ianwremmel]

It's been a while since I opened this issue, so I don't remember all of the context, but if I recall correctly, I was think that if it used IndexedDB's native JSON support, there might be possibilities to take advantage of IndexedDB's indexing as well.

[mcollina]

Here is the problem: by using IndexedDB native indexing on top of LevelUp, you are kind-of violating all possible encapsulation and abstraction layers we placed upon it. If you need some IndexedDB special feature, just use IndexedDB. LevelUp exposes an ordered view of your data, where keys cannot be objects (well, only with typewise), and you code your own indexes.

I think we should keep this simple as we are all spread very thin in maintaining all those modules. Any other opinion on this?

[dominictarr]

I think this will probably work:

function id (e) { return e }
var nothing = {encode: id, decode: id, buffer: false}
levelup({db: levelJs, valueEncoding: nothing})

this should pass values into leveldown without serializing them.
keys still need to be strings.

[dominictarr]

I think it's okay, although a little bit dirty, to have a module that implements indexeddb like indexing that also detects if it is running on indexeddb and in that case uses native indexes.
As long as it still presents a consistent abstraction to the user.

[Ivshti]

@mcollina good point.

I also have a performance concern about this, but it will be good to see if it actually hurts something. It's possible that it doesn't

@ianwremmel do you have anything on this?

[ianwremmel]

Hi @Ivshti, we ended up going a different direction. Our primary motivation for using indexeddb was performance rather than offline access. Instead of focusing on building a database engine (because a year ago, that's really what you had to do to practically use indexeddb, even with helper libraries), we instead worked on improving API response times and adjusting UI feedback to improve the initial-load experience.

[juliangruber]

if you want level.js to get the raw values, +1 on this pull request: Level/abstract-leveldown#85

[juliangruber]

abstract-leveldown@2.5.0 is out! so if you want to bypass encoding completely, simply do this:

db._serializeKey =
db._serializeValue = function (data) { return data }

[dominictarr]

by the way, since we added custom codecs you could do valueEncoding: {encode: id, decode: id} where id is the id function: function id (e) { return e }

[juliangruber]

The problem was that abstract-leveldown would still (!) convert everything to a string of buffer. That can be bypassed now. So yeah you need to set those options on levelup and leveldown for now.

[dominictarr]

oh I see. well, we have encodeing.buffer: boolean right, maybe we need encodeing.string: boolean and then you can have naked encoding {encode: id, decode: id, buffer: false, string: false} ?

[vweevers]

level-js now defines its own _serializeKey and _serializeValue (#83). The above hack is no longer required, and level-js does not stringify objects.

So when using levelup and encoding-down, you can use the id encoding to put and get raw values and keys. If you do this with binary data, you'll get back a pure Uint8Array instead of Buffer:

https://github.com/Level/level.js/blob/eb9b63727597913b7e8c3825ffe2de62228fdfb9/test/test-levelup.js#L48-L51

Since encoding-down is an optional layer nowadays, you can also do:

https://github.com/Level/level.js/blob/eb9b63727597913b7e8c3825ffe2de62228fdfb9/test/custom-tests.js#L92-L96