Buffers and custom encoding examples

[dcousens]

Is there a canonical set of examples for no string encoding and or custom encodings?

The first example I found was https://github.com/Level/levelup/blob/master/test/inject-encoding-test.js but that requires learning about msgpack-js.

Second example I found was #412, which described binary (a string encoding format, aliasing latin1 as per nodejs documentation) somehow returning a Buffer...

I just want to put(keyBuffer, valueBuffer), get(keyBuffer, (err, valueBuffer) => {}) and it isn't entirely clear how to do that...

At least with a Buffer, I could then just push it through something like https://github.com/dominictarr/varstruct for type checking.

[dcousens]

It seems something is off here, and I'm not sure if its just the binary special case, or maybe some kind of strange encoding going on internally...

[dcousens]

It should probably also document the lifetime of any Buffer passed, since the arguments are not copied-on-use the underlying data committed can change... 😨

[peakji]

I just want to put(keyBuffer, valueBuffer), get(keyBuffer, (err, valueBuffer) => {}) and it isn't entirely clear how to do that...

Set keyEncoding: "binary" and valueEncoding : "binary" in options of any main interface method.

It should probably also document the lifetime of any Buffer passed, since the arguments are not copied-on-use the underlying data committed can change... 😨

We can't document it in levelup because the lifetime depends on the implementation of storage backend. For the put method in the default leveldown backend, it is handled by database_async.cc#L175, so its not safe to modify buffer values until callback:

db.open(function(err) {
	var buf = Buffer.from([0x00, 0x01, 0x02]);
	db.put("test", buf, {
		valueEncoding: "binary"
	}, function(err) {
		buf[0] = 0x04;
		db.get("test", {
			valueEncoding: "binary"
		}, function(err, val) {
			console.log(val);	// <Buffer 03 01 02>
		});
	});
	buf[0] = 0x03;
});

[dcousens]

@peakji but why doesn't binary return a latin1 encoded string? Similar to how you would expect a utf8 encoded string if you used utf8.

To illustrate my example:

> Buffer.from('test', 'binary').toString('binary')
'test'

Notice how it isn't a Buffer... why was something so ambiguous with an actual toString encoding format chosen for 'raw' Buffers?
What if I wanted a binary encoded string?

[peakji]

@dcousens the encoding here is actually the codec name of your custom-encoding , which represents a pair of encode and decode functions, and a buffer option. utf8 and binaryare just built-in bidirectional codecs, in fact, theres also one called json.

[peakji]

@dcousens you should write your own custom encodings for anything other than utf8, json, binary.

If you need binary strings:

{
    encode : val => Buffer.from(val, 'binary'),
    decode : val => val.toString('binary'),
    buffer : true,
    type   : "mybinary"
}

[dcousens]

@peakji in which case, the documentation is mis-leading:

The encoding of the keys and values passed through Node.js' Buffer implementation (see Buffer#toString()).