Skip to content

Commit b725dff

Browse files
committed
Initial, forked from level-codec
1 parent 2aa01c6 commit b725dff

File tree

17 files changed

+726
-198
lines changed

17 files changed

+726
-198
lines changed

.github/workflows/test.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,9 @@ jobs:
1818
run: npm install
1919
- name: Test
2020
run: npm test
21-
- name: Coverage
22-
run: npm run coverage
23-
- name: Codecov
24-
uses: codecov/codecov-action@v1
25-
with:
26-
file: coverage/lcov.info
21+
# - name: Coverage
22+
# run: npm run coverage
23+
# - name: Codecov
24+
# uses: codecov/codecov-action@v2
25+
# with:
26+
# file: coverage/lcov.info

CONTRIBUTORS.md

Lines changed: 0 additions & 12 deletions
This file was deleted.

LICENSE.md renamed to LICENSE

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
# The MIT License (MIT)
1+
The MIT License (MIT)
22

3-
**Copyright © 2012-present [Contributors](CONTRIBUTORS.md).**
3+
Copyright © 2012 The contributors to level-transcoder and level-codec.
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

README.md

Lines changed: 51 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,62 @@
1-
# level-codec
1+
# level-transcoder
22

3-
> Encode keys, values and range options, with built-in or custom encodings.
3+
**Encode data with built-in or custom encodings.** The (not yet official) successor to `level-codec`, that introduces "transcoders" to translate between encodings and internal data formats supported by a db. This allows a db to store keys and values in a format of its choice (Buffer, Uint8Array or String) with zero-effort support of all known encodings.
44

55
[![level badge][level-badge]](https://github.com/Level/awesome)
6-
[![npm](https://img.shields.io/npm/v/level-codec.svg?label=&logo=npm)](https://www.npmjs.com/package/level-codec)
7-
[![Node version](https://img.shields.io/node/v/level-codec.svg)](https://www.npmjs.com/package/level-codec)
8-
[![npm](https://img.shields.io/npm/dm/level-codec.svg?label=dl)](https://www.npmjs.com/package/level-codec)
9-
[![Test](https://github.com/Level/codec/actions/workflows/test.yml/badge.svg)](https://github.com/Level/codec/actions/workflows/test.yml)
10-
[![Coverage Status](https://codecov.io/gh/Level/codec/branch/master/graph/badge.svg)](https://codecov.io/gh/Level/codec)
11-
[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)
12-
[![Backers on Open Collective](https://opencollective.com/level/backers/badge.svg?color=orange)](#backers)
13-
[![Sponsors on Open Collective](https://opencollective.com/level/sponsors/badge.svg?color=orange)](#sponsors)
6+
[![Standard](https://img.shields.io/badge/standard-informational?logo=javascript&logoColor=fff)](https://standardjs.com)
7+
[![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)
8+
[![Donate](https://img.shields.io/badge/donate-orange?logo=open-collective&logoColor=fff)](https://opencollective.com/level)
149

1510
## Usage
1611

17-
**If you are upgrading:** please see [`UPGRADING.md`](UPGRADING.md).
12+
**This is in POC stage.**
1813

1914
```js
20-
const Codec = require('level-codec')
21-
const codec = Codec({ keyEncoding: 'json' })
22-
const key = codec.encodeKey({ foo: 'bar' })
23-
console.log(key) // -> '{"foo":"bar"}'
24-
console.log(codec.decodeKey(key)) // -> { foo: 'bar' }
15+
const Transcoder = require('level-transcoder')
16+
17+
// Create a transcoder, passing a desired format
18+
const transcoder1 = new Transcoder(['view'])
19+
const transcoder2 = new Transcoder(['buffer'])
20+
const transcoder3 = new Transcoder(['utf8'])
21+
22+
// Uint8Array(3) [ 49, 50, 51 ]
23+
console.log(transcoder1.encoding('json').encode(123))
24+
25+
// <Buffer 31 32 33>
26+
console.log(transcoder2.encoding('json').encode(123))
27+
28+
// '123'
29+
console.log(transcoder3.encoding('json').encode(123))
30+
```
31+
32+
If given multiple formats (like how `leveldown` can work with both Buffer and strings), the best fitting format is chosen. Not by magic, just hardcoded logic because we don't have that many formats to deal with.
33+
34+
For example, knowing that JSON is a UTF-8 string which matches the desired `utf8` format, the `json` encoding will return a string here:
35+
36+
```js
37+
const transcoder4 = new Transcoder(['buffer', 'utf8'])
38+
39+
// '123'
40+
console.log(transcoder4.encoding('json').encode(123))
2541
```
2642

43+
In contrast, the `view` encoding doesn't match either `buffer` or `utf8` so data encoded by the `view` encoding gets transcoded into Buffers:
44+
45+
```js
46+
// <Buffer 31 32 33>
47+
console.log(transcoder4.encoding('view').encode(Uint8Array.from([49, 50, 51])))
48+
```
49+
50+
Copying of data is avoided where possible. That's true in the last example, because the underlying ArrayBuffer of the view can be passed to a Buffer constructor without a copy.
51+
52+
Lastly, the encoding returned by `Transcoder#encoding()` has a `format` property to be used to forward key- and valueEncoding options to an underlying store. This way, both the public and private API's of a db will be encoding-aware (somewhere in the future).
53+
54+
For example, on `leveldown` a call like `db.put(key, 123, { valueEncoding: 'json' })` will pass that value `123` through a `json` encoding that has a `format` of `utf8`, which is then forwarded as `db._put(key, '123', { valueEncoding: 'utf8' })`.
55+
56+
---
57+
58+
**_Rest of README is not yet updated._**
59+
2760
## API
2861

2962
### `codec = Codec([opts])`
@@ -125,19 +158,11 @@ See the [Contribution Guide](https://github.com/Level/community/blob/master/CONT
125158

126159
## Donate
127160

128-
To sustain [`Level`](https://github.com/Level) and its activities, become a backer or sponsor on [Open Collective](https://opencollective.com/level). Your logo or avatar will be displayed on our 28+ [GitHub repositories](https://github.com/Level) and [npm](https://www.npmjs.com/) packages. 💖
129-
130-
### Backers
131-
132-
[![Open Collective backers](https://opencollective.com/level/backers.svg?width=890)](https://opencollective.com/level)
133-
134-
### Sponsors
135-
136-
[![Open Collective sponsors](https://opencollective.com/level/sponsors.svg?width=890)](https://opencollective.com/level)
161+
Support us with a monthly donation on [Open Collective](https://opencollective.com/level) and help us continue our work.
137162

138163
## License
139164

140-
[MIT](LICENSE.md) © 2012-present [Contributors](CONTRIBUTORS.md).
165+
[MIT](LICENSE)
141166

142167
[level-badge]: https://leveljs.org/img/badge.svg
143168

index.d.ts

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import { Encoding, EncodingOptions } from './lib/encoding'
2+
3+
declare class Transcoder<T = any> {
4+
/**
5+
* Create a Transcoder.
6+
* @param formats Formats supported by consumer.
7+
*/
8+
constructor (formats: Iterable<string>)
9+
10+
/**
11+
* Get the types of supported encodings, including transcoded encodings.
12+
*/
13+
types (): string[]
14+
15+
/**
16+
* Get the given encoding, creating a transcoder if necessary.
17+
* @param encoding Named encoding or encoding object.
18+
*/
19+
encoding<TIn, TFormat, TOut> (
20+
encoding: Encoding<TIn, TFormat, TOut>|EncodingOptions<TIn, TFormat, TOut>
21+
): Encoding<TIn, T, TOut>
22+
23+
encoding (encoding: 'utf8'): Encoding<string | Buffer | Uint8Array, T, string>
24+
encoding (encoding: 'buffer'): Encoding<Buffer | Uint8Array | string, T, Buffer>
25+
encoding (encoding: 'view'): Encoding<Uint8Array | string, T, Uint8Array>
26+
encoding (encoding: 'json'): Encoding<any, T, any>
27+
encoding (encoding: 'hex'): Encoding<Buffer | string, T, string>
28+
encoding (encoding: 'base64'): Encoding<Buffer | string, T, string>
29+
encoding (encoding: 'id'): Encoding<any, any, any>
30+
encoding (encoding: string): Encoding<any, T, any>
31+
}
32+
33+
export = Transcoder

index.js

Lines changed: 110 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -1,109 +1,130 @@
11
'use strict'
22

3+
const ModuleError = require('module-error')
34
const encodings = require('./lib/encodings')
4-
const rangeOptions = new Set(['lt', 'gt', 'lte', 'gte'])
5-
6-
module.exports = Codec
7-
8-
function Codec (opts) {
9-
if (!(this instanceof Codec)) {
10-
return new Codec(opts)
11-
}
12-
this.opts = opts || {}
13-
this.encodings = encodings
14-
}
15-
16-
Codec.prototype._encoding = function (encoding) {
17-
if (typeof encoding === 'string') encoding = encodings[encoding]
18-
if (!encoding) encoding = encodings.id
19-
return encoding
20-
}
21-
22-
Codec.prototype._keyEncoding = function (opts, batchOpts) {
23-
return this._encoding((batchOpts && batchOpts.keyEncoding) ||
24-
(opts && opts.keyEncoding) ||
25-
this.opts.keyEncoding)
26-
}
27-
28-
Codec.prototype._valueEncoding = function (opts, batchOpts) {
29-
return this._encoding((batchOpts && (batchOpts.valueEncoding || batchOpts.encoding)) ||
30-
(opts && (opts.valueEncoding || opts.encoding)) ||
31-
(this.opts.valueEncoding || this.opts.encoding))
32-
}
33-
34-
Codec.prototype.encodeKey = function (key, opts, batchOpts) {
35-
return this._keyEncoding(opts, batchOpts).encode(key)
36-
}
37-
38-
Codec.prototype.encodeValue = function (value, opts, batchOpts) {
39-
return this._valueEncoding(opts, batchOpts).encode(value)
40-
}
41-
42-
Codec.prototype.decodeKey = function (key, opts) {
43-
return this._keyEncoding(opts).decode(key)
44-
}
45-
46-
Codec.prototype.decodeValue = function (value, opts) {
47-
return this._valueEncoding(opts).decode(value)
48-
}
49-
50-
Codec.prototype.encodeBatch = function (ops, opts) {
51-
return ops.map((_op) => {
52-
const op = {
53-
type: _op.type,
54-
key: this.encodeKey(_op.key, opts, _op)
5+
const { Encoding } = require('./lib/encoding')
6+
const { BufferFormat, ViewFormat, UTF8Format, NativeFormat } = require('./lib/formats')
7+
8+
const kFormats = Symbol('formats')
9+
const kEncodings = Symbol('encodings')
10+
11+
/** @template T */
12+
class Transcoder {
13+
/**
14+
* @param {Iterable<string>} formats
15+
*/
16+
constructor (formats) {
17+
/** @type {Map<string|Encoding<any, any, any>|EncodingOptions<any, any, any>, Encoding<any, any, any>>} */
18+
this[kEncodings] = new Map()
19+
this[kFormats] = new Set(formats)
20+
21+
// Only support aliases in key- and valueEncoding options (where we already did)
22+
for (const [alias, { type }] of Object.entries(aliases)) {
23+
if (this[kFormats].has(alias)) {
24+
throw new ModuleError(`The '${alias}' alias is not supported here; use '${type}' instead`, {
25+
code: 'LEVEL_ENCODING_NOT_SUPPORTED'
26+
})
27+
}
5528
}
56-
if (this.keyAsBuffer(opts, _op)) op.keyEncoding = 'binary'
57-
if (_op.prefix) op.prefix = _op.prefix
58-
if ('value' in _op) {
59-
op.value = this.encodeValue(_op.value, opts, _op)
60-
if (this.valueAsBuffer(opts, _op)) op.valueEncoding = 'binary'
29+
30+
// Register encodings (done early in order to populate types())
31+
for (const k in encodings) {
32+
try {
33+
this.encoding(k)
34+
} catch (err) {
35+
if (err.code !== 'LEVEL_ENCODING_NOT_SUPPORTED') throw err
36+
}
6137
}
62-
return op
63-
})
64-
}
38+
}
6539

66-
Codec.prototype.encodeLtgt = function (ltgt) {
67-
const ret = {}
40+
types () {
41+
const types = new Set()
6842

69-
for (const key of Object.keys(ltgt)) {
70-
if (key === 'start' || key === 'end') {
71-
throw new Error('Legacy range options ("start" and "end") have been removed')
43+
for (const encoding of this[kEncodings].values()) {
44+
const type = encoding.type.split('+')[0]
45+
if (type) types.add(type)
7246
}
7347

74-
ret[key] = rangeOptions.has(key)
75-
? this.encodeKey(ltgt[key], ltgt)
76-
: ltgt[key]
48+
return Array.from(types)
7749
}
7850

79-
return ret
80-
}
51+
// TODO: document that we don't fallback to 'id' anymore if encoding is not found
52+
/**
53+
* @param {string|Encoding<any, any, any>|EncodingOptions<any, any, any>} encoding
54+
* @returns {Encoding<any, T, any>}
55+
*/
56+
encoding (encoding) {
57+
let resolved = this[kEncodings].get(encoding)
58+
59+
if (resolved === undefined) {
60+
if (typeof encoding === 'string' && encoding !== '') {
61+
resolved = lookup[encoding]
62+
63+
if (!resolved) {
64+
throw new ModuleError(
65+
`Encoding '${encoding}' is not found`,
66+
{ code: 'LEVEL_ENCODING_NOT_FOUND' }
67+
)
68+
}
69+
} else if (typeof encoding !== 'object' || encoding === null) {
70+
throw new TypeError('Encoding must be a string or object')
71+
} else if (encoding instanceof Encoding) {
72+
resolved = encoding
73+
} else if (encoding.format === 'view') {
74+
resolved = new ViewFormat(encoding)
75+
} else if (encoding.format === 'utf8' || encoding.buffer === false) {
76+
resolved = new UTF8Format(encoding)
77+
} else {
78+
resolved = new BufferFormat(encoding)
79+
}
8180

82-
Codec.prototype.createStreamDecoder = function (opts) {
83-
if (opts.keys && opts.values) {
84-
return (key, value) => {
85-
return {
86-
key: this.decodeKey(key, opts),
87-
value: this.decodeValue(value, opts)
81+
const { type, idempotent, format } = resolved
82+
83+
if (this[kFormats].has(type)) {
84+
// If idempotent, run data through it to normalize
85+
if (!idempotent) resolved = new NativeFormat(type)
86+
} else if (!this[kFormats].has(format)) {
87+
if (this[kFormats].has('view')) {
88+
resolved = resolved.transcode('view')
89+
} else if (this[kFormats].has('buffer')) {
90+
resolved = resolved.transcode('buffer')
91+
} else {
92+
throw new ModuleError(
93+
`Encoding '${type}' or '${format}' is not supported`,
94+
{ code: 'LEVEL_ENCODING_NOT_SUPPORTED' }
95+
)
96+
}
97+
}
98+
99+
for (const k of [encoding, type, resolved.type]) {
100+
if (k) this[kEncodings].set(k, resolved)
88101
}
89102
}
90-
} else if (opts.keys) {
91-
return (key) => {
92-
return this.decodeKey(key, opts)
93-
}
94-
} else if (opts.values) {
95-
return (_, value) => {
96-
return this.decodeValue(value, opts)
97-
}
98-
} else {
99-
return function () {}
103+
104+
return resolved
100105
}
101106
}
102107

103-
Codec.prototype.keyAsBuffer = function (opts) {
104-
return this._keyEncoding(opts).buffer
108+
module.exports = Transcoder
109+
110+
/**
111+
* @typedef {import('./lib/encoding').EncodingOptions<TIn,TFormat,TOut>} EncodingOptions
112+
* @template TIn, TFormat, TOut
113+
*/
114+
115+
/**
116+
* @type {Object.<string, Encoding<any, any, any>>}
117+
*/
118+
const aliases = {
119+
binary: encodings.buffer,
120+
'utf-8': encodings.utf8,
121+
none: encodings.id
105122
}
106123

107-
Codec.prototype.valueAsBuffer = function (opts) {
108-
return this._valueEncoding(opts).buffer
124+
/**
125+
* @type {Object.<string, Encoding<any, any, any>>}
126+
*/
127+
const lookup = {
128+
...encodings,
129+
...aliases
109130
}

0 commit comments

Comments
 (0)