[WIP] feat(repository): add KVRepository impl using legacy juggler #1527

raymondfeng · 2018-07-13T16:32:57Z

The PR adds a default implementation of KVRepository with legacy juggler

Depends on loopbackio/loopback-datasource-juggler#1608

Connects to #1448

Checklist

npm test passes on your machine
New tests added or existing tests modified to cover all changes
Code conforms with the style guide
API Documentation in code was updated
Documentation in /docs/site was updated
Affected artifact templates in packages/cli were updated
Affected example projects in examples/* were updated

bajtos

Great start!

Please add tests to cover all newly added APIs, see packages/repository/test/unit/repositories/legacy-juggler-bridge.unit.ts.

I am concerned about the growing size of legacy-juggler-bridge.ts file, is it perhaps time to split it into multiple files, one for each different repository?

bajtos · 2018-07-16T06:51:07Z

packages/repository/src/repositories/kv.repository.ts

@@ -38,7 +35,7 @@ export interface KVRepository<T extends Model> extends Repository<T> {
   * @param options Options for the operation
   * @returns A promise of the entry
   */
-  get(key: string, options?: Options): Promise<T>;
+  get(key: string, options?: Options): Promise<DataObject<T>>;


IMO, repositories should always return model instances, not data objects. Our CRUD repository already does that (see e.g. findById method), let's do the same for KeyValue too please.

bajtos · 2018-07-16T06:53:49Z

packages/repository/src/repositories/kv.repository.ts

   */
-  expire(key: string, ttl: number, options?: Options): Promise<boolean>;
+  expire(key: string, ttl: number, options?: Options): Promise<number>;


Do we expect the returned TTL set to be different from the ttl requested? How do you expect the callers to handle that situation? If the returned TTL is always the same as requested, then what's the benefit of returning that value?

I think it would be better to change this API to return Promise<void>.

Thoughts?

bajtos · 2018-07-16T06:55:50Z

packages/repository/src/repositories/kv.repository.ts

   */
-  iterateKeys?(filter?: Filter, options?: Options): Promise<Iterator<T>>;
+  iterateKeys?(filter?: Filter, options?: Options): Iterator<Promise<string>>;


IIRC, the key-value access API provided by LB 3.x returns AsyncIterator<string> which should be consumable via for await in Node.js 10.x:

for await (const key of model.iterateKeys()) { // ... }

Could you please verify my claim and then fix the typescript definitions accordingly?

See also https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-3.html#async-iteration

And https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-3.html#the-for-await-of-statement

LB3 does not fully implement AsyncIterator. See https://github.com/strongloop/loopback-datasource-juggler/blob/ade9f0fd9c1b133b3e6fc8dfec73fa82b5f0ab70/types/kv-model.d.ts#L210-L219

Interesting. When I was implementing KeyValue data-access API, my intention was to design iterateKeys in a way that make it usable as an async iterator, at least in the way how async iterators were specified at that time.

What are we missing in our implementation that makes iterateKeys incompatible with AsyncIterator and for-await-of? I would like to fix it in juggler before the first release of KeyValue API in LB4.

I suppose iterateKeys can be fixed after this initial pull request is landed in a follow-up work.

bajtos · 2018-07-16T06:58:17Z

packages/repository/src/repositories/relation.factory.ts

@@ -43,10 +44,12 @@ export function createHasManyRepositoryFactory<
        'The foreign key property name (keyTo) must be specified',
      );
    }
+    // tslint:disable-next-line:no-any
+    const constraint: any = {[fkName]: fkValue};


Have you tried using object instead of any, or perhaps leave out the type information altogether?

const contraint = {[fkName]: fkValue};

bajtos · 2018-07-16T07:03:25Z

packages/repository/src/repositories/legacy-juggler-bridge.ts

+    throw new Error('Not implemented');
+  }
+
+  deleteAll(options?: Options): Promise<void> {


AFAICT, juggler 3.x provides both delete and deleteAll operations, although they seems like not documented in our apidocs.

KVAO.prototype.delete(key[, options], cb)

KVAO.prototype.deleteAll([options, ]cb)

See the tests for usage examples:

test/kvao/delete-all.suite.js

test/kvao/delete.suite.js

bajtos · 2018-07-16T16:33:16Z

packages/repository/src/repositories/kv.repository.ts

@@ -68,7 +64,7 @@ export interface KVRepository<T extends Model> extends Repository<T> {
   * @param options Options for the operation
   * @returns A promise of the TTL value
   */
-  ttl?(key: string, ttl: number, options?: Options): Promise<number>;
+  ttl?(key: string, options?: Options): Promise<number>;

  /**
   * Fetch all keys


Loading all keys into memory is a foot-gun that opens doors to a DoS attack where the datasource is queried with a pattern that matches too many keys.

The only reason why I added the method keys(): string[], was to work around the limitations of LB 3.x - the way how it was directly exposing juggler methods via REST APIs and a lack of support for streaming responses.

We don't have that limitation in LB4 because REST API is exposed via custom Controller classes that can work around missing support for streaming responses in LB4.

I am proposing to remove keys(): string[] API from LB4 and provide only one method based on AsyncIterator. Ideally, this single method should be called keys instead of iterateKeys for simplicity.

Until our REST layer supports AsyncIterators, we can provide a helper function to convert AsyncIterator into a static array that people can use in their controllers.

Thoughts?

I am ok to leave this change out of this pull request if you prefer, as long as we make it happen before the first public release of KeyValue feature in LB4.

raymondfeng · 2018-07-16T16:38:18Z

@bajtos Please review loopbackio/loopback-datasource-juggler#1608 1st.

raymondfeng · 2018-07-16T19:02:27Z

github seems to be messed with the PR - two files are missing.

raymondfeng requested a review from bajtos as a code owner July 13, 2018 16:32

raymondfeng force-pushed the kv-repository branch from a86a764 to 2138def Compare July 13, 2018 16:34

bajtos suggested changes Jul 16, 2018

View reviewed changes

bajtos reviewed Jul 16, 2018

View reviewed changes

raymondfeng force-pushed the kv-repository branch 2 times, most recently from d23051d to bc0d9b1 Compare July 16, 2018 16:37

raymondfeng force-pushed the kv-repository branch from bc0d9b1 to ef5ddb9 Compare July 16, 2018 17:09

feat(repository): add KVRepository impl using legacy juggler

ca55097

raymondfeng force-pushed the kv-repository branch from ef5ddb9 to ca55097 Compare July 16, 2018 17:14

raymondfeng closed this Jul 16, 2018

raymondfeng mentioned this pull request Jul 16, 2018

feat(repository): add KVRepository impl using legacy juggler #1539

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] feat(repository): add KVRepository impl using legacy juggler #1527

[WIP] feat(repository): add KVRepository impl using legacy juggler #1527

raymondfeng commented Jul 13, 2018 •

edited

Loading

bajtos left a comment

bajtos Jul 16, 2018

bajtos Jul 16, 2018

bajtos Jul 16, 2018

bajtos Jul 16, 2018

bajtos Jul 16, 2018

raymondfeng Jul 16, 2018

bajtos Jul 16, 2018

bajtos Jul 16, 2018

bajtos Jul 16, 2018

bajtos Jul 16, 2018

raymondfeng Jul 16, 2018

raymondfeng commented Jul 16, 2018

raymondfeng commented Jul 16, 2018

[WIP] feat(repository): add KVRepository impl using legacy juggler #1527

[WIP] feat(repository): add KVRepository impl using legacy juggler #1527

Conversation

raymondfeng commented Jul 13, 2018 • edited Loading

Checklist

bajtos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raymondfeng commented Jul 16, 2018

raymondfeng commented Jul 16, 2018

raymondfeng commented Jul 13, 2018 •

edited

Loading