Deno support #12397

vkarpov15 · 2022-09-05T18:04:59Z

Summary

Tests are passing against Deno 👍 Still with --unstable, so there's some cleanup there, but this is still a big step.

For the most part, no noteworthy runtime issues beyond #12313. Just a bunch of test issues due to our tests being a bit sloppy with schematype global state cleanup, and a quirk with dns resolution throwing an uncatchable error. We'll sort those out a little more later.

Examples

Re: #9056

…st failures re: #9056

… async/await for better debug

…rkflows re: #9056

… done re: #9056

…t failure

…9056

test/.eslintrc.yml

hasezoey · 2022-09-06T09:39:33Z

tested it locally and seems to be fine, though there are some errors which dont seem to be consistent, but dont really happen in normal node:

  2862 passing (5m)
  31 pending
  5 failing

  1) Model
       dropDatabase() after init allows re-init (gh-6967):
     MongoNetworkError: connection 3 to 127.0.0.1:27017 closed
      at Connection.onClose (file:///mnt/projects/nodejs/mongoose/node_modules/mongodb/lib/cmap/connection.js:135:19)
      at Socket.<anonymous> (file:///mnt/projects/nodejs/mongoose/node_modules/mongodb/lib/cmap/connection.js:62:46)
      at Socket.emit (https://deno.land/[email protected]/node/_events.mjs:391:35)
      at https://deno.land/[email protected]/node/net.ts:1417:14
      at TCP.close (https://deno.land/[email protected]/node/internal_binding/handle_wrap.ts:37:5)
      at Socket._destroy (https://deno.land/[email protected]/node/net.ts:1409:20)
      at _destroy (https://deno.land/[email protected]/node/internal/streams/destroy.mjs:96:25)
      at Socket.destroy (https://deno.land/[email protected]/node/internal/streams/destroy.mjs:58:5)
      at Timeout.<anonymous> (file:///mnt/projects/nodejs/mongoose/node_modules/mongodb/lib/cmap/connection.js:145:27)
      at https://deno.land/[email protected]/node/timers.ts:21:21
      at Object.action (deno:ext/web/02_timers.js:149:13)
      at handleTimerMacrotask (deno:ext/web/02_timers.js:66:12)

  2) Model
       "after each" hook for "dropDatabase() after init allows re-init (gh-6967)":
     Error: Timeout of 8000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/mnt/projects/nodejs/mongoose/test/model.test.js)
      at https://deno.land/[email protected]/node/timers.ts:21:21
      at Object.action (deno:ext/web/02_timers.js:149:13)
      at handleTimerMacrotask (deno:ext/web/02_timers.js:66:12)

  3) Query
       deleteOne/deleteMany
         handles deleteMany:
     MongoNetworkError: connection 2 to 127.0.0.1:27017 closed
      at Connection.onClose (file:///mnt/projects/nodejs/mongoose/node_modules/mongodb/lib/cmap/connection.js:135:19)
      at Socket.<anonymous> (file:///mnt/projects/nodejs/mongoose/node_modules/mongodb/lib/cmap/connection.js:62:46)
      at Socket.emit (https://deno.land/[email protected]/node/_events.mjs:391:35)
      at https://deno.land/[email protected]/node/net.ts:1417:14
      at TCP.close (https://deno.land/[email protected]/node/internal_binding/handle_wrap.ts:37:5)
      at Socket._destroy (https://deno.land/[email protected]/node/net.ts:1409:20)
      at _destroy (https://deno.land/[email protected]/node/internal/streams/destroy.mjs:96:25)
      at Socket.destroy (https://deno.land/[email protected]/node/internal/streams/destroy.mjs:58:5)
      at Timeout.<anonymous> (file:///mnt/projects/nodejs/mongoose/node_modules/mongodb/lib/cmap/connection.js:145:27)
      at https://deno.land/[email protected]/node/timers.ts:21:21
      at Object.action (deno:ext/web/02_timers.js:149:13)
      at handleTimerMacrotask (deno:ext/web/02_timers.js:66:12)

  4) Query
       "after each" hook for "handles deleteMany":
     Error: Timeout of 8000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/mnt/projects/nodejs/mongoose/test/query.test.js)
      at https://deno.land/[email protected]/node/timers.ts:21:21
      at Object.action (deno:ext/web/02_timers.js:149:13)
      at handleTimerMacrotask (deno:ext/web/02_timers.js:66:12)

  5) Query
       "after all" hook for "skips applying default projections over slice projections (gh-11940)":
     Error: Timeout of 8000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/mnt/projects/nodejs/mongoose/test/query.test.js)
      at https://deno.land/[email protected]/node/timers.ts:21:21
      at Object.action (deno:ext/web/02_timers.js:149:13)
      at handleTimerMacrotask (deno:ext/web/02_timers.js:66:12)

normal node run on my system

  3246 passing (3m)
  47 pending
  6 failing

  1) model: querying:
       lean
         casts $elemMatch (gh-2199):
     Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/mnt/projects/nodejs/mongoose/test/model.querying.test.js)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

  2) Model
       can be created using _id as embedded document:
     Uncaught AssertionError [ERR_ASSERTION]: undefined == 'Daniel'
      at /mnt/projects/nodejs/mongoose/test/model.test.js:117:16
      at /mnt/projects/nodejs/mongoose/lib/model.js:5135:18
      at process.processTicksAndRejections (node:internal/process/task_queues:77:11)

  3) Model
       getters
         nested
           pre-existing null object re-save:
     Uncaught AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Operation `Test.findOne()` buffering timed out after 10000ms
      at /mnt/projects/nodejs/mongoose/test/model.querying.test.js:2477:18
      at /mnt/projects/nodejs/mongoose/lib/model.js:5135:18
      at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
      at Timeout.<anonymous> (lib/drivers/node-mongodb-native/collection.js:153:23)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

  4) Model
       bug fixes
         createCollection() enforces expireAfterSeconds (gh-11229):
     Uncaught AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Operation `Test.updateOne()` buffering timed out after 10000ms
      at /mnt/projects/nodejs/mongoose/test/model.test.js:1889:20
      at /mnt/projects/nodejs/mongoose/lib/model.js:5135:18
      at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
      at Timeout.<anonymous> (lib/drivers/node-mongodb-native/collection.js:153:23)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

  5) Model
       bug fixes
         createCollection() enforces expires when set by Schema (gh-11229):

      AssertionError [ERR_ASSERTION]: 12 == 0
      + expected - actual

      -12
      +0
      
      at Context.<anonymous> (test/model.test.js:7203:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  6) Model
       bug fixes
         createCollection() enforces expireAfterSeconds (gh-11229):
     done() called multiple times in test <Model bug fixes createCollection() enforces expireAfterSeconds (gh-11229)> of file /mnt/projects/nodejs/mongoose/test/model.test.js; in addition, done() received error: MongooseError: Operation `TestGH11229Var1.count()` buffering timed out after 10000ms
    at Timeout.<anonymous> (/mnt/projects/nodejs/mongoose/lib/drivers/node-mongodb-native/collection.js:153:23)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7)
  Error: done() called multiple times in test <Model bug fixes createCollection() enforces expireAfterSeconds (gh-11229)> of file /mnt/projects/nodejs/mongoose/test/model.test.js; in addition, done() received error: MongooseError: Operation `TestGH11229Var1.count()` buffering timed out after 10000ms
      at Timeout.<anonymous> (lib/drivers/node-mongodb-native/collection.js:153:23)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

AbdelrahmanHafez

LGTM, thanks 👍

Uzlopak

I have no objections.

hasezoey

also looks good to me, though i still dont know why in deno so many timeout / connection close errors are happening locally (see earlier comment)

vkarpov15 · 2022-09-08T18:48:45Z

@hasezoey what version of Deno are you using?

hasezoey · 2022-09-08T19:21:24Z

@hasezoey what version of Deno are you using?

deno 1.25.1
installed through cargo install deno (no changed features)
compiled with rust 1.63.0 (stable toolchain)
on a x86_64 manjaro-linux system (same system and mongodb with which i run node)
tests run through yarn run test-deno (and node with yarn run test)

PS: sometimes i also get the timeouts with node, but it seems to be more common in deno because deno somehow runs slower than node (see the time in the comment, where deno was 5m and node only 3m)

Edit: i am on the machine again and can provide deno --version:

deno 1.25.1 (release, x86_64-unknown-linux-gnu)
v8 10.6.194.5
typescript 4.7.4

README.md

vkarpov15 · 2022-10-21T19:41:41Z

Moving this to 6.8, still waiting on feedback from Deno team.

Uzlopak · 2022-10-27T20:54:19Z

@vkarpov15
Deno 1.27 is out. I guess it has now the necessary compatibility?

.github/workflows/test.yml

README.md

test/deno.js

…vkarpov15/deno

Co-authored-by: Uzlopak <[email protected]>

rojvv · 2022-11-29T17:14:05Z

Does this really need a node_modules?

hasezoey · 2022-11-29T17:37:28Z

Does this really need a node_modules?

@roj1512 could you explain what you mean?

rojvv · 2022-11-29T17:45:25Z

Do we need NPM and node_modules to use this Deno support?

hasezoey · 2022-11-29T17:49:04Z

Do we need NPM and node_modules to use this Deno support?

read the deno documentation at NPM Specifiers

TL;DR:

When doing this, no npm install is necessary and no node_modules folder is created. These packages are also subject to the same permissions as Deno applications. At the moment though, there are some unnecessary permissions that get asked for, but in the future the above program will only require network permissions.

unless option --node-modules-dir is used (though from what i can tell, it is not required for mongoose)

though note that i did not use this yet so i cannot verify that this is actually true (probably deno still has a cache somewhere)

rojvv · 2022-11-29T18:10:20Z

Sad.

vkarpov15 · 2022-12-02T00:12:09Z

Currently yes, you do need npm and node_modules. We'll consider improving this in the future.

hasezoey · 2022-12-02T09:23:08Z

test/connection.test.js

+    if (typeof Deno !== 'undefined') {
+      // In Deno dns throws an uncatchable error here.
+      return this.skip();
+    }


btw, are there any news regarding this "uncatchable dns error"?

@hasezoey nope, I don't think this issue is significant enough to block shipping deno support.

didnt mean to block with this question, just wanted to ask if there are any known updates regarding this problem to re-enable the test for deno

vkarpov15 added 18 commits September 4, 2022 16:25

feat: run tests on Deno

3f79b18

Re: #9056

chore: correct action name

c1a4823

chore: setup MongoDB server on deno tests

52110b6

Re: #9056

chore: correct download path re: #9056

8697ce5

test: convert #3973 test to async/await to try to work around deno te…

32e8055

…st failures re: #9056

test: ignore deno.js for linter, convert a few more populate tests to…

f4d8f61

… async/await for better debug

test: print error to debug why #3973 test only fails on Deno in GH wo…

0698d9a

…rkflows re: #9056

test: print some more errors re: #9056

356a132

avoid using elvis operator

2bcebe0

test: try cleaning up castNonArrays option to fix Deno tests re: #9056

79fb5b2

test: correctly reset castNonArrays

f89f833

test: skip child_process test in Deno, explicitly exit when tests are…

5045b5b

… done re: #9056

test: check if #8222 error is caught in Deno re: #9056

c13f2a6

test: temporarily skip strictQuery tests to see if they fix #9056 tes…

d6f6ce8

…t failure

test: increase test timeout in deno and ensure strictQuery is set re: #…

72124b8

…9056

test: add back connection and index tests, skip because of dns issue #…

b3c31f1

…9056

test: quick test fix re: #9046

1fb268e

test: correctly clean up model re: #9056

17e2831

vkarpov15 requested review from Uzlopak, hasezoey and AbdelrahmanHafez September 5, 2022 18:04

hasezoey reviewed Sep 6, 2022

View reviewed changes

test/.eslintrc.yml Show resolved Hide resolved

hasezoey mentioned this pull request Sep 6, 2022

Improve tests #12401

Merged

AbdelrahmanHafez approved these changes Sep 6, 2022

View reviewed changes

Uzlopak approved these changes Sep 7, 2022

View reviewed changes

hasezoey approved these changes Sep 7, 2022

View reviewed changes

vkarpov15 added this to the 6.7 milestone Sep 8, 2022

vkarpov15 added 3 commits October 5, 2022 16:40

Merge branch '6.7' into vkarpov15/deno

3fb3ccb

docs: add deno instructions and badges to README

759e47d

docs: more info on required deno flags

872b9a3

hasezoey reviewed Oct 6, 2022

View reviewed changes

README.md Show resolved Hide resolved

vkarpov15 modified the milestones: 6.7, 6.8 Oct 21, 2022

Uzlopak reviewed Oct 27, 2022

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

Uzlopak reviewed Oct 27, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

Uzlopak reviewed Oct 27, 2022

View reviewed changes

test/deno.js Show resolved Hide resolved

Uzlopak reviewed Oct 27, 2022

View reviewed changes

test/deno.js Outdated Show resolved Hide resolved

Uzlopak and others added 3 commits October 28, 2022 20:44

Apply suggestions from code review

bd5e28a

Merge branch '6.8' into vkarpov15/deno

5517fbc

Merge branch 'vkarpov15/deno' of github.com:Automattic/mongoose into …

566c6e1

…vkarpov15/deno

vkarpov15 changed the base branch from 6.7 to 6.8 November 28, 2022 17:10

vkarpov15 and others added 4 commits November 28, 2022 12:11

Update README.md

e4e26f1

Co-authored-by: Uzlopak <[email protected]>

fix: use deno 1.28 in tests, use --allow-sys, no more --unstable

0a37232

test: clear data between query cursor tests

b41f2de

quick note about alpha support for deno

68ec07d

vkarpov15 merged commit 0c98cbb into 6.8 Nov 28, 2022

hasezoey deleted the vkarpov15/deno branch November 29, 2022 10:16

hasezoey reviewed Dec 2, 2022

View reviewed changes

vkarpov15 mentioned this pull request Jan 23, 2023

fix(query): correctly pass context when casting $elemMatch #12915

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deno support #12397

Deno support #12397

vkarpov15 commented Sep 5, 2022

hasezoey commented Sep 6, 2022

AbdelrahmanHafez left a comment

Uzlopak left a comment

hasezoey left a comment •

edited

Loading

vkarpov15 commented Sep 8, 2022

hasezoey commented Sep 8, 2022 •

edited

Loading

vkarpov15 commented Oct 21, 2022

Uzlopak commented Oct 27, 2022

rojvv commented Nov 29, 2022

hasezoey commented Nov 29, 2022

rojvv commented Nov 29, 2022

hasezoey commented Nov 29, 2022 •

edited

Loading

rojvv commented Nov 29, 2022

vkarpov15 commented Dec 2, 2022

hasezoey Dec 2, 2022

vkarpov15 Dec 4, 2022

hasezoey Dec 5, 2022

Deno support #12397

Deno support #12397

Conversation

vkarpov15 commented Sep 5, 2022

hasezoey commented Sep 6, 2022

AbdelrahmanHafez left a comment

Choose a reason for hiding this comment

Uzlopak left a comment

Choose a reason for hiding this comment

hasezoey left a comment • edited Loading

Choose a reason for hiding this comment

vkarpov15 commented Sep 8, 2022

hasezoey commented Sep 8, 2022 • edited Loading

vkarpov15 commented Oct 21, 2022

Uzlopak commented Oct 27, 2022

rojvv commented Nov 29, 2022

hasezoey commented Nov 29, 2022

rojvv commented Nov 29, 2022

hasezoey commented Nov 29, 2022 • edited Loading

rojvv commented Nov 29, 2022

vkarpov15 commented Dec 2, 2022

hasezoey Dec 2, 2022

Choose a reason for hiding this comment

vkarpov15 Dec 4, 2022

Choose a reason for hiding this comment

hasezoey Dec 5, 2022

Choose a reason for hiding this comment

hasezoey left a comment •

edited

Loading

hasezoey commented Sep 8, 2022 •

edited

Loading

hasezoey commented Nov 29, 2022 •

edited

Loading