Discussion: Use of NPM modules within deno

[reggi]

I made an example repo that allows someone to convert npm modules for use with deno.

https://github.com/reggi/node-to-deno

Not sure if anyone is interested or if this has been done before.

Is there any discussion for making polyfills node -> deno and deno -> node for standard lib? That's the next chunk needed to make some node modules work with deno.

Is there any discussion about something like a mixture of what I built and https://unpkg.com? A whole CDN registry that has npm modules ready to use with deno?

[kevinkassimo]

Nice!

I don't think there are discussions about polyfills yet, but I did create some org aiming to host ported packages from node to deno. Polyfills would definitely be an interesting topic to work on and we could probably create a repo specially for that.

About registry, there is the deno.land/x/ for redirection of scripts hosted elsewhere, but they are specifically for modules written solely for deno and is similar to golang.org/x/. Maybe an alternative registry could be created?

[reggi]

@kevinkassimo Is there any chat (slack, discord, gitter) service we're using where I can bounce ideas. I create a new set of proposals and thoughts and I'd like some feedback. Seems like this issue should be closed but I'd like to talk about polyfills, npm -> deno, and deno -> npm.

https://github.com/reggi/deno-thoughts

[bartlomieju]

@reggi we hang around on gitter: https://gitter.im/denolife/Lobby

[jgierer12]

I discovered https://jspm.io/ which is basically unpkg but modifies the code to be usable with browser-native ES modules. It's not yet compatible with deno because it uses extensionless URLs but that could be pretty easily fixed. See jspm/project#44.

[reggi]

Idea for NPM modules to work with DENO (optimally)

I'm trying to find a way to bundle a NPM module but preserve it's integrity, not just one webpack bundle for all the deps within, possibly duplicating the dep tree.

Essentially, I've been trying to figure out to have my cake and eat it too.

What I want is to be able to take a node_modules folder, and export a copy for a deno registry server.

For instance, I install graphql with npm. I can wire up all files and node module resolvers to (package main, etc) so that the module would convert all require calls to local file system calls. but in a way where require('graphql-dep/x.js') works etc.

The thing that is hard to figure out is a module graphql has a version lets say 1.0.0 but the dep-tree can always be different. If I install today, tomorrow it can be different.

I created a webpack bundler to export bundles of every individual file. SO it's not one massive bundle of the package I want, the idea being that if two packages use the same module, It's smaller file sizes used.

With this method, the issue being if I snapshot graphql today with version 1.0.0 and hard link all the dependencies, tomorrow the snapshot can be different because if graphql has a dep that has been updated it will be different in the chain.

What I want is to be able to target:

• https://ts.reggi.com/npm/graphql/1.0.0.js?8f60c8102d29fcd525162d02eed4566b
• https://ts.reggi.com/npm/graphql/1.0.0.js?312ccdad4f70cf1fb05926ca7d73d028

Where the hash after the version is [ a hash, of a JSON file, (sorted keys and minified) of all the absolute version dependencies in the entire tree ].

The idea here being is that https://ts.reggi.com/npm/graphql/1.0.0.js should be atomic, there shouldn't be hundreds of versions of this on my server, because we want to cache it and not have to download a whole other copy that points to other dependencies.

I've developed a way to wrap every JS file and "uptranspile" (js to ts) it to use import statements using webpack and work out a way to preserve node module resolve, and entry points.

What I'm thinking about is some dynamic way to pull in this hash from the server

import DenoDependency as 'https://ts.reggi.com/DenoDependency/1.0.0.json'
import * from depencencies as 'https://ts.reggi.com/lodash/tree/8f60c8102d29fcd525162d02eed4566b.json'
export default import(DenoDependency(depencencies)).then({
    lodash, graphql, etc
} => {

})

But I can't export from within this import, and import can't be a template string/variable directly.

This is where top-level async await can be utilized because nobody is going to want to await lodash or graphqlthat doesn't make sense.

Been thinking non-stop about deno->npm and npm->deno would love to talk with someone else who shares these thoughts.

Idea For Deno authored files to work with NPM

For every typescript file on a server, we have a .git file, we use a webpack bundler to convert ts -> node, dynamically create a package.json from webpack with the dependencies of the file assuming that all local dependencies will also have a .git file, and Bam!

• https://ts.reggi.com/meow_mix.ts
• https://ts.reggi.com/meow_mix.git

npm install https://ts.reggi.com/meow_mix.git

import meowMix from https://ts.reggi.com/meow_mix.ts

Reality

Of course, this doesn't help if there are native node modules, or deno_std modules because that code will not be compatible until polyfills are made.

[reggi]

Ok, I just had an interesting thought. My main issue is that I'm creating a version of [email protected] and it could use [email protected] one day and [email protected] another if there's a * in package.json. [email protected] alone is not immutable, from the semver / npm spec. But what if we flip the resolve. Rather then get the latest version to satisfy a dependency, what if we just get the first? Then [email protected] is immutable and would always resolve lodash to say 1.0.0 if it was the first version. In reality, this isn't the way that npm works, so it could break a lot when people were lazy and didn't specify dependency restraints correctly.

[Macil]

@reggi Go 1.11's new module system works like that. (Also, if you have two dependencies that both depend on a third dependency with the same major version number, like A depends on C 1.1.0 and B depends on C 1.2.0, then the only C 1.x version that will be used will be 1.2.0.)

[markstos]

@reggi Resolving to the oldest dependency version is still prone to inconsistencies as well, as the oldest version of a module might disappear over time, so that someone asking for the oldest version in the future might get a result. Modules might be removed for serious security issues, legal issues, or major bugs.

The whole goal of minor version bumps is to add compatible features and bug fixes, so it generally seems safer to resolve towards newer versions. I think yarn.lock and package-lock.json do a good job keeping the resolution stable while also considering newer versions.

But to address the original issue about NPM compatibility, I'll ask a naive newercomer question: Is it possible to add a top-level require() keyboard to Deno, allowing NPM modules to be loaded directly?

[biels]

I think deno should do whatever it takes to support the npm registry and all its packages out of the box. I think it's going to be hard to gain traction and make the project successful otherwise.

[markstos]

Agreed. Without a clear migration path, This could drag on forever like Perl 5 vs Perl 6 or Python 2 vs Python 3.

The io.js fork was successful and forced some changes in the Node.js management in large part because it was experiencing a quick adoption rate. If the fork had not been so compatible that story could have turned out differently.

[mckenziewagner]

@biels I mostly agree, but at the same time maybe adopting npm packages isn't the best idea for the overall scope of deno? Maybe starting fresh isn't the worse idea. It would be cool if their was some sort of portability between the two in the meantime to get steam rolling.

I am sure @ry has addressed this somewhere else or has a better explanation than I.

[jgierer12]

With GitHub packages just begging to be the 'next big thing' in the JavaScript ecosystem, I think it becomes even more important to gain at least some level of compatibility with the system.

[askbeka]

Import maps and unpkg, IMHO, seems like a shim for npm.

…

On Wed, 15 May 2019, 17:09 Jonas Gierer, ***@***.***> wrote: With GitHub packages just begging to be the 'next big thing' in the JavaScript ecosystem, I think it becomes even more important to gain at least some level of compatibility with the system. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1397?email_source=notifications&email_token=AAW4RGVRC67Y4NNQ6E4DKODPVQRRBA5CNFSM4GL5G5WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVO7AXI#issuecomment-492695645>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAW4RGXOYR5T2RQX27X6XT3PVQRRBANCNFSM4GL5G5WA> .

[kitsonk]

Deno clearly doesn't want to inherit the existing Node.js ecosystem and be encumbered with it. There are some things that can be done to make things easier, but Deno is Deno. If people want a better Node.js then they should use Node.js v12. Deno has different goals and objectives as Node.js and adoption, at this stage, is not a key goal.

As it has been mentioned import-maps is the almost official way of package mapping on the web, therefore is being adopted by Deno.

[zhaoyao91]

Hi, importmap is supported by deno, but how can we use it to leverage the existing packages?

[dheeraj-br]

@kitsonk disregarding the largest wealth of open source projects and calling it a hindrance is not at all wise, 10 years of active contribution that makes these collection the life blood of node and you choose to leave it all on the table and not make it a top priority, this interoperability would greatly benefit developers from the get go of v1 deno, it would do loads of good for deno the product of the next decade. im sure if not the core team the community would make this possible very shortly

[markstos]

@dheeraj-br When Node.js was started, it disregarded the largest repo of open source projects-- Perl's 20,000 CPAN modules and made no attempt to be compatible with it.

Today Perl has 40,000 CPAN modules and Node.js has 1,135,000 modules despite having to start a module repository from scratch.

[biels]

But what is the upside of starting the module ecosystem from scratch? Is it having all codebases using typescript? Would the compatibility prevent deno from providing some features or optimizations?

…

On Thu, Nov 21, 2019 at 4:09 PM Mark Stosberg ***@***.***> wrote: @dheeraj-br <https://github.com/dheeraj-br> When Node.js was started, it disregarded the largest repo of open source projects-- Perl's 20,000 CPAN modules and made no attempt to be compatible with it. Today Perl has 40,000 CPAN modules and Node.js has 1,135,000 modules despite having to start a module repository from scratch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1397?email_source=notifications&email_token=ABHLQMDFBOH4PXA6OG2ICDLQU2QDJA5CNFSM4GL5G5WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE2RLOQ#issuecomment-557127098>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHLQMCONAUK4GD3XWF7PNTQU2QDJANCNFSM4GL5G5WA> .

-- Biel Simon Rojas [email protected] <[email protected]>

[markstos]

I don't know what the right decision is for deno here, but I have helped maintain open source software for about two decades and have seen what happens when projects are too aggressive about maintaining backwards compatibility. This "backcompat" adds complexity for the developers and the users as both the old and new ways of doing things are supported.

Eventually another project comes along that drops all the old baggage. It is clearly simpler and more pleasant to use. New users flock to the new choice because they don't care about backwards compatibility. More time passes and the new users outnumber the old users. The backward-compatibility that was once seen as a feature is now seen as legacy technical debt.

Apache used to rule as the dominant web server, adding and more and more features, plugins, and an "MPM" layer to support different ways to serve web pages.

Nginx learned from the features and mistakes of Apache, completely dropping backwards compatibility with Apache config files, creating a new simpler way to serve web pages. Apache tried to respond to by adding yet more complexity, while maintaining backwards compatibility. Nginx's market share took off like a rocket at Apache's expense.

The question for Deno is whether it can disrupt Node.js like that, or whether it's strives to do well in a niche and not compete directly. If Deno were to be highly compatible, that it might be seen as only incrementally better. If so, then why switch?

[kevinkassimo]

As a small update to the situation, std/node now provides a way to create a require_ function for loading CJS modules (extensionless, index.js, modules like fs, and likely npm packages). The remaining problem is adding more polyfills for Node.js APIs.

Since it is part of the standard library, it is independent from the Deno binary. The general idea is to promote use of ES Module and Deno APIs, while people still can import from Node.js modules if no alternatives are available.

[Ragav-d]

wouldn't harm to have the compatibility since its a case of adding more polyfils and would not affect the Deno architecture in any way, in fact it would be detrimental to the switching over from node to deno only thing that would keep me stuck on nodejs is the wealth of packages. i wouldn't switch to deno even with its javascript(for prototyping) and typescript (for scaling), wasm, its speed, es modules, the developer user experience etc. i can't switch because im chained to the node ecosystem. The way you suggest would most definitely restrict deno to a niche market in the shadow of well established node, im staying with node and keeping my 1,135,000+ packages.

[David-Else]

@kevinkassimo Hi, I am not sure if I have misunderstood, I am not very familiar with node and coming straight to deno. I have a common JS module module-name.js that has exports.default = {} in my project root, and:

import { createRequire } from 'https://deno.land/std/node/module.ts';
const require_ = createRequire(import.meta.url);
const cjsModule = require_('./module-name.js');

The result is:

error: Uncaught Error: Cannot find module './module-name.js'

It does not matter if I add the '.js' or not. Is this a bug, or am I doing something wrong? Thanks.

[kevinkassimo]

@David-Else can you print out your import.meta.url and confirm if ./module-name.js relative to it points to the file you want to import?

[David-Else]

@kevinkassimo when I do console.log(import.meta.url) it prints:

file:///home/xxx/sites/TypeScript/extract-rules/calculate.ts

This is the name of the file that is running the console.log and is in the same directory as the file I am trying to import. Here are the files I am running right now in case you find a bug:

import data from './output.json';
import { createRequire } from 'https://deno.land/std/node/module.ts';
const require_ = createRequire(import.meta.url);
console.log(import.meta.url);
const cjsModule = require_('./eslint-recommended.js');

./eslint-recommended.js

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
/**
 * This is a compatibility ruleset that disables rules from eslint:recommended
 * which are already handled by TypeScript.
 */
exports.default = {
    overrides: [
        {
            files: ['*.ts', '*.tsx'],
            rules: {
                //Checked by Typescript - ts(2378)
                'getter-return': 'off',
                // Checked by Typescript - ts(2300)
                'no-dupe-args': 'off',
                // Checked by Typescript - ts(1117)
                'no-dupe-keys': 'off',
                // Checked by Typescript - ts(7027)
                'no-unreachable': 'off',
                // Checked by Typescript - ts(2367)
                'valid-typeof': 'off',
                // Checked by Typescript - ts(2588)
                'no-const-assign': 'off',
                // Checked by Typescript - ts(2588)
                'no-new-symbol': 'off',
                // Checked by Typescript - ts(2376)
                'no-this-before-super': 'off',
                // This is checked by Typescript using the option `strictNullChecks`.
                'no-undef': 'off',
                // This is already checked by Typescript.
                'no-dupe-class-members': 'off',
                // This is already checked by Typescript.
                'no-redeclare': 'off',
            },
        },
    ],
};
//# sourceMappingURL=eslint-recommended.js.map

$ deno --allow-write calculate.ts 
Compile file:///home/xxx/sites/TypeScript/extract-rules/calculate.ts
file:///home/xxx/sites/TypeScript/extract-rules/calculate.ts
error: Uncaught Error: Cannot find module './eslint-recommended.js'
Require stack:
- /home/xxx/sites/TypeScript/extract-rules/calculate.ts
► module.ts:251:19

251       const err = new Error(message);
                      ^

    at _resolveFilename (module.ts:251:19)
    at _load (module.ts:357:29)
    at require (module.ts:112:21)
    at require (module.ts:1096:16)
    at calculate.ts:5:19
$ 

[kevinkassimo]

@David-Else Figured it out. This is because our CJS loader will try to stat the directory to get information, therefore it requires --allow-read permission.

Try deno --allow-read --allow-write calculate.ts

I'll see if I could make the loader throw an error that is more relatable with --allow-read

[David-Else]

@kevinkassimo Thanks, it worked!

I hope someone can add an appropriate error message for this situation, I think it is important. Deno often suggests the --alllow-xxx options, but it seems to have been overlooked in this case.

[aadamsx]

How am I to find drivers to "talk" to Azure Cosmos DB or MongoDB or MySQL or MSSQL? If there's no database story with Deno, what can we realistically do server side? I need these drivers for databases.

EDIT:
I don't mean to be down on Deno, I'm actually very excited about the prospects. So far I've built a simple web server that calls out to an external API. I've packaged this inside a docker container and deployed to Azure Container Service.

My next step was to create an auth service agains Azure Cosmos DB when I ran into the fact I don't see any database libs out there really. I see this one but it does me no good: https://deno.land/x/mysql/

I guess it's still early days and I'll have to wait to use Deno for more than toying around or write something myself.

[hayd]

There are mysql, dynamodb, postgres libraries already (I've tried the last two and they worked for me). It looks like there is a javascript (browser) sdk for azure which has a cosmodb directory... https://github.com/Azure/azure-sdk-for-js it's possible it will "just work" 😬

[aadamsx]

@hayd so you think just copy /src from here https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/cosmosdb/cosmos into my project and follow the examples it it might "just work"?

[aadamsx]

@hayd I tried to copy the cosmos directory over and work through the issues. Deno errors have a lot to be desired.

For example: > deno --allow-net main.ts

» deno --allow-net main.ts                                                                                                                                                                                                                                                                       ~/P/deno.proto.01 1
Compile file:///Users/aadams/Projects/deno.proto.01/main.ts
error: Uncaught Other: Is a directory (os error 21)
► $deno$/dispatch_json.ts:40:11
    at DenoError ($deno$/errors.ts:20:5)
    at unwrapResponse ($deno$/dispatch_json.ts:40:11)
    at sendAsync ($deno$/dispatch_json.ts:91:10)

NOTE: in the above project directory, main.ts is in root and a /cosmos directory is there to contain the files from the cosmos sdk.

And this is such a big library, it will take a lot of time to go through the lib... uhh

Second thought, might be easier for me just to wrap up MySQL or PostgreSQL into a container and wire up my Web App to Db container inside azure instead of using integrated Cosmos...