feat: allow generator to pull from private npm repo #1061

AayushSaini101 · 2023-11-07T12:35:22Z

Resolves #538
Resolves #1100

Description:

Allow the Generator to pull the template from the private repo. [ Check docs/installing-privateTemplate.md]
Adding new testing to verify the Generator pulls template from the private Repository.
Integrate project tests now run in isolation, each separately
Now tests do not make any changes to the file system of the host (entrypoint script) - they run longer but yeah, there is no other way to do it
The code of html-template no longer needs to be stored inside the test project
Some await/async bugs and removal of redundant promises
update arborist to as latest as possible without breaking node 12 support

github-actions

Welcome to AsyncAPI. Thanks a lot for creating your first pull request. Please check out our contributors guide useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

AayushSaini101 · 2023-11-17T10:04:59Z

@derberg Hello Sir. I updated the PR according to your instructions. Can you please review :)

derberg · 2023-11-22T08:57:05Z

sorry, few days delay from my side, not feeling well

@aeworxet fyi ☝🏼

AayushSaini101 · 2023-11-22T15:29:53Z

sorry, few days delay from my side, not feeling well

@aeworxet fyi ☝🏼

@magicmatatjahu @jonaslagoni Can you please review this

derberg · 2023-12-12T09:06:18Z

Hey there, sorry a lot for delay.

There are some improvements we need here:

please solve merge conflicts
sonarcloud shows 3 code smells - please address them
in documentation it is not enough to just explain how to configure to use another NPM registry, we should have an example there on how to do it. I think we need a new, separate markdown file that explains things
there is a huge amount of code changes that do not seem like related to the goal of this PR which makes review super hard. I'm not sure if they are automated by some of the tools you use or what. Please make sure PR focus on the topic it addresses and do not refactor irrelevant parts
generator now uses https://github.com/smoya/multi-parser-js which contains all custom schema parsers. And you see in code https://github.com/asyncapi/generator/blob/master/lib/generator.js#L32. So why do you add all the parsers to generator.js again?

AayushSaini101 · 2023-12-13T10:22:13Z

https://github.com/smoya/multi-parser-js

Thanks for the suggestions working on them

AayushSaini101 · 2023-12-13T16:06:21Z

https://github.com/smoya/multi-parser-js

Thanks for the suggestions working on them

@derberg I have done some changes how can i test it locally i checked the documentation but cannot able to find the suitable method how to test the changes locally can you tell me some points . So that i can ensure to test before pushing

AayushSaini101 · 2023-12-14T14:32:08Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

3 New issues 0 Security Hotspots No data about Coverage 0.0% Duplication on New Code

See analysis details on SonarCloud

Working on fix it

AayushSaini101 · 2023-12-14T17:21:15Z

@derberg Done, Do we need to add documentation in generator or update in AsyncapiDoc

AayushSaini101 · 2023-12-18T16:44:37Z

Thanks working on fix

derberg

added few comments

also please see why tests are failing

last but not least please specify how did you test new functionality to make sure it works - like integration test I mean

lib/generator.js

docs/template.md

lib/generator.js

AayushSaini101 · 2023-12-19T03:43:59Z

added few comments

also please see why tests are failing

last but not least please specify how did you test new functionality to make sure it works - like integration test I mean

Fix Test cases and added new test cases:

derberg

left some comments

I'm wondering how we could test it to make sure private registry works. Setting up automated test will be huge, but maybe we could at least have a short instruction for manual testing whenever we modify this functionality 🤔

so basically:

one line command with docker to start local npm with custom auth -> https://developer.entando.com/v6.1/tutorials/ecr/how-to-create-local-npm-registry.html
one line command to publish https://github.com/asyncapi/cli/tree/master/test/fixtures/minimaltemplate to this local npm
and then one liner to start generation using private registry 🤔

you know, like developer guide

test/generator.test.js

lib/generator.js

AayushSaini101 · 2023-12-19T17:10:33Z

left some comments

I'm wondering how we could test it to make sure private registry works. Setting up automated test will be huge, but maybe we could at least have a short instruction for manual testing whenever we modify this functionality 🤔

so basically:

one line command with docker to start local npm with custom auth -> https://developer.entando.com/v6.1/tutorials/ecr/how-to-create-local-npm-registry.html

one line command to publish https://github.com/asyncapi/cli/tree/master/test/fixtures/minimaltemplate to this local npm

and then one liner to start generation using private registry 🤔

you know, like developer guide

Added small doc @derberg

aeworxet · 2023-12-19T21:36:05Z

@asyncapi/bounty_team

AayushSaini101 · 2023-12-20T12:08:10Z

left some comments
I'm wondering how we could test it to make sure private registry works. Setting up automated test will be huge, but maybe we could at least have a short instruction for manual testing whenever we modify this functionality 🤔
so basically:

one line command with docker to start local npm with custom auth -> https://developer.entando.com/v6.1/tutorials/ecr/how-to-create-local-npm-registry.html

one line command to publish https://github.com/asyncapi/cli/tree/master/test/fixtures/minimaltemplate to this local npm

and then one liner to start generation using private registry 🤔

you know, like developer guide

Added small doc @derberg

@derberg I used verdaccio to create private repository in the local.

AayushSaini101 · 2023-12-21T00:22:21Z

@derberg Can you please check before you go for holidays 🙏🏼

derberg

Sorry for not reviewing before holidays, I just had to many topics to close

lib/generator.js

derberg · 2024-01-03T13:42:16Z

docs/installing-privateTemplate.md

+
+**Step 1:** Settuping Private Repository in Local:
+
+Installing [verdaccio](https://verdaccio.org/) using below command:


I was thinking, wouldn't it be better to hide all of this and just extend https://github.com/asyncapi/generator/tree/master/test/test-project that is responsible for testing generator in isolated environment 🤔

If I'm correct you would just have to:

extend https://github.com/asyncapi/generator/blob/master/test/test-project/docker-compose.yml to handle startup of the verdaccio registory

here you have example https://verdaccio.org/docs/docker#using-docker-compose

you could add htpasswd file with username and password https://github.com/juanpicado/verdaccio-end-to-end-tests/blob/main/verdaccio/htpasswd and pass to it in docker compose command

you could also add packages that are available there by default https://github.com/juanpicado/verdaccio-end-to-end-tests/blob/main/verdaccio/verdaccio.yaml - that could be html-template that is already used in other tests (the specific version )

so with above you are getting rid of step 1-4 from this instruction

and to get rid last step:

you would just add new file, like we already have test-global.test.js, that would do what you have in step 5, so programatically trigger generator and then evaluate if generation worked like a charm

you would have to add a script in https://github.com/asyncapi/generator/blob/master/test/test-project/package.json#L5-L6 that would be fired after step 15 in https://github.com/asyncapi/generator/blob/master/test/test-project/test.sh

Thoughts? this way we get one document less to maintain, and out of the box good quality integration test that would work out of the box as we have GitHub Action workflow for it -> https://github.com/asyncapi/generator/blob/master/.github/workflows/pr-testing-with-test-project.yml

I was thinking, wouldn't it be better to hide all of this and just extend https://github.com/asyncapi/generator/tree/master/test/test-project that is responsible for testing generator in isolated environment 🤔

If I'm correct you would just have to:

extend https://github.com/asyncapi/generator/blob/master/test/test-project/docker-compose.yml to handle startup of the verdaccio registory

here you have example https://verdaccio.org/docs/docker#using-docker-compose

you could add htpasswd file with username and password https://github.com/juanpicado/verdaccio-end-to-end-tests/blob/main/verdaccio/htpasswd and pass to it in docker compose command

you could also add packages that are available there by default https://github.com/juanpicado/verdaccio-end-to-end-tests/blob/main/verdaccio/verdaccio.yaml - that could be html-template that is already used in other tests (the specific version )

so with above you are getting rid of step 1-4 from this instruction

and to get rid last step:

you would just add new file, like we already have test-global.test.js, that would do what you have in step 5, so programatically trigger generator and then evaluate if generation worked like a charm

you would have to add a script in https://github.com/asyncapi/generator/blob/master/test/test-project/package.json#L5-L6 that would be fired after step 15 in https://github.com/asyncapi/generator/blob/master/test/test-project/test.sh

Thoughts? this way we get one document less to maintain, and out of the box good quality integration test that would work out of the box as we have GitHub Action workflow for it -> https://github.com/asyncapi/generator/blob/master/.github/workflows/pr-testing-with-test-project.yml

@derberg I have tried using docker but we have to manually publish the template the repository, after providing the username and password in config, we have to still login, i have tried https://github.com/verdaccio/verdaccio/tree/master/docker-examples/v4/https-portal-example this example but also we have to manually publish the repo, Event the official video is publishing the repo manually [ We have to manually ]login https://www.youtube.com/watch?v=zRI0skF1f8I&t=234s&ab_channel=Docker

Also i cannot pass parameters in the cli for login and publishing the repository.

I guess existing the small doc is sufficient to help developers

derberg

Please push to repo what you have in regards to verdaccio so I can try to help on my side. But first I need to see what configs you already tried

lib/generator.js

AayushSaini101 · 2024-01-04T11:48:49Z

Please push to repo what you have in regards to verdaccio so I can try to help on my side. But first I need to see what configs you already tried

Sure @derberg

Extend https://github.com/asyncapi/generator/blob/master/test/test-project/docker-compose.yml to handle startup of the verdaccio registry [Done]
Startup the verdaccio and config the password i created new because mentioned one is not working. [Done]
Add package name in the conf that is html-template and the default package**[Done]**
I have to provide login and password credentials and packages are not present after building the services.

fix: refactor of int tests

AayushSaini101 · 2024-01-17T15:41:07Z

Sonar Cloud issue can be avoided because it is suggested to use optional changing that will cause lint issue

derberg · 2024-01-17T17:43:35Z

docs/using-private-template.md

+title: "Using private templates"
+weight: 180
+---
+[Generator](https://www.asyncapi.com/tools/generator) allows to fetch the template from the private repositories like verdaccio, nexus, npm etc. Let's understand how can use the Generator to fetch the templates from the private repositories.


Suggested change

[Generator](https://www.asyncapi.com/tools/generator) allows to fetch the template from the private repositories like verdaccio, nexus, npm etc. Let's understand how can use the Generator to fetch the templates from the private repositories.

Generator allows to fetch the template from the private repositories like verdaccio, nexus, npm etc.

no need to have one more sentence + link to generator - as long these are generator docs 😄

Done @derberg

derberg · 2024-01-17T17:44:07Z

docs/using-private-template.md

+
+* **token:**  Optional parameter to pass npm registry auth token that you can grab from .npmrc file
+
+## Example to pull Private template:


Suggested change

## Example to pull Private template:

## Pulling private template using library

Done @derberg

derberg · 2024-01-17T17:45:25Z

docs/using-private-template.md

+const generator = new Generator('@asyncapi/html-template', outputDir,
+      { 
+        debug: true,
+        install: true, 
+        forceWrite: true, 
+        templateParams: { 
+          singleFile: true 
+        },
+        registry: {
+          url: 'http://verdaccio:4873',  
+          auth: 'YWRtaW46bmltZGE=' 
+            // base64 encoded username and password 
+            // represented as admin:nimda
+
+        }
+      });


Suggested change

const generator = new Generator('@asyncapi/html-template', outputDir,

{

debug: true,

install: true,

forceWrite: true,

templateParams: {

singleFile: true

},

registry: {

url: 'http://verdaccio:4873',

auth: 'YWRtaW46bmltZGE='

// base64 encoded username and password

// represented as admin:nimda

}

});

const generator = new Generator('@asyncapi/html-template', 'output',

{

debug: true,

registry: {

url: 'http://verdaccio:4873',

auth: 'YWRtaW46bmltZGE='

// base64 encoded username and password

// represented as admin:nimda

}

});

Done @derberg

derberg · 2024-01-17T17:49:25Z

docs/using-private-template.md

+        }
+      });
+```
+Let's suppose the template name as @asyncapi/html-template is present in the private repository, In order to pull the template from the private repository, we need to pass the url and auth as a parameters. 


Suggested change

Let's suppose the template name as @asyncapi/html-template is present in the private repository, In order to pull the template from the private repository, we need to pass the url and auth as a parameters.

Let's assume you host `@asyncapi/html-template` in private package registry like Verdaccio. In order to pull this template you need to provide `registry.url` option that points to the registry URL and `registry.auth` as base64 encoded value that represents username and password. Instead of username and password you can also pass `registry.token`.

Done @derberg

derberg · 2024-01-17T17:49:54Z

docs/using-private-template.md

+[Generator](https://www.asyncapi.com/tools/generator) allows to fetch the template from the private repositories like verdaccio, nexus, npm etc. Let's understand how can use the Generator to fetch the templates from the private repositories.
+
+
+## Parameters that needs to pass in Generator to pull private template:


Suggested change

## Parameters that needs to pass in Generator to pull private template:

## Private registry options

Done @derberg

derberg · 2024-01-17T17:55:38Z

docs/using-private-template.md

+* **URL:** The URL of the registry where is private template is present. 
+* **Auth:** Optional parameter to pass npm registry username and password encoded with base64, formatted like username:password value should be encoded. 
+
+**For example**: if the username and password is admin and nimda, we need to base64 encoded of admin:nimda.
+
+* **token:**  Optional parameter to pass npm registry auth token that you can grab from .npmrc file


Suggested change

* **URL:** The URL of the registry where is private template is present.

* **Auth:** Optional parameter to pass npm registry username and password encoded with base64, formatted like username:password value should be encoded.

**For example**: if the username and password is admin and nimda, we need to base64 encoded of admin:nimda.

* **token:** Optional parameter to pass npm registry auth token that you can grab from .npmrc file

* **registry.url**: The URL of the registry where is private template is present. Defaults to `registry.npmjs.org`

* **registry.auth**: Optional parameter to pass npm registry username and password encoded with base64, formatted like `username:password`. For example if the username and password is `admin` and `nimda`, you need to encode with base64 value like `admin:nimda` which results in `YWRtaW46bmltZGE=`.

* **registry.token** : Optional parameter to pass to npm registry auth token. To get the token you can first authenticate with registry using `npm login` and then grab generated token from `.npmrc` file.

Done @derberg

derberg

Well done 👏🏼
That was a big task, something that I definitely did not expect will be so advanced

but as a result we have lots of bugs solved, much better testing setup and very good reliable integration tests

derberg · 2024-01-17T18:07:15Z

@Florence-Njeri this PR has some docs, please have a look and approve 🙏🏼

Florence-Njeri

Editorial review

docs/using-private-template.md

sonarqubecloud · 2024-01-18T10:00:09Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

derberg · 2024-01-18T11:06:52Z

/rtm

asyncapi-bot · 2024-01-18T11:12:53Z

🎉 This PR is included in version 1.17.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

AayushSaini101 requested review from Florence-Njeri, derberg, asyncapi-bot-eve, magicmatatjahu and jonaslagoni as code owners November 7, 2023 12:35

github-actions bot reviewed Nov 7, 2023

View reviewed changes

pierrick-boule mentioned this pull request Nov 18, 2023

feat!: template pulling with npm config #1069

Merged

derberg mentioned this pull request Dec 12, 2023

feat: add support to pull templates from private npm repository #877

Closed

AayushSaini101 force-pushed the issue-538 branch from 5da6c56 to b36c18e Compare December 14, 2023 14:12

AayushSaini101 force-pushed the issue-538 branch from 678ceec to 4af7923 Compare December 14, 2023 17:19

derberg requested changes Dec 18, 2023

View reviewed changes

lib/generator.js Outdated Show resolved Hide resolved

docs/template.md Outdated Show resolved Hide resolved

lib/generator.js Outdated Show resolved Hide resolved

derberg reviewed Dec 19, 2023

View reviewed changes

test/generator.test.js Outdated Show resolved Hide resolved

test/generator.test.js Outdated Show resolved Hide resolved

lib/generator.js Outdated Show resolved Hide resolved

lib/generator.js Outdated Show resolved Hide resolved

lib/generator.js Outdated Show resolved Hide resolved

AayushSaini101 requested a review from derberg December 22, 2023 08:03

derberg requested changes Jan 3, 2024

View reviewed changes

derberg reviewed Jan 4, 2024

View reviewed changes

lib/generator.js Outdated Show resolved Hide resolved

lib/generator.js Outdated Show resolved Hide resolved

lib/generator.js Outdated Show resolved Hide resolved

lib/generator.js Show resolved Hide resolved

derberg and others added 6 commits January 17, 2024 14:08

jest and lint

9aea5f9

Merge pull request #3 from derberg/asdadsfljkadsflkjadlk

73750c5

fix: refactor of int tests

Fix testing

618f64b

Remove sonar issues

4fe6585

Fix Sonar issues

f32478e

Fix sonar issues

2d4de58

aayushRedHat added 2 commits January 17, 2024 21:59

update the doc for private template

169157e

update private doc

522f863

derberg reviewed Jan 17, 2024

View reviewed changes

update doc

111966d

derberg mentioned this pull request Jan 17, 2024

feat!: use latest arborist 6 and drop support for node 12 and 14 #925

Closed

derberg previously approved these changes Jan 17, 2024

View reviewed changes

derberg mentioned this pull request Jan 17, 2024

Improve arborist (npm installation) to have no hacks #1102

Open

Florence-Njeri requested changes Jan 18, 2024

View reviewed changes

update doc

c60dff9

AayushSaini101 dismissed derberg’s stale review via c60dff9 January 18, 2024 09:59

Florence-Njeri approved these changes Jan 18, 2024

View reviewed changes

derberg approved these changes Jan 18, 2024

View reviewed changes

asyncapi-bot added the ready-to-merge label Jan 18, 2024

asyncapi-bot merged commit b5ff633 into asyncapi:master Jan 18, 2024
14 checks passed

asyncapi-bot added the released label Jan 18, 2024

AayushSaini101 mentioned this pull request Mar 24, 2024

Convert OpenAPI 3.0 to AsyncAPI 3.0 asyncapi/converter-js#233

Closed

AayushSaini101 mentioned this pull request Jun 1, 2024

Improve design of CLI commands structure asyncapi/cli#551

Closed

AayushSaini101 mentioned this pull request Jun 11, 2024

Create create-template in AsyncAPI organization asyncapi/cli#337

Closed

asyncapi-bot added the bounty AsyncAPI Bounty program related label label Aug 21, 2024


		Step 1: Settuping Private Repository in Local:

		Installing [verdaccio](https://verdaccio.org/) using below command:

	[Generator](https://www.asyncapi.com/tools/generator) allows to fetch the template from the private repositories like verdaccio, nexus, npm etc. Let's understand how can use the Generator to fetch the templates from the private repositories.
	Generator allows to fetch the template from the private repositories like verdaccio, nexus, npm etc.


		* token: Optional parameter to pass npm registry auth token that you can grab from .npmrc file

		## Example to pull Private template:

	## Example to pull Private template:
	## Pulling private template using library

	Let's suppose the template name as @asyncapi/html-template is present in the private repository, In order to pull the template from the private repository, we need to pass the url and auth as a parameters.
	Let's assume you host `@asyncapi/html-template` in private package registry like Verdaccio. In order to pull this template you need to provide `registry.url` option that points to the registry URL and `registry.auth` as base64 encoded value that represents username and password. Instead of username and password you can also pass `registry.token`.

		[Generator](https://www.asyncapi.com/tools/generator) allows to fetch the template from the private repositories like verdaccio, nexus, npm etc. Let's understand how can use the Generator to fetch the templates from the private repositories.


		## Parameters that needs to pass in Generator to pull private template:

	## Parameters that needs to pass in Generator to pull private template:
	## Private registry options

feat: allow generator to pull from private npm repo #1061

feat: allow generator to pull from private npm repo #1061

Conversation

AayushSaini101 commented Nov 7, 2023 • edited by derberg Loading

github-actions bot left a comment

Choose a reason for hiding this comment

AayushSaini101 commented Nov 17, 2023

derberg commented Nov 22, 2023

AayushSaini101 commented Nov 22, 2023

derberg commented Dec 12, 2023

AayushSaini101 commented Dec 13, 2023

AayushSaini101 commented Dec 13, 2023

AayushSaini101 commented Dec 14, 2023

Quality Gate passed

AayushSaini101 commented Dec 14, 2023

AayushSaini101 commented Dec 18, 2023

derberg left a comment

Choose a reason for hiding this comment

AayushSaini101 commented Dec 19, 2023

derberg left a comment

Choose a reason for hiding this comment

AayushSaini101 commented Dec 19, 2023

aeworxet commented Dec 19, 2023

AayushSaini101 commented Dec 20, 2023

AayushSaini101 commented Dec 21, 2023 • edited Loading

derberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thoughts? this way we get one document less to maintain, and out of the box good quality integration test that would work out of the box as we have GitHub Action workflow for it -> https://github.com/asyncapi/generator/blob/master/.github/workflows/pr-testing-with-test-project.yml

Choose a reason for hiding this comment

Thoughts? this way we get one document less to maintain, and out of the box good quality integration test that would work out of the box as we have GitHub Action workflow for it -> https://github.com/asyncapi/generator/blob/master/.github/workflows/pr-testing-with-test-project.yml

derberg left a comment

Choose a reason for hiding this comment

AayushSaini101 commented Jan 4, 2024

AayushSaini101 commented Jan 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derberg left a comment

Choose a reason for hiding this comment

derberg commented Jan 17, 2024

Florence-Njeri left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 18, 2024

Quality Gate passed

derberg commented Jan 18, 2024

asyncapi-bot commented Jan 18, 2024

AayushSaini101 commented Nov 7, 2023 •

edited by derberg

Loading

AayushSaini101 commented Dec 21, 2023 •

edited

Loading