Caching on the server?

[glen-84]

Is it possible to set up caching on the server, so that requests for public packages are stored after being retrieved from the "source" registry?

For example, given the following "network diagram":

[Client 1] \
             [CNPM] – [SOURCE]
[Client 2] /

If client 1 requests a public package, is it possible to have that package stored on the CNPM server so that when client 2 installs the same package, it is loaded from there instead of being loaded from the source?

This should make the installation process a lot faster when the client has no cache of its own (f.e. a new Docker container).

Thanks. =)

[alsotang]

[image: 内嵌图片 1]

when use exist but not none, then cnpmjs would cache installed package,
and sync from source intervally.

2016-01-14 1:14 GMT+08:00 Glen [email protected]:

Is it possible to set up caching on the server, so that requests for
public packages are stored after being retrieved from the "source" registry?

For example, given the following "network diagram":

[Client 1]
[CNPM] – [SOURCE]
[Client 2] /

If client 1 requests a public package, is it possible to have that package
stored on the CNPM server so that when client 2 installs the same package,
it is loaded from there instead of being loaded from the source?

This should make the installation process a lot faster when the client has
no cache of its own (f.e. a new Docker container).

Thanks. =)

—
Reply to this email directly or view it on GitHub
#793.

[glen-84]

Great! I have some feedback/questions, if that's okay?

1. Looking at the log file, it did a sync of a package named constructor – is this normal/correct? (I don't think that I installed this)
2. It logs send email with type: log, subject: Sync Finished, html: Sync packages from official registry finished. and send email with type: undefined, subject: undefined, html: undefined, but I don't have any e-mail notifications configured (also note the undefined) – should this still be logged?
3. Is there a minimum syncInterval? I set it to 2m (for testing), but it seems to only sync every 5 minutes.
4. Is it possible to do manual (on install) syncing only?
5. We have much better pings to NPM than to taobao.org, do you think that it's okay for us to use NPM as a source, if we're only using exists sync with not a lot of packages? I assume that you only need to contact NPM if you're gonna do full & frequent syncs?
6. Is there any automated log file clean-up, or should we manage that ourselves?

Sorry about all the questions! 😟

Thanks. =)

[alsotang]

1. Perhaps a pkg you installed depend on constructor
2. There are some email-send logic in code, whether you config smtp or not.
   It would log info anyway.
3. Nope. Perhaps in 2m, there is no pkg update, so cnpmjs dont sync
   anything. Just check, but no sync.
4. You can inspect the code to see that if in none mode(means just proxy
   to source but dont store anythig), how cnpm sync behave. But I dont
   think it's a good strategy. I use cnpmjs in http://qq.com/, we are top3
   internet company in China. And after running for 3 month in exist mode.
   cnpm only use less than 50G in disk.
5. Not sure. Use taobao source is much faster than official source in
   China, so we use it. You can use official source up to your network
   environment. Maybe official is faster.

But there is a risk, official npm seem not hope a 3rd source to sync from
it. So maybe they would block you if they detect you are syncing?? Not
sure, at least npm.taobao.org would not.
6. Do it yourself.

You are welcome!
Forgive my poor English.

2016-01-14 17:29 GMT+08:00 Glen [email protected]:

Great! I have some feedback/questions, if that's okay?

1. Looking at the log file, it did a sync of a package named
   constructor – is this normal/correct? (I don't think that I installed
   this)
2. It logs send email with type: log, subject: Sync Finished, html:
   Sync packages from official registry finished. and send email with
   type: undefined, subject: undefined, html: undefined, but I don't have
   any e-mail notifications configured (also note the undefined) – should
   this still be logged?
3. Is there a minimum syncInterval? I set it to 2m (for testing), but
   it seems to only sync every 5 minutes.
4. Is it possible to do manual (on install) syncing only?
5. We have much better pings to NPM than to taobao.org, do you think
   that it's okay for us to use NPM as a source, if we're only using
   exists sync with not a lot of packages? I assume that you only need to
   contact NPM if you're gonna do full & frequent syncs?
6. Is there any automated log file clean-up, or should we manage that
   ourselves?

Sorry about all the questions! [image: 😟]

Thanks. =)

—
Reply to this email directly or view it on GitHub
#793 (comment).

[glen-84]

1. Hmm. I will try to find out later, it's not a big problem. =)

2. I see. Maybe I will look at the code and consider making a change to stop the logging, but it's also not urgent.

3. Hmm. I asked because I see logging every 5m, like this:

   [2016-01-14 10:27:01.608][2460] send email with type: undefined, subject: undefined, html: undefined
   
   {"successes":[],"fails":[]}
   
   [2016-01-14 10:32:01.716][2460] send email with type: undefined, subject: undefined, html: undefined
   
   {"successes":[],"fails":[]}
   
   [2016-01-14 10:37:01.603][2460] send email with type: undefined, subject: undefined, html: undefined
   

   I was just curious, but it's not important because we are likely to use a large value.

4. We do want it to store, but the synching seems a bit unnecessary, since the packages we use probably won't change that much. From what I understood, if the package and/or version is not available in the CNPM registry, it will look at the source, right? So the first client to request a new version would cause the package to be synched/cached, and subsequent clients would benefit from that cache. Anyway, if this is not possible we will maybe just set a high value, like 1440m (24 hours). I hope that my understanding is correct, and that I am not missing other benefits of synching. =)

5. For us the ping can change from ~18ms (npm) to ~234ms (taobao), so there is quite a difference. However, you make a very good point about possibly being blocked, so we might just stay with taobao.

6. Okay. =)

Your help is greatly appreciated, and there is no need to apologize. =)

[alsotang]

1. After you set the interval, did you restart the server?
2. em...You dont benefit from sync real time, then set a large value is
   fine.

2016-01-14 20:44 GMT+08:00 Glen [email protected]:

1. Hmm. I will try to find out later, it's not a big problem. =)

2. I see. Maybe I will look at the code and consider making a change
   to stop the logging, but it's also not urgent.
   3.

   Hmm. I asked because I see logging every 5m, like this:

   [2016-01-14 10:27:01.608][2460] send email with type: undefined, subject: undefined, html: undefined

   {"successes":[],"fails":[]}

   [2016-01-14 10:32:01.716][2460] send email with type: undefined, subject: undefined, html: undefined

   {"successes":[],"fails":[]}

   [2016-01-14 10:37:01.603][2460] send email with type: undefined, subject: undefined, html: undefined

   I was just curious, but it's not important because we are likely to
   use a large value.

3. We do want it to store, but the syncing seems a bit unnecessary,
   since the packages we use probably won't change that much. From what I
   understood, if the package and/or version is not available in the CNPM
   registry, it will look at the source, right? So the first client to request
   a new version would cause the package to be synched/cached, and subsequent
   clients would benefit from that cache. Anyway, if this is not possible we
   will maybe just set a high value, like 1440m (24 hours). I hope that my
   understanding is correct, and that I am not missing other benefits of
   synching. =)

4. For us the ping can change from ~18ms (npm) to ~234ms (taobao), so
   there is quite a difference. However, you make a very good point about
   possibly being blocked, so we might just stay with taobao.

5. Okay. =)

Your help is greatly appreciated, and there is no need to apologize. =)

—
Reply to this email directly or view it on GitHub
#793 (comment).

[glen-84]

3. Yes, and I just tested it again.
4. Okay, cool.

[glen-84]

Looking at the log file, it did a sync of a package named constructor – is this normal/correct? (I don't think that I installed this)

This happened again with a clean installation, before installing any packages.

[alsotang]

perhaps some pkg your installed depend on constructor

2016-01-29 20:44 GMT+08:00 Glen [email protected]:

Looking at the log file, it did a sync of a package named constructor – is
this normal/correct? (I don't think that I installed this)

This happened again with a clean installation, before installing any
packages.

—
Reply to this email directly or view it on GitHub
#793 (comment).

[glen-84]

I said that I didn't install any packages. I installed CNPM, started the server, and then it started to sync this constructor package.

[taoqf]

when use exist but not none, then cnpmjs would cache installed package,
and sync from source intervally.

where is the image [image: 内嵌图片 1]?
i am sorry i can not understand after reading the topic Deploy-a-private-npm-registry-in-5-minutes