jsipfs add -Qr <directory-containing_1206 _files> doesn't work

[hhfeng]

Version: 0.35.0
Platform: Linux ip-XXX-aws #42-Ubuntu SMP Fri May 17 13:47:10 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Type: Bug

Severity: High

Description: I have a folder containing 1200+ files and takes 4.1G disk space. When I did

jsipfs add -Qr <folder_name>

It returns a Hash QmedigS67VJr71NZs7U1DBhrWXmBFwEHRVTWHQ6f624PMm. But
jsipfs ls /ipfs/QmedigS67VJr71NZs7U1DBhrWXmBFwEHRVTWHQ6f624PMm
doesn't return anything (I expect it to return a list of 1200+ files).
Local gateway also doesn't work.

Steps to reproduce the error:

See above

[alanshaw]

@hhfeng are you able to add directories with fewer files or are you seeing this issue for all directories you try to add?

The latest JS IPFS version is 0.37.0-rc.0 - would you mind installing it to verify if the problem still exists on this new version?

[hhfeng]

I can add directories with 100 files with no problem. Only for this 1200+ file folder. I will give it a try with 0.37.0-rc.0. Should the following installation work? npm install [email protected]

…

On Jul 30, 2019, at 09:38, Alan Shaw ***@***.***> wrote: @hhfeng are you able to add directories with fewer files or are you seeing this issue for all directories you try to add? The latest JS IPFS version is 0.37.0-rc.0 - would you mind installing it to verify if the problem still exists on this new version? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[alanshaw]

No, 0.37 is not out yet, but please try the RC!

[hhfeng]

How do I install the RC?

…

On Jul 30, 2019, at 12:03, Alan Shaw ***@***.***> wrote: No, 0.37 is not out yet, but please try the RC! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[hhfeng]

This is only js-ipfs issue. I tested using go-ipfs, and it worked fine.

[alanshaw]

@hhfeng npm i [email protected]

@achingbrain any chance you can look into this one?

[achingbrain]

I can't replicate this. I've tried adding:

• A folder with 1206 empty files
• A folder with 1206 small files with contents from /dev/random
• A folder with 1206 files with contents from /dev/random totalling about 5GB

All work as expected, producing the same hashes as go-ipfs and I can list the contents of them.

@hhfeng can you tell us more about your setup? Are you running a daemon? Is there anything out of the ordinary about the files you are adding - contents, names, etc?

I can't get the contents of QmedigS67VJr71NZs7U1DBhrWXmBFwEHRVTWHQ6f624PMm, I'm guessing there's nothing serving it?

[hhfeng]

Thanks @alex, it seems it doesn't matter whether running daemon or not. jsipfs 0.36.4 still fails as 0.35. I haven't tried 0.37 RC.The foler contains a bunch HLS movie files likes the following:

…

-rw-r--r-- 1 root root 3428932 Jul 29 12:41 master997.ts -rw-r--r-- 1 root root 7040036 Jul 29 12:41 master998.ts -rw-r--r-- 1 root root 1154696 Jul 29 12:41 master999.ts I worked around the problem by switching to go-ipfs for now.

On Mon, Aug 5, 2019 at 10:16 AM Alex Potsides ***@***.***> wrote: I can't replicate this. I've tried adding: - A folder with 1206 empty files - A folder with 1206 small files with contents from /dev/random - A folder with 1206 files with contents from /dev/random totalling about 5GB All work as expected, producing the same hashes as go-ipfs and I can list the contents of them. @hhfeng <https://github.com/hhfeng> can you tell us more about your setup? Are you running a daemon? Is there anything out of the ordinary about the files you are adding - contents, names, etc? I can't get the contents of QmedigS67VJr71NZs7U1DBhrWXmBFwEHRVTWHQ6f624PMm, I'm guessing there's nothing serving it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2310?email_source=notifications&email_token=AESOVZKC63BGFW47R2UZDPLQDAY4PA5CNFSM4IHX4G32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3R6Q2Y#issuecomment-518252651>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESOVZJ7K2DRFNJESKHC7ZTQDAY4PANCNFSM4IHX4G3Q> .

[alanshaw]

@hhfeng JS IPFS 0.37 is out now so installing should be a simple as npm i [email protected]. I'd really like to get to the bottom of this so any more information you can provide would be really beneficial. Thank you!

[hhfeng]

I just tried 0.37, and it couldn't even finish. The process was killed (the only display was "Killed") jsipfs add -Qr hls-lw_june6_CC_v2 Killed

…

On Tue, Aug 6, 2019 at 12:42 PM Alan Shaw ***@***.***> wrote: @hhfeng <https://github.com/hhfeng> JS IPFS 0.37 is out now so installing should be a simple as npm i ***@***.*** I'd really like to get to the bottom of this so any more information you can provide would be really beneficial. Thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2310?email_source=notifications&email_token=AESOVZMHX5RF7OAHXIIOBETQDGSXHA5CNFSM4IHX4G32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3VX2FI#issuecomment-518749461>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESOVZOSIO5T6ZRYMNU3MOTQDGSXHANCNFSM4IHX4G3Q> .

[achingbrain]

Linux sometimes displays Killed when the kernel has killed a process that's consuming too much memory - what kind of machine are you running this on?

[hhfeng]

It's a EC2 instance with 2 CPU core and 4G RAM Linux machine. It could be the memory issue as I saw the available memory dropped to ~20M when the jsipfs was running. But why it consumes so much memory as each file is at most 10M? Also, go-ipfs has no problem.

…

On Wed, Aug 7, 2019 at 6:31 AM Alex Potsides ***@***.***> wrote: Linux sometimes displays Killed when the kernel has killed a process that's consuming too much memory - what kind of machine are you running this on? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2310?email_source=notifications&email_token=AESOVZJ4P3WFWVUMH75NNATQDKQAVA5CNFSM4IHX4G32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3X6XPI#issuecomment-519039933>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESOVZKBOP3QU6Q4DINLVV3QDKQAVANCNFSM4IHX4G3Q> .