Integrate SingleFIle into NodeJS

[daniel31x13]

Hello,
I'm currently working on an app and wanted to know how can I run SingleFile from NodeJS.

Cheers,
Daniel

[gildas-lormeau]

Hi Daniel,

I'm glad to hear that you would like to integrate SingleFile in your app :). There are 2 main ways to run SingleFile from NodeJS.
You can use JSDOM, see https://github.com/gildas-lormeau/single-file-cli/blob/master/back-ends/jsdom.js, and run SingleFile directly in Node. In this case, the JavaScript won't be executed and the resulting page might be altered. For example, lazy-loaded images might be missing.
Otherwise, you can run SingleFile in a headless browser via puppeteer for example, see other files here: https://github.com/gildas-lormeau/single-file-cli/tree/master/back-ends. I do not recommend to use Selenium though. With puppeteer, you should get more or less the same result as the extension.
An overview of the integration of SingleFile is explained here: https://github.com/gildas-lormeau/SingleFile/wiki/How-to-integrate-SingleFile-library-code-in-%22custom%22-environments%3F

Please let me know if you need more details.

[gildas-lormeau]

I'm closing and moving this issue in the "Discussions" tab.

[bryanph]

Having tried both JSDOM and puppeteer, my suggestion would be to avoid JSDOM. In my experience It's too slow for doing anything more than basic tasks. Seeing as SingleFile does a lot of parsing of content, it will probably be way too slow.

[CRC32EX]

Puppeteer supports headful mode
So, we can integrate chrome extensions easily, I think.

Refs

• https://twitter.com/ChromiumDev/status/1628439484043001856
• https://pptr.dev/guides/chrome-extensions

[gildas-lormeau]

Thank you for the info! That's very interesting :)

[daniel31x13]

@gildas-lormeau Hey there it’s been a while, sorry for not responding.
Long story short, I did a full rebuild since the initial version lacked many features from the ground up.
So anyway everything has been changed/improved…

Coming back to this, what is the best way you’d suggest me to implement SingleFile into Linkwarden? Worth noting that I’m using Playwright behind the scenes.

The closest thing I found is gildas-lormeau/SingleFile#723 (comment) which didn't work out for me.

[gildas-lormeau]

FYI, I did a test with Ubuntu Server 22.04.4 (64-bit ARM) in a VM (with the user name "ubuntu", 4GB of RAM and 32GB of disk) downloaded from https://cdimage.ubuntu.com/releases/jammy/release/. It worked as expected on my end. I ran these commands:

sudo apt install chromium-browser
sudo apt install npm
sudo apt install unzip
curl -fsSL https://deno.land/install.sh | sh
export DENO_INSTALL="/home/ubuntu/.deno"
export PATH="$DENO_INSTALL/bin:$PATH"
npm install single-file-cli
cd node-modules/single-file-cli/
deno run --allow-run --allow-write --allow-net --allow-read ./single-file https://liftvault.com/programs/strength/ --dump-content

[daniel31x13]

Hmm, this is pretty strange since I'm following the exact steps and encountering the same issue. The config of the dummy server is also pretty much the same as yours (with the exception that it's not ARM-based... Idk if that really matters.)

Email me your SSH public key and I can give you access to the dummy server to take a look if you're okay with that... It's a fresh server I created a couple of minutes ago.

Also thanks for putting your time till now!

[gildas-lormeau]

Sorry for the late reply, I fixed some issues meanwhile that could be the cause of the problem. Can you test the last version please? If it still does not work, I'll give you my public key.

[daniel31x13]

Unfortunately the exact same issue is persisting.

[gildas-lormeau]

Sorry for the late reply, do you still need help?