-
Notifications
You must be signed in to change notification settings - Fork 967
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve support for users that want to deploy their own backened #25
Comments
Hi, thanks for trying to make Omnivore self-hostable ! |
Google built this go library that abstracts gcp services and has plugs for aws, azure, etc. https://gocloud.dev/ I wonder if a similar library exists for nodejs. 🤔 |
Thanks for a great tool - making this a nextcloud app would make this easy to deploy for many non technical users who have access to nextcloud instances. The app fits well with the open nextcloud ecossystem and meet a need not adressed by any nextcoud app. |
Could we get a progress update? Omnivore is exactly what I am looking for – only production-level self-hosting is missing. |
Please update to fully open source. Thank you. |
Hi @Coo-ops the only piece that isn't open source is a PDF viewer library that we license, in the future we will try to replace this with pdf.js. |
any new progress on this issue? |
I really want to go all-in on Omnivore. Coming across from Wallabag (and tried out a load of other options) the UX seems great, and happy to see it be open source! So been keeping an eye on this - right now both the lack of ability to self-host (or at least do so easily!) paired with the lack of any data export function (locking you in) makes me really hesitant.. If data export existed I'd at least be more willing to go with your hosted version for now, knowing it'll either be easy to migrate over when self-hosting is properly available (or to migrate my data to another solution altogether in the event self-hosting failed to materialise). Wondered if there are any updates on this at present? |
deployment with docker won't do the trick ? |
Is there a detailed tutorial for docker self-hosting? |
@jerryzhang721 All I've been able to find are the extremely basic instructions in the readme (https://github.com/omnivore-app/omnivore#how-to-setup-local-development-computer), but I was simply unable to get this working when using a custom domain rather than local IP/port. And I'm still extremely unclear on if all the external cloud dependencies have been refactored out yet allowing proper self-hosting or not.. |
I'm very much hoping to self-host Omnivore as well! I didn't see these docs posted in this issue so I'll post them here:
|
For the record, @lawrencegripper did contribute on k8s setup in #2966. Unfortunately, it is in WIP and he is unavailable. |
Thanks for this pointer! I'm currently thinking about/planning to work on a Helm chart. Will probably start over the Christmas days. Feel free to ping me or connect if you would like to support. |
FYI, not sure if you heard about https://bjw-s.github.io/helm-charts/docs/app-template/, it is pretty popular and it probably can save you quite a lot of time in this case. Also, lots of homelab users already have deployed cloudnative-pg or something equivalent, script to bootstrap PG is likely not needed to them. |
Hi, sorry for the late response there were some hurdles I had to overcome. As @grapemix suggested, I use the bjw-s helm chart to setup a functioning instance (Web, API, content-fetch). You can found the current (WIP) version here: https://github.com/se-jaeger/omnivore There are some things I want to improve. However, in the meantime, I'd love to get feedback from you:
One more remark. I built and pushed the images to my Docker Hub account: https://hub.docker.com/u/sejaeger |
FYI: already merged #3385 |
Any news on docker self-hosting? |
Sharing my experience attempting to self-host this, not a coder at all so trying to fix some things is out of my expertise.
|
Hi @mbhkoay, thanks for this write up! Here are some pointers that may help.
In the https://github.com/omnivore-app/omnivore/blob/main/self-hosting/helm/values.yaml file, I added some hard coded credentials (
I added a environment variable that allows to turn-off the creation of this default user:
Also stumbled across this. If you rollback to this comment (e44616b01), which is the latest before redis is required for content-fetch, it should be possible to run it. I plan to dive into these changes and propose a solution for self-hosted instances. Hope it helps. Cheers. |
Hi @mbhkoay and @se-jaeger, is still unclear to me if elastic is needed. thanks in advance! |
Hey @mariusrugan we actually just dropped the elastic requirement recently, we're also in the middle of pulling out most of the GCP requirements and getting things down to two images (backend which will both process async jobs and run the API, and content-fetch which is the standalone service for fetching page content). its in a bit of flux right now though as we wrap up this work. |
Docker images available here: https://github.com/orgs/omnivore-app/packages?repo_name=omnivore |
I think a lot of this is improved with our move to bullmq jobs instead of cloud functions. The backend service has health checks for both the api server and the queue-processor server that also handle graceful shutdown via SIGTERM. We've started running both in k8s for our services as well. |
When using docker, is the extension for chrome able to connect to it ? Thanks |
You'd have to build the extension yourself. For security, the extension includes a content security policy that specifies the domains it can connect to. |
would be awesome to have settings for that to bypass default :) |
thanks a lot @r0bbie ! much appreciated ! |
Anyone got a docker-compose file setup using the new images? |
Thanks to mbhkoay's work. I give a try on deploying omnivore in my k8s + fluxcd cluster. Here is a few feedback:
I cannot fully test my deployment since the web img is missing. But I can see the api page. Finally, thanks all for the hardwork. It is a good start. |
Thanks, this is what i use for our probes:
regarding the web image currently you need to build it, because of the way next.js works you have to "bake" the environment variables like the API endpoint into the javascript bundle. There's some workarounds with replacing them in the docker image that I'd like to try but haven't had time yet. |
@grapemix can i ask how you are managing your secrets? Are you able to just store the entire CONTENT_FETCH_URL as a secret? Also i am wondering if we can remove that token in self hosted scenarios anyways. |
|
I think it would be a fantastic possibility for user to have their own Omnivore instance on their hardware running, thus controlling their own data! I run a small number of services in docker containers on my Synology NAS, and I'm sure that this is quite a common way for people like me who don't know enough how (or simply don't want to) to run a fully fledged web server to self-host stuff. If a way (and how-tp steps for getting it set up) to accomplish running Omnivore (front and backend) in docker containers, would there be a way to export / migrate my existing data (which right now is on https://Omnivore.app) to the new instance? If I could run the backend myself, I'd gladly support the creation of the "self-hosting-capable extension variant" financially. |
I have tried self hosting.But,to be honestly,it's not easy to be deployed on my VM.I'll wait and try the updated deployment method.😥😥😥 |
Regarding Cloud Storage: What is the currently envisioned solution? Has WebDAV been considered? |
@Mikilio I've been working on making GCS optional, you can track progress here. So far, I've got puppeteer (content-fetch), YouTube AI transcripts and PDF working (which fits basically all my own usecases). If anyone is keen to help please let me know. This is rather outside my skillset so if you see anything looking a bit off do tell me. Oh and design wise, I was thinking of switching my custom storage abstraction for this one: https://flystorage.dev/ which would allow using a variety of storage backends. |
@Mikilio gcs-s3-adapter is a library to replace GCS. Hope it helps. |
I've just made a quick attempt at running in a LAN server, that hosts other services, with the provided Docker Compose file. And my quick spotlights are:
I'll make a next attempt with moving what I can to an Update: |
At least, now it becomes clear, why there was no real effort towards making self-hosting easier. |
Well, so much for that. Guess we can always fork it and figure it out on our own, because I for one am not paying for Readwise Reader again. |
Linkwarden for a moment on my side |
Does it support highlighting? |
Try readeck! They just added omnivore Import yesterday |
IDK |
Deleted my docker last week for linkwarden |
Linkwarden vs Readreck vs Wallabag? |
Wouldn't recommend Wallabag. Go for Linkwarden or Linkding. |
Why not wallabag? |
I use wallabag since almost 10 years with success. Still a few pages not well scrapped but it works well. Didn't try the others though. |
I have also decided to switch to Readeck. It's a great piece of software — free, self-hosted, and “it just works”! It comes with browser extensions for Firefox and Chrome. And for those like me who want to self-host on their Synology NAS, here's a really easy how-to. I find it remarkable that Readeck's author added an Omnivore import option in just one day! 🤯 You can use said import functionality by clicking the “three dots”-button next to the “add new link” bar and choose “import bookmarks”: You can then choose Omnivore import: You need an API key (which can be added via Omnivore settings > API keys). The process worked flawlessly for me and imported all my content, including all tags, without any errors whatsoever. Overall, highly recommended! |
Do you know if there is a way to save from iPhone? Maybe a shortcut? |
I'm not an Apple user. As far as I know, there's no dedicated mobile app (neither for Android nor iPhone), but Readeck itself is a mobile friendly web app, and as I said, there are extensions for Firefox and Chrome, which are usable from the iPhone as well, right? So, you could just do that: Use the browser extension or the web app itself, or install it as PWA. |
I am subscribed to this issue to receive news on the state of the actual issue in the title. For anyone looking to self-host Omnivore: At the moment, the PR #4465 looks promising and would close this issue. |
Currently Omnivore relies on a few GCP services to run, but open source users will likely want to deploy the api, web, and content fetching (puppeteer-parse) service to another platform. We need to come up with a list of target platforms and supported deployment configurations that are realistic for users wanting to deploy a minimalistic configuration.
Some of the services we currently rely on:
Other services we are using:
The text was updated successfully, but these errors were encountered: