This project is bigger than me

[danielquinn]

...and I don't have a whole lot of time to donate to it.

Since I started it back in 2015, Paperless has grown far beyond what I ever expected it to be. With 5000+ stars on Github and a surprising popularity in Germany, issues are posted or commented on on an almost daily basis.

With the little time that I have (baby on the way!), I try to roll out fixes & features, but every change tends to add instabilities on platforms I don't personally use. There are people using this project on bare-metal Linux installs, Docker installs on Synology boxes, cloud installations, Rasberry Pis, and FreeBSD machines -- and that's only the ones I know about from issues users have posted.

If this project is going to survive & grow, there has to be more people driving it. Primarily, we need people who the rest of us can trust, but also people willing to do the boring work of adding tests and testing frameworks for the various platforms.

I'd like to transition Paperless to more of a collective effort, either as part of the @jazzband, or some other means -- I'm open to suggestions. I'm happy to stay on to audit pull requests or make suggestions, but it's becoming clear that Paperless needs more from me than I'm able to give it, and so I'm thinking we need to expand the pool.

Here's what I'm suggesting: I believe GitHub can be configured to require that merges be reviewed by one or more other collaborators before code is merged. If I can figure out how to do this (maybe I need to create an organisation? Advice on this is appreciated.) then we could have a system where we have more people who can create & merge more stuff, but where merges require review from at least one other person -- that'll keep people honest.

If GitHub can't do this, then I'm also happy to move to GitLab unless there are strong objections.

IMHO, the first mission of this collective should be to get us to 100% test coverage and to update Travis (or something else?) to execute those tests in different environments (if possible). After that, I imagine some of you have your own priorities regarding where this should go. I'm excited to see what you have in mind.

I'm tagging a few people specifically who have been more heavily involved with Paperless than others just to give them a heads-up. If any of you lot are interested in taking a more primary role in this project, please comment below. Maybe we can get an idea of what kinds of resources/interest is out there. This is not an exclusive list, so if you don't appear here but feel you've contributed significantly (or just want to) it's because I didn't recognise your username from the autocomplete:

👍 @Strubbl @philippeowagner @jat255 @ahyear @avichalp @BastianPoe @CkuT @ddddavidmartin @erikarvstedt @MasterofJOKers @pitkley @ovv

If no one steps up, nothing will change. That is to say, issues will continue to pop up and people from the community will (hopefully) try to help people with questions. I will try to keep on top of pull requests and chat in the issue queue when I have time, but as we've already seen, I don't have a lot of time.

[ahyear]

Hello @danielquinn ,

I'm happy to participate, unfortunatly i am not a good dev ( not a dev at all!).
I can help on testing et beta testing the docker part but i won't be on any help en correcting bugs or managing the repo...

I certainly hope that this project will grow to be a new standard as i have not seen anything as efficient as this one.

[philippeowagner]

@danielquinn congratulations! and thank you very much for all the efforts and time you invested in paperless. Maintaining open source projects is hard, especially in the long run.

Unfortunately I can hardly keep up with my own open source projects like django-hijack, django-background-tasks, etc.

Even if I would like to, it will not be possible for me to participate as a lead in this project at the moment.

Thanks again Daniel for all the work on and around paperless.

[ovv]

Thanks for everything @danielquinn. Sadly I have to resonate with the others I don't have time (for the moment) to follow the project as much as I want.

[danielquinn]

Thanks for responding guys. @philippeowagner & @ovv: I know exactly where you're coming from. I'm trying to keep the bulk of my Free software contributions focused on Aletheia, and with the kid due in a few days, I figure that even that sounds optimistic. Your contributions thus far have been greatly appreciated :-)

@ahyear it's cool that you can't do dev stuff, but testing is super important too! How do you feel about me tagging you on the occasional pull request to ask for a review? That is to say, you'd have to pull the changes and run them to check that things are working as expected?

[CkuT]

Congratulations @danielquinn ! Thanks for this great project ;)

I really want to improve paperless, and I have some ideas but don't have many time at the moment. I think I'll have more time available in few months to work on paperless. However, I would be glad to be tagged for reviews in the meantime.

[JensPfeifle]

Thanks as well from a recent convert from MayanEDMS. I’ve been looking for a project I can contribute to and would like to help. As a student I’m not very experienced but may have a little more time than others ;)

I was just reading through some of the open issues, and think that quite a few of them could be closed right away, e.g. #241, #292, #327, or #446. Removing some of the “clutter“ in the issue list and making it more clear where contributions are needed may help in getting more people onboard.

[syntonym]

Configuring github so that any pull requests needs X reviews should be straightforward and should not need a new organization. The documentation mentions a setting in the protected branch settings where the behavior can be configured. Users with write access or admin access to the repository count as reviewers.

Congratulations on the baby!

[ahyear]

@danielquinn you can add me, I will try my best to help the project.
But even if you slow down your involvement, you will still be there right?

By the way I just had my second baby as well 9 days ago ;)

So I understand your situation....

[bmsleight]

testing frameworks for the various platforms

bare-metal Linux installs, Docker installs on Synology boxes, cloud installations, Rasberry Pis, and FreeBSD machines

I would like to give this a crack.

Enjoy the baby! Although I think it just a way of generating a Birth Certificate, so your can store it in paperless ;)

[danielquinn]

Thanks for all of your advice everyone, especially to @syntonym who pointed me in the right direction regarding branch protection and merge rules.

I've tweaked this project to require a minimum of 2 👍 reviews from "collaborators" before a merge to master is permitted. However, I don't have any collaborators on the project yet, so the behaviour remains the same (as an admin, I can merge whatever I want).

So, who here wants to be a collaborator? This would mean that you could be tagged for a request to review a pull request and that you could be one of the two required reviewers to merge something.

As for @ahyear's question: yes, I will still be around. I'm just short on time/resources and I don't want this project to die just because I'm busy. This is basically an attempt to improve the bus factor of Paperless, not a means of abdicating my role in it.

The fixing of the bus factor starts with other people volunteering to be collaborators/reviewers, and until then, this project will move forward... slowly, but it'll still be moving.

[jat255]

@danielquinn I had thus far failed to pipe up, but I'd be happy to be a collaborator/reviewer. I've been using the project regularly for the past few months, and I have a few ideas on how to improve some things (if I can find the time!)

[MasterofJOKers]

I'm happy to help with reviewing stuff whenever time permits.

[ahyear]

@danielquinn asI said, I will be happy to help to. (as much as i can...)

[kevin2107]

@danielquinn I'd love to start helping, I've recently made the decision to go paperless and would like to begin contributing bug fixes and features

[ovv]

I was thinking about this and #479 . I believe it could be a good idea to create a paperless organization that way some non-essential part could be taken out of the primary repository and maintain by other people more easily. It will also reduce the load on @danielquinn.

By non-essential part I'm thinking about documentation, docker setup, other installations method like the script in #479 if we decide to expand on it or the ansible role I maintain.

[JensPfeifle]

@danielquinn, If you can give me permission to assign labels I would start by sorting through the issue queue.

[danielquinn]

Alright it sounds like I'll have to convert this project into an organisation. I'll try to do that this weekend and will keep this issue updated with whatever I figure out.

If anyone here has some tips regarding how to do this, by all means, share it :-)

[philippeowagner]

Create a new organisation https://github.com/organizations/new, eg. paperless.
Transfer the paperless repo to paperless. That's it.

See https://help.github.com/articles/transferring-a-repository/ for more info.

[danielquinn]

Ok I've now done the following:

• Created a new Paperless organisation. It turns out paperless was taken, as was paperless-project, and the-paperless (pretty crowded space I guess), so I ended up with the-paperless-project :-)
• I added a bunch of the people who spoke up here as members: @ovv, @ahyear, @MasterofJOKers, @jat255, @bmsleight, and @CkuT. As for @JensPfeifle & @kevin2107, I appreciate your enthusiasm, but as you haven't submitted any pull requests yet, I'm not ready to add you as a member. Feel free to find an issue you wanna help out with, send a PR and once your contributions come through, we can talk about granting write permissions.

If I've done this right, each one of the organisation members will have write permissions to the project to create branches or even additional projects, but as the paperless/master branch is protected, requiring 2+ approvals to merge, a member shouldn't be able to arbitrarily merge stuff into master. Please let me know if that's correct.

Additionally, I know that @JensPfeifle offered to go around and work on labelling stuff in the issue queue. That's awesome and really appreciated, but I can't figure out how to do that without giving you write access to the code, and I'm not sure how I feel about offering that for someone who hasn't submitted a PR yet. Thoughts from the rest of you on this are appreciated.

Let me know if anyone thinks I've screwed this up, eh?

[bmsleight]

Cool [Paperless organisation]https://github.com/the-paperless-project)!

a member shouldn't be able to arbitrarily merge stuff into master

Most of the open pull requests are for docker, guess I have to start learning docker!

[pitkley]

Hey @danielquinn, sorry for not responding sooner. I'm glad you reached out to the great community of users and contributors of Paperless in aiding you and the project itself. I think that adopting additional maintainers is a great step in the right direction, and requiring 2+ approvals is the right way to still ensure the quality of Paperless, even if you personally can't check or "vouch" for every change introduced.

I have seen that you already have added quite a few members to the organization, which I find awesome! 👍 Should you think that one more helps, feel free to add me. In any case I'll continue contributing to this project as I have in the past (and as time allows), either if I stumble upon something where I feel like I can support, or if I'm specifically tagged on something where the community thinks my input could help!

Also a big thank you to everyone in this thread, without the support and contributions extended by all of you Paperless wouldn't be as great of a tool! 🎉

[danielquinn]

I was hoping you'd be interested @pitkley! I'll send your invite now :-)

[ovv]

If it's ok with everyone, and after a bit of cleanup, I plan to move ovv/ansible-role-paperless to the organization.

[philippeowagner]

@ovv I personally think it's a good idea to find all loose parts in one place.
But we should consider have the licences aligned. I noticed ansible-role-paperless is licensed unter the MIT License and paperless itself under the GNU General Public License v3.0.

[bmsleight]

HI
Any chance of a review of my PRs ?
@ovv @ahyear @MasterofJOKers @jat255 @CkuT @pitkley ?

#499 #500 #501 #502

[pitkley]

@danielquinn apparently the move to the organization broke Travis CI, at least it doesn't seem like any of the PRs opened since then are building. I have requested approval from you for Travis CI to be able to access the organization.

I'm not 100% sure if that was the right thing, maybe double-check on your end if there is something else needed to be able to access the organization. 🙂

[ddddavidmartin]

What the status of the Docker image?
Is some help needed?

I believe the automated build is not working and it is somewhat out of date. I don't quite remember who was last looking into it though.

[danielquinn]

Until we can figure out how to let Docker do an automated build, I'm afraid the only option is to do the build yourself with docker build . -t paperless. We could automate the push to Docker via Travis, but that wouldn't be an "automated build" according do Docker, which is less desirable.

[ahyear]

is it the same than docker-compose build ?

[danielquinn]

Sorry @ahyear, yes it's the same thing. docker-compose is just a smart wrapper around docker that automates the process of standing up an image into container along with all of the other configurations we need to make use of that container.

[sbrunner]

I just open a Pullrequest for a proposal to deploy the Docker image ...
#532

[pitkley]

@danielquinn unfortunately the change to the permissions didn't seem to have any effect, I am still unable to configure the repository. 🙁 What I did though is try to request access for the "Docker Hub Builder" app to the-paperless-project organization. If you can approve that (you should have received an e-mail), maybe that helps in solving the issue.

[danielquinn]

I've just approved "Docker Hub Builder" so hopefully that'll get things rolling. If there's anything else, just ask.

[ddddavidmartin]

Hi all,

#532 has been merged in but I don't think any builds have made it to Dockerhub yet. Is this something for @danielquinn to arrange?

It would be great to get this going as the current official build still has the convert error that stops documents from being parsed.

Thanks everyone for looking into this!

[danielquinn]

From the looks of #532, my understanding is that we now have a way of making Travis push updates to Docker Hub, and all that's required is that someone set DOCKER_USERNAME and DOCKER_PASSWORD into Travis so that those values are available to the deploy script.

So as a test, I went ahead and added these two values to the Travis secrets and re-ran the most recent build. The build script exited with a 0 status, but nothing appears to have been run :-(

Additionally, this isn't a good permanent solution, as anyone can get my docker credentials now simply by pushing a commit that says something like:

echo ${DOCKER_USERNAME}, ${DOCKER_PASSWORD}

...so I'm going to delete these secrets from Travis before something like that happens ;-)

We could have a role account -- one whose only powers include writing to the Paperless organisation, or someone with admin rights to this repo on Docker Hub could add Github integration, though there are some scary security concerns with that option (which is why I haven't done this myself).

I'm open to suggestions here.

[sbrunner]

... anyone can get my docker credentials now simply by pushing a commit that says something like ...

This is wrong, on a fork the secret will not be set, then this is true only for committers.

Event if I think that's good to have a technical account on Docker hub only for that :-)

[danielquinn]

I'm not sure I understand. Surely, if we create a role account and set the user/pass into the Travis secrets, those secrets could be exposed if one were to echo them in the ci/deploy-docker script. Otherwise, they couldn't be used in said script.

[sbrunner]

If it's pushed on a fork it can't be echo on the and the ci/deploy-docker can't works because the variable are not interpreted.

If it's pushed on the the-paperless-project we can do an echo and the ci/deploy-docker will work but only committers can do that (not anyone), We can say that's not completely safe but it's not so simple to get the secrets :-)

[danielquinn]

How 'bout this:

1. I create a dummy role account on github, something like paperless-dummy
2. I then add said account to Docker Hub as one of the contributors and use it to automate the build process.

[colinfrei]

Any progress on getting new releases built and to docker hub?

[ddddavidmartin]

Any progress on getting new releases built and to docker hub?

Ha. In the meantime I looked into it and pushed my own image to docker hub as I had a growing backlog of documents that were failing to be consumed. And it had to be on docker hub as that is easiest to get onto my NAS. I pushed it here.

In the end it was not that difficult. For me on OSX it was basically a matter of

• follow steps for the Docker setup in the Paperless readme; basically install Docker and adjust the docker-compose files same as the existing docker image
• build, tag and push the image

start Docker.app
$ docker build --tag=latest .
$ docker image ls
$ docker tag d38ca0c3fcd0 molecareer/paperless_consumer:latest
$ docker push molecareer/paperless_consumer:latest

[pitkley]

@colinfrei I don't think there was any progress on the official build. Something I have done now is "revive" my pitkley/paperless Docker Hub image. I have currently configured it as such:

• It is built from my GitHub fork of Paperless.
• The latest tag is built from master.
• The master in my fork is kept in-sync with upstream master (i.e. this repository) automatically using https://github.com/wei/pull. pull ensures that the master is identical, i.e. the Docker Hub image in my namespace will never contain changes that are not part of the official repository.

What this should mean is that whenever something is merged into master of Paperless, my fork will receive this change too, automatically rebuilding Docker Hub image in my namespace from the official sources.

Now returning to my namespace is far from ideal, but at least it gives people a pre-built image to depend on again. I commit to maintaining the Docker Hub repository, configuring any changes the community might need.

I'm not sure if we will ever resolve the organization-integration with Docker Hub, maybe we should contact their support? (cc @danielquinn)

[pitkley]

@danielquinn hm, do you maybe see Paperless in the repository selection here: https://cloud.docker.com/repository/docker/the-paperless-project/paperless/builds/edit?

AFAICT Docker Hub should be authorized to see the GitHub repositories, I can't configure the build on Docker Hub though since I seem to lack those permissions.

[colinfrei]

It seems like some builds are platform-specific - trying to run with the build from @pitkley causes this error for me:

standard_init_linux.go:211: exec user process caused "exec format error"

It seems it would be possible to build cross-platform docker images, I found this: https://www.sanisimov.com/2019/03/building-docker-images-for-multiple-architectures/

[danielquinn]

Hello all from Athens! I'm on holiday and the grandparents are taking care of my kid, so I finally have time to spend on Paperless again!

Thanks to @pitkley for picking up the torch on this one and Making Things Work on Docker Hub until we can get something "official" working properly. I think I may have figured this out, so I'm interested in some feedback. The permissions system between GitHub and Docker is a dumpster fire, but I believe I've managed to get it working. Here's what I did:

• For security reasons I was unwilling to give Docker write access to all of my public & private repos just to automate a build, so I created a GitHub role account called @paperless-dq. This account is a member of the Bots team, which has read-only access to the code.
• I logged into Docker Hub and linked my Docker Hub account with this new GitHub role account. The way I figure it, this was better than creating another role account on Docker Hub, and I don't intend to use Docker Hub for automated builds on anything else for the above reasons anyway. If someone thinks I should create a role account on Docker Hub however, I can do that too.
• As part of the linking process, Docker added a read-only deploy key to the paperless project. I suspect that this is why they needed write access, though it's still unclear why this public key couldn't simply be made available to those of use more concerned about security :-(
• Setup the autobuild to build master any time it's pushed to. I believe anyone in the Managers group on Docker Hub can tinker with these settings, now that the link is in place and there's something to see. As an aside, @pitkley in answer to your question of what I saw at your provided link, the answer was a blank page. Just a header, footer, and nothing in between. I think the Docker people have some more work to do here.
• I added, removed, re-added, and removed a file called .docker-hub-test to the project root, trying to get the autobuild process to run.

The result is that builds appear to be happening as-expected, and you can now pull from the "official" repo:

$ docker pull thepaperlessproject/paperless

Please, can some of you take a crack at this, specifically @jat255 and @pitkley who are in the managers group on Docker Hub. I want to know that you have everything you need should you want to expand things to do stuff like building for multiple architectures as @colinfrei suggests.

[sbrunner]

I will try it during my holiday :-)

[jlauwers]

It's working well.
Used https://github.com/the-paperless-project/paperless/blob/master/docker-compose.yml.example as base and replaced

build: ./   

by

image: thepaperlessproject/paperless

env file is:

PAPERLESS_OCR_LANGUAGES=fra                                                                                                                                                                                   
USERMAP_UID=1001                                                                                                                                                                                              
USERMAP_GID=102  

[ranomier]

@danielquinn Do you have a patreon or something similar?

Maybe using the mony not for you, but for bug hunters.

[danielquinn]

Aww, that's nice :-)

I think this project has grown far beyond this mostly being my gig, so accepting donations myself feels pretty uncool. With that said, if you're looking to throw money around, the README still recommends donating to the UNHCR 'cause frankly, they need it more than any of us.

If you're looking to target some money toward certain feature development however, you might want to have a look at BountySource or something like it. Put some money up there, and then mention it in the Github issue here. I don't know how those systems work, but I have to assume there's some way for people who fix your issue to get reimbursed for that work through such platforms.

[r-kells]

Hey all, Im a senior developer with python experience. I haven't used Django much but Im interested in helping where I can. Feel free to tag me in PRs.

Cc @danielquinn