Why 1 million jobs per day?

[mockdeep]

We're looking at switching away from Sidekiq due to issues with job loss and duplication. For our part, it's pretty critical that a job runs once and only once, and it seems like it's hard to get that sort of guarantee without a database backed solution. GoodJob looks pretty promising based on our needs, but I noticed the mention of 1 million jobs in the README. Right now we're at ~300k jobs on a peak day, but I could see that approaching ~1 million by the end of the year. Where does this number come from? And what might we expect to see if we pushed GoodJob above that?

[bensheldon]

Thanks for asking! There isn't any science behind the 300k jobs/day; that's primarily for focusing feature development in the gem's early days (one of those "let's walk before we run" sort of things) as well as the limit of my own ability to provide real-world empirical data.

These are the issues I would highlight in a tech review:

• Incompatible with connection pooling: GoodJob's connection-based advisory locks are not compatible with transaction-based connection pooling (PgBouncer) issue
• Not DBA reviewed: I would sure love for an experienced DBA to look at the fetch-and-lock query. issue / issue
• Single maintainer (me): I don't have any co-maintainers (yet), so there is a low bus factor. I'll keep asking.

[mockdeep]

@bensheldon thanks for the thoughtful response! I have lots more questions based on that, but probably the biggest thing is maintainership. What would it take for you to want to add someone as a contributor? We're going to be exploring alternatives in the next engineering arc, and if it's the right fit we should be able to offer some resources to help maintaining it.

[bensheldon]

What would it take for you to want to add someone as a contributor?

I'm imagining following the well-trodden path from solo maintainer, to benevolent dictator, with a much longer term goal of achieving a consensus-based collective.

So for moving from solo maintainer to benevolent dictator stage, I'm looking for:

1. A commitment to my vision of GoodJob, as well as the values of creating a friendly, responsive, inclusive environment for users and contributors.
2. A willingness to do the work of gem maintainership: helping out in managing and responding to issues, working in the codebase, and helping carry the ineffable burden of being relied upon by people from the Internet.

[bensheldon]

I did some slight rewording to hopefully cause less questions about the 1-million jobs/day limit: 70fbfe6