Project Lifecycle Document: Impact Stage, Well-established Project Communities

[Naomi-Wash]

Comment moved from Project Lifecycle Document Section 3. Stages - Definitions & Expectations - Impact Stage

Impact Stage projects are commonly used in enterprise production environments and have large, well-established project communities.

Discussion

Suggest to qualify/quantify, e.g., "comprising at least 20 committers active in the last year" (with committers being the sort I suggested elsewhere: People that have been judged/voted to be valuable committers by existing committers, not just folks that have done a small doc update) "and 1000 discussion or code issue items raised and resolved in the last year".

I think it's generally the TAC's job to "judge who has been valuable committers", and this is implicit in their voting.
I think your metrics might be a little much. 1000 issues seems to be quite a lot.

I absolutely do not agree that it is or can be the TACs job to judge this. Just the top two reasons below: 1) Self-governance: If we'd subscribe to your view we could do away with GOVERNANCE.md files claiming self-rule (e.g., files documenting contributors/maintainers and rules/responsibilities for them to adhere to) by the projects themselves -- or add caveats to them along the lines "Final decisions on contributors are taken by the TAC". This seems to be in marked contrast to your initial statements from last year that "LinuxFoundation is there to help OSS projects & maintainers": Is changing this really your intention? If so, you might even do away with TSCs (that I understood you set up to delegate project-specific decision-taking to), right? Isn't the mere existence of TSCs an acknowledgement of the fact that the TAC is too far removed from actual work(ers) most able to make good judgement calls for the project? Isn't the design of the TAC rather one of a political body, not one (mostly) comprised of people doing real work? Looking at the rules stated here, it has 1 technical savvy (TSC) representative per project; the majority of the TAC in turn looks like a mirror-image of the companies funding PQCA. Accordingly, its majority decisions are most likely equally going to be politically motivated, not based on technical experience or actual contribution activity, e.g. along the lines of "I fund this, so I decide who is a contributor I trust" (e.g., those I employ). In the PQCA TAC there even seems to be one person/company that I do not recall having contributed anything technically (considering money as a non-technical item), right? And looking into the future where PQCA is going to adopt projects that only have a "stretch relation" to PQ topics it's only going to get hard(er) for the TAC to arrive at meaningful decisions as to who is a "good" contributor/maintainer in any single project.

2. Proper care and attention: Except for Brian, not a single TAC member has cared (or time) to leave even a single comment in this discussion -- since months. "TAC-level people" mostly very probably live from every 1 hour meeting slot in their agenda to the next -- how can they be expected to gain the hands-on experience required to make proper decisions as to who is a good contributor/maintainer (beyond using subjective or self-serving rationales)? Finally, by way of example, your oft-cited "sample project" Hyperledger has 144 sub projects. Are you really convinced that the TAC members there know and personally blessed and keep tracking the activities of all people contributing to and maintaining those 144 sub projects?
   And regarding the metric: 1000 items (incl. discussions/questions) per year are less than 3 per day. That's what a single contributor may handle -- and we're in agreement that a single contributor/maintainer project is not "healthy".

I think 1000 is far excessive since depends on the project. Some project might not even have 1000 issues in their lifetime. We need to calibrate this per project but have a minimum bar.

Also we are basing this on the current best practice at the LF which has had tons of successful projects and communities.
I propose we close (Resolve) this issue and reopen if a case can be made for a more stringent set of requirements.

So what's the minimum bar you'd suggest? What is a "large, well-established project community"? The case for stating this more precisely is two-fold: 1) Users should be able to rely on this designation, e.g., knowing that more than a single over-worked and underpaid maintainer looks after the code they rely on. 2) Contributors should be able to trust they get help, are part of a community that alleviates them of the pressure to deliver, that can rely on (sufficient numbers of) others to not quote "other obligations" as a reason to do other things first

Also, your reference to "tons of successful projects" is a bit vague: Is there any scientific analysis that tells successful from less successful projects apart in this regard? If LF has so many projects such data should exist and inform this wording, no?

See previous comment. I suggest we start capturing some metrics to see if that shows us ways we can improve.
If we want to proactively go further we could look into results from the CHAOSS project ( https://chaoss.community/ ) who for example have a number of community related metrics ( https://chaoss.community/kbtopic/community/ )
There's clearly a lot of relevance in community health - but it's a difficult problem. I don't see we can adopt specifics until we have some more insight.
Beyond starting with lf insights, if there's community interest in pursuing more chaoss metrics that could be done too.. but I'd start with insights as it should be easy to enable & will help us understand next steps.
WIth that we could close this discussion?

[baentsch]

@Naomi-Wash Sorry for the repeat question if I posted it somewhere else, but it's here as valid as anywhere else: What's the plan of action to move this and the various "sibling issues" to resolution? The arguments are captured above (some since months) but how and by when is this supposed to lead to a generally agreeable lifecycle document?

[Naomi-Wash]

@baentsch Please forgive the delay in my response. There is an 8/28/2024 TAC agenda item to review the deferred Lifecycle Document comments.

Adding @KennyPaul for any future questions as he is now the PM for PQCA.

[KennyPaul]

@baentsch My apologies if I am misunderstanding the context here but a PM typically wouldn't take a leadership role related to community strategy discussions. I believe that Naomi took breaking out specific topics into a series of GitHub Issues as an action item at a prior TAC meeting. That was pretty much be the extent of our responsibility there. Any time line for resolving those open tickets is fully up to the community to triage, prioritize and execute against.
-kenny

[baentsch]

@baentsch My apologies if I am misunderstanding the context here but a PM typically wouldn't take a leadership role related to community strategy discussions. I believe that Naomi took breaking out specific topics into a series of GitHub Issues as an action item at a prior TAC meeting. That was pretty much be the extent of our responsibility there. Any time line for resolving those open tickets is fully up to the community to triage, prioritize and execute against. -kenny

@KennyPaul Thanks for the explanation. Given the lack of feedback by the TAC members on my GH input they had requested I had been assuming (hoping, really) for you to operate as project managers (the "PM" moniker led me to that assumption), i.e., "stewards", but it seems you're really assistants to the TAC then, right? If so (?), apologies, my mistake, I won't bother you again then.

@maximilien Can I ask you then what the meaning is of the formal approval/"6-0 vote" of the lifecycle document -- a document in key areas debated and IMO dangerously unsuitable and misleading to PQCA users? How does this vote relate to the open GH issues?

[brian-jarvis-aws]

@baentsch the intent here was to move the remaining areas of open discussion into GitHub issues. As they reach their natural conclusion, they may result in changes to the lifecycle document or perhaps get transferred to decisions that the TSC of PQCA projects would make. That document is not set in stone - it is a living document. But in the interest of establishing a baseline for how new projects are proposed, admitted, and evolved through the PQCA we needed to formalize this initial version of the document.

As for this particular discussion thread (actually, all of the ones which were migrated from the google doc to GH issues), I apologize that I have not been able to follow up on it. My intent was to grok the thread once more and try to determine if the edits that I made to the doc already addressed the concerns raised in the thread. For example, this thread gets into particular metrics and what value they should be. Well, in the doc we took a different approach and said that the point is for projects to establish their own set of metrics that they use to signal the health of the project, and the PQCA TAC would use those in considering the project's proposal to move to a new lifecycle stage. The idea here being that no single set of metrics would suffice for all projects, so it makes sense (to me at least) to push that decision down to the projects' own maintainers.

[baentsch]

Thanks very much @brian-jarvis-aws for a sign of life/activity in GH! I already started to think you guys are really bound to only talk about the project as you find time for meetings (typically in your time zone and when folks in other parts of the world are already dozing off).

But in the interest of establishing a baseline for how new projects are proposed, admitted, and evolved through the PQCA we needed to formalize this initial version of the document.

That's understandable -- the drawback of this, though, is that (people working on) established projects are getting the impression that PQCA rides roughshod on them: Feedback is solicited but then disregarded; decisions unsuitable for established projects are thus getting "sanctioned": I'm deeply, deeply worried that unscrupulous corporate marketing teams will use this document now to declare software like OQS fit for productive use based on this document and/or to justify internally that it is not necessary to invest/contribute (as the project is deemed "healthy" as per this document).

And just one thought about adding new projects: Wouldn't it be more healthy to add those only when the existing projects are alive and kicking? May it be that PQCA only worries about the number of projects it can list on the website, not about their quality? If (there's any chance that this is) so, please consider "upgrading" some OQS sub projects to PQCA projects in their own right: This would also make unnecessary the question in the next section:

The idea here being that no single set of metrics would suffice for all projects, so it makes sense (to me at least) to push that decision down to the projects' own maintainers.

In this case, that would be a solution I'd immediately second. The original version had a ring to it such that the (mostly) non-technical folks in the TAC would be able to overrule the project's maintainers. That said, whom do you/TAC consider as "project's maintainers"? The folks doing the real work of maintaining (the concrete sub projects in the case of OQS) or the sometimes more politically chosen members of a TSC?

[brian-jarvis-aws]

I'm deeply, deeply worried that unscrupulous corporate marketing teams will use this document now to declare software like OQS fit for productive use based on this document and/or to justify internally that it is not necessary to invest/contribute (as the project is deemed "healthy" as per this document).

So to play out this scenario, you're saying that hypothetically if a project proposes being a Production-track project in the Incubation stage, that users of that software would see these titles and jump to conclusions about the project's maturity? The document does explain how this stage is for those on a track towards production-level maturity, and that artifacts like a growth plan and maturity signaling metrics (among others) are needed to move forward.

And just one thought about adding new projects: Wouldn't it be more healthy to add those only when the existing projects are alive and kicking?

I don't believe this statement can be applied for all types of projects. If a project wants to come in directly to the Production::Impact stage, for example, then of course it would be expected to show the signals of that level of maturity. But this is also a framework that allows brand new projects with production-level ambition to join (e.g. the Incubation stage) and grow up through the subsequent stages.

May it be that PQCA only worries about the number of projects it can list on the website, not about their quality? If (there's any chance that this is) so, please consider "upgrading" some OQS sub projects to PQCA projects in their own right

Nothing I have seen would indicate this. To the contrary, I believe the lifecycle document establishes the framework for projects to join at various stages of maturity, taking maturity and quality metrics into consideration as proposed by each project. If OQS or the maintainers of one of its sub-projects would like to propose splitting out a sub-project to become a separate PQCA project, they are welcome to make that proposal.

whom do you/TAC consider as "project's maintainers"? The folks doing the real work of maintaining (the concrete sub projects in the case of OQS) or the sometimes more politically chosen members of a TSC?

The projects are expected to have their own documented governance practices, and that is where I would turn to identify those who make the strategic decisions of the project. In the case of OQS, there is a charter document that identifies the TSC as this decision-making body.

[baentsch]

if a project proposes being a Production-track project in the Incubation stage, that users of that software would see these titles and jump to conclusions about the project's maturity?

No; my concern is that of a project being labelled "Impact" without having gone through those stages, thus misleading and putting at risk its (end-)users trusting the highest possible classification ("Impact" or whatever its name).

If a project wants to come in directly to the Production::Impact stage, for example, then of course it would be expected to show the signals of that level of maturity.

Exactly those signals are not generally agreed-upon (as being documented by the many open GH issues like this one). In consequence, there is no agreement as to what is a suitably conscientious classification for OQS. Hence my complaint that the TAC pre-maturely approved the lifecycle document. I'd indeed really urge the TAC to adopt clear-cut criteria for security software such as Common Criteria or FIPS 140 and less touchy-feely "we use these criteria in other LF projects" (where they may be completely suitable, no question).

If OQS or the maintainers of one of its sub-projects would like to propose splitting out a sub-project to become a separate PQCA project, they are welcome to make that proposal.

Given the committee inter dependencies this is arguably only an option for projects that have a solid contributor base and can afford to trade time by technical people to participate in the various committees and groups established by LF. That's IMO not the case for a single OQS sub project; hence also my assertion that OQS is far away from Impact stage, even if applying the most shallow definition of a "healthy" project community. Of course, this assumes technical know how is relevant for TSC and/or TAC discussions: Please let me know if that is optional (would put into question the "T" a bit, though :).

In the case of OQS, there is a charter document that identifies the TSC as this decision-making body.

The document exists -- the actual setup is pretty far away from that documentation, though (as you arguably know as a TSC member), so in turning there you may face a void. Something else to discuss at the TSC...

[brian-jarvis-aws]

No; my concern is that of a project being labelled "Impact" without having gone through those stages, thus misleading and putting at risk its (end-)users trusting the highest possible classification ("Impact" or whatever its name).

I’m not seeing the distinction. In both cases, to reach impact stage - either by growing from an earlier stage or by coming in directly - they would need to provide the justification of health and maturity that warrants acceptance at that level. Is there a suggestion for how to reword something to make this more clear?

Exactly those signals are not generally agreed-upon (as being documented by the many open GH issues like this one). In consequence, there is no agreement as to what is a suitably conscientious classification for OQS. Hence my complaint that the TAC pre-maturely approved the lifecycle document. I'd indeed really urge the TAC to adopt clear-cut criteria for security software such as Common Criteria or FIPS 140 and less touchy-feely "we use these criteria in other LF projects" (where they may be completely suitable, no question).

But wasn’t there alignment on the idea that the TAC cannot and should not set forth specific signals since they will not possibly apply to every single project, and that the TSC of the project proposing this stage will need to propose the signals that they use to measure the maturity of their own project?

—-
I’d like to try and distill this down to a concrete suggestion for a change to the doc, if there is one to be made. I think that I understand your concerns from this thread, but I also think that the document was already updated to address the issues you had raised. Are there additional modifications you are proposing?

[baentsch]

Is there a change-tracking mode that I can activate on the Google doc to see what has changed since I initially raised the concern, @brian-jarvis-aws ?

[KennyPaul]

@baentsch A limitation of G-docs is that there is no way to view history as a viewer or commenter.

[baentsch]

@baentsch A limitation of G-docs is that there is no way to view history as a viewer or commenter.

@KennyPaul That's unsatisfactory. I'd then suggest not using that tool any more for purposes such as this as it causes wasted efforts.

I understand your concerns from this thread, but I also think that the document was already updated to address the issues you had raised

@brian-jarvis-aws Can I ask you then to point out all concerns and their respective updates/resolutions in the doc for me to confirm (or not) your statement?

I'd like to avoid starting from scratch again: On request by the TAC I already spent dozens of hours of work and I'm not willing to keep wasting any more (it's all my free, unpaid time: Please be considerate of this).

[brian-jarvis-aws]

I agree that the lack of a track-changes type feature is disappointing here. However, it does appear that we can view the resolved comment queue and use that as a proxy for the edits that were made, since I was editing in response to comments and then commenting back to summarize what I did. If you click the "Show all comments" button in the upper-right, then change the drop-down to only show resolved comments, you can scroll through to the comments and suggestions you raised and see the correspondence I added about those changes. I didn't keep a local exhaustive list, but I know there were changes to the language around who proposes changes in lifecycle stage versus who accepts those proposals and who defines the criteria to be used for such stage-change proposals. Which is why I say I believe that the current state of the document captures the concerns that were raised. There were some remaining comment threads on the document which were transferred over to GH issues like this one, but as I look through the correspondence here I don't know that any further change is warranted. Unless you see something in the doc that you are suggesting to change?

[baentsch]

we can view the resolved comment queue and use that as a proxy for the edits that were made

This assumes that comments marked resolved have been preceded by an actual change. This has not always been the case: There have been instances when IBM&LF personnel did close, sometime completely delete comments I made. So if you are certain that the promise to not do it again has been kept (?) I'll give your proposal a try, so please provide a pointer to the doc.

[brian-jarvis-aws]

The document is here: https://docs.google.com/document/d/1NV-0vNgXWdc81oqT0jv0C-9Funb8dySS06u90ghF-X4

But I don't think we should continue to add comments directly onto that google doc. I'd like to see it migrated over to a markdown file in the PQCA GH, and then proposed changes raised directly on that md file. But since that's not ready yet, in the meantime, please raise any proposed changes here.

[KennyPaul]

That's unsatisfactory. I'd then suggest not using that tool any more for purposes such as this as it causes wasted efforts.

@baentsch I agree that limiting the red-line view to editors can be frustrating at times. I get impacted by that on occasion as well and when that happens it irks me. That said, as a collaborative tool for documentation it has proven its worth a gazillion times over. Typically comments/suggestions are things such as pointing out typos, poorly formed sentences, asking simple clarifying questions and such that are both easily viewed, addressed and tracked within the context of the comments. Where that falls apart is on the rare cases where the commentary evolves into a running dialog. While still trackable via the comments as @brian-jarvis-aws points out, it certainly isn't ideal. That is exactly why the running dialog(s) about the policy were moved to be GHIs as it is better equipped for such things.

For what it is worth, PR #50 which is the approved version of the policy is pending to be merged.

[baentsch]

But I don't think we should continue to add comments directly onto that google doc. I'd like to see it migrated over to a markdown file in the PQCA GH, and then proposed changes raised directly on that md file

Absolutely agree. Will do. Thanks @brian-jarvis-aws ! When is that migration to be done? Sounds like a reasonably quick formatting thing: I'd probably wait for that then given the horse is out of the barn anyway.

Where that falls apart is on the rare cases where the commentary evolves into a running dialog.

Well, then why did the TAC not react immediately when it saw this happening (pretty much from day 1 (first long comments on Mar 13) and surely way before the community was asked to comment (June 26), @KennyPaul? What happened is that (not doing) this created a feeling (at least on my side) of complete and utter disrespect towards the input given -- constantly and over months. And it may also be a reason why so few people chose to comment at all: It's just been a very bad, bordering on unprofessional experience.

For what it is worth, PR #50 which is the approved version of the policy is pending to be merged.

Proof point to my point of view: "Approved" by people not ever having contributed a single line of code to the project, not having stood in the line of fire of user and support questions & expectations and thus not being impacted by its contents at all. All onus to sort it out in the interest of the community actually doing the work put onto a volunteer. Wild.

[baentsch]

And fwiw, @KennyPaul and @brian-jarvis-aws just was made aware of a nice article explaining my concerns from a more general perspective: https://www.theregister.com/2024/09/18/open_source_maintainers_underpaid/. So far I only thought I represented the majority of OSS maintainers, now I know.

[KennyPaul]

It's just been a very bad, bordering on unprofessional experience.

Unquestionably any community member is entitled to express their dissatisfaction with a process or decision to the parties involved.

However community members electing to make disparaging comments about the perceived qualifications of those serving on a community's governing body certainly does not foster a professional experience. This type of interaction has no place in any community regardless of anyone's status as a maintainer.

I would remind all community members of the LF's Code of Conduct and kindly request that any discussions stay focused on the technical or policy issues at hand and not focused on the individuals, organizations, groups or governing bodies that comprise this community.

Thank you all for your cooperation in this regard.
-kenny

[baentsch]

It's just been a very bad, bordering on unprofessional experience.

Unquestionably any community member is entitled to express their dissatisfaction with a process or decision to the parties involved.

However community members electing to make disparaging comments about the perceived qualifications of those serving on a community's governing body certainly does not foster a professional experience. This type of interaction has no place in any community regardless of anyone's status as a maintainer.

I would remind all community members of the LF's Code of Conduct and kindly request that any discussions stay focused on the technical or policy issues at hand and not focused on the individuals, organizations, groups or governing bodies that comprise this community.

Thank you all for your cooperation in this regard. -kenny

Thanks for making me check the connotations the term "(un)professional" carries particularly in the Anglo-Saxon world, namely a reference to personal qualifications. I duly apologize to anyone feeling personally offended by this: This was not my intention.

My intent was to summarily characterize usage of the tool in light & despite its known shortcomings, the (consequent?) application of (TAC voting) power-driven top-down decision making, and the impact this has on the community. Please let me learn how you would call this "in short/summarily", @KennyPaul and what you/the TAC is going to do to improve this in the interest of a mutually supportive relation.

[brian-jarvis-aws]

The document is now merged and available here: https://github.com/PQCA/TAC/blob/gh-pages/Processes/Project_Lifecycle.md

The original comment was regarding a suggestion to add specific metrics that describe a well-established project community. That phrasing still remains in the overall description of the stage. But the acceptance criteria was adjusted to indicate that the project itself must provide the metrics they are using to gauge the proposed maturity level. I assert that this is an adequate compromise where the document is not overly prescriptive on specific metrics that inevitably would not apply to every single project, while still establishing an expectation that objective measures are required for admission into the stage. If there is a suggestion for a better way to phrase these criteria, please let me know. If not, I believe we can resolve this issue.

[brian-jarvis-aws]

I don’t believe anything further is needed on this topic. Resolving.