WG Data proposal #673
WG Data proposal #673
Conversation
|
I am very strongly opposed to using the name My proposal for the name is: Where "data" can mean both actual data (spark) and metadata (model registry). We can also split it up in the future, if the members who are maintaining these components diverge. |
very well noted @thesuperzapper , as also marked here: I just wanted to have a branch where to start collecting this kind of feedback in a non-sparse way and also to report back to you and the group on the progress on Tuesday meetings. |
|
@thesuperzapper how about we make it more explicit |
|
As it currently stands, this WG does not meet the requirement for diverse leadership given all chairs come from one company (IBM - which owns RedHat). |
|
@thesuperzapper Andrey is listed as a Chair, he's from Apple |
|
noticing only now it was not marked as Draft PR despite being my intent:
my sincerest apologies. Marked as Draft PR per original message in thead. |
|
@thesuperzapper Is there a minimum number of companies to compose the chair to make the WG eligible? |
|
While there is no specific number requirement, the steering comity must approve the new WG (currently, @jbottum @james-jwu) in line with the community's interests. I would expect at least some concern with having 4 leads from one company and only 1 from another. For reference, here is the lifecycle and other info about forming a working group: Also, there are only meant to be 2-3 chairs, some other WGs have more, but in most cases, there are 2 active members and we just need to formally clean up the inactive chairs. |
|
Also, some of the proposed chairs are not even current Kubeflow org members, so are ineligible unless they go through that process first: |
|
Thank you for the references! Those are valid points though, and I'll see how we can work on the eligibility topic as well as your concerns. |
|
As Ricardo noted, thanks ! Is there guidance for deputies to keep work WG ongoing during leaves, please? As noted, will work out to account all the feedback received; thank you those are very helpful |
|
Thank you for starting this @tarilabs! Let's collaborate together on this PR for the WG Charter and Name. Please provide your suggestion on how we should name this WG that initially will have Spark Operator and Model Registry component. A few initial suggestions if WG Lifecycle is too ambitious:
This is valid concern @thesuperzapper. We can add folks from Spark Operator maintainers to this WG |
|
cc @kubeflow/wg-training-leads |
|
I would request "WG ML Lifecycle" if the purpose of the group is to house things in the MLOps orbit that don't have a more specific working group yet so they can "incubate". Data Preparation, Feature Store, and Model Registry being 3 examples that have been recently discussed that likely aren't big enough yet to have their own working group. I guess one key aspect here is to consider how new efforts can happen without the overhead of setting-up a new working group for each one until it is truly merited and bandwidth is available. Is there a process that exists for refactoring a topic out of one working group to a new working group? |
|
Kubeflow seems to be entering a new growth phase. The community needs a structure to support add-on components (Spark, Ray, Model Registry, Feature Store, etc). We want to encourage contributors and users to meet, discuss, experiment, decide, store code and produce documentation with a goal that integrations will help both Kubeflow and the add-on projects. We need to minimize overhead. We need to set expectations (of support...to/from Kubeflow and for users) especially if we are experimenting and trying to find market acceptance. Most importantly, we need active user participation, comment and leadership. I want to move this forward...I am a +1 to adding a single umbrella WG for all of these projects to get things moving. @james-jwu would you please provide your thoughts |
|
I think that the name
Also, I am still very against Separately to the discussion around names, I think we should confirm that the maintainers of these various components are actually overlapping, otherwise it will make it difficult for this "mega working group" to function. |
|
+1 to @thesuperzapper I would suggest voting for |
|
New commit ae188fe incorporates some feedback received around:
will keep posted during KF Community meeting on any further updates. |
|
Just so we are clear, I think |
|
Would this working group be relevant for the minio replacement (seaweedfs) as well? I am currently working on a PoC in Kubeflow/manifests. |
address: kubeflow#673 (comment) Signed-off-by: tarilabs <[email protected]>
as suggested. Signed-off-by: tarilabs <[email protected]> Co-authored-by: Francisco Javier Arceo <[email protected]>
|
I've added all comments pertaining to Feast in a single commit with fa3c318, so to more easily manage that addition to this wg charter if required or by feedback from SC. |
not entirely sure, that to me is more a "storage"-related concern, while "data"-related concern expressed here are more orthogonal to the actual medium.
I'm very happy however to engage in discussions, since "storage" is also a dimension we're exploring for Model Registry (bringing in OCI as first class, but potentially others with an abstraction layer). Let me know your thoughts! |
|
Thank you for addressing the feedback @tarilabs! Given that we still have discussion around WG governance and what projects WGs should maintain: #673 (comment), should we include Feast addition as a separate PR after followup discussion ? From my point of view, initially we should just establish the Data WG with 2 Kubeflow components: Spark Operator and Model Registry, and after that we can update charter to include Feast and other projects that we want to maintain under this WG. Any thoughts @franciscojavierarceo @kubeflow/kubeflow-steering-committee @tarilabs ? |
I agree with you @franciscojavierarceo, but should we include Feast in the Data WG once we make Feast as part of Kubeflow core components ? |
|
Per my comment in the Community meeting, I support Feast as part of the WG Data and as a core KF component. I am glad to pursue that path or another, if that cannot be accomplished (as I believe a defined relationship would help both communities). |
|
@andreyvelich I am okay including Feast before making it a core component. :) |
Then kubeflow/manifests#2826 and kubeflow/pipelines#10998 might be interesting for you. |
@juliusvonkohout This issue is related to Kubeflow Pipelines (e.g. Pipelines WG), isn't ? |
Anyone who needs S3 storage in Kubeflow, but especially pipelines. |
|
Bumping this PR. What is missing to get this merged? |
I think, we need to make a decision with Feast. |
I'm following up on this @andreyvelich to follow up with the rest of the KSC. |
|
I would love to get this finally merged. @franciscojavierarceo What do you think about keeping the existing charter of Data WG with Model Registry and Spark Operator projects, given that they are already part of Kubeflow ecosystem ? |
|
My preference is to include Feast in this working group. Feast now supports RAG (alpha) and I believe that would help boost Kubeflow's place in the GenAI conversation. Updating the PR on adding new projects to KF (https://github.com/kubeflow/community/pull/741/files) is on my to do list I've just been heads down working on the Milvus integration. Of course, I don't want to block anything but, based on my knowledge of the field and my conversations with startups, Kubeflow not having a solution for RAG (and for Data Scientists to feel empowered with RAG) is a reasonably large gap as most AI Applications rely on some form of RAG. |
|
Given the KSC is being decided this weekend, can we wait until next week? |
I totally agree with you @franciscojavierarceo, but it will take time for Kubeflow Community to migrate Feast into Kubeflow Core Components, isn't ? No one stopping us to update Data WG charter once FEAST will be officially part of Kubeflow Core Components. Given that this Working Group will be already established, we don't need to create a new Working Group for that. |
Yeah, that sounds good. 👍 Can Feast be included in the WG before it is officially part of the Core Components (since it's an add-on)? |
The main concern that I see is that Working Groups don't maintain add-on components since they live outside of Kubeflow organization. Given the limited number of active contributors, working groups were created to maintain components living under Kubeflow GitHub organization. KServe is an exception given that this project was part of Kubeflow GitHub org before. As I mention before, in my personal opinion, is that we remove the concept of add-ons to not confuse our users. We even don't have any clear explanation what is add-ons in the WG governance docs: https://github.com/kubeflow/community/tree/master/wgs The only reference that I found is this proposal from @kubeflow/wg-manifests-leads to introduce contrib components: https://github.com/kubeflow/manifests/blob/master/proposals/20220926-contrib-component-guidelines.md |
|
It's important that Kubeflow Working Groups only try and manage components that we own as an organization. It would be inappropriate for us to claim ownership of external projects. Previously, we had the concept of a SIG (special interest group) for anything which was not directly maintaining one of the components of Kubeflow but which still had a community of users in Kubeflow. PS: we should still consider splitting Also, the concept of an "external add-ons" allows us to create an "ecosystem" of tools, beyond our own, for things that we don't have a competitor to, it's important we don't get rid of that concept. We should even expand it to include model frameworks (PyTorch, TensorFlow), and other things that we already integrate with. This lets us create formal pages on the website for them, and help new users better understand our ecosystem. |
💯 |
@thesuperzapper Shouldn't this be a responsibility of distributions to allow users to deploy these additional tools on top of Kubeflow Core Components ? If Kubeflow Core Component needs to have integration with 3rd party tools, they can always have dedicated page in their documentation. |
I'm following up on action item: raise WG proposal to Kubeflow per yesterday's Model Registry meeting (recording timestamp).
As discussed in KF community meeting.
Main links:
👉 I'm starting to raise a draft PR in order to "seed/bootstrap" the work in raising the request to form the WG--using a draft PR give us a branch we can collaborate on between stakeholders @andreyvelich @Tomcli @dhirajsb @rimolive
This also give us a medium we can keeps-tab-on so to report back on progress during Tuesdays' community plenary meetings, wdyt?