Automate Slack user groups + channels creation and management

[smoya]

Reason/Context

I think having user groups in Slack like the @coc_committee and others is really useful because:

• People can mention all the members without having to know them.
• People can know the members of such group by just checking the user list of the group.

The issue with that is that at this moment everything is being managed manually by either @thulieblack or @derberg. I feel it is not a big deal, but imagine if we want that Slack user group for the TSC members, a really big list of users that is alive and it changes from time to time, with new additions or rather deletions.

The same happens with channels. At this moment "nobody" can create new channels (which is intended); those should be suggested to @thulieblack or @derberg and if makes sense (sometimes under the voting process) will create them.
The point in this case is that the process is a bit not pretty transparent to the community as there is no chance to discuss about why name A or B, etc etc. Also, there is no official place to discuss about creation of new channels, temporary or not.

In my opinion, this could be solved by automating the creation of those user groups and channels. Having a process where people can submit a simple PR for creating a new user group (could be just in a JSON format), adding a new member, creating a new channel, or renaming it, etc.

Description

We can use the following Terraform provider that looks pretty simple. It allows to create channels, manage them, and the same with user groups.
In the example of the TSC user group (if we want it), we could even fill the data of the group by using the TSC_MEMBERS.json as source. That Terraform could run every time that list changes as well.

Obviously, this is not a high priority one IMHO but still nice to have. I believe this project could be a cool Bounty Program issue.

cc @thulieblack @derberg @aeworxet

[aeworxet]

I would go further and propose groups consisting of one person ed, cmanager etc. (names discussable,) so if a new person is elected, they could be added to such a group and automatically receive all permissions and privileges accorded to that group (meaning 'role'.)

It also seems to me that it's time to introduce new types of Bounty Issues.
Issue asyncapi/spec#957 looks to me as devops (because CI,) and this one I'd shoot as infrastructure (because Slack's tuning is 'DevOps' for the internal Slack's team that CI/CDs it as a product, but AsyncAPI is the user of Slack which it uses simply for communication, or an infrastructural entity.) But it can be a devops too, of course, if I'm missing something.

[derberg]

sounds good, just I cannot commit to have time for review and implementation. Not is close future

[Shurtu-gal]

I can help out here 😁. We can have a single CI/CD workflow which looks for changes in files of one folder and runs terraform accordingly.

[Amzani]

It would be great to have an automation working group :)

[Shurtu-gal]

@smoya maybe could be submitted here, https://github.com/orgs/asyncapi/discussions/963#discussioncomment-8628690

[smoya]

@smoya maybe could be submitted here, https://github.com/orgs/asyncapi/discussions/963#discussioncomment-8628690

I believe a maintainer of this repo should do it if they consider. I think could be a fit indeed

cc @alequetzalli @thulieblack @derberg

[quetzalliwrites]

Sorry, this feels a little beyond my scope of knowledge, feeling a little lost here 😁

[derberg]

I'm having to many thing on my plate atm. I can submit this issue to bounty but only:

• if @Shurtu-gal will work on it as he is already very experienced with our CI/CD and I know do not require any mentoring
• I can do final review and merge but I really cannot be involved in discussion about technicalities, how the best do the things, how to get that integrated, etc. I can commit to review of GitHub Action technicalities

So if anyone can help with discussion and probably participation in testing, that would be great. Otherwise that will have to wait for Q3

[smoya]

if @Shurtu-gal will work on it as he is already very experienced with our CI/CD and I know do not require any mentoring

+1. I also believe @Shurtu-gal won't need that much mentoring atm.

I can do final review and merge but I really cannot be involved in discussion about technicalities, how the best do the things, how to get that integrated, etc. I can commit to review of GitHub Action technicalities

Additionally, I would cut the requirements on the most simple features, leaving @aeworxet suggestion for a future improvement.

I can be that person. Eventually asking @derberg for GH Actions advise/review. 👍 However, I think only maintainers can be mentors? Didn't read that explicitly in the rules, but seems to suggest it. cc @aeworxet sorry for bothering you mate

[derberg]

you can be mentor as long as you have a maintainer that is aware and can approve the work 😄

[derberg]

Additionally, I would cut the requirements on the most simple features, leaving @aeworxet #1072 (comment) for a future improvement.

@smoya you mean it because there is no concept yet of storing the info about it somewhere? like in case of working groups, we have all members listed in working groups yaml?

[smoya]

Additionally, I would cut the requirements on the most simple features, leaving @aeworxet #1072 (comment) for a future improvement.

@smoya you mean it because there is no concept yet of storing the info about it somewhere? like in case of working groups, we have all members listed in working groups yaml?

No, not concerned about that. We have that issue with channels as well anyway, so the same solution will be applied for users. Just to cut the scope somewhere.

[aeworxet]

However, I think only maintainers can be mentors? Didn't read that explicitly in the rules, but seems to suggest it.

No. No. No. There are no mentors in the Bounty Program; there are only AsyncAPI Maintainers who are responsible for the resolution of the Bounty Issue from the AsyncAPI's side.
To what extent an AsyncAPI Maintainer stretches their responsibility is their own business and good will, but by default it is assumed that the Bounty Program Participants are ALREADY qualified enough to perform with little to no supervision, and 'Mentors' as such exist only in 'Mentorship' programs.

[aeworxet]

Is there a final decision on whether this issue will be submitted for the Bounty Program 2024-Q2?

[smoya]

So if anyone can help with discussion and probably participation in testing, that would be great. Otherwise that will have to wait for Q3

I'm in on this. @derberg seems to agree as well. So I don't see why this shouldn't be there. @derberg do you want me to submit it?

[aeworxet]

Please keep the chain of responsibility in accordance with CODEOWNERS.

[smoya]

Please keep the chain of responsibility in accordance with CODEOWNERS.

Sorry but It is so hard to me to have in mind all the tons of rules of the bounty, plus other programs. I guess you mean @derberg or any other maintainer should do the submission.

[derberg]

@smoya you have to submit. I'm just merging, but you offered to run work with contributor

[smoya]

@smoya you have to submit. I'm just merging, but you offered to run work with contributor

It seems I can't due to the strict rules of the Bounty Program:

by one of the GitHub users whose GitHub handles are written in the CODEOWNERS file of the given repository (one of the AsyncAPI Maintainers of this repository

[derberg]

then we need more flexible rule as in the past we had non codeowners submitting issues and it was all fine if maintainers of give repo were aware and happy to help merging

[aeworxet]

Changed the Bounty Program Rules to

Bounty Issues ... are submitted ... by an AsyncAPI Maintainer who will be responsible for the resolution of the given Bounty Issue from the AsyncAPI's side

Implying that any AsyncAPI Maintainer can submit any issue from any repository as a Bounty Issue (still two issues per repository, in order not to bite more than you can chew,) but they will take full ownership over this Bounty Issue on all stages, from submission to completion.

And now there is a quite well-shaped system:

• On Bounty Issue's submission:
  • AsyncAPI Maintainer who will be responsible for the resolution of the given Bounty Issue from the AsyncAPI's side
• During Bounty Issue's resolution:
  • AsyncAPI Maintainer who is responsible for the resolution of the given Bounty Issue from the AsyncAPI's side
• After Bounty Issue's completion:
  • AsyncAPI Maintainer who was responsible for the resolution of the given Bounty Issue from the AsyncAPI's side

Meaning that now @smoya can submit this issue as a Bounty Issue, but accompany it on all stages.

[smoya]

Meaning that now @smoya can submit this issue as a Bounty Issue, but accompany it on all stages.

It's a thing now 🎉 https://github.com/orgs/asyncapi/discussions/963#discussioncomment-8799664
Btw, used devops as its type.

[aeworxet]

Introduced in the Bounty Program Rules a new type of DevOps (Capital D, Capital O.)

[aeworxet]

Bounty Issue's service comment

Text labels: bounty/2024-Q2, bounty/medium, bounty/devops
First assignment to third-party contributors: 2024-03-22 00:00:00 UTC+12:00
End Of Life: 2024-08-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

[aeworxet]

Bounty Issue's Timeline

Complexity Level	Assignment date (by GitHub)	Start date (by BP rules)	End date (by BP rules)	Draft PR submission	Final PR submission	Final PR merge
Medium	2024-03-19	2024-04-01	2024-05-10	2024-04-12	2024-04-26	2024-05-10

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

[asyncapi-bot]

❌ @asyncapi-bot is not authorized to use the Bounty Program's commands.
These commands can only be used by members of the Bounty Team.

[asyncapi-bot]

❌ @asyncapi-bot is not authorized to use the Bounty Program's commands.
These commands can only be used by members of the Bounty Team.

[asyncapi-bot]

❌ @asyncapi-bot is not authorized to use the Bounty Program's commands.
These commands can only be used by members of the Bounty Team.

[asyncapi-bot]

❌ @asyncapi-bot is not authorized to use the Bounty Program's commands.
These commands can only be used by members of the Bounty Team.

[fmvilas]

I disabled the Bounty commands workflow for now so it stops spamming. cc @aeworxet

[smoya]

Testing phase is now done. More info in #1131 (comment)

The PR is now waiting for the final aproval from @derberg and @thulieblack.

cc @aeworxet

[smoya]

Reopening as It got merged but It shouldn't. Still configuration is missing: #1131 (comment)

[smoya]

We are dealing with Slack token and permission issues. We had to ask AsyncAPI Slack administrators to do the following work and they did some but still waiting for some more extra burocracy needed regarding Slack app installation processes. All of this is being followed via a closed Slack DM group due to the sensitive data (basically just in case we leak any token).

Delays are something we couldn't really influence in. Unexpected stuff that can happen 🤷

[smoya]

Now that I have a proper token (thanks @thulieblack and @derberg), I ran the terraform apply but I got the following errors:

╷
│ Error: couldn't set conversation purpose This is the channel for the Maintainers Growth Working Group.: not_in_channel
│
│   with module.channels.slack_conversation.wg_channels["Maintainers Growth"],
│   on channels/channels.tf line 70, in resource "slack_conversation" "wg_channels":
│   70: resource "slack_conversation" "wg_channels" {
│
╵
╷
│ Error: couldn't set conversation purpose The Conference Coordination Working Group aims to plan, manage, and create a seamless way to enhance the conference experience. GitHub Project: https://github.com/orgs/asyncapi/projects/43/views/2: not_in_channel
│
│   with module.channels.slack_conversation.wg_channels["Conference Coordination"],
│   on channels/channels.tf line 70, in resource "slack_conversation" "wg_channels":
│   70: resource "slack_conversation" "wg_channels" {
│

The following scopes are in place:

I'll keep investigating anyway. @Shurtu-gal did you have this problem in your own Slack workspace? What is missing?

BTW, there is an issue that got stale in the TF provider github repo... pablovarela/terraform-provider-slack#161 🤷

[smoya]

It seems the only solution to that is to invite the bot app into all managed channels by writing: /invite @Eve and Chan. Worth to add this into the README @Shurtu-gal.

[smoya]

@Shurtu-gal Another issue I'm facing now:

╷
│ Error: couldn't get usergroups: slack rate limit exceeded, retry after 30s
│
│   with module.groups.slack_usergroup.groups["TSC Members"],
│   on groups/groups.tf line 36, in resource "slack_usergroup" "groups":
│   36: resource "slack_usergroup" "groups" {
│

Not only with that group but with tons. And is very random, because even after waiting such seconds, I still have the issue.
The problem is that I can't run the apply because of this being hit all the time. The current state is not completed.

[Shurtu-gal]

Nope, I didn't face this issue but this maybe due to me not having many repository leading to less no. of groups.

[smoya]

The rate limit issue is a blocker for this automation. WIthout solving it, we are not even able to generate a terraform plan via terraform plan.

There was once a PR on the TF provider, but it got stale. See pablovarela/terraform-provider-slack#72.

I tried modifying the parallelism with terraform plan -parallelism=1 but got no success.
I see no other option than contributing to the provider or making our own fork...

I don't know how to handle this regarding the bounty, because the work "is done" but not working. Any idea @asyncapi/bounty_team ?

[aeworxet]

On the one hand, the Bounty Program Participant fulfilled their obligations to perform the work well, correctly, and in full.
On the other hand, the result of work does not perform due to a third-party infrastructure's issue. This issue is beyond control, it could not be foreseen at the time of the conclusion of the agreement, and it is not possible to avoid it or overcome.
So it's a classic force majeure.

I view this situation as similar to the one when the plumber has built a fully functioning water supply system but the inlet tap is turned off. Providing water supply is not the plumber's responsibility.
So, since apples are not related to oranges, I do not see any factual reason why the Bounty Program Participant, after full completion of the assignement, should be denied a reward, while the AsyncAPI Maintainer, who was responsible for the resolution of this Bounty Issue from the AsyncAPI's side, should continue to search for a solution on their own.

@derberg, @thulieblack ?

[smoya]

I do not see any factual reason why the Bounty Program Participant, after full completion of the assignement, should be denied a reward, while the AsyncAPI Maintainer, who was responsible for the resolution of this Bounty Issue from the AsyncAPI's side, should continue to search for a solution on their own.

I completely agree 👍

[derberg]

don't get the part about rate limit

https://api.slack.com/apis/rate-limits#:~:text=Limits%20when%20posting%20messages,that%20limit%20for%20short%20periods.

so which limit we hit and why to do a retry? we have the same with GitHub API and retry helps

[Shurtu-gal]

@derberg we are hitting the tier 2 limit.
https://api.slack.com/methods/usergroups.create

[smoya]

don't get the part about rate limit

The point is that I believe the TF provider is not optimized at all. I have the feeling this code is being executed per each managed Usergroup whenever TF wants to refresh its state: https://github.com/pablovarela/terraform-provider-slack/blob/master/slack/resource_usergroup.go#L108-L128, so potentially we are calling the usergroups.list API method on each usergroup we have.

I believe we could do some work on the provider repo (it's written in go, seems easy to read and understand at a glance) so we can implement some caching or whatever mechanism we decide. But in short term, I can't see how to fix it.

[smoya]

I opened a new follow up issue so we can work on a fix at some point (not for this bounty, maybe next one?) #1238

For me, this issue can be closed and considered done @aeworxet @Shurtu-gal

[derberg]

great work folks, imho work completed

we can improve validation and retry for timeouts, but the main thing is done and it is amazing work!!! Automation always rocks!

[aeworxet]

Bounty Issue Completed 🎉

@Shurtu-gal, please go to the AsyncAPI page on Open Collective and submit an invoice for USD 200.00 with the expense title Bounty community#1072, tag bounty, and full URL of this Bounty Issue in the description.