Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: move second-tier MVs into dbt project (FC-0033) #34

Merged
merged 1 commit into from
Dec 18, 2023

Conversation

SoryRawyer
Copy link
Contributor

@SoryRawyer SoryRawyer commented Nov 16, 2023

Now that dbt-clickhouse supports materialized views, this PR moves MV creation and maintenance to the Aspects dbt project. MV definitions are included for:

  • xapi_events_all_parsed
  • completion_events
  • enrollment_events
  • forum_events
  • grading_events
  • problem_events
  • video_playback_events

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Nov 16, 2023
@openedx-webhooks
Copy link

openedx-webhooks commented Nov 16, 2023

Thanks for the pull request, @SoryRawyer! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

  • supporting documentation
  • Open edX discussion forum threads
  • timeline information ("this must be merged by XX date", and why that is)
  • partner information ("this is a course on edx.org")
  • any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

@SoryRawyer SoryRawyer marked this pull request as ready for review December 8, 2023 15:14
@SoryRawyer
Copy link
Contributor Author

@bmtcril this is ready for review. The dbt version upgrades seem to work without issue and the MVs seem to match what I have with the alembic-created tables.

@bmtcril
Copy link
Contributor

bmtcril commented Dec 12, 2023

Sorry I was waiting for this to merge, can you pull in these new ones in as well? Then we should be able to make the changeover.
openedx/tutor-contrib-aspects#529

@SoryRawyer
Copy link
Contributor Author

Just to be sure, are you talking about the user profile table? I don't see any views in that PR and I don't think dictionaries are supported by dbt-clickhouse.

@SoryRawyer
Copy link
Contributor Author

Sorry, I just noticed that the user profile table is mentioned in this issue as being included in the alembic migrations. I think the only MV in our migrations that isn't listed in that issue is the most recent blocks MV. Do you want me to include that in this PR?

@bmtcril
Copy link
Contributor

bmtcril commented Dec 13, 2023

Hmmm yeah I think we should move everything we can that's not a "base needed to write to" table. If you could confirm about the dictionaries that would be great. Seems like another thing ripe for an upstream PR.

@bmtcril
Copy link
Contributor

bmtcril commented Dec 15, 2023

(FWIW I don't think the dictionaries thing is blocking this at all, just curious about it for future work)

@SoryRawyer
Copy link
Contributor Author

Apologies for the delay - dictionaries aren't supported at the moment, but that does seem like something we could contribute to dbt-clickhouse.

@Ian2012
Copy link
Contributor

Ian2012 commented Dec 15, 2023

@SoryRawyer can you handle Brian comment around the Json extract float, and squash the commits?

@SoryRawyer
Copy link
Contributor Author

@bmtcril were you imagining the xapi_events_all_parsed being part of this PR as well? I think that might be the last MV in the alembic migrations.

@bmtcril
Copy link
Contributor

bmtcril commented Dec 15, 2023

@SoryRawyer sure, let's move it 👍

This delegates all MV creation and maintenance to the dbt project now
that materialized views in ClickHouse are supported by the dbt adapter.
@bmtcril bmtcril merged commit d74b350 into openedx:main Dec 18, 2023
2 checks passed
@openedx-webhooks
Copy link

@SoryRawyer 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
open-source-contribution PR author is not from Axim or 2U
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants