Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post view: move cursor pagination to separate library, add backward pagination to PostQuery #4320

Merged
merged 26 commits into from
Jan 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
3ec3ce1
stuff
dullbananas Dec 17, 2023
4ebf37d
Merge remote-tracking branch 'upstream/main' into i-love-jesus
dullbananas Dec 21, 2023
6144861
stuff
dullbananas Dec 22, 2023
281c61f
crates.io
dullbananas Dec 22, 2023
d41bdcc
Merge branch 'main' into i-love-jesus
dullbananas Dec 22, 2023
d858b87
Update up.sql
dullbananas Dec 22, 2023
ef80e5d
Rerun federation tests
dullbananas Dec 22, 2023
fb28321
Update post_view.rs
dullbananas Dec 27, 2023
4ed77aa
Merge branch 'main' into i-love-jesus
dullbananas Dec 27, 2023
9bc6534
Update post_view.rs
dullbananas Dec 27, 2023
2fe52ef
Merge branch 'main' into i-love-jesus
dullbananas Jan 4, 2024
027a9b1
Merge branch 'main' into i-love-jesus
dullbananas Jan 4, 2024
cf81ddf
Update up.sql
dullbananas Jan 5, 2024
b276cdc
Update utils.rs
dullbananas Jan 5, 2024
81958cf
Merge branch 'main' into i-love-jesus
dullbananas Jan 5, 2024
b8be479
Merge branch 'main' into i-love-jesus
dullbananas Jan 6, 2024
efe8a8f
Fix precision loss
dullbananas Jan 7, 2024
074fe2c
Merge branch 'main' into i-love-jesus
dullbananas Jan 13, 2024
a3844ec
Update up.sql
dullbananas Jan 13, 2024
b554218
Update down.sql
dullbananas Jan 13, 2024
c2721c3
remove unwrap
dullbananas Jan 13, 2024
bc77ae1
Merge branch 'main' into i-love-jesus
dullbananas Jan 18, 2024
74fdb4b
Merge branch 'main' into i-love-jesus
dessalines Jan 19, 2024
0424bf6
Update post_view.rs
dullbananas Jan 19, 2024
e0418d1
Merge branch 'main' into i-love-jesus
dullbananas Jan 23, 2024
f06308c
Merge branch 'main' into i-love-jesus
dessalines Jan 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ tokio-postgres = "0.7.10"
tokio-postgres-rustls = "0.10.0"
enum-map = "2.7"
moka = { version = "0.12.1", features = ["future"] }
i-love-jesus = { version = "0.1.0" }
clap = { version = "4.4.11", features = ["derive"] }
pretty_assertions = "1.4.0"

Expand Down
2 changes: 2 additions & 0 deletions crates/db_schema/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ full = [
"tokio-postgres",
"tokio-postgres-rustls",
"rustls",
"i-love-jesus",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably be renamed to something more descriptive? like diesel-tuple-pagination maybe. Also not sure what dessalines/nutomic think of depending on an external library like this, since it does seem fairly specific to lemmy.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will not rename my library

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just don't like the extra complication cursor pagination requires in general, with all of these pre-fetching of bounds, macros, etc. It makes post_view.rs so much harder to read and understand even to me.

Every example I can find online of cursor pagination is where you are sorting based off a single id or time column, rather than multiple filter and sort columns like we have. Its unfortunate that limit and offset are such a performance burden, since they require none of this complication, but I spose there's no way around it.

Whether that complicated cursor logic lives in lemmy, or elsewhere where it can be used by other projects, doesn't matter too much to me, but the latter seems better if it helps other diesel projects besides ours. We're dependent on a lot of diesel libraries that aren't made by its maintainer.

As far as the name, it doesn't bother me personally, but I could see other people not wanting to use it because of the name. For limited-scope libraries it makes sense to name the thing what it does, and it seems like a lot of other diesel projects are named things like diesel-.... or ...-diesel .

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarification, the prefetching is unrelated to the cursor pagination - I just added it in the same PR because it was a convenient related change. It's a performance boost for the post query in general with the "subscribed" filter, even for the first page where the pagination method is irrelevant. For the page-based pagination the same method could in fact be used for both sides of the page so the db doesn't have to join everything while trying to find the page start and end.

]

[dependencies]
Expand Down Expand Up @@ -76,6 +77,7 @@ tokio-postgres = { workspace = true, optional = true }
tokio-postgres-rustls = { workspace = true, optional = true }
rustls = { workspace = true, optional = true }
uuid = { workspace = true, features = ["v4"] }
i-love-jesus = { workspace = true, optional = true }
anyhow = { workspace = true }

[dev-dependencies]
Expand Down
12 changes: 11 additions & 1 deletion crates/db_schema/src/aggregates/structs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ use crate::schema::{
site_aggregates,
};
use chrono::{DateTime, Utc};
#[cfg(feature = "full")]
use i_love_jesus::CursorKeysModule;
use serde::{Deserialize, Serialize};
#[cfg(feature = "full")]
use ts_rs::TS;
Expand Down Expand Up @@ -93,13 +95,21 @@ pub struct PersonAggregates {
#[derive(PartialEq, Debug, Serialize, Deserialize, Clone)]
#[cfg_attr(
feature = "full",
derive(Queryable, Selectable, Associations, Identifiable, TS)
derive(
Queryable,
Selectable,
Associations,
Identifiable,
TS,
CursorKeysModule
)
)]
#[cfg_attr(feature = "full", diesel(table_name = post_aggregates))]
#[cfg_attr(feature = "full", diesel(belongs_to(crate::source::post::Post)))]
#[cfg_attr(feature = "full", diesel(primary_key(post_id)))]
#[cfg_attr(feature = "full", diesel(check_for_backend(diesel::pg::Pg)))]
#[cfg_attr(feature = "full", ts(export))]
#[cfg_attr(feature = "full", cursor_keys_module(name = post_aggregates_keys))]
/// Aggregate data for a post.
pub struct PostAggregates {
pub post_id: PostId,
Expand Down
24 changes: 23 additions & 1 deletion crates/db_schema/src/utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ use diesel::{
query_dsl::methods::LimitDsl,
result::{ConnectionError, ConnectionResult, Error as DieselError, Error::QueryBuilderError},
serialize::{Output, ToSql},
sql_types::{Text, Timestamptz},
sql_types::{self, Text, Timestamptz},
IntoSql,
PgConnection,
};
Expand All @@ -32,6 +32,7 @@ use diesel_async::{
};
use diesel_migrations::EmbeddedMigrations;
use futures_util::{future::BoxFuture, Future, FutureExt};
use i_love_jesus::CursorKey;
use lemmy_utils::{
error::{LemmyError, LemmyErrorExt, LemmyErrorType},
settings::SETTINGS,
Expand Down Expand Up @@ -153,6 +154,25 @@ macro_rules! try_join_with_pool {
}};
}

pub struct ReverseTimestampKey<K>(pub K);

impl<K, C> CursorKey<C> for ReverseTimestampKey<K>
where
K: CursorKey<C, SqlType = Timestamptz>,
{
type SqlType = sql_types::BigInt;
type CursorValue = functions::reverse_timestamp_sort::HelperType<K::CursorValue>;
type SqlValue = functions::reverse_timestamp_sort::HelperType<K::SqlValue>;

fn get_cursor_value(cursor: &C) -> Self::CursorValue {
functions::reverse_timestamp_sort(K::get_cursor_value(cursor))
}

fn get_sql_value() -> Self::SqlValue {
functions::reverse_timestamp_sort(K::get_sql_value())
}
}

/// Includes an SQL comment before `T`, which can be used to label auto_explain output
#[derive(QueryId)]
pub struct Commented<T> {
Expand Down Expand Up @@ -424,6 +444,8 @@ pub mod functions {
fn controversy_rank(upvotes: BigInt, downvotes: BigInt, score: BigInt) -> Double;
}

sql_function!(fn reverse_timestamp_sort(time: Timestamptz) -> BigInt);

sql_function!(fn lower(x: Text) -> Text);

// really this function is variadic, this just adds the two-argument version
Expand Down
2 changes: 2 additions & 0 deletions crates/db_views/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ full = [
"tracing",
"ts-rs",
"actix-web",
"i-love-jesus",
"lemmy_db_schema/full",
]

Expand All @@ -37,6 +38,7 @@ serde_with = { workspace = true }
tracing = { workspace = true, optional = true }
ts-rs = { workspace = true, optional = true }
actix-web = { workspace = true, optional = true }
i-love-jesus = { workspace = true, optional = true }

[dev-dependencies]
serial_test = { workspace = true }
Expand Down
Loading