Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add constructor for Tablechanges to get a CDF scan #459

Draft
wants to merge 46 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
c7913dc
Move log segment into separate module
OussamaSaoudi-db Oct 28, 2024
6b331ac
Fix tests, make fields pub
OussamaSaoudi-db Oct 28, 2024
f1f9886
Improve comments
OussamaSaoudi-db Oct 28, 2024
8122113
Remove table changes
OussamaSaoudi-db Oct 28, 2024
46185ae
Merge branch 'main' into snapshot_cleanup
OussamaSaoudi-db Oct 28, 2024
471a858
change visibility
OussamaSaoudi-db Oct 28, 2024
4ac35da
Merge branch 'main' into snapshot_cleanup
OussamaSaoudi-db Nov 6, 2024
6297805
Merge branch 'main' into snapshot_cleanup
OussamaSaoudi-db Nov 6, 2024
e569719
Merge remote-tracking branch 'refs/remotes/origin/snapshot_cleanup' i…
OussamaSaoudi-db Nov 6, 2024
5edf4db
Merge branch 'main' into snapshot_cleanup
OussamaSaoudi-db Nov 6, 2024
0b8463a
Remove old log segment
OussamaSaoudi-db Nov 6, 2024
5300a7b
fix failing tests
OussamaSaoudi-db Nov 6, 2024
81d0de0
Get rid of warnings
OussamaSaoudi-db Nov 6, 2024
6b85932
Fix failing tests
OussamaSaoudi-db Nov 6, 2024
1384ea3
Apply suggestions from code review
OussamaSaoudi-db Nov 6, 2024
0182326
Address more pr comments
OussamaSaoudi-db Nov 6, 2024
d053a77
fix imports
OussamaSaoudi-db Nov 6, 2024
52f57e5
rebase onto git changes
OussamaSaoudi-db Nov 6, 2024
aa6c9f4
address nits
OussamaSaoudi-db Nov 6, 2024
dca491c
fix visibility issue
OussamaSaoudi-db Nov 6, 2024
bf5cdd4
Use LogSegmentBuilder
OussamaSaoudi-db Oct 30, 2024
748bab9
Introduce start and end versions
OussamaSaoudi-db Oct 30, 2024
2a6eb3e
Remove old code
OussamaSaoudi-db Nov 1, 2024
841f17f
Fix failing tests
OussamaSaoudi-db Nov 1, 2024
1ce29d8
Most up to date logsegment
OussamaSaoudi-db Nov 6, 2024
a2f9810
Fix failing test and remove unnecessary code
OussamaSaoudi-db Nov 6, 2024
5f7a680
remove table changes from this commit
OussamaSaoudi-db Nov 6, 2024
6d8e35f
remove table_changes
OussamaSaoudi-db Nov 6, 2024
ea7029a
Fix failing tests
OussamaSaoudi-db Nov 1, 2024
9464d4e
Basic Table Changes construction
OussamaSaoudi-db Nov 1, 2024
f320416
Add beginning of the Scannable trait
OussamaSaoudi-db Nov 1, 2024
ecd6cd7
table_changes constructor
OussamaSaoudi-db Nov 4, 2024
585c663
Revert "Add beginning of the Scannable trait"
OussamaSaoudi-db Nov 4, 2024
b2af9e6
Constructor
OussamaSaoudi-db Nov 4, 2024
6e814cf
TableCHangesScan copied from Scan
OussamaSaoudi-db Nov 4, 2024
412f551
CHange log segment builder's names
OussamaSaoudi-db Nov 4, 2024
1a6a630
Basic iteration for CDF
OussamaSaoudi-db Nov 5, 2024
de206c7
Read path until scanfile
OussamaSaoudi-db Nov 6, 2024
298c70c
Remove non-constructor code
OussamaSaoudi-db Nov 7, 2024
3a043b0
Remove change_data testing file
OussamaSaoudi-db Nov 7, 2024
759f537
Remove unnecessary changes
OussamaSaoudi-db Nov 7, 2024
42896ec
Fix comments
OussamaSaoudi-db Nov 7, 2024
c558715
Add preliminary test for table changes
OussamaSaoudi-db Nov 7, 2024
44db9ea
fix failing test
OussamaSaoudi-db Nov 7, 2024
ab672e6
Add test for valid and invalid cdf ranges
OussamaSaoudi-db Nov 7, 2024
6241ed1
address pr comments
OussamaSaoudi-db Nov 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions kernel/src/actions/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ pub(crate) const METADATA_NAME: &str = "metaData";
pub(crate) const PROTOCOL_NAME: &str = "protocol";
pub(crate) const SET_TRANSACTION_NAME: &str = "txn";
pub(crate) const COMMIT_INFO_NAME: &str = "commitInfo";
#[allow(unused)]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume when we're ready to go here this allow won't be needed?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I'll go ahead and add a comment

pub(crate) const CDC_NAME: &str = "cdc";

static LOG_ADD_SCHEMA: LazyLock<SchemaRef> =
LazyLock::new(|| StructType::new([Option::<Add>::get_struct_field(ADD_NAME)]).into());
Expand Down
5 changes: 4 additions & 1 deletion kernel/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ use std::{
str::Utf8Error,
};

use crate::schema::DataType;
use crate::{schema::DataType, Version};

/// A [`std::result::Result`] that has the kernel [`Error`] as the error variant
pub type DeltaResult<T, E = Error> = std::result::Result<T, E>;
Expand Down Expand Up @@ -171,6 +171,9 @@ pub enum Error {
/// The file already exists at the path, prohibiting a non-overwrite write
#[error("File already exists: {0}")]
FileAlreadyExists(String),

#[error("Change data feed is disabled in range: {0} to {}", _1.map_or("end".into(), |ver| ver.to_string()))]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error message has some confusing syntax, but I think we should clarify here. The error is not because it's disabled in the range (or the entire range perhaps), there is nothing wrong with doing that but that CDF doesn't allow you to disable it anywhere in the range and still work.

TableChangesDisabled(Version, Option<Version>),
}

// Convenience constructors for Error types that take a String argument
Expand Down
2 changes: 2 additions & 0 deletions kernel/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,12 @@ pub mod path;
#[cfg(not(feature = "developer-visibility"))]
pub(crate) mod path;

pub mod log_segment;
pub mod scan;
pub mod schema;
pub mod snapshot;
pub mod table;
pub mod table_changes;
pub mod transaction;
pub(crate) mod utils;

Expand Down
Loading
Loading