Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request - Add bulkload edits to flat_edit_history #43

Open
camwebb opened this issue Jul 9, 2024 · 2 comments
Open

Feature Request - Add bulkload edits to flat_edit_history #43

camwebb opened this issue Jul 9, 2024 · 2 comments
Labels
Enhancement I think this would make Arctos even awesomer! Priority-Normal (Not urgent) Normal because this needs to get done but not immediately.

Comments

@camwebb
Copy link

camwebb commented Jul 9, 2024

As far as I understand (via several bulkload experiments), edits to records made via the bulkloaders are not recorded in flat_edit_history, but only edits via the GUI. I use flat_edit_history frequently to monitor progress of assistants but can't see their edit activity via the bulkloaders. Not sure how much work it would be to add this capability, but here's a request for it.

Alternatively, some SQL to generate reports about bulkloader would suffice. Searching on "flat_edit_history" in ArctosDB repos I have not been able to discover how that table is created.

Thanks

@camwebb camwebb added Priority-Normal (Not urgent) Normal because this needs to get done but not immediately. Enhancement I think this would make Arctos even awesomer! labels Jul 9, 2024
@dustymc
Copy link
Contributor

dustymc commented Jul 9, 2024

Your timing is impeccable. I've been flailing around (ineffectively) with authentication and such while failing to address @acdoll 's basically identical issue for a while. A bot lead to https://github.com/ArctosDB/PG/issues/29 (same thing for a different class of user, sorta), I haven't yet found any fatal flaws (plenty of minor expense and quirks; defaults don't work well so triggers are necessary, back-to-back updates all merge together, etc.), and I think that's going to pave the way to this.

This does require work for every table, it's not the 'figure it out from the environment' global/seamless solution I've been seeking. Please let me know if you've got something to prioritize.

flat_edit_history ... discover how that table is created

Generally: The scope of this is 'something involving {table} changed' - I can't tell if a user added and removed the . on the end of an attribute remark 500 times, or added 500 high-quality determinations which have completely changed the nature of the record. The mechanism should be adaptable to more detail (I know what changed for locality, where this is developing, because I use it to I save the olds), but that would involve - uhh, something, probably a proposal, some storage, etc.

Specifically: I grab the node name that's requested a flat update when I'm refreshing the cache. (Which magnifies the above caution - those 500 high-quality updates could be completely missed by someone else updating some other node at the right - or wrong, depending on your view! - time.)

Very tentatively going next task, if I don't melt anything in test in the next couple days and next release survives exposure to reality, I might know how to do this.

Also ping https://github.com/ArctosDB/internal/issues/326, I've changed a LOT of stuff and having the ability for someone besides me to test/look for what I've broken would be pretty sweet.

@camwebb
Copy link
Author

camwebb commented Jul 12, 2024

I don't understand all the internal complexity and load bottlenecks, but would it not be possible simply to add a edit_timestamp and edited_by_user field on every major table that someone might want to track? If the bulkloaders always know who the submitting user was they could pass this to the table.

@dustymc dustymc transferred this issue from ArctosDB/arctos Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement I think this would make Arctos even awesomer! Priority-Normal (Not urgent) Normal because this needs to get done but not immediately.
Projects
None yet
Development

No branches or pull requests

2 participants