Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialize smalldata to a blob-less entry in CGDFG #509

Closed
dehann opened this issue Jun 17, 2020 · 6 comments · Fixed by #962
Closed

Serialize smalldata to a blob-less entry in CGDFG #509

dehann opened this issue Jun 17, 2020 · 6 comments · Fixed by #962
Labels
data: entry=>blob Previously bigdata, suggested over `smalldata` designdecision enhancement New feature or request
Milestone

Comments

@dehann
Copy link
Member

dehann commented Jun 17, 2020

Quirky decision, how about dropping the separate smalldata container, and only maintaining the Data plumbing but allow entries that do not have an associated Blob? We can flag the Entry object as either having a Blob or not (as immutable field in an entry)?

We can use the Entry.description::String field to store a JSONed string that represents the Smalldata interface, and limit its size...

Smalldata functions are then fuss-free wrappers around entries without worrying about what data types, MIME, Entry or Blob.

(bit like Macklemore, bought a broken keyboard): I have a short string, please store the short string...

cc @GearsAD @Affie

Related #403

@dehann dehann added enhancement New feature or request designdecision data: entry=>blob Previously bigdata, suggested over `smalldata` labels Jun 17, 2020
@dehann dehann added this to the v0.9.0 milestone Jun 17, 2020
@dehann dehann changed the title Smalldata, how about fake Entry as wrapper? Smalldata, how about using Blob-less Entry wrapper? Jun 17, 2020
@GearsAD
Copy link
Collaborator

GearsAD commented Jun 17, 2020

Not a bad idea, I'm about to separate out the big data as separate nodes though, so that may make this difficult.

@GearsAD
Copy link
Collaborator

GearsAD commented Jun 17, 2020

I guess we could still do it, we would just have to collect the entries around the node, but it would be a bit more difficult than a JSON blob inside the actual node.

@Affie
Copy link
Member

Affie commented Jun 17, 2020

It fits in with why I originally created the AbstractBigDataEntry (renamed AbstractDataEntry) in our application.
I just stored the data in a custom structure directly. For example, tags were something like:

struct TagData <: AbstractDataEntry
    key::Symbol
    family::String
    tagids::Vector{Int}
    bTt
end

The problem was a concern that the payload could get too big. That's why the in-memory data store was created. I don't like and never used smalldata because of the String, but it can perhaps be consolidated as:

struct SmallData <: AbstractDataEntry
    key::Symbol
    data::String
   #OR 
    smallDataDict::Dict{String, String}
end

I guess smallDataDict::Dict{String, String} would work better as it would just be a JSON blob inside the satellite node if you implement it that way.

@dehann
Copy link
Member Author

dehann commented Jul 28, 2020

Affects JuliaRobotics/Caesar.jl#514

@GearsAD GearsAD changed the title Smalldata, how about using Blob-less Entry wrapper? Serialize smalldata to a blob-less entry in CGDFG Aug 12, 2020
@GearsAD GearsAD modified the milestones: v0.10.0, v0.11.0 Aug 12, 2020
@Affie Affie modified the milestones: v0.11.0, v0.12.0 Dec 4, 2020
@dehann dehann modified the milestones: v0.12.0, v0.13.0 Feb 13, 2021
@dehann dehann modified the milestones: v0.15.0, v0.0.x Jun 29, 2021
@dehann
Copy link
Member Author

dehann commented Feb 28, 2023

we could if we really wanted to store something like this using a combo of the BlobEntry.description, BlobEntry.metadata, and BlobEntry.mimeType fields.

@dehann
Copy link
Member Author

dehann commented Mar 1, 2023

Also,

julia> UUID("00000000-0000-0000-0000-000000000000")
UUID("00000000-0000-0000-0000-000000000000")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data: entry=>blob Previously bigdata, suggested over `smalldata` designdecision enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants