Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create - STATE #30

Open
5 of 8 tasks
alyhxn opened this issue Oct 11, 2024 · 22 comments
Open
5 of 8 tasks

Create - STATE #30

alyhxn opened this issue Oct 11, 2024 · 22 comments

Comments

@alyhxn
Copy link

alyhxn commented Oct 11, 2024

@todo

#19


  • Create - STATE
    • Implement base
    • Integrate into components
    • Providing secure data to components
    • Listen for changes and trigger relevant response
    • Store and access all data from DB
    • Improvements to better support data operations
    • Finalize
    • @outputSTATE_v0.0.?
@alyhxn
Copy link
Author

alyhxn commented Oct 17, 2024

Tasks - 2024.10.18

  • Update - STATE_v0.0.1
    • Implement the fallback management - 4h
    • Planning and discussions - 15h
    • Debugging - 2h
    • Logged tasks - 5min
    • Recoreded Worklog - 10min
    • @output 📦 STATE_v0.0.2

Worklog

worklog-238

@alyhxn
Copy link
Author

alyhxn commented Oct 23, 2024

Tasks - 2024.10.24

  • Update - STATE_v0.0.2
    • Updated fallback data structure - 1h
    • Changes due to change in data structure - 3h
    • Planning and discussion - 5h
    • Implementing level-1 override - 2h
    • Authored docs for fallbacks - 3h
    • Logged tasks - 5min
    • Recorded Worklog - 10min
    • @output 📦 STATE_v0.0.3
    • @output 📦 Fallback_docs

Worklog

worklog-239

@serapath
Copy link
Member

feedback 2024.10.24

for worklog 238 and 239

First, I feel the worklog video is missing a lot of things.

  1. It does not have a section about DB and you posted worklog238 on that issue too
  2. It does not have a section about parser and you posted worklog238 there too (Refactor - playproject #19 and Create - DB #31)
  3. It does not have a section to walk through the doc/... updates, but fallback docs are linked as an output.

I would be happy if you could structure future worklogs to cover all the outputs and changes you worked on. Either one after another, or if it doesnt fit into 5 minutes, then multiple worklog videos.

Having said that, here my FEEDBACK: 😺

STATE.js

snapshot
STATE.js should not require a hard coded snapshot.json file

status.module_index = {}
status.fallback_handlers = {}

graph_explorer

+1 for highlight color for currently focused entry
+1 for color highlighting duplicate names, it is a bit much though

  • could we just give duplicates a "pale grey font color"?
  • the original entry should not have any special color
  • when any pale grey duplicate is selected, it shows the highlight color for
    • the original entry (shortest path from root to entry)
    • and all its expanded duplicates

+1 for showing state when component instance is selected
+1 for showing javascript source code files

snapshot.json

Only in the theme widget, there should be special action buttons to:

  • export/dump the entire database into a snapshot.json file to download
  • import and populate the database from a user provided snapshot.json file

There needs to be a special admin... method to allow such an operation

require

The require function will be customized by us, but we have to implement it all without that.
Basically, once we patch the require function, we wont need to require the STATE module anymore. The lines

const STATE = requuire('STATE')
const statedb = STATE(__filename)

will be automatically prefixed into every module.

Apart from that, the main plan is to get rid of budo, but thats it for now.

fallback

yes, fallback system is really simple in its core idea, but quite challenging and complex to implement for sure.

fallback fallbacks
It is confusing that this fallback is called index.
Shouldnt there be a proper module name instead?
What is the file required? is it just "."? In that case, let that be the name instead of index.

idx
I agree that idx:1 doesnt tell anything, neither does type:1, but then again,
all it takes is scrolling in the same file, because fallback_instance and fallback_module
are both part of the same file, so it should be easy to figure out.
Also usually, the "default fallback data" does not include hundreds of instances or modules, but even then you can still figure it out. All it needs is a CTRL+F and seach for e.g. 1: and it will probably quickly find the key of the required module entry to say type: '...' and you know.

We could of course add the name of the module, but it makes everything more verbose as well.
I'm not convinced it improves anything. I feel unsure about idx though looking at it.

When i see idx, i would assume it is an index into an array or something.
I constantly have to remind myself, that this is the "module type" of that specific instance.

sub fallbacks
This seems fine.
A fallback override for a direct sub component makes sense.

Just for the sake and in case we dont have another good example, those
topnav entries could be a "button instance", so that we get the chance to try out
more complex fallback structures, but maybe we get that with contributors or projects already.

QUESTION:
I am watching the files and the video and every time you open e.g. a module.json or instance.json, i need to always think for which module or component is it.
If I'm inside a file and I watch the code, the fallback function gives me no insights,
apart from e.g. require('module.json').
Now i get it, if the instance of module fallback data is large, maybe we want the ability to
have a separate file for it, BUT - if it is as little data as in all your examples,
inlining the data makes it so much easier to quickly read what is going on imho.

That fallback is a function and doesn't really care how you structure your code, apart from
you having to return the "default data" is clear, so people can be creative how they use it.
I think for us having the data inside the code file itself is quicker for now.

Do you agree? And if so, could we try to change it and inline all fallback data and skip external fallback data like "module.json" or "instance.json" for now? :-)

prefix (153)
We need that in the localStorage.
Think of it as a process tree id.
We could technically have an option later to restart the entire app a second time
Or to start and modify the app under lets say 154.
We could then switch processes between 153 and 154 - both using different database data.
In order to be able to play with that, we need all data stored in such a way that it is it's own "process tree".

@alyhxn
Copy link
Author

alyhxn commented Oct 28, 2024

Tasks - 2024.10.28

  • Update - STATE_v0.0.3
    • Removed populate and added fetch_save - 15min
    • xtype removed from data - 6h
    • Now we are using module's ID instead of name, not using type + idx - 1h
    • Batch is now array - 10min
    • Version code updated - 1min
    • Code updated with replaceChildren - 1min
    • Updated subs as mentioned - 25min
    • Moved the content file to web - 1min
    • Updated the docs a bit - 30min
    • Moved default data inside the module - 30min
    • Implemented multi-level fallback - 5h
    • Discussion - 3h
    • Cleaned STATE.js code - 3h
    • Logged tasks - 10min
    • Recorded Worklog - 40min
    • @output 📦 STATE_v0.0.4

Worklog

worklog-240
worklog-240.2
worklog-240.3

@alyhxn
Copy link
Author

alyhxn commented Nov 2, 2024

Tasks - 2024.11.02

  • Update - STATE_v0.0.3
    • Unified module and instance fallbacks - 2h
    • Updated code accordingly - 2h
    • Logged tasks - 5min
    • @output 📦 STATE_v0.0.4

Feedback

  • Implemented the code as asked so not much to explain in a video.

@alyhxn
Copy link
Author

alyhxn commented Nov 7, 2024

Tasks - 2024.11.07

  • Update - STATE_v0.0.4
    • Code snippet - 2h
    • Discussion - 3h
    • Updated footer.js and header.js with the latest code fallback DS - 3h
    • Logged tasks - 5min
    • Recorded Worklog - 10min
    • @output 📦 STATE_v0.0.5

Worklog

worklog-242.2

@serapath
Copy link
Member

serapath commented Nov 8, 2024

feedback 2024.11.08

Lets compare our current and potentially new approach
Given: demo > app > foo > head > nav > menu > (btn | btm.small) > icon

1

// ----------------------------------------------------------------------------
// 1. make icon set its default fallback (=`FB_II`) using the `image.svg` in the above snippet
// icon.js
/***********
  NOW:
***********/
function FB_II () {
  return { 'image.svg': `<svg>▶️</svg>` }
}
/***********
  vs. NEW:
***********/
function FB_II () {
  return { 'image.svg': `<svg>▶️</svg>` }
}

This is the same, no changes.


2

// ----------------------------------------------------------------------------
// 3. make button override icon `image.svg`
// 2. set button default fallback (=`FB_IB1` + `FB_IB2`) to `label/size`
// btn.js
/***********
  NOW:
***********/
function FB_IB () {
  return {
    0: { subs: [1], data: { label: 'button', size: 'small' } },
    1: { type: 'icon', override: [override_icon] }
  }
}
function override_icon (data) {
  data[0].data['image.svg'] = `<svg>🧸</svg>`
  return data
}
/***********
  vs. NEW:
***********/
function FB_IB1 () {
  return {
    _: { icon: { 0: override } }, // icon === 'icon:0'
    'data.json': { label: 'button', size: 'large' }
  }
}
function FB_IB2 () {
  return {
    _: { icon: { 0: override } }, // icon === 'icon:0'
    'data.json': { label: 'button', size: 'small' }
  }
}
function override (data) {
  data['image.svg'] = `<svg>🧸</svg>`
  return data
}

This is different.

  1. the override data[0].data['image.svg'] = seems more verbose in current state
  2. your version does not overridethe normal "large" button, only the small one.
  3. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

3

// ----------------------------------------------------------------------------
// 5. make menu code require 2 button module instances, one for small button, one for normal button
// 4. make menu (=`FB_IM`) override button label + reset icon back to original from what button changed
// menu.js
/***********
  NOW:
***********/
function FB_MM () {
  return { 0: { subs: ['btn'] } }
}
function FB_IM () {
  return {
    0: { subs: ['1', '2'] },
    '1': { type: 'btn:small', override: [override_btn] },
    '2': { type: 'btn:normal', override: [override_btn] }
  }
}
function override_btn (data) {
  data.label = 'beep boop'
  Object.entries(data).forEach(([id, value]) => {
    if (value.type.includes('icon')) data[id].override = null
  })
}
/***********
  vs. NEW:
***********/
// FB_MM
function FB_IM () {
  return { _: { btn: { 0: override }, 'btn:1#small': { 0: override } } } // btn === 'btn:0'
}
function override (data) {
  data['data.json'].label = 'beep boop'
  data._['icon:0/0'] = ([icon]) => icon()
  return data
}
  1. the FB_IM in the new version is much smaller
  2. the override does not need to test for type 'icon', because its in the key
    • again it is also smaller in code
  3. data.label next to data[id] seems weird, because in button:
    • the label is part of the data[id=0] attribute, so this is confusing
  4. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

4

// ----------------------------------------------------------------------------
// app.js
// foo.js
// head.js
// ----------------------------------------------------------------------------
// 6. make nav (=`FB_IN`) undo the menu overide for button module instances
// nav.js
/***********
  NOW:
***********/
function FB_IN () {
  return {
    0: { subs: ['1'] },
    '1': { type: 'menu', override: [override_menu] }
  }
}
function override_menu (data) {
  Object.entries(data).forEach(([id, value]) => {
    if (value.type.includes('btn')) data[id].override = null
  })
  return data
}
/***********
  vs. NEW:
***********/
function FB_IN () {
  return { _: { menu: { 0: override } } } // menu === 'menu:0'
}
function override (data) {
  data._['btn:0/0'] = ([btn]) => btn()
  data._['btn:1#small/0'] = ([btn]) => btn()
  return data
}
  1. again, FB_IN in new version is shorter
  2. override in new version does not need to loop and check for type
  3. also your check might fail if there is a module called "debtnote"
    • maybe a component card to visualize the debt of a user?
    • so more a heuristic than a very exact way to do it
  4. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

5

// ----------------------------------------------------------------------------
// 7. make demo (=`FB_MD`) redo the menu override
// demo.js
/***********
  NOW:
***********/
function FB_MD () {
  return {
    0: { subs: ['app/foo/head/nav/0'] },
    // PROBLEM_1: this does **not** work: '0' key will overwrite 0 key
    '0': { type: 'menu', override: [override_menu] }
  }
}
function override_menu (data) {
  // PROBLEM_2: how does this "redo" the original "menu override" from menu default data?
  data.menu.items.push('demo')
  return data
}
/***********
  vs. NEW:
***********/
function FB_MD () {
  return { _: { app: { 0: override } } } // app === 'app:0'
}
function override (data) {
  data._['foo/head/nav/menu/0'] = ([menu]) => menu()
  return data
}

There seem to be some issue with your version:

  1. see PROBLEM_1
  2. see PROBLEM_2
  3. how do you KNOW that app/foo/head/nav/0 exists without looking at source code?
    • the NEW version can console.log(data._) to learn about available paths
  4. The NEW version has a much shorter FB_MD function
  5. the override: [...] array needs a user to know the paths of sub components to be able to fill it out with more than just a single function, so they have to know the source code

@alyhxn
Copy link
Author

alyhxn commented Nov 13, 2024

Tasks - 2024.11.13

  • Update - STATE_v0.0.5
    • Converted fallback DS to tree. 1h
    • Made relevant changes in the code 3h
    • Constructed the tree to show the path - 3h
    • Update fallbacks/override with new proposal - 2h
    • Logged tasks - 5min
    • Recored worklog - 5min
    • @output 📦 STATE_v0.0.6

Worklog

worklog-243
worklog-243.1

@alyhxn
Copy link
Author

alyhxn commented Nov 20, 2024

Tasks - 2024.11.20

  • Update - STATE_v0.0.6
    • Discussion for creating multiple modules of the same type - 3h
    • Experiment - 5h
    • Created new repo - 30min
    • Implemented example with the boot.js solution - 1h
    • Debugging the hidden issues - 2h
    • Logged tasks - 5min
    • Recorded Worklog - 20min
    • @output 📦 STATE_v0.0.7

Worklog

worklog-244
worklog-244.1

@serapath
Copy link
Member

feedback 2024.11.20

  1. regarding const default_slots = ['hubs', '_', 'inputs', 'outputs'], developers using the graph-explorer are supposed to define slots, but the STATE module is not supposed to have custom slots, apart maybe from the data defined in fallbacks and the "dataset groups/mapping/switching" we havent talked about yet.

  2. you havent refactored yet, but just to comment on some things:

    • getname
    • In my opinion this screams for a short helper function to get the module name from a filepath or whatever. The exact splitting and extracting of relevant string parts can be details of that function. That way, i can read the function call name and know what it does and if interested in details, i can go to that function and check how it does it.
  3. another thing that is confusing is localdb and the api. i expected a simple key value store

    • pic
    • where is data defined? i cant find any var/const/let data = ...
    • why do we need those "search_filters"? you say because modules dont pass sid down, but:
      1. we are supposed to know the require path, e.g. app/head/menu/button
      2. when require('button') runs we can count how often we see app/head/menu/button
      3. so we can count and based on execution order make it button:0 and then button:1
      4. this is already true for super modules, thus: app:0/head:0/menu:0/button:0
      5. this can serve as unique ID and make passing a sid irrelevant for modules
  4. i dont get status.fallback_check so i assume that when it is true, then the database is empty and we execute fallbacks? ...otherwise i am confused, because we execeute the fallback and preprocess when the if check is true... so ehh..yeah. let me know.

@serapath
Copy link
Member

serapath commented Nov 21, 2024

feedback 2024.11.20

Looks good. I have no opinion yet on whether name:x or name#x or name:x:y. I agree we need some convention. If yours work, lets roll with it for now and lets make everything work and feature complete. Once it works, we can do another iteration about changing names or symbol conventions, etc...

I didnt understand what or why we need something extra for export/import, ...like a $ sign, but if there is a need, lets use it. Any sign is okay for now, lets just minimize what we need to make all features work and then we take another look.

I am not yet super happy with the path argument for override functions. I was more thinking those would already be part of the data returned by e.g. topnav() or any other function provided as an argument ...e.g. [topnav, menu, button] if we try to override button in head of the app that has head/topnav/menu/button.
I have not understood yet what the advantage of path is or why we need to have it like that.


issue: topnav#1.json

I think this is a slight misunderstanding.
The idea is fallbacks/overrides execute when the database is empty, so no files exist yet in the database.
files
Now when topnav() executes, it is supposed to create a "fresh instance" of data.inputs['topnav.json'] which is distinct from the one in topnav#1 even though it has the same name.

Imagine EVERY COMPONENT HAS ITS OWN VIRTUAL DISK :-)

So e.g.

  • localStorage['app#0/head#0/topnav#0'] = {}
  • localStorage['app#0/head#0/topnav#1'] = {}
    They will both contain a key called topnav.json: ... with the content and if one components edits their version, it should not change the other component.

In fact, every single "component instance" will also have ITS OWN VIRTUAL DISK too :-)

Now IF we want to have many component "sharing" a file, then

  1. now a module developer can give all instances of that module access to the same data that belongs to the module, not to any specific instance
  2. otherwise, if multiple instances should share a different file, that is not in the data of the module they belong to or where instantiated from, then there needs to be a super component instance to override the defaults of multiple different instances with a pointer to the same file, which can be a file stored in that super component for example.

I guess based on use cases - requesting and sharing files or data between multiple components is something we have to work on, but for now, by default, everything has to have its own file copies.

This means, the components:

  • app:0/head:0/topnav:0/menu:0/button:0
    • has app:0/head:0/topnav:0/menu:0/button:0#button.json
  • app:0/head:0/topnav:0/menu:0/button:1
    • has app:0/head:0/topnav:0/menu:0/button:1#button.json

where app:0/head:0/topnav:0/menu:0/button:0 is the id of the button instance 0 and also the id of the "virtual disk" and e.g. #button.json is a file stored in that disk. This is also how the database can be structured.

Now the above is not yet meant to make a decision how we use: /, :, # or maybe $ or other kinds of separators to structure those name spaces... lets use something and later, when everything works, we can see what seems best.

Looking forward for multiple named exports and what notation you choose there :-)


regarding fallback data:

  • JSON.parse(JSON.stringify(...))
  • or require('./instance.json') with `delete require.cache[require.resolve('./instance.json')]

The fastest and most appropriate way to do this these days is ti use https://developer.mozilla.org/en-US/docs/Web/API/Window/structuredClone

BUT ...in the case you mentioned, i would highly recommend to maybe just define the entire fallback in a module instead, like:

// instance.json.js
module.exports = () => {
  return {
    // content of ./instance.json
  }
}

instead of
old

Because that means for every instance we can just call instance() to get a fresh copy of the data.

I personally would probably just define the fallback functions at the end of the current module file and keep it there instead of an external file, but the good thing with our system is:

=> People can do it in their module however they want and people programming other modules dont need to really care and things still work :-) ...great that there are so many ways to do it :P
Thats a feature.


Regarding chatgpt - i strongly feel, that neither for our STATE module, nor for our fallbacks, nor for the details about the require function and many more things, like the stack trace parsting we tried to do and plenty more to come - will be properly answered by chatgpt.

If it helps you, that's great, but my past experiment is, that apart from basic stuff, it fails and you are then quite busy fixing all the mistakes that in the end you didnt even save time at all.
Chatgpt seems to be only good in solving generic beginner problems in a helpful manner, but yeah...

Also openAI already announced that progress isnt that fast anymore and they hit a limit where progress slows down significantly, so i expect no change in the situation any time soon.

well - kinda irrelevant feedback. If it feels helpful to you, go ahead - whatever works :P

It might have told you about structuredClone perhaps :D

@alyhxn
Copy link
Author

alyhxn commented Nov 25, 2024

Tasks - 2024.11.25

  • Update - STATE_v0.0.7
    • Worked on feedback - 3h
    • Added example modules - 5h
    • Debugging - 2h
    • Logged tasks - 5min
    • @output 📦 STATE_v0.0.8

Worklog

worklog-245
worklog-245.1

Feedback

Oh, I forgot to talk about the graph explorer. It has been updated as proposed. Clicking the line expands top and clicking the emoji expands bottom.

@serapath
Copy link
Member

serapath commented Nov 25, 2024

feedback 2024.11.24


regarding slots

you answered, but i'm unclear why we need slots at all in the STATE module. I have a feeling we dont.


regarding .find and search_filters

for modules we dont know the sid, but, we can know the path and the order in which they get required, e.g.

  1. /page/app/header/topnav/menu/button/icon could be the location to require the "icon module".
  2. When we detect that order inside of a call to statedb(...), we can:
const modules = {}
// ...
module.exports = STATE
function STATE (__filename, modulepath) {
  const prefix = modulepath.join('/')
  return statedb
  function statedb (...fallbacks) {
    const m = modules[prefix] || (modules[prefix] = { counter: 0 })  const id = `${prefix}:${m.counter++}`
    const sid = Symbol(id)
    // ...
  }
}

This way we can just NOT use an id, but instead a modulepath to identify a specific module instance.
The "modulepath" is unique per module, but if 2 instances are required in the same modulepath, we can count the order.

Also, the modulepath is NOT the __filename, because each segment of that modulepath has its own filename. In order to track that we might need to modify the reuqire function, but luckily we now have boot.js where we can exactly do that and also pass a unique require function to each module, which means, we can patch it like this:

function patch_cache_in_browser (source_cache, module_cache) {
  const meta = { modulepath: [], paths: {} }
  for (const key of Object.keys(source_cache)) {
    const [module, names] = source_cache[key]
    const dependencies = names || {}
    source_cache[key][0] = patch(module, dependencies, meta)
  }
  function patch (module, dependencies, meta) {
    const MAP = {}
    for (const [name, number] of Object.entries(dependencies)) MAP[name] = number
    return (...args) => {
      const original = args[0]
      require.cache = module_cache
      require.resolve = resolve
      args[0] = require
      return module(...args)
      function require (name) {
        const identifier = resolve(name)
        if (name.endsWith('node_modules/STATE')) {
          const modulepath = meta.modulepath.join('/')
          const original_export = require.cache[identifier] || (require.cache[identifier] = original(name))
          const exports = (...args) => original_export(...args, modulepath)
          return exports
        } else if (require.cache[identifier]) return require.cache[identifier]
        else {
          const counter = meta.modulepath.concat(name).join('/')
          if (!meta.paths[counter]) meta.paths[counter] = 0
          const localid = `${name}:${meta.paths[counter]++}`
          meta.modulepath.push(localid)
        }
        const exports = require.cache[identifier] = original(name)
        if (!name.endsWith('node_modules/STATE')) meta.modulepath.pop(name)
        return exports
      }
    }
    function resolve (name) { return MAP[name] }
  }
}

Using the above as a strategy, we can skip the searching i'd say. What do you think?


regarding the path argument

I do think your analysis that the data returned from an override function would potentially contain all data down to the leaf nodes. Would that be different with using path though?

And even though the data would contain data down to the leaf node, we dont have to execute or apply it yet. We can store it in a cache inside the STATE module until we actually need it and then we apply it.

Maybe you can explain a bit more in depths how it would work with path instead. Is there an example? I dont yet understand it.


example.html

can we try to mimic exactly the structure defined in the discord chat? I like the way you made it work, but i would prefer the following structure, so we can have more edge cases.

/* page structure:
 - page:0
   - app:0
     - head:0
       - (foo) // does NOT call `statedb(...)`
         - nav:0
           - menu:0
             - btn:0
               - icon:0
             - btn:1#small
               - icon:0
     - foot:0
       - text:0
*/

Of course, it makes sense to keep the extra css and html to make the example look like something :-)
The idea with the hover vs click menu is pretty cool though.

So maybe you can add a second:

           - menu:1#hover
             - btn:0
               - icon:0
             - btn:1#small
               - icon:0

So if you could extend the whole thing a bit, that would help, because i am coding against the same example already for some additional features that we need to integrate it with the sandboxed runtime.

But also - after watching your worklogs, i have not seen the kinds of overrides defined in the example on discord. Could you do that? Because thats the whole point, to see if the overrides really work :-)

Finally, when the whole page is initialized, can you inside of STATE module make sure to define, e.g. window.STATEMODULE or something like that, so i can open the devtools and console.log(window.STATEMODULE) which contains all relevant state thta is being built up by the STATE module when initializing and executing the entire page?

That way i could explore the structure that keeps track of everything and explore it and compare it after the page has successfully loaded, e.g. all the different instances and "virtual file systems", etc...

@alyhxn
Copy link
Author

alyhxn commented Nov 30, 2024

Tasks - 2024.11.30

  • Update - STATE_v0.0.8
    • Integrated modulepath - 1h
    • Created example components - 2h
    • Merging data and path - 7h
    • Logged tasks - 5min
    • Recorded worklog - 10min
    • @output 📦 STATE_v0.0.9

Worklog

worklog-246

@serapath
Copy link
Member

serapath commented Dec 1, 2024

feedback 2024.12.01

slots

Okay, lets ignore slots for now. I think we need to revist them when we finally get to "dataset group mapping and switching" :-)


find/search

I didnt entirely understand how this works.
If you only pass in modulepath, then how do you load the right state from the database? each module instance (e.g. 0, 1, 2...) if it gets required multiple times, will have a different state.

In my mind, this can become a simple lookup, where type or path is basically a "unique instance id" and can serve as a key in a key value store to lookup associated data state to be loaded when a module initializes with a non-empty database


data/path

you said it is merged.
i still would love to see fallbacks as shown in the discord snippet where page overrides a sub sub sub dependency, e.g. like page/app/head/menu/btn/icon or something like that.

How does it work and how does it play with other icon overrides defined e.g. in app or head, etc...

Basically:

  1. we could define all kinds of complex fallbacks/overrides
  2. we should comment in each one why or what it is testing
  3. when the app has loaded, we run a last script at the end of page.js to analyse and test window.STATEMODULE to see if the state that was built by the execution of all fallbacks/overrides is the correct one

Of course, that testscript should have a hard coded hand crafted version of the final state we expect for all the instances and modules in our app and it could walk window.STATEMODULE to compare if our expectations are met.

So we first have to "run" all modules and fallbacks in our head and use the comments to build that hand crafted hard coded expectation in the test script.
The hard coded expectations in test script we compare against should also have comments to mention which module and which fallbacks are meant to be responsible for our expectation.


STATEMODULE

Cool that it has a .tree property to check if the constructed version is the correct one :-)


Looking forward for also fixing the "pre execution" you mention.

@alyhxn
Copy link
Author

alyhxn commented Dec 10, 2024

Tasks - 2024-12-09

  • Update STATE_v0.0.9
    • Implemented fallback of sub sub sub entry - 4h
    • Made foo visible to the system - 3h
    • Recorded Worklog - 10min
    • @output 📦 STATE_v0.0.10

Worklog

worklog-247

@alyhxn
Copy link
Author

alyhxn commented Dec 26, 2024

Tasks - 2024-12-25

  • Update STATE_v0.0.10
    • Implemented foo.nav mechanism - 2h
    • Cleaned the code - 8h
    • Created example2 along with new STATE.js file for it - 5h
    • Updated fallback structure - 5h
    • Logged tasks - 5min
    • Recoreded Worklog - 1h30min
    • @output 📦 STATE_v0.0.11

Worklog

worklog-248
worklog-248.1
worklog-248.2
worklog-248.3
worklog-248.4

@serapath
Copy link
Member

serapath commented Dec 27, 2024

feedback 2024.12.27

feedback for worklog 248

Sounds good so far. I need to check the example code how the module is used to really understand some of what you explain, such as: preprocess function, which you describe before you showed where or how it is used. You say it is important, but when you explain it, it hasnt been used anywhere yet, so at that point it is unclear what problem it solves and there are a few more situation like that.

I would have been a bit better i believe, if you had started with the example demo app source code, that runs and initialized one statedb after the next and requires dependencies and so on, all the way down and at each step, you could explain what is already known to the STATE module and what it stores, calculates or internally uses to provide the requested state to the diffrent modules and instances. I feel that way it would be easier for me to follow in which order things execute and when we preprocess and why.

Other than that - i would request a bit of refactoring:

function statedb (defaults) {
  const { api, drive, on, _ } = get_module_data(defaults)

  const data = drive
  local_state.api = api // local_status.fallback_instance

  // ...
}

// button.js
function defaults () {
  const drive = {} // inputs, files, css, data stuff
  const api = function fallback_instance () {} // module.exports
  Object.assign(api, { small: function fallback_instance_small () {} }) // module.exports.small, ...
  const on = function (subpath, overrides) { return overrides[0]() }
  const _ = {
    'foo:0': { $: function module_override () {}, 0: { $: function instance_0_override () {} } }
  }
  return { drive, api, on, _ }
}

Basically to change statedb(function defaults () {}) to only have one argument, which is the "module fallback" and that returns an object with 4 properties

  1. drive (contains all the data)
  2. _ (contains all the subs)
  3. on (a callback we dont use yet, but later to react to dynamically created modules and instances at runtime - i guess you can add it without implementing what it does for now)
  4. api (to store the "instance fallback" function and potentially additional "instance fallback" functions for other module.exports[name] = ...) (e.g. such as small button, large button, etc... in our button module example)

for example:

example

The above example refactored would look like:

function fallback_module () {
  return {
    // drive: {},
    api: fallback_instance,
    // on (subpath, overrides) {},
    _: {
      "app": {},
    }
  }
  function fallback_intsance () {
    return {
      drive: {
        inputs: {
          "page.css": { data: `body { font-family: 'system-ui'; }`
        }
      },
      _: {
        "app": {
          0: override_app
        }
      }
    }
  }
}

following the pattern or schema:

function fallback () { return { drive, api, on, _ } }

comparison

Could you do another kind of refactor by literally making the file the same line by line where it is supposed to be the same?

comparison
Here for example:

  1. line 1 should in both files be // STATE.js i suggest
  2. followed by require localdb and instantiating it into db
  3. then the constants version an default_slots
  4. then the status object
  5. then the check_version function or writing it out but in both the same
  6. then symbol mappings, etc...

In these, both version, example and example2 are technically absolutely exatly the same, yet it is written differently, which means when i read through the file its additional confusion.

Ideally i can only focus on the lines that are DIFFERENT to see where and why things are different with the different approaches.

Could you fix that throughout the file? :-)


your explanation

thank you for all the explanation, but still, whether it is clean_subs or extract_data or clean_node or preprocess, etc... it would become a whole lot easier if instead of just explaning STATE.js, you go through the execution of example page.js and example2 page.js and all the dependencies step by step and then go to the STATE module and explain in each step and phase of the initialization of the page what code runs and how it all works.

It is otherwise difficult to follow if you arent yet deeply familiar with the code you wrote - which at this point only you are :-)

Also, the "example" and "example2" page.js is ideally the same code, where you would first write the "deep overwrite" version and then copy the code and just comment out the "deep overwrites", so it is easy to spot where the differences are when going through the code myself, which i plan to do after the next worklog with the above changes :-)

You do explain the page.js code and the overrides and show the logs in the console, but what is unclear is, in which phase and order STATE module functions get triggered by those overrides to build up the internal cache or state step by step, based on what it already knows and so on ....

That would be most important to describe and compare between example and example2 :-)

@serapath
Copy link
Member

serapath commented Dec 30, 2024

feedback 2024.12.29

for worklog 248.2

get_module_data

1.

db.find is problematic, because db as a module is meant to be a very primitive database, thus either a filesystem or a key value store. The reason is, that it will be a "BUILT IN" and we want to follow the idea:

A concept is tolerated inside the core modules, such as STATE, only if moving it outside, i.e., permitting competing implementations, would prevent the implementation of the system's required functionality

In essence - we want to keep it modular and minimal and advanced databases should be built on top of the more primitive core instead.

2.

get_module_data is explained before the explanation of how the data is getting collected is explained. You mention it while explaining and that when modules execute it will store data in the database for direct sub nodes, but - it wasnt shown at this point yet, so i still struggle to understand in which order you execute things at this point.

  • So, does data here contain fallbacks/overrides from super modules?
  • Is it an object or an array of ids? How does it work?

update (243.3):
So, data is actually pre_data which is better but still vague. I see later it contains { id: pre_id, hubs }, and that is what is returned from db.find(['state', { path: modulepath }), which still doesnt feel super intuitive, but it is better.

I do think what would help is a comment or an explanation of the "database schema" (or how it is used, e.g. db.find(['state'], ...) and what kind of data that can return and how it is structured. Maybe an example in comments so one could better imagine what we are dealing with here :-)

For example - in the code i learn, pre_data.name is a thing too. What else can be part of it?
Oh and also i immediately learn, that pre_data.name is actually local_id as i can see from the first parameter in the get_instance_path function. This is another advantage of what is explained later down... basically of passing a single opts object to functions instead of many params, because it favors/incentivizes to keep things named the same for the most part :-)

3.

I couldn't completely understand things, because the preprocess and find_super functions were not explained yet.

Maybe the next videos will answer some of these questions already.

Update (248.3): Okay, so find_super internally uses preprocess :-) Great - that helps.

Generally, it would be good to avoid side effects as much as possible and keep those limited to a few functions, such as sending over the network or saving to the database.

The preprocess and find_super functions would ideally do everything they need to do but without creating side effects, such as updating the database for example.

The call data = db.find(['state'], search_filters) tells me those functions DID update the database. It would be nicer if they would just return all the relevant data and then we update the database here instead of querying the updates that were implicitly created ...but nobody know where or what else they might have updated.

So if we refactor more, let's take those things into account :-)


for worklog 248.3

And if we are already at coding style :-)

example

1.

The preferred pattern of defining and calling function is giving them only a single parameter, maybe sometimes 2 or 3, but never more. An example for 2 or 3 parameters would be to pass a callback to a function or maybe even a callback and an "errback" (but 3 is really rare, because a callback by convention anyway should have function callback (error, data) { } the first argument be an error and the second one data.

All other data, such as the preprocess function should instead be wrapped in an opts object, so it would change from:

function preprocess (fallback, xtype, pre_data, fun_status = local_status, orphan) { ... }

// to
function preprocess ({ fallback, xtype, pre_data, fun_status = local_status, orphan }) { ... }

That is not a big change at all, but it is significant when calling the function, such as:

preprocess(fallback, 'module', {id:0})
// or
preprocess(fallback, 'module', data)

Because having many parameters makes it difficult to always remember in which order to provide them. If some params are optional but we want to provide "orphan", we have to pass a bunch of undefined to get to that last param, but with the opts convention we can now write:

preprocess({ fallback, xtype: 'module', pre_data: {id:0} })
//or
preprocess({ fallback, xtype: 'module', pre_data: data })

This will help the reader by providing some context of what the preprocess function expects and it also avoid counting the position of a provided argument, like: preprocess(a, b, c, d, e) ...now i have to count to make sure d means fun_status and not maybe pre_data or orphan ...the problem gets worse with longer parameter lists.

Same here:
clean_node
i would otherwise never know what '' means, but calling

clean_node({local_id: '', entry: host_data, path: modulepath})

is more meaningful and also immediately gives us the question, should host_data be called entry instead or the other way around? ...or do we call it state instead?

One more naming thing:
clean_node should be called sanitize_entry() or sanitize_state(), because "clean" is often used when deleting or removing things, but reading the function, it seems we are standardizing into a canonical form, thus that is known as "sanitizing" and because above i shared we should probably rename data to state, then host_data would become host_state, but in general, instead of entry, we are sanitizing (=standardizing or canonizing) the state of a module or instance. ...helps a lot. The name was very confusing.

side effects
Here again we have 2 functions with side effects and it is unclear what they return or what data they delete or modify or in general what they do.

By providing arguments, its easier to see on which data a function can act and by returning data, its more clear what the result is ...the result can then be send over the network or stored in a database ...basically side effects should be as direct as possible, so storing in the database is a direct side effect, but storing in a database inside of a function, such as clean_subs or extract_data (if that is happening) is indirect and thus way more obscure...

Hope we can change or improve this as well to some degree.

database schema
here again, it would be great to have a comment to explain what shape entry can have when it gets added to the database.

IDEALLY: we would validate that data provided in fallbacks actually has the correct size and shape, so defining generic function validate (entry) { ... } to run before/after storing stuff in the database ...and maybe even when reading from it, because maybe a user IMPORTED a customly edited snapshot.json file which might contain invalid data, would help a lot.

These validate function are essentially some sort of schema for a specific entry and reading that function and also writing that function with good names and clean code style, will maybe be even better than the comment showing an example of a data entry.

Although, that kind of comment could be added inside that validate function, so that apart from abstract data or state validation logic, we also have this comment to show us an example of how aprox. those data or state items could look like :-)

2.

preprocess details
I understand this, but given that we have fallback and fun_status, what function is state.overrides[path] and when or where does it get set?
At this point, that wasnt explained, only that it is now being used...

...oh ...i guess you now explain it, by sharing, that
state.overrides[path] could be:
override_app
But if that is the case, then i can now derive, that merge_trees(...) returns an array of functions?

I do think "merge trees" is an implementation detail of how the function might achieve this, but get_fallbacks might be more appropriate then?

Basically, that is the difference between "override" and "fallback". The override_app or current status.overrides[path] override function gets executed, but all the other "override" functions get provided as an array of fallback functions ...so those fallback functions are also overrides, but they here get passed as fallback to the current override if that makes sense and if i interpreted correctly what you explained here 😅 ...if not, let me know.

I guess, doing another refactoring of renaming things or changing parameter lists to single parameter objects and the likes would be useful to steadily improve readability.

People don't say for nothing that "naming" is one of the hardest problems in computer science :P


fallback/override

In the past, data was what is now drive, only that there was also data._ for subs, but now that we changed it to { drive, api, on, _ }, i think data is confusing and it would make more sense to refactor and change it everywhere to state, thus: const state = app() in the example above.

Another refactor we should do to improve the overall situation :-)

You also say: register_overrides stores the appropriate overrides in status.overrides[path], but does it also update fun_status? ...because where do you find all the overrides if a path has multiple overrides provided by different supers?
Those should be provided to the active override as an array of fallback functions :-)

...again, maybe i misunderstand something here. let me know.


3.

page boot
This is confusing. get(sid) is how we defined things and sid is meant to be a symbol and never an empty string '', so i dont understand what that means.

Especially, because above we had const { sdb, subs: [get] } = statedb(fallback_module), where we did not use the sdb returned from the statedb call. Is that why we are using the empty string here? ...if so, i'd prefer to use the sdb from above :-)


for worklog 248.4

orphans

So i'm confused, because foo module would pass through a sid provided to it, e.g. head > foo > nav
where

// head
foo(sid)
// foo
module.exports = sid => nav(sid)
// nav
module.exports = function nav (sid) { ... }

and we have a sid now :-) Otherwise, it would really be weird and i cant really imagine how nav would be useful or how it would even work.

regarding watch function:

watch

Maybe you shared that this needs to be updated anyway, but in case it is supposed to work already, it needs some explanation, because the Promise.all loop is a bit confusing to me.

Again, a validate function to guarantee some basic structure and some sort of example comment to show how data.inputs (which should now be state.drive.inputs i guess) could look like, would help a lot.


FINALLY

1.

If the above feedback is included/worked on and fixed in the next worklog, including answers for any questions above, then we can use it to finalize the refactoring of playproject and update the theme widget with everything.

Before approaching dataset group mapping and switching i would like to ensure that what we have works fully.

The dataset group mapping and switching will anyway only affect the state.drive part and not the other parts.

2.

Now one thing, the watch(onbatch) will only ever have ONE LISTENER (=one onbatch function) and if ever a component calls watch() with an empty function, it will remove the current listener and if it calls watch(onbatch2)with a new function, it will **replace** the current listener ....so persdb.watch()`, there can always only be a single listenre at any given time - just to clarify that :-)

3.

Of course, the key part is still missing, which is - similar to a "debugger" where you "step through" the code execution, to walk through the main important parts of each module that runs and how it calls statedb to provide fallback and receive data and how STATE builds up its internal data structure step by step to fill the database.

THE IDEA:

is to start with page and how it calls STATE statedb(...), then how it requires app, then how app calls statedb, then how app requires head, then how head calls statedb, then how it requires foo ... nav ... menu ... btn ... icon and then how icon calls statedb and then doesnt require anything anymore and runs the rest of the icon module.
After that it returns to btn and continues with the rest of the button module after it required icon and then,
after that it returns to menu and continues with the rest of the menu module after it required button and then,
after that it returns to nav ... foo ... head ... app ... page, and then continues with the rest of the page module, which means it calls sdb.watch in page, to get a sid and then calls:

  • app(sid)instance, which continues in the app module to lookup the sid and get an sdb instance object and calls .watch() to get another sid to call
  • head(sid) instance, which continues in the head module to lookup the sid and get an sdb instance object and calls .watch() to get another sid to call
  • foo(sid) passthrough ...
  • nav(sid) instance ...
  • ...
  • icon instance ...

Inside every statedb() call or sdb.watch() call or get(sid) call, etc... the STATE module will slowly update the database and its internal structure to provide the correct state data to each module and i would hope that some high level walkthrough can be given to show the dynamics of the main ideas of how the STATE module manages all of that and builds up the state.

Next ... a second walkthrough using the exact same scenario i tried to explain above, but instead with example2 would be good.

In the end ...a third video to discuss the differences in both approaches as well as what i started to share on discord about extending it with a dynamic require and what implications this has in terms of security and how we could deal with that could be talked about :-)

@alyhxn
Copy link
Author

alyhxn commented Jan 4, 2025

Tasks - 2025.01.03

  • Update STATE_v0.0.11
    • Refactored the STATE - 2h
    • Removed and replaced db.find() - 3h
    • Added validate() - 4h
    • Logged tasks - 5min
    • Remove db.find
    • get_module_data questions -> answered in update from 248.3
    • Remove side effects from preprocess and find_super
    • Combine function args into opts and validate function
    • Renaming requests and return of array of overrides
    • Recorded Worklkog - 1h10min
    • @output 📦 STATE_v0.0.12

Worklog

worklog-249
worklog-249.1

Proposals

  • how foo passes through sid (Needs discussion)
  • Empty sid of root issue (Needs discussion)
  • Dataset grouping and mapping
  • Complete the theme_widget
  • kv-idb integration

@serapath
Copy link
Member

serapath commented Jan 7, 2025

feedback 2025.01.07

Some comments based on what you shared.

  1. if pre_id means the id provided by the fallback/override, which will "inspire" the real but different id used in the database, then "pre id" is okay i think. it is also shorter.
  • maybe a short comment next to the code that introduces "pre_id" to explain what it means would be helpful though.
  1. thanks for the validate functions and comments inside. this looks very helpful :-) ...errors are a side effect again though.
    And style wise, there should not be any ; semicolons. Could you stick to standardjs code formatting (but with snake_case naming allowed.
const errors = []
// ...
validate_shape(data, expected_structure)

if (errors.length > 0) // ...

better would be:

const errors = validate_shape(data, expected_structure)
if (errors.length > 0) // ...

Because that makes it a lot more clear where those errors come from in general. If there would be MORE than one function call between errors and the if check, then it would be unclear which function populated it apart from reading the source.

Same feedback basically as shared previously about:

  • preprocess & find_super - avoid side effects

What is still outstanding is the discussion about:

  • get('') in module
  • or foo.nav when we have a module not using the state and whether it passes through a sid from foo's super module through the foo to the nav as a target, which uses it to load state.
  • watch(onbatch) explanation i guess :-)

Lets discuss on discord maybe :-)

validate
I havent yet executed the code and checked the debugger or console output, but just from a first glimpse.

  1. the expected_structure starts with {drive, ...} keys
  2. the validate_shape knows it as expected and starts with Object.entries(expected), which would give you [['drive', ...}], ... and .forEach means expected_key='drive', expected_value={..} and **how would the .split(':') do anything useful here? Does this really work?

But you show in the console when changing fallback data, that it seems to work, so i will dig a bit into the source code to understand how. Thanks.

@serapath
Copy link
Member

serapath commented Jan 7, 2025

feedback 2025.01.07

The longer worklog video (26 minutes) is processed in great detail on discord, but is too much information for a worklog comment here, so rather the processed results of that will be added after the next worklog to save some space here :P

But it was a good video, quite informative and definitely lots of progress. Thanks :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants