Stores. Pagination. Frustration

[ivan-kleshnin]

Not exactly a flummox question, but would be great to discuss it somewhere.
It's up to @acdlite to close this as offtopic if he wants to keep issues clear (some people do).
Some people also prefer to discuss on Gitter. But those threads are temporal and non indexable.

---

Pagination is a good example to illustrate some thoughts. Everyone makes pagination so
we should have countless implementations, libraries and competition here, right?
Well... actually no.

When I've come from backend programming I just wanted to reimplement basic CRUD stuff. All that features I had before: filters, sorts, pagination. I expected to find multiple solutions, smart decisions for this. So naive! I started with Backbone. I read through a lot of Backbone books. I watched a couple of Backbone video courses. About zero enlightment. They didn't even approach the questions of state management. Load all your data, then do frontend-only filters, frontend-only sorts, frontend-only pagination. Backend versions were more scalable!

I switched to Ampersand. Same things there. Just improved Backbone, still bloody low-level.
Logicless templates. No, thanks. Then React. Better render, same widgets / todo apps.
Still no state management solutions.

[ivan-kleshnin]

For pagination we want to get backend + frontend hybrid solution which merges scalability of the first and usability of the second. I think the first fluxy lib which will document this will get popularity boost. I'm not really a flux architecture fan, but you've got the point.

The only reasonable attempt to discuss it I have found in 30 minutes is:
https://github.com/gaearon/flux-react-router-example.

That's something.
@gaearon classifies stores to 3 types:

• Content
• List
• Indexed lists

"Content" stores keep app data as {id: model} objects. "List" and "Indexed lists" stores keep id relations.

First things first. Most tutorial examples represent app state as Arrays of models. Array is extremely unsuitable for get / update / delete operations. You have to implement even most primitive access methods in store, have to mess with offsets, ohmy... This is extremely low-level and stupid.

So, I agree here, Map is better choice if we limit ourselves for simplest types. It support one-line basic ops and can be iterated (e.g. used for INDEX) as well. I talk in general, but you can imagine ImmutableJS objects as a reference. They have enough methods to get rid of Lodash helpers for this stuff.

But what about those 3 types of stores? We've got data duplication concerns as soon as we copied data relations to separate stores. What are stores after all? They are poor-man's database. Reactive database, and so what? There are already a lot of them. And if we agree to see stores as tables of database, we should apply same well-known, time-proven rules and criterias to them: normalization, rich query language, ability to join, import / export questions, history, etc. Referential integrity features would be highly desirable in this case of Indexed lists. But we have none.

So the set of stores feels and behaves like the most uncomfortable database ever.
Why should it be so hard to use? Does everyone find normal to write stupid helpers like getLastMessage for data access, data aggregation again and again?

I still can't believe that for 6+ years of frontend evolvement we have nothing reuseful here.
Countless jQuery plugins. Countless widget libs. Zero reliable pagination solutions :(
It was really Dark Ages. 😞

[ivan-kleshnin]

Should we just wait for Relay to solve this for us in some magic way? It won't be REST on backend, so I'm personally not very excited. But it's interesting to inspect their direction, at least.

They merge Stores in one single Store. That's even closer to DB concept. Joins should be easier as well as imports / exports. They add GraphQL language: queries over data. That's DB aspect too.

There are other very interesting approaches. DataScript is a lightweight frontend DB. It's in ClojureScript with JS bindings. Sadly, no JS documentation, only test suite with rather obscure examples. Received very good feedback from top guys.

RxJS gives another approach. Same immutable database we want can be viewed as stream.
Every next stream value – new snapshot of the whole app state. There is rx-flux. There is cycleJS whose author had provided very sound React criticism. There is Om with distinct "cursor" concept. There is Reagent with "ratoms" (another kind of observables). There are coming WebComponents.

Anyone wants to classify and explain all this? 😃

[nambrot]

So I don't have solutions for you, but I'm in the same boat if that helps. It seems so weird that it is so hard to do a basic CRUD app with all this "new and amazing technology", something that takes 0 time in Rails. I'm currently working on getting the simplicity of simple_form, kaminari etc. up the React + Flummox, but it's still quite a lot of work.

[tappleby]

Yeah I'm running into the same issues, still haven't come up with a good solution yet.

Things are not so bad in simple apps but as soon as you have complex relationships it gets a bit more complicated.

[acdlite]

I'll keep this open because I agree that Flux doesn't have a well-established pattern for solving this. For complex stuff, you really need some sort of query language.

[acdlite]

@ivan-kleshnin Btw I appreciate you laying out your frustrations here. They are shared by many of us, including me. I haven't said anything on this thread because I also don't have a good solution.

[tappleby]

Yeah I wonder if its a query language or perhaps a base set of stores for avoiding all the boilerplate (similar to @gaearon has).

[acdlite]

Yeah, creating a base store that takes implements the ideas in @gaearon's example project could conceivably work pretty well. I've been using a base store for stuff like serializing Immutable data, but not really for pagination yet.

[ivan-kleshnin]

@nambrot, @tappleby, @acdlite thank you guys. It's very helpful to know we share common feelings about this. There is always a chance one missed something, considering today's velocity of changes.
I didn't expect this will be handed on a silver platter. But, as we descbribed alot, there is a chance for someone to find this thread and bring more mental food.

I'm sure many of us are working on CRUD demo apps, so I propose to share their examples here (as soon as they are more or less "ready") and discuss them. I want to detail my complaints about them as well. Most of such apps are TODO-apps. They are too-specific and not really similar to what most of us build for business. Those CRUD apps should really account next pitfalls:

• Pseudo-models. Most of TODO apps present only message field, replacing real model objects with string values. That's drastically oversimplified, cause in real world even todos will have timestamps.
• Single page. I bet not many of our production apps are single-paged... Really need to demonstrate routing aspect.
• Data loading. That's most painful. Reflux guys propose to create special load actions. React guys describe both non-flux approach, which is not scalable and mention they don't have load actions at all... But don't answer how exactly they load it. I think that's because they don't really have any established pattern (which is again frustrating), at least at the moment of the interview.

Relay direction implies component should declare it's external data dependencies. With data load actions we can emulate this to some degree in imperative way. Rather primitive solution is next.
Component mounts and sends loadOne / loadMany action with optional params. Action or Store receives this and unsubscribes immediately until task will be resolved or rejected. This will prevent consequent re-asking and duplicate loads. Store updates it's state and component refreshes.

I think it's crucial to have responsive page without blank screens. Gradual page load is better as soon as page doesn't jerk and shows some sort of spinner. So I consider the way of data loading in this flummox demo as non-mainstream solution.

• Data loader. Who loads data? Previous point was about requesting. This one is about performing.
  Real Zoo here. Components? Noticed in simpliest non-flux solutions. Responsibility overload.
  Stores? In forementioned interview with React guys they say they load data in stores... Actions? Reflux guys are sure that is
  flux-way ^_^

As soon as we load data in actions we:

• make stores simplier (good?)
• make actions not just agents, but something useful (good?)
• make it easier to see action result for other system parts, because actions are more visible (good?)
• pull data-caching questions to actions (bad?)

If I understand correctly, Relay will bring some "magic" declarative solution which will undercover load this data for us. I see a lot of benefits and drawbacks with this approach so it will be at least controversial. Anyway that's enough arguments to mark data load as additional pain point.

[nambrot]

Think this might be slightly relevant here. I just listened to TheChangelog #131 with @wycats and @tomdale, and especially found their comments on Ember Data fascinating. http://emberjs.com/blog/2014/03/18/the-road-to-ember-data-1-0.html

I think what we are really lacking is the best practice to managing our "models" and their relationships. I so heartedly agree with their assessment that basically everyone reinvents syncing data. Like Flux is very helpful in making the "how" work, but it's kind of a pain to be able to declaratively specify how we should sync the data in the stores with the various requests being made from the App. I think the Ember team showed pretty well that they can get the right abstraction with the Router and I'm excited to see whether they can achieve the same with Ember Data and what that possibly means for us Fluxxors.

[ivan-kleshnin]

@nambrot 👍 I think that's pretty relevant.

[ivan-kleshnin]

Moar inspirational read: http://christianalfoni.github.io/javascript/2015/02/06/plant-a-baobab-tree-in-your-flux-application.html

[MrEfrem]

Look Swarm JS?

[ivan-kleshnin]

@MrEfrem, that's M of MVC with a lot of magic. Not inspirational from the first sight.

[nambrot]

@ivan-kleshnin Thanks for the link, very good read. I actually found http://christianalfoni.github.io/javascript/2015/02/06/plant-a-baobab-tree-in-your-flux-application.html#comment-1902000761 to be most reflective of what I'm looking for.

Not sure about Swarm JS either. Personally, I'm looking for a CRUD/REST compatible solution as 90% of all my apps fit greatly in within that abstraction

[ivan-kleshnin]

Personally, I'm looking for a CRUD/REST compatible solution as 90% of all my apps fit greatly in within that abstraction

Yes, I also keep wondering how often CRUD/REST is used and how few examples of it are published.

[snickell]

I'm interested in this too. I've used Ember for the past few years, and wanted to check out React. The React layer itself is amazing, but it seems like there's huge missing pieces/patterns for implementing basic DB-backed CRUD model stuff.

[ivan-kleshnin]

I've created a schematic view of most popular frontend archs https://github.com/Paqmind/reactive
If there is something I missed – please notify.

[netgusto]

For complex stuff, you really need some sort of query language.

There's something in the making at Facebook : React + GraphQL + Flux + Relay: https://www.youtube.com/watch?v=9sc8Pyc51uU

[nambrot]

Hey Guys, sorry to revive this without a constructive solution, but I just finished my CRUD example for anyone interested https://github.com/nambrot/rails-webpack-react-flux/

[gcanti]

Still no state management solutions.

This is the elephant in the room.

I'm looking for solutions to the problems that modern SPAs must face but what I find is the nth way to increment a counter.
And weird terms: "isomorphic", "HOC", "stateless stores"

Some thoughts:

• once you have a stateful client you have, by definition, a distributed system
• hence the frontend development has become a serious matter, maybe we should learn from the backend guys
• let's call that state with its proper name: a database on the client which is also a cache
• cache invalidation is a huge problem: we have "stores" but nobody carefully explains
  • how the data is loaded
  • how to invalidate the cache
• additionally, you may be required to write the code so that it can be shared between client and server
• for sure elementary examples, tutorials and todo apps don't help
• solutions without a single line of code (relays) or unexplained use cases (data as graphs in the case of Facebook) are useless
• what if my APIs are REST and I don't want to rewrite my whole backend?
  • what if my data is relational? must I join, filter, groupby on the client by hand?
  • how to manage multiple requests where one request depends on the result of a previous one?
  • how to optimize them?

Let's take modern frontend development seriously. ES6/7, cursors (other hyped terms follow...) are cute and I have fun playing with them in my spare time but in order to get my daily work done I must solve the big problem of state management in all my apps. If you can point me to resources that I'm missing, please (please!) share them. I'll be extraordinarily happy to be wrong and take it all back.

[clearjs]

The Case for Flux by @gaearon touches this topic briefly.

Perhaps Dan might want to elaborate on usage scenarios like this, either in general, or in context of his new redux project and its recent update: reduxjs/redux#46.

[gaearon]

With Redux, I'd implement a generic store (or a higher-order store) for pagination and use store functional composition to reuse this functionality across different stores.

[nambrot]

I agree with @gcanti I think there are two issues at hand.

1. Consistent state management within the domain of the client.
2. Globally consistent state across all clients and servers.

The whole Flux movement is very focus on 1), making sure that data state on the client itself is easy to reason about and one change in one part of the app is very predictable on its effect on the client side state.

I 100% agree that the next step, and imo more difficult but also more important problem is that once you introduced stateful clients how to make sure they they are consistent. I think the reason why this is not a much-debated topic is because you can get away with "eventual consistency". Most people simply can afford to just load data once and assume it's relatively consistent due to session length and other factors.

If you are interested in truly globally distributed and consistent state management, I imagine you don't have much choice than to look at Meteor or RethinkDB as far as I can see. Otherwise I think we are milking REST as far as we can already.

[gcanti]

@clearjs Yes I follow @gaearon and his article is like water in the desert. In particular I find valuable:

• the series of real world use cases
• the reference to the Command Query Responsibility Segregation pattern

[clearjs]

Yes I follow @gaearon

Who doesn't? 💯 But he produces content faster than I'm able to consume, and I'm trying hard... So I thought it was useful to provide another reference, even though he was mentioned here a few times already.

[blainegarrett]

I too am attempting to figure out these issue for a Reflux app. Namely, the best approach to getting a single entity from a store w/o having to load the entire database clientside first (re: Pagination) yet also avoiding having duplicate data spanning separate Stores to keep in sync. Reflux seems to introduce the other 'feature' of Stores trigger() indifferentiable events. Reflux moved Flux's case statements of actions from the Store level to the consuming View. Now if I attempt to have a single store and load "all" vs. a single entity not already in the store, the View has to case out what is going on in the listener callback.

[gaearon]

FWIW Redux today has an example with pagination.
It is implemented as a reducer factory.

[davertron]

@gcanti hits this one so hard on the head for me it hurts :) I'm struggling with the problems he outlines EXACTLY right now in a React + Reflux application, and looking around for examples almost everything is completely contrived; they either keep all their application state in the store as an object (i.e. Todos) that don't load anything from the server and just add to it over time or they load data in the most naive way possible at the beginning as a "starter" set of data and then manipulate thereafter, but I'm having a really hard time finding something realistic where they a) load data from the server that ideally is kept maximally consistent with the server and b) has multiple dependent requests. On top of that I'd love to see something that handles this once you add in routing but that's a little orthogonal to this discussion I suppose.