[proposal] provide benchmarks for the claims in changelog 3.16.0 and against other state managements

[iapicca]

I noticed these 2 conflicting PR's pro and against adding getX to the list state management options

I find the allegations in the against issue quite concerning

It looks like the package author makes some pretty wild claims* on social media. Removing the links, at least until we can verify that these claims are true.

*) The claims are about the package's official support from Google and/or usage by Google Product Managers — neither of which I'm aware of. I'd also like to verify the claims about performance (currently, they are backed only by a benchmark written by the author).

I've seen often on social media
claims about GetX being faster than Bloc, MobX and so on
and if this seems to be the only benchmark available at the moment

could you provide a benchmark against the aforementioned state managemets?

in you changelog you claim

• [...] There are cases where the performance gain exceeds 9000%,
  [...] which due to the low latency will have performance of low level languages ​​close to C, C ++, Rust and GO.

could you provide a benchmark that backs these claims?

thank you

[orestesgaolin]

Here are some other benchmarks https://github.com/jonataslaw/flutter_state_managers

[jonataslaw]

Hi @iapicca
The performance gain refers to the difference in the use of standard Streams and GetStreams.
Here is a benchmark that you can start on your own that measures the latency difference between GetStreams and Streams.
Note: this does not mean that dart streams are bad, just that we are looking for options with better latency, because in addition to the Get client, we have the GetServer which, as it is a server side, it is essential that all resources used have the best time possible answer.
https://github.com/jonataslaw/getx/blob/master/test/benchmarks/benckmark_test.dart

As for the performance of GetX and Bloc, you can run the application from this repository and check with the dart inspect tools.
https://github.com/jonataslaw/flutter_state_managers

There are tests performed by other users, like https://twitter.com/r_FlutterDev/status/1274146807073902599 there were also prints about the tests from the above repository, but I removed any reference to the names of other packages from the readme, because I don't see this as a product rivalry as people are putting it, even because you can use GetX in conjunction with other tools, other state managers, other dependency injectors, etc.
If necessary, I build a new test based on the above repository that makes it possible to run the cascade tests to see the difference in frame rate and RAM consumption. Unfortunately there is no way to create benchmarks automatically with Flutter, or at least I don't know of any tools that can measure fps and memory usage without resorting to dart inspect.

As for the first claim that Google and product managers use GetX, I never said that, and that claim doesn't make any sense, and I believe they should check the authenticity of it, because it's a really serious accusation about me about a fact that never happened , and well, besides being a programmer i am a lawyer and this type of slander is a crime against image. I would really like the Google team to analyze the package, both the performance issues (which, however trivial and manual, are not surprising), as well as the veracity of the source of this claim. If necessary, I can link my social media to check the comments, because if there was any comment in that sense, it is fake.

[iapicca]

thanks for the answer, what about the claims in the changelog?
9000% and C++ level performances seems too good to be true

[jonataslaw]

thanks for the answer, what about the claims in the changelog?
9000% and C++ level performances seems too good to be true

Hi, you're welcome.
In changeLog I think it was clear that:
The performance difference is between the use of streams.
The benchmark shows this, the difference between standard streams, and getStreams.
Dart when compiled for AOT has an incredible performance, and GetStream is performing very well on AOT. Obviously, dart will hardly be like low-level languages, but I'm struggling to create some low-latency resources to equip get_server (the server_side part of the framework).
Is there something wrong with what I wrote? If I hinted otherwise, please let me know, I am human and I am susceptible to errors, especially in another language that I do not speak.
GetStream = A class that mimics the dart stream api, using simple voidcallbacks
GetStream = has better latency (which doesn't mean that dart streams are bad, just that GetStream responds faster to a high number of requests).
GetServer = Flutter syntax on the server-side, GetX is a framework, state management is just a small fragment of it.
9000% does not refer to the difference between state management, but GetStream, which is easy to verify, just run a dart file and see the response time for each request.
GetStream has such a low latency that it is not far from streams used in low level language. This does not mean that dart has been turned into another fast language when adding get to your dependency, just that if you depend on a resource that uses Stream, the difference between using dart or another language will be very small.
If you have any other questions, feel free to ask, or suggest, if I misspelled, English is not my first, second, or third language.

[escamoteur]

you compare apples to pears. If you create a Stream with synchronous=true, the Dart Stream is faster than your implementation. Because then it's operates like your implementation.
comparing Value to ValueNotifier has also a Problem. the reason ValueNotifier is slower is that it always compares if the value has changed and notifies its listeners only then while Value notifies everytime a value is assigned which can lead to worse performance if it triggers unneccessary rebuillds-

[jonataslaw]

you compare apples to pears. If you create a Stream with synchronous=true, the Dart Stream is faster than your implementation. Because then it's operates like your implementation.
comparing Value to ValueNotifier has also a Problem. the reason ValueNotifier is slower is that it always compares if the value has changed and notifies its listeners only then while Value notifies everytime a value is assigned which can lead to worse performance if it triggers unneccessary rebuillds-

Value also checks whether the value has been changed.
https://github.com/jonataslaw/getx/blob/master/lib/get_state_manager/src/rx_flutter/rx_notifier.dart#L44
The reason Value is faster is that ValueNotifier uses LinkedList, which is slower to iterate than the standard List.

[jonataslaw]

Well, I'm honestly tired of it. I just received an email threatened by GetX, and I never thought I could get through it.
GetX's goal is to make life easier for people, and it has done so successfully and will continue to do so. I want to dedicate my energy to evolving this, and not discussing whether A or B is better.
State manager looks like a soccer team, and GetX is not only "State management", it is a framework.
Well, as of today I will remove all references to benchmarks, just as references to other packages have been removed, I will remove everything that may sound like 'marketing', because I have seen that this is the biggest reason for complaints, I will transfer GetX from my account to an organization, and I will continue to maintain and improve that lib as a contributor.
I sincerely hope that all this hate will stop, because I am already being threatened by the creation of a package, and this is serious. I never thought that I could receive threats to myself, for contributing to opensource, but maybe I was to blame for that, for wanting to 'publicize' it with such euphoria.
All I want is the community united again, without wars, without discussions about SM, and without so much hate.
Well, Thomas, I know you have a personal problem with me, and I wanted to resolve this - really. I have nothing against you, I admire and respect your work, and I would like to have a chat with you to resolve this at once.
To the rest, if you have had any problems with me during these years, my email [email protected] will be available for a personal conversation.
I'm sorry you're reading this too, for having entered this war Getx x anything, and I wanted to ask you, not to underestimate the work of the other, be it GetX, be Mobx, Be Bloc, Provider, get_it, or anything else. Use what you feel comfortable with, you have nothing to do with what someone else uses in your projects, so don't get in trouble for it.
I am closing this issue, and I will not touch on that again. Let's move on, and try to unite this very fragmented Flutter community.
If you want to help me redesign the readme for something better, and especially if you speak English, your help is welcome.

[jonataslaw]

For future reference only:
I was wrong when I mentioned other state managers in the past. Today I see that this has caused many problems, from the aversion to people I admire, to how it has attracted some toxic people. State managers are nothing less than one of your project's resources, this is much more a matter of personal taste and project requirements than exact math. There is no better or worse here.
I removed all references from the docs to other SMs, and also removed (at least the ones I found) all references that can be interpreted with an air of superiority.
2- I respect and admire Thomas, Remi and Felix a lot. For me they do an incredible job in the opensource world. So if you speak ill of what they did, from now on, you will also have a problem with me. Use the SM you like without trying to denigrate the other, after all, it is another dev who will use the state manager he wants, you do not pay his bills, you should not meddle in it. I believe that there is room for everyone in the opensource world. Stop this fight for State Manager, this is ridiculous.
3- I changed the changeLog to be more clear, because definitely, the presentation of the update was ambiguous. There was no lie in it, but the tone of sensationalism that it was written on could lead people to draw wrong conclusions.
4- I just read Thomas's comment about Streams Synchronized, and he is right about their performance which has very low latency, but there is a good reason why I didn't use them.

Dart also has a Synchronous Streams api that has very low latency, however, it is not suitable for use in state management for two reasons:
1- Synchronous Streams can only have one listen (see the issue opened by Hixie on dart lang for reference: dart-lang/sdk#22240).
This means that we cannot use this api for more than one listener, which is the basis of global state management, where we aim to change the state of more than one location. You can test this with this simple snippet:

void main() {
  var controller = StreamController.broadcast(sync: true);
  var stream = controller.stream;
  stream.listen((data) {
    print('$data');
    if (data == 'test4') controller.add('test5');
  });

  print('test1');
  controller.add('test2');
  stream.listen((event) {}); // second listen throws a exception
  print('test3');
  controller.add('test4');
  print('test6');
  controller.add('test7');
  print("test8");
}

2- Even with a single listener, the dart's Synchronous Streams api cannot deliver events in the exact order. We plan to work on a PR in the future at dart-lang to address this. So if we remove the line above that causes the exception, we will have the following output in the log:

void main() {
  var controller = StreamController(sync: true);
  var stream = controller.stream;
  stream.listen((data) {
    print('$data');
    if (data == 'test4') controller.add('test5');
  });

  print('test1');
  controller.add('test2');
  // stream.listen((event) {}); // second listen throws a exception
  print('test3');
  controller.add('test4');
  print('test6');
  controller.add('test7');
  print("test8");
}
///////////////////// log:
test1
test2
test3
test4
test6
test8
test5

As we can see, test 4 skips to test 6, which skips to test 8, which skips to test 5. Note that test 7 didn't even appear in the log.

However, if we work with GetStream, everything works as expected:

void main() {
  var controller = GetStream();
  var stream = controller.stream;
  stream.listen((data) {
    print('$data');
    if (data == 'test4') controller.add('test5');
  });

  print('test1');
  controller.add('test2');
  // stream.listen((event) {}); // second listen throws a exception
  print('test3');
  controller.add('test4');
  print('test6');
  controller.add('test7');
  print("test8");
}
///////////////////// log:
test1
test2
test3
test4
test5
test6
test7
test8

The dart documentation is clear that this api should be used with caution, and in view of these tests, we were sure that it is not stable enough to be used as the core of our state management, nor of the websockets notifications and get_server requests.

Clarification about the controversy over benchmarks:
In a version prior to changeLog, we talked about the 9000% difference in performance between Streams, and GetStreams that ended up causing a lot of controversy in the community.
Initially, we would like to clarify that this does not mean that you will have mobile applications 9000% faster. Only that one of our main resources, showed itself with a high rate of requests, 9000% faster than using traditional streams. In a real world scenario, you will hardly have so many simultaneous requests.
Skia renders frames on new devices at up to 120fps. This means that if you have a 10 second animation, you will have 1200 reconstructions. Unless you are working with animations, or something that requires rendering at the skia boundary, you won't need that much power. So this does not mean that we are revolutionizing the mobile world, only that we have created an alternative to Sync Stream, which works as expected to own proposity, and which has satisfactory performance for low latency resources. The benchmarks are real, but that does not mean that you will have mobile applications 9000% faster, but that you have a new feature that performs at this level to use.
For reference only, the old benchmark can be found (HERE) (removed from docs)

In short: asynchronous streams from dart work perfectly, but add a latency that we want to remove on Get_server.
Synchronous dart streams have unexpected behaviors, cannot have more than 1 listener and do not deliver events in the correct order, which completely prevents their use in mobile state managements, since you run the risk of displaying data on the wrong screen, since the last event will not always be the last event entered by the sink.
The 9000% figures are real, however, they refer to the gross performance between Streams and GetStreams. This does not mean that this number will impact your applications, because you are unlikely to use all of that power.
This does not mean that the dart streams are bad, the truth is that I have worked with many languages, and for me, reactive programming in dart is the most satisfactory. I'm just saying that this should not be a priority for the dart team, which has a lot of work to do, this language has evolved at a rapid pace with each update, and probably improving an api to support multiple listeners or delivering events in the exact order is not a concern now.

I want to apologize if I caused turbulence in the community, and I hope that from today our community will be more united, without discussions about Getx, Provider, Bloc, Get_it_mixin, Mobx is better, without people denigrating each other's work, without people with a tone of superiority. And I wanted to ask for respect in discussions by everyone in the community, including GetX users.
Let us together create a more united and healthy community.