Leaderboard Gradio or Dash?

[x-tabdeveloping]

Hey guys! I have been working on the new leaderboard for MMTEB for the last bit, and I have a couple of considerations that I want to put out here so that we can take an informed decision together, as I expect that some of these things might affect some of you just as much as me.

I started working on the leaderboard in the Gradio framework, which is thee one that was used for the original leaderboard too.
I must say, that I find it quite limiting and I wanted to ask you if it would be okay with you to base the new leaderboard on Dash instead.

Here are a couple of advantages of switching to Dash:

1. Writing UIs in Dash is waaaay more convenient. For one, it is much more flexible, and the components available to you are of way higher quality. I, for instance use the Mantine component library quite a bit, which has a rich array of high-quality components. Gradio is missing a number of components that I would like to work with (for instance a double-ended slider, or a switch), and doesn't have the same flexibility when something needs to be modified. Some component behaviours are also quite weird. For instance, a multi-item dropdown never becomes scrollable, even when the number of elements grows quite large. I just have a feeling that most of it is rather rough around the edges and gets tedious to work with when more complex interfaces or behaviours are needed.
2. Interactivity is also much more clear-cut in Dash. All abstractions are rather explicitly built on dependency graphs, and the dependencies are very clearly spelled out. It might just be my inexperience, but I feel that Gradio is quite a bit more opaque in this regard.
3. I have quite a bit of experience with Dash, and I know for a fact that you can build complex apps in it without things getting out of hand (topicwizard, embedding-explorer). I have a gut-feeling that Gradio is incredibly convenient for smaller apps, but might not be a good fit for the complexity that a multilingual, build-your-benchmark style of leaderboard requires.
4. Optimization is Dash is simpler. This is simply because it is, yet again, very clear what runs on the server and client sides, and Dash also allows you to write client-side operations in Javascript, which can be quite a significant speedup in certain cases. I have only been working on the leaderboard for a week and I have already encountered performance bottlenecks, that I found a bit hard to iron out.

On the other hand there are also quite a few disadvantages to switching to Dash, namely:

1. Dash code is significantly less readable, and mostly relies on you already being familiar with common concepts in web development. This might admittedly make it harder for Python developers (like most of us) to work with.
2. Deployment is more complicated. Gradio apps are incredibly easy to deploy on HF Hub, and Dash apps require a bit more work, which again, requires people to understand Docker and gunicorn. I would not consider this a deal-breaker, but it's certainly an additional inconvenience.
3. A good Dash app might need more code and man-hours. I'm 100% convinced that it would result in a much better leaderboard, but this is definitely something to take into account.

I'm curious to hear your opinions about this. I don't have terribly strong feelings about the issue, and I'm sure that I could still push out a decent leaderboard in Gradio, but I feel like it would make my job easier if I was allowed to use Dash.

[x-tabdeveloping]

I'm just gonna tag you, so that you can chime in @KennethEnevoldsen @orionw @Muennighoff @vaibhavad @imenelydiaker @isaac-chung @tomaarsen

[orionw]

Thanks @x-tabdeveloping for the great work!

I think one highlight of the current leaderboard is that anyone can come in and edit it/make PRs. I am a bit hesitant to move off of the simple supported libraries like Gradio and Streamlit.

OTOH, I'm certain you would create a very slick UI in Dash though and we could probably pick it up -- plus how much do we really change the leaderboard beyond adding models and/or new tabs?

So I would say I probably lean against switching, but could be convinced if the UI is that much better for Dash/some other framework. But I haven't been working with the leaderboard for long so I defer to others.

[Muennighoff]

Great overview! I'm not sure about the details of the 4 Gradio issues so tagging @abidlabs & @pngwn & @dawoodkhan82 - It'd be great to get your thoughts (it is about changing the mteb lb space - https://huggingface.co/spaces/mteb/leaderboard).

[abidlabs]

Hi @x-tabdeveloping and folks! Thanks for the great issue, this is super helpful for us to learn what the pain points for Gradio users are. I'll address the points that you mentioned, but ultimately will leave to you all to decide the best path forward for the leaderboard:

Writing UIs in Dash is waaaay more convenient. For one, it is much more flexible, and the components available to you are of way higher quality. I, for instance use the Mantine component library quite a bit, which has a rich array of high-quality components. Gradio is missing a number of components that I would like to work with (for instance a double-ended slider, or a switch), and doesn't have the same flexibility when something needs to be modified

If there is a component that is missing in Gradio, you actually can create a custom component, which is a combination of Python + Svelte-flavored JavaScript. The effort required in writing a custom component varies, but for small changes like adding scrolling to the Dropdown choices, its quite straightforward as you can template off of the existing Dropdown component and add the modification. You can then publish it so its available to other users. You can browser ~80 custom components here: https://www.gradio.app/custom-components/gallery, which actually includes a double-ended slider.

Interactivity is also much more clear-cut in Dash. All abstractions are rather explicitly built on dependency graphs, and the dependencies are very clearly spelled out. It might just be my inexperience, but I feel that Gradio is quite a bit more opaque in this regard.

It is true that Gradio events are not built on dependency graphs, but this is an explicit decision, as we want our developers to be able create interactive web apps without needing to delve deeply into underlying structures like dependency graphs. That being said, we are open to suggestions on how we can improve the developer experience for events, e.g. we could provide an alternative way to define an event for complex apps. Speaking of which...

I have quite a bit of experience with Dash, and I know for a fact that you can build complex apps in it without things getting out of hand (topicwizard, embedding-explorer). I have a gut-feeling that Gradio is incredibly convenient for smaller apps, but might not be a good fit for the complexity that a multilingual, build-your-benchmark style of leaderboard requires.

Gradio has been successfully used for quite complex apps! Some examples include Automatic1111, Fooocus, and oobabooga, and kotaemon.

Optimization is Dash is simpler. This is simply because it is, yet again, very clear what runs on the server and client sides, and Dash also allows you to write client-side operations in Javascript, which can be quite a significant speedup in certain cases. I have only been working on the leaderboard for a week and I have already encountered performance bottlenecks, that I found a bit hard to iron out.

This is possible as well in Gradio. Every event includes a js parameter, which can be used to define functions that should run client-side to modify the value of a component. Again, open to feedback here -- if you see specific limitations about what cannot be done in Gradio, let us know and we are totally open to finding a way to support it.

[KennethEnevoldsen]

Given that we are expecting many new changes going forward (MIEB etc.) I am generally leading toward "easier to change" > "better leaderboard". Ideally, however, it should be both :)

I do believe both frameworks can solve both problems though. For me, it is fine to leave the decision up to @x-tabdeveloping (we can also discuss it further in person tomorrow/monday).

[pngwn]

Hey folks. As a maintainer of gradio I thought it would be good to add some other details that may be relevant in addition to what @abidlabs mentioned.

In v5 (the release of which is imminent) gradio will set itself apart from most similar tools like dash with a couple of features.

Accessibility

available today but constantly improving

We have made a huge effort to ensure that gradio is as accessible as possible to as many users as possible including making gradio applications navigable with both keyboards and other assisted devices and adhering to best practices around color contrast and element sizing.

Security

available at v5 launch and constantly improving

We have been working tirelessly over the past few months to make gradio the most secure way to build machine learning applications. This includes working with an independent third party to make everything as good as it can be.

We will have more extensive details at launch (very soon).

Server Side Rendering

available at v5 launch

Starting in version 5 gradio applications will be rendered in the server as well as the client. This leads to significantly better behaviour in a number of ways:

• perf: initial content rendering - this will happen almost instantly now with no loading spinner.
• perf: time to interactivity - this will also happen significantly faster as there are fewer server round trips required.
• SEO - while this depends how you structure your app to some degree, as gradio apps are fully rendered on the server (and semi functional as a static view) seo will be much better. In theory search engine crawlers are now able to run JavaScript but in practice static or server rendered sites are much more compatible with good seo.

Multiple pages

coming shortly after the v5 release

In an update post v5 launch, users will have the ability to create multiple pages. This allows far more flexibility in how an application is structured as it will allow more nuanced information architecture and UX design.

Instead of needing to put everything in tabs (which have now been redesigned and are much cleaner) you will have the option of using pages which will be great for detailing more information about the project or providing instructions for submissions.

These are not rudimentary FAST API routes either, gradio will be using SvelteKit under the hood, so sub pages will also be server rendered if a user visits them directly but subsequent navigations will be highly optimised client side navigations with intelligent asset preloading, leading to a seamless user experience.

---

I'm not in the business of convincing people to use tools if they don't want to but hopefully this is helpful in some way. I fully support your decision to use whatever tool seems best to you.

Gradio certainly isn't perfect and we are always hoping to improve, but we do believe we focus on certain details far more than most other frameworks which can sometimes stretch us a little more than if we just focused on the UI. While there are a few rough edges that we need to smooth out, with v5 and the changes planned shortly after v5, we don't really think any other tool has a comparable feature set without dropping down to a JavaScript based meta-framework.

Regarding the pain points you have encountered, I'd love to hear more about them regardless of what framework you end up choosing. Would you be willing to chat more about them so we can figure out how to improve those issues? We could jump on a call sometime or we can do it asynchronously. Leaderboards are a core use case for us so we do want to make the experience as good as possible and your feedback would be invaluable, we are also happy to work with you to help resolve any issues if you do end up going with gradio.

[KennethEnevoldsen]

Thanks for these detailed messages @pngwn and @abidlabs having support from the gradio team and knowing that multiple improvements are coming up is def. something that we value greatly. I will grab a discussion with @x-tabdeveloping later today -- we will get back with a more detailed response

[x-tabdeveloping]

Hey all! Thanks for the insightful comments, I really appreciate them! (also sorry for the delay, I have been a bit busy with other stuff) As some of the responses have demonstrated, at least some of my problems were stemming from me being inexperienced with Gradio, thanks for pointing these out.
I will quickly try to explain with some examples what types of things made me frustrated as a new user, and what made me gravitate towards Dash:

Example 1:

The examples are semi-fictional

You have two dropdowns, one of them is multi-choice and contains languages, the other one is single-choice and contains language families.
These dropdowns influence each other. When you select a language family, the languages in the other dropdown get changed to the languages that belong in that family. When you change the languages, the language family dropdown either changes to "custom" or if the languages match a family, it changes to the appropriate family.
This is a circular dependency. In Dash, users are very much encouraged to resolve dependencies like this, and there are some very explicit tools, that aid you in this (PreventCallback exception, being able to specify whether a dependency is an Input or a State etc.)
In Gradio, if I specify two callbacks like this:

@gr.on(inputs=[lang_dropdown], outputs=[fam_dropdown])
def update_fam():
    ...

@gr.on(inputs=[fam_dropdown], outputs=[lang_dropdown])
def update_langs():
    ...

it's not immediately clear to me how this gets resolved, or if it results in an infinite loop, and I'm also unsure whether these callbacks are only called upon user interaction or also when other callbacks trigger them.
This is of course not the end of the world, and I could probably figure it out if I spent a bit more time in Gradio's docs and played around with it for a little bit, but I do think this syntax is a bit unintuitive, and I honestly prefer the explicit nature of Dash's dependency graphs and callback definitions.

Example 2:

I want to have multi-select dropdowns, that become scrollable if they reach a certain height.
I tried implementing this in Dash and it took me about five minutes (I'm using dash-mantine-components and Tailwind in this example, because it's what I'm used to, but I'm 100% certain this would be just as easy in vanilla Dash)

dmc.MultiSelect(
      label="Select Languages",
      data=[...],
      clearable=True,
      searchable=True,
      value=[...],
      classNames={
             "pillsList": "overflow-y-scroll max-h-32"
      },
)

While I haven't put too much effort into figuring this out in Gradio, judging by the fact that @abidlabs recommended me to write a custom component, my immediate judgement is that this would be a tad more complicated to do in Gradio.

I think this can in general be attributed to the fact, that Gradio does not expose html and css as much as Dash does (in Dash, you are literally using html components as the default).
This is a huge plus for usability, since people don't have to have experience with css and html to be able to write nice UIs (which to be frank are two of the most horrendous technologies I have learnt in my lifetime, I wouldn't wish it to my worst enemy), on the other hand, it limits how flexibly one can manipulate layout and elements.

Example 3:

I want a double-ended slider.

Dash	Gradio
Comes prepackaged with the library (dcc.RangeSlider) or I can use Mantine, which has loads of users and is very actively maintained (dmc.RangeSlider)	I can use gradio-range-slider which is a package, that only contains this and is maintained by a single person.

While I don't have any doubts that gradio-range-slider is a high quality component, and that the maintainer has the best intentions and puts effort into keeping the package up to date, it is nowhere near the level of convenience and security of just being able to use a range slider out-of-the-box and being rather sure that I will be able to use it as long as I need it.

Again, these are not unfixable problems, but for a new user, like me, they add up quite quickly.
I'm also sure you guys will have meaningful answers to some of these, and they will present less of an issue as I get more experience with Gradio.
I hope I could be helpful.

[x-tabdeveloping]

That being said, I can feel that most of the team is leaning towards sticking with Gradio, and I see reasons for why this would be a good idea.
I also hope that my inputs can help you either explain me where I do something wrong or iron out the inconveniences that I'm facing.