Benchmark/comparison against MiniGPT-4?

[152334H]

Or just thoughts about which architecture should be expected to perform better with the same training data

[ChunyuanLI]

It is great to see the community's recognition and excitement about this direction; Both pieces of work are taken independently during the same period.

IMHO, LLaVA is unique at three aspects, see below.

1. More rigorous results: LLaVA has rigorous quantitative results, including the level of similarity with Visual Chat and GPT-4, the SoTA accuracy on Science QA, and ablation studies on data iteration and model design. Mini GPT-4, on the other hand, lacks quantitative results.
2. Quality of Chat Demo: LLaVA can reproduce results for visual reasoning examples in GPT-4 paper and has strong OCR capabilities. These features are impressive and unique, making it possibly the closest demo to Multimodal GPT-4. Check results: https://llava-vl.github.io
3. Lastly, it should be clarified that the focus of this line of work is data-centric, not model-centric. As the differences in models are diminishing, data quality has a greater impact on results. We released our multi-modal instruction following data, to replicate Multimodal GPT-4. The high quality data is all you need (compare which, the architecture is secondary).

[Pilot-LH]

I agree with the author that LLaVA is better than MiniGPT-4 in terms of demo quality and comprehensive analysis.
Regarding the last point, I attempted to fine-tune the BLIP-2 model (based on Flan-T5) using high-quality data provided here, but did not achieve outputs as interesting as LLaVA or MiniGPT-4. While it's possible that I didn't execute the process properly, I'm curious if architecture is indeed a secondary factor. Can Vicuna be substituted with the original LLaMA or other open-sourced models to yield comparable results? Such a development could result in a fully open-sourced model with greater impact, in my opinion.

[ChunyuanLI]

Comparison of LLaVA and mini-GPT4 on "Extreme Ironing" example from the OpenAI GPT-4 technique report.

LLaVA

---

mini-GPT4

Run 1:

Run 2:

[xdevfaheem]

Can You Show The Configs You Set to Get this Result?

i really like LLaVA, I really Wants LLaVA To Perform Better Than mini GPT-4.

[ldfandian]

I feel it is hard to believe LLaVa is better.

miniGPT-4 use "BLIP-2 QFormer + Project Layer" vs LLaVa use "purely Project Layer".
Technically, miniGPT-4 is able to handle more sophisticated scenarios.

[kyugorithm]

Regarding the vision encoder, it is true that BLIP-2 QFormer outperforms CLIP significantly in terms of performance. But LLaVA fine-tunes the LLM (Vicuna) model. Moreover, I believe LLaVA employs a more effective data-acquisition approach.

[xdevfaheem]

How Will We Mitigate Hallucination issue?

[Glavin001]

My understanding of hallucinations is that training datasets often include answers which the LLM may not know how to get, such as expecting them to reply with a fact the LLM actually doesn't know, thus training it to make up reasonable looking information instead of accurate.
In the case of vision models, I wonder if the visual encoding doesn't provide enough detail for all of the examples so the LLM learns "make up something reasonable in this scene" instead of "I see this from the visual input".

[mostafa-adel]

when I run "python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload
". I got this issue "2023-12-09 23:02:03 | ERROR | stderr | Traceback (most recent call last):
2023-12-09 23:02:03 | ERROR | stderr | File "", line 198, in _run_module_as_main
2023-12-09 23:02:03 | ERROR | stderr | File "", line 88, in _run_code
2023-12-09 23:02:03 | ERROR | stderr | File "/Users/mostafaabdo/Downloads/LLaVA-main/llava/serve/gradio_web_server.py", line 22, in
2023-12-09 23:02:03 | ERROR | stderr | no_change_btn = gr.Button.update()
2023-12-09 23:02:03 | ERROR | stderr | ^^^^^^^^^^^^^^^^
2023-12-09 23:02:03 | ERROR | stderr | AttributeError: type object 'Button' has no attribute 'update'". Could you please help me running it?