Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

195 Open 338 Closed
195 Open 338 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

SafetyBench数据集评测bug
#1622 opened Oct 18, 2024 by shutttttdown
2 tasks done
@tonysy
[Bug] 使用Circular方式评测，few-shot数据拼接错误
#1615 opened Oct 17, 2024 by Alexyuanfun
2 tasks done
@acylam
[Feature] 请求添加BigCodeBench评测集 backlog
#1613 opened Oct 16, 2024 by BinBrent
1 task
@MaiziXiao
1
[Bug] 评估wikitext ppl数据集时候由于没有reference无法计算结果
#1608 opened Oct 15, 2024 by LanDisen
2 tasks done
@liushz
[Bug] hf推理与vllm推理评测结果不一致
#1594 opened Oct 9, 2024 by luhairong11
2 tasks done
@MaiziXiao
[Bug] AttributeError: 'ConfigDict' object has no attribute 'reader_cfg'
#1591 opened Oct 9, 2024 by xtuhcy
2 tasks done
@MaiziXiao
1
[Bug] 评估过程中的oom 问题
#1587 opened Oct 8, 2024 by tonney007
2 tasks done
@liushz
3
[Bug] Silent GPU failures, works with --debug
#1585 opened Oct 4, 2024 by anuragprat1k
2 tasks done
@tonysy
[Bug] pid_params.py dumps not successfully.
#1580 opened Sep 29, 2024 by tonysy
2 tasks done
@tonysy
[Bug] It seems run wrong number of prompts when max-num-workers > 1.
#1579 opened Sep 29, 2024 by zhulinJulia24
2 tasks done
@MaiziXiao
[Bug] debug不显示日志
#1578 opened Sep 29, 2024 by HaltonJiang
2 tasks done
@tonysy
1
[Bug] mtbench101：batch_size设置
#1571 opened Sep 27, 2024 by zhang-junjian
2 tasks done
@tonysy @bittersweet1999
1
[Feature] Support MMMLU backlog
#1560 opened Sep 24, 2024 by tonysy
1 task
@liushz
1
[Bug] HuggingFacewithChatTemplate
#1557 opened Sep 24, 2024 by ZCzzzzzz
2 tasks done
2
[Bug] 试图添加mlc后端遇到问题，找不到cuda
#1551 opened Sep 23, 2024 by XJY990705
2 tasks done
@bittersweet1999
[Bug] SciCode miss with_background Bug Something isn't working
#1540 opened Sep 18, 2024 by tonysy
2 tasks done
@MaiziXiao
[Bug] mbpp_plus.jsonl 在哪里，下载的zip中没有这个东西
#1538 opened Sep 18, 2024 by kkwhale7
2 tasks done
@MaiziXiao
[Feature] About a subset of academic rankings
#1537 opened Sep 18, 2024 by mary-0830
1 task
@MaiziXiao
3
[Bug] PreTrainedTokenizerFast inference with unexpected parameter token_type_ids Bug Something isn't working
#1533 opened Sep 18, 2024 by kkwhale7
2 tasks done
@MaiziXiao
3
[Bug] MTBench数据集读取出现拼接bug
#1531 opened Sep 18, 2024 by zhang-junjian
2 tasks done
@MaiziXiao
2
使用Qwen2-72B-Base测试GPQA数据集时报错：NotImplementedError: OpenAI does not support ppl-based evaluation yet, try gen-based instead.
#1526 opened Sep 12, 2024 by 13416157913
2 tasks done
@acylam
2
[Bug] Cibench generate matplotlib has a wrong weights which should be [175, 2, 1, 156]
#1524 opened Sep 12, 2024 by imaginistLi
2 tasks done
@acylam
您好，请教一个问题，请问OpenCompass官网公开学术榜单中，测试的Qwen2模型中temperature、repetition_penalty、、top_p这些参数是设置多少的？模型部署的方式是采用什么方式部署的？
#1523 opened Sep 12, 2024 by 13416157913
2 tasks done
@acylam
3
[Feature] 数据集版本问题
#1509 opened Sep 9, 2024 by bjzhb666
1 task
@bittersweet1999
2
[Feature] How to config to support auto awq int4 llama3 model
#1506 opened Sep 9, 2024 by FlexLaughing
1 task
@bittersweet1999
ProTip! Adding no:label will show everything without a label.