Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codeparrot/codecomplex #806

Open
wants to merge 5 commits into
base: eval-hackathon
Choose a base branch
from
Open

Codeparrot/codecomplex #806

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
619daa1
Merge pull request #4 from bigscience-workshop/eval-hackathon
Muennighoff Jul 19, 2022
a3ed6a3
Add cp complexity
Muennighoff Jul 22, 2022
0b99615
Adapt names
Muennighoff Jul 22, 2022
ae7f431
complexity -> time complexity
Muennighoff Jul 28, 2022
27f6a43
Add clue @07e1c20b2f4e8ac1af98d9a7a9cf3d05f007f36d
Muennighoff Jul 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion promptsource/templates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,21 @@
# These are users whose datasets should be included in the results returned by
# filter_english_datasets (regardless of their metadata)

INCLUDED_USERS = {"Zaid", "craffel", "GEM", "aps", "khalidalt", "shanya", "rbawden", "BigScienceBiasEval", "gsarti"}
INCLUDED_USERS = {
"Zaid",
"craffel",
"GEM",
"aps",
"khalidalt",
"shanya",
"rbawden",
"BigScienceBiasEval",
"gsarti",
"Helsinki-NLP",
"Muennighoff",
"facebook",
"codeparrot",
}

# These are the metrics with which templates can be tagged
METRICS = {
Expand Down
45 changes: 45 additions & 0 deletions promptsource/templates/codeparrot/codecomplex/codeparrot--codecomplex/templates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
dataset: codeparrot/codecomplex
subset: codeparrot--codecomplex
templates:
5b108b1c-7514-488f-99ed-3ca5da70e103: !Template
answer_choices: null
id: 5b108b1c-7514-488f-99ed-3ca5da70e103
jinja: '{{ code }}
What is the complexity of the previous code?
Muennighoff marked this conversation as resolved.
Show resolved Hide resolved
|||
{{ complexity }}'
metadata: !TemplateMetadata
choices_in_prompt: false
languages:
- en
metrics:
- Other
original_task: true
name: whatcomplexity
reference: ''
1d85c898-70fe-4a51-be37-5111be357762: !Template
answer_choices: null
id: 1d85c898-70fe-4a51-be37-5111be357762
jinja: "Identify the complexity of the following code as constant, linear, quadratic, cubic, log(n), nlog(n) or NP-hard. {{ code }} |||{{ complexity }}"
metadata: !TemplateMetadata
choices_in_prompt: false
languages:
- en
metrics:
- Other
original_task: false
name: identifycomplexity
reference: ''
5d85c898-70fe-4a51-be37-5111be357762: !Template
answer_choices: null
id: 5d85c898-70fe-4a51-be37-5111be357762
jinja: "{{ code }} Which one is the correct complexity of the code snippet: constant, linear, quadratic, cubic, log(n), nlog(n) or NP-hard? |||{{ complexity }}"
metadata: !TemplateMetadata
choices_in_prompt: false
languages:
- en
metrics:
- Other
original_task: false
name: whichcomplexity
reference: ''