-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[wip] updating reproschema commands to the new pydantic model #36
Conversation
looks like a reasonable approach. a few comments.
|
ok, I understand that all the changes that we were doing to the pydantic model, including changes the type |
you already have that in the meaning/mapping/schema_uri components of the linkml schema. the generator does not add that to pydantic, and that may be good to add to the metadata. i thought you or puja was going to look into how to add other metadata to the generated code on the linkml side. more specifically, those tests are really not helpful at the moment :) |
Yes, indeed, there is a current PR that should add |
…anges in formating
for more information, see https://pre-commit.ci
update conversion tools and their tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we also do the things we discussed earlier in the thread (#36 (comment))?
raise Exception(f"expected a list or dictionary, got {data_el}") | ||
|
||
|
||
def fixing_old_schema(data, copy_data=False): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we call this migrate_schema
, include a version check, and set schemaVersion
to 1.0.0
?
reproschema/models/reproschema
Outdated
@@ -0,0 +1,252 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this file needed in this repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought that first we will release reproschema
and after this I will point to the specific version
@@ -57,13 +55,11 @@ def validate_dir(directory, shape_file, started=False, http_kwargs={}): | |||
return True | |||
|
|||
|
|||
def validate(shapefile, path): | |||
def validate(path): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should this also check schemaVersion
and either ask user to migrate or ask them to use the older shacl-based validator?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but I thought that the idea was to use fixing_old_schema
(or migrate_schema
) for the older version. This is why I was checking entire reproschema-library
with the new validation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but this function doesn't check anything, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it initialize the pydantic class, so will get an error if the arguments are off
yes, I will work from the new redcap2reproschema scripts that @yibeichan updated |
@yibeichan - can you check if the fields with compute makes sense. I checked the phq9 as Satra suggested and there is indeed no isVis there |
@djarecka yes, but the newly generated compute items are not correct. there are many repetitive items, you can see a lot of
|
@yibeichan - ok, will check the repetition. btw., I believe I also found some error with |
@djarecka for |
@yibeichan - just to be clear, we are talking about Also not sure what are other options that you might have in other redcap cvs. I will add perhaps some exceptions, because now there are ignored |
the demo protocol has a file upload item. perhaps check that out. |
can you check now. I think the loops in the codes was wrong and it went multiple times through rows. Now the codes runs much faster... |
yessss! it's way faster and all repetitive results have gone!! thank you!! |
@djarecka what are the remaining issues we need to fix before merging this PR? |
yes, I found what about I the "Field Annotation" I can see multiple mysterious values, that I'm pretty sue are not human-readable descriptions, but have no ide what to do with them, e.g. "@hidden @NOW-SERVER" or "@noneoftheabove=5" |
yes, it's in the UI: https://github.com/ReproNim/reproschema-ui/blob/05096f07b3c27e82ee9382849b073c0896448a47/src/components/Inputs/DocumentUpload/DocumentUpload.vue#L2 (is this what you need?)
for
sorry about this, yes, it should not be translated to
yes, for row 1877, it has row 1879, it has row 1886 row 2455 that's a lot of variations, we probably need a dictionary about how to map them. I can take a look at all HBCD and bridge2ai CSV files and find unique values of |
I want to make two main points from HBCD:
These issues are completely unrelated to my changes. I've been fixing multiple old issues with the code, since I'm getting newer examples, and I can keep doing this (if someone helps me with it), but perhaps this should be done in another PR? |
you mean SQL cannot be directly put into
so among all HBCD csv files I have,
I agree that those are complicated but critical issues. we should get this PR merged, release out, and open another PR to fix those issues |
@djarecka anything else we need to check/update in this PR? If not, I guess we can merge it and move on (to get the release out) |
Don't have good internet today, but working on a couple more commits and
later today I will add them to the pr and will create issues with things
that need to be done in addition
…On Mon, Jun 10, 2024, 9:49 AM Yibei Chen ***@***.***> wrote:
@djarecka <https://github.com/djarecka> anything else we need to
check/update in this PR? If not, I guess we can merge it and move on (to
get the release out)
—
Reply to this email directly, view it on GitHub
<#36 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMV6GT2V32AFQGC53KJRSLZGW4IZAVCNFSM6AAAAABFZZZXJKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYGU3TIMZQGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
…ors if not found); fixing choices and adding slider; adding sql to compute types (this does not work properly right now); adding many comments
@yibeichan - at the end I added many command to do the script re. If that looks goo, one thing we should change for sure, is the path to contextfile |
…fixing preamble (can be either activity level or issue level
for more information, see https://pre-commit.ci
@djarecka thanks for the work!! I tested nimh-minimal with a roundtrip: reproschema2redcap and redcap2reproschema. Everything looks good, except the branching logic. I'm going to merge this PR now |
* adding pydantic model from the reproschema ref/linkml branch * fix altLabel * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updates to the model: updating AllowedType, adding MissingType to value, updated to ResponseOption and image * running pre-commit * ENH: updating reproschema commands to the new pydantic model (#36) * add print for testing * update clear_header * remove print * fix order and other errors * change ui yesno to radio * fix typo * update context, field->item, fix isVis * remove useless due to failed validation * remove visibility at the item level & remove matrixInfo * fix choice * remove identifier * updating validate command to the new pydantic model * updating/fixing the tests; updating the model to use CreativeWork; changes in formating * fix conversion tests * remove test output * change test output directory * final improvments on tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * model version after adding Thing class * updating model after removing CreativeWork and ImageUrl * adding tests to initialize the model classes * fixing load_file; adding write_obj_jsonld function and expanding test_schema * changing redcap2reproschema to use ned pydantic classes; some small changes to the pydantic model * changing name from string to landstring with en as language * fixing jsonld files * Adding option to return compact schema to load_file * fixing the protocol jsonld file * changing reproschema2redcap to use the new model * adding contectfile to write_obj_jsonld function and improving test; improving compact option for load_file * fixing reproschema2redcap and tests * removing file with the context and fixing references to the context_url (for now the link rfom the ref/linkm branch * updating the reproschema2redcap to work for activity/items from urls * improving error message for file_load and validate; checking the suffix of the file before treating it as jsonld * fixing identify_model_class, so Item and Field are treated the same * fixing reproschema2redcap so it reads responseOptions from another file * rewriting parts of redcap2reproschema, fixing some bugs[wip] * fixing compute: removing isvis condition * fixing process_csv so it doesn't go multiple time through the same condition * changes to input and value mapping (mapping explicitly or raising errors if not found); fixing choices and adding slider; adding sql to compute types (this does not work properly right now); adding many comments * adding output for redcap2reproschema command; removing argparse * model without decimal; revert changes to valueType in the model * adding migrade command * fixing multiple issues with redcap2rp and rp2redcap: adding compute, fixing preamble (can be either activity level or issue level * WIP: addining test to test rp2redcap and redcap2repo using nimh exampl * fixing paths in the tests, should run now * ignore .DS_Store files in validation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: yibeichan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rename to be consistent --------- Co-authored-by: Dorota Jarecka <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@satra - for now I updated only
validate
, but can you please take a look if this is what you had in mind?also, I wasn't sure where the context should be, for now I added to
models