-
Notifications
You must be signed in to change notification settings - Fork 4
Meeting Records
Previous meeting records can be found here: https://github.com/petermr/openVirus/wiki/Records-and-Reports
- Schematron - A tool to validate dictionaries.
- Everybody should come up with a set of rules that their dictionary should comply to.
- Briefly discussed the progress and the future directions for the project.
- GY: New Inters are to join us from January.
- Test-Driven Development
- Write tests that the dictionary editors should comply with.
- Discussed the structure of projects in Python. We looked at the project structure of 'Pymatgen', a project in which PMR is involved.
- Welcome Radhu, the new NIPGR intern!
- History and a bit of Context:
- 1.5 years ago, NIPGR-ContentMine internships started.
- Plant literature into a database.
- openVirus started in March with a focus on 'viral epidemics.
- Gita ma'am's group has developed EssOilDB which is now being converted to use for the newer technology. New interns will be working on this.
- We will continue to work on openVirus dictionaries as well.
- We follow 'open notebook philosophy'
- Review the EssOilDB and Emanuel work to make sure it's compatible.
- Review:
- Plant Taxonomy
- A typical paper about a plant has
- Place
- Part
- Chemicals
- Project Management
- Regular Meetings
- Agenda
- Run the Meetings
- Meeting Account
- Brief recap of the 'openVirus work' so far. (By Shweata, Ayush, PMR and Dheeraj)
- Requirments for
pygetpapers
is documented here https://github.com/petermr/openVirus/wiki/pygetpapers -
pyamidict
We discussed the general workings of the code PMR developed. https://github.com/petermr/dictionary/blob/main/pythoncode/pyamidict/editor/amidict.py
- Coordinate the review for
pygetpapers
-> Ayush - Revise dictionaries
- prototype code to search using Dictionary
- Proper labelling and naming of rules for dictionary validation
- Introduction about Julia Arsuffi, PhD Plant Science, Cambridge & Emanuel Faria, Brazil and their work
- Discussion about Manny works so far
- Dictionaries
- (Country, Disease, Drug, Organization)
- Plant(essential oil-producing), Extraction Methods, Phytochemicals
- Brief about tigr2ess Program to Radhu (https://github.com/petermr/tigr2ess)
- Explanation of
pygetpapers
by Ayush - Test-Driven Development: Collectively drafted some tests to implement for
pyamidict
- Branching: As multiple people are going to be involved in the development of software, it's important to make sure that we don't change each others' codes without knowing. Branching, therefore, becomes important. Radhu and Ayush, together are going to write a guide for all of us to help understand how branching works.
- Welcome Kanishka, Manisha the new NIPGR intern!
- Getting to know each other's computational backgrounds.
- Introduction given to each intern and brief overview of their role in project.
- We got an overview of the people, the project, the resources, how they are used, and how to build them.
- Discussion about the agenda behind openVirus (wikicite presentation) By PMR, to build a system so that anybody can understand the science behind the current pandemic.
- Gita ma'am's group has developed EssOilDB which is now being converted to use for the newer technology. New interns will be working on this.
- Aim to start "4-projects" - for new interns with different criteria including Parts of plant, Chemicals, Extraction method & Analytic method.
- Introduction about working of "Slack" and interns to new interns.
- Introduction about working of Github.
- PMR introduced 3 Software Project (getpapers, pygetpapers, Ami Dictionary).
- learn how to use the wiki and contribute.
- Discussion on running getpapers to retrive papers from medrxiv.
- Running maven to build ami.
- be able to run getpapers and ami with existing dictionaries.
- explore dictionaries in https://github.com/petermr/dictionary/tree/main/openVirus202011
- Welcoming our new intern, Prashant
- A brief introduction of new interns
- A short review of the CEV Open project
- Brainstorming and discussing potential project ideas, feasible in 6 months time
- Project Ideas generated at the end of the session:
-
Phytochemistry-specific Projects (tentative):
- Medicinal activities of plant essential oils
- Volatile Terpenes and genes
- Essential oils from invasive species
-
Tech/Independent Projects:
pygetpapers
pyamidict
search
- Location (Country)
- Organization
- Disease
- Drug(?)
-
Phytochemistry-specific Projects (tentative):
- Interns' standup
-
pygetpapers
Reviewing Ayush's code
- Shweata: Email all the volunteers of openVirus team informing the new developments and project ideas.
- Ambreen, Anugrah, Rajan, Vanisha, Vaishali, Aishwarya, Mukul
- Priya, Kareena, Sana
- Scoping Review: Preliminary searching and readings. Mainly to figure out the feasibility.
- Radhu: Medicinal activities of plant essential oils
- Kanishka: Volatile Terpenes and genes
- Prashant: Essential oils from invasive species
- Brief introduction by each intern and getting started.
- Briefly discussed the progress about Ambreen's wrok.
- A short review of the CEV Open project.
- Discussion about last 3-4 week's works so far.
- 4 core dictionaries:
- country (Ambreen) => location => geolocation services
- disease (Dheeraj) - human disease
- drugs (Rajan)
- organizations (Shweata, Vaishali)
-
pygetpapers
Reviewing Ayush's code. - Explanation of pygetpapers by Ayush to Ambreen and Vaishali.
- Review of PMC papers to explain the interns about sectioning and output of xml files.
- Review of dictionaries(Plants and openvirus).
- Review of shweata's and dheeraj's excellent developments on "instanceof" and OPTIONAL work.
- Discussion on SPARQL queries and downloading the .xml file.
- Extended discussion on dictionaries, editing the wikipage dict schema, different terms and elements of wikidata.
- Review on Poster of "WikiFactMine for Phytochemistry".
Thanks @Radhu. good start. There are some typos (wrok, Pythochemistry) Some things are case sensitive or space sensitive ("instance of", "CEVOpen"). This matters because machines can't correct errors. Can you remind us of the core projects? Did we decide on the fourth project for the new intern? I think we had "pheromones" . We'll need to agree this with Gita.
We need a little more detail (perhaps an extra line or two) where decisions were made. Where we have "extended discussion" we certainly need more. We have to assume that some people won't have been able to join and the record is important.
- Briefly discussed the progress and the future directions for the project.
- Discussed on the new and exciting directions to the project.
- Standup by interns (those who are present)
- Extended discussion on dictionaries, editing the wikipage dict schema, different terms and elements of wikidata
- Some requirements we identified
- merge entries with same WikidataID
- detect and eliminate scholarly articles, books, etc.
- add language wikipedia pages from wikidataID
- (SH) post-SPARQL filtering, or query refinement
- translate attributes into wikidata properties where possible (crossrefid => _p3153_crossrefid)
- remove unwanted terms (term value or wikidataID)
- Introduction about planner for new interns
- Dictionaries
- Plant(Radhu)
- Compound(Kanishka)
- Gene(Prashant)
- Incomplete dictionaries(?)- Activity, extraction method, plant parts [Ask Emanuel] - Missing Wikidata items
- Revise description and extract synonym.
- Some terms have wikidata id of scholarly articles
- Does Wikidata id of terms still exist? (Some items might have either be moved or deleted since the dictionary was created)
- Write python code to go through the ids and check if they exist
- PMR: Write software to convert SPARQL output into the dictionary.
- volatile_compound
- PROBLEM: Chemicals have commas in them. AltLabel gives synonyms separated by a comma.
- PMR: Query all the chemicals automatically and look them up.
- Find out if the compounds are in CheBI
- found in taxon property - P703
- Genes dictionary - Contact Guilia
- PMR's Network has dropped and he has a meeting so no meeting on Thursday.
- Welcome Vasant, Talha the new NIPGR interns!
- Getting to know each others Backgrounds.
- General instruction given to all by Gitanjali Ma'am.
- We got an overview of the people, the project, the resources, how they are used, and how to build them.
- Discussion about the agenda behind openVirus By PMR, to build a system so that anybody can understand the science behind the current pandemic.
- Introduction about working of "Slack" and interns to new interns.
- Introduction about working of Github.
- Mini Projects: Each project should have a scientific target. It must involve technology development.
- Medicinal oils (Emanuel (Manny) Faria) (Radhu Ladani)
- Genes (Giulia Arsuffi) (Talha Hasan)
- Invasive species ( Gitanjali Yadav) (Kanishka Parashar)
- PMR introduced 3 Software Project (getpapers, pygetpapers, Ami Dictionary).
- Briefly discussed the progress and the future directions for the project.
- Explanation of a dictionary given by PMR , "Ocimum sanctum". Introduction to 'xml' markup language and other components such as 'elements', 'attributes' and Q number for wikidata.
- Scientific strategy discussion over "Why we are doing these Projects?" To build an organized system using technological tools
- Review of each intern's dictionary. Each one to create their dictionary's own wiki page.
- Review of each miniproject (update your project pages with progress made, Create pages for tools which you use)
- Everyone to commit their miniproject data on Github ( https://github.com/petermr/CEVOpen/wiki)
- Main components of intern activity:
- Technology - (getpapers, ami, wikidata/SPARQL) - search
- Dictionaries (1 existing dictionary, 1 new dictionary) approx.
- Miniproject chemotype, genotype, activities (medicinal) phenotype - invasive species
- Integration - how these fit together - an atlas
- All new topics of project will be discussed in slack and create Wiki pages on CEVOpen as Slack is for immediate conversations and GitHub is for structured technical conversions.
- Brief project review by PMR to GY regarding the progress made and upcoming tasks.
- Standup by each interns.
- Introduction to interns about
getpapers
andami
search query by PMR. - Review of Vasant and Talha's miniprojects.
- Review of dictionary- if each contains name, term, wikidata ID, wikidata label, description, Wikipedia URL.
- Update the dictionary-Do all entries have synonyms? Wikipedia pages? If not we have to add them and how to retrieve these using VALUES.
- Briefly discussed the progress and the future directions for the project.
- Interns' standup
- Discussion of organization documents of Github and blockage of interns.
- Explanation of Shuttleworth Foundation by PMR including their fellows, fellowships as well present the ideas of people who are open, innovative, global, etc.
- Open: “A piece of data or content is open if anyone is free to use, reuse, and redistribute it - subject only, at most, to the requirement to attribute and/or share-alike.”
- Discussed the outline of "FlashForward presentation"
- ISC report and overview (PMR)
- Shweata overview
- Tigr2ess -> open climate -> open virus -> CEVopenPlant (-> crops, -> plant technology)
- Ayush pygetpapers (launch)
- demo of Wikidata/SPARQL - Dheeraj? + interns?
- (miniprojects) -> matplotlib displays
- (background photos - farms, landscapes, etc.) - stress mobile?
- Theme finalization about event: "Scientific knowledge for Global Challenges"
- Review of
pygetpapers
and explanation of Flowchart by Ayush and PMR. - difference between
pygetpapers
andgetpapers
https://github.com/petermr/dictionary/blob/main/pygetpapersdev/oldgetpapers.js
- Standup by interns (those who are present)
- Planning for the event (Flash Forward Workshop). It is going to be a hands-on session for people to try out our software and give us feedback. We are also going to discuss briefly about our current projects, its motives, and so on.
- Presentation Discussion Regarding OpenVirus which is a team of volunteers who build software to query the scientific literature automatically in large amount.
- We will explore the technology and the issues:
- where can you find science?
- what's its value? hoe can it be used?
- how much is hidden by the punishers?
- how can you do this yourself?
- Flash Forward Workshop. This will be a workshop where
- we will demo the technology
- discuss what the world would like to be able to do. This includes non-English languages.
- PMR will introduce current project EO, past projects climate and epidemics ? demo is specific topic of invasive species ? theme invasive species ? introductions ? [St Edmunds Game - Gita. Present science to non-scientists. Fun! Involved. May 2021]
- Presentation discussion of openVirus Overview of the project:
- Shweata (Textmining software and community - Introduction) - 5 min.
- Ambreen (Her experience, initial results and machine learning 101 - 5 min.
- What interns can present (CEVOpen - Invasive species theme):
- Kanishka - Intro to Invasive species project - 2 min. recorded video
- Talha - Megapublishers and Manual search of scientific literature - 2 min.
- Radhu - Intro to Wikidata, Invasive species dictionary creation demo - 2 min.
- Vasant - Jupyter Notebook demo - data display (histograms, cooccurrence) - 2 min
- Ayush - pygetpapers demonstration - 5 min.
- Discuss the time slot for the presentation
- interns gave demo for their particular topic
- PMR and other members suggest points and correction so presentation would me more precise.
- get familiars with the BigBlueButton
- Day of Flash Forward Workshop
- review of FlashForward presentations
- review of personal commitments (interns) including their College Schedule, Exams and other things.
- alpha testing -> Preliminary tests for software.
- Every interns have to do testing of
pygetpapers
and recording in detail what they did. Good documented testing is one of the absolutely essential aspects to software.
- Every interns have to do testing of
-
pyamisearch
works and needs alpha-testing. It will be used onsections
. Currently we can extract given sections quite well . There are a number of short, important subsections and each intern can take one.- acknowledgements
- conflict of interest
- ethics statements
- author contributions
- CEVOpen at StEds. Event in Cambridge ca 2021-05-01 showing our projects and software to Cambridge students, researchers and faculty. Many are NOT scientists. This will be interactive and may have a game format. We need all interns to be completely fluent in
- installation
- tutorial materials and examples
- management (perhaps within teams)
- There may be a "dry run" about 2021-04-01 with volunteers (e.g. from Wikipedia)
- Everyone must be prepared to alpha-test
pygetpapers
and give feedback pygetpapers: test reports.
- review Ayush's work: Updated pygetpapers with options to make json and make csv on demand. Also added time elpased to pygetpapers.
- review of Wikipages Dictionary wikipages for dictionaries. Everyone should have 1-2 dictionaries and summarize progress and problems.
- Radhu:
plant
andactivity
- Kanishka:
Invasive species
- Talha:
compound
andplant material history
- Vasant:
Plant parts
andgene
- Radhu:
- some skills will frequently need. so we have to can learning them by web tutorials and community self-help and set up Wiki pages for each:
- Programming: Regular Expressions
- Programming: Globbing
- Programming: Xpath
- Programming: JSON
- Programming: XML
- Programming: grep
- Each page should help newcomers learn these techniques. we don't have to write a tutorial - it's more useful to point to good (often interactive) tutorials. It's also useful to point out the things which we found difficult.
- Discussed the structure of projects in Python. We looked at the project structure of
pyami.ini
, a project in which PMR is involved. - Kanishka: Project Manager( maintain a wikipages for the testing of
pyamisearch
and asking people to put their tests and update this documents)
- Every intern's have to check whether their dictionaries are capable to do
ami search
or not. - verify you can run /physchem/python/util.py
- install
pyami.ini
to your "home" directory. I don't know where this is on Windows. This will need to be customised to define- where your dictionaries are
- where your project/s are
- try to use symbols where possible
- run:
python <your path>/util.py
- keep editing till it shows your dictionaries
- If any one has a any error then create issue on Github https://github.com/petermr/dictionary/issues and give a link of issue on Slack.
- After issue has been resolved please add the problem encountered and the accepted solution to the wiki. Also please don't delete issues even after issue has been resolved because other fellows might encounter the same problem.
- Standup by interns
- Review of the individual tasks, Interns to come up with their own dictionaries and projects.
- creation with SPARQL
- editing and problems of existing dictionaries
- Potential use of dictionary of genes.
- identifying articles (IR)
- annotating genes in articles (IE)
- linking to Wikidata
- translating synonyms
- Discussed the recent problem that we encountered with SPARQL query, as reported by Talha & Radhu( We can't downloads Large URL of endpoints so we have to download endpoints into the different part and then merge manually)
- Every intern's dictionary wikipages created on "CEVOpen" repository and should records update and issues related dictionary
- Standup by each interns.
- Review of dictionaries
- whether all SPARQL results have been downloaded and have been converted with
amidict
- each entry should have attributes:
- term
- name
- wikidataID
- wikidataURL
- en-wikidataURL
- en-description
- each entry may have children:
- EN-
synonym's
(optionalxml:lang
attribute) - non-EN
synonym's
(xml:lang
mandatory) - non-EN
description
(one per language, withxml:lang
) -
related
- e.g. non-EN Wikipedia pages
- EN-
-
entry's
may also have-
p
attributes for properties -
q
attributes for items
-
- whether all SPARQL results have been downloaded and have been converted with
- please checkout
openDiagram
and run latestsearch_lib.py
:cd physchem/python
python search_lib.py
- This should create graphs of occurrences of chemicals. click on the destroy-window button to move to next edit the file
search_lib
to reference your dictionary and your corpus choose sections that are likely to contain words then run your search and be prepared to demonstrate to us. We want to see all 4(6) dictionaries in action.
- Discussed the current status of each member's Dictionary
- Review of each dictionary if each contains name, term, wikidata ID, wikidata label, description, Wikipedia URL
- Reviewed
invasive species
dictionary updated by Kanishka. Synonyms, IUCN status and taxon id need to be added. - Reviewed
Activity
dictionary updated by Radhu. Synonyms and language equivalents need to be added. - PMR Debugging people's problems while running
search_lib.py
using share screen - A brief discussion on Regular expressions (RegExp)
- Communal review of pygetpapers including installation and alpha testing review of each interns:
-
--update
option is unclear with what is meant to do - downloading additional types of file such as
--pdf
could download PDF file to an existing repository
-
- it is essential that all the options common to
getpapers
andpygetpapers
are IDENTICAL. -Ingetpapers
the-f <filename>
option creates a LOG file. Inpygetpapers
this has a completely different operation "frompickle" . -Log levels and messages: -The currentpygetpapers
is verbose, relatively uninformative, and cannot be altered. Most of it would bedebug D
ortrace T
. - Users of
getpapers
will expect 16 flags to be present inpygetpapers
.- if present in
pygetpapers
these should have the SAME syntax asgetpapers
. The operation should ideally be the same. If enhanced or restricted this should be noted - if NOT present in
pygetpapers
these flags should be reserved for future use. - Flags should NOT be used for different purposes.
- if present in
- Standup by interns (those who are present)
- Review of
pygetpapers
with new version and new flag addition - Review of
search_lib
in (openDiagram
) -
ami
search_lib
is working with facets of dictionary, corpus and section and runs quickly on small corpus - We tested Talha Hasan and Radhu Ladani and Shweata Hegde dictionaries and they worked excellently!
- We understand Concepts of Data Science in our project
- Explanation of Natural Language ToolKit and Natural Language Processing by PMR
- Talha Hasan please make Wiki page for EPMC(Explore synonyms on EPMC)
- Radhu Ladani make wiki page for Natural Language ToolKit(NLTK) https://github.com/petermr/openDiagram/wiki/Natural-Language-Toolkit-(NLTK)
- Standup by each interns
- Minicorpora Review with each intern's dictionary
- We downloads the 200 paper for each dictionary's topic using
getpapers
and then did section forami search
- Radhu: Test
Plant
&Activity
dictionary withami search_lib
hereActivity
dictionary worked perfectly but We need to updatePlant
dictionary to work withami search_lib
- Kanishka: Test
Invasive_species
dictionary with Minicorporaoil186
&Invasive Plant species
- Vasant: Test
Plant Parts
- Talha: Test
Plant Compound
- Radhu: Test
- Understand The Database system and the Five Laws of Library Science of S. R. Ranganathan by PMR
- Kanishka & Radhu work's Review Regarding their dictionary
- Review of
pygetpapers
by Ayush - Review of
Search_lib
by PMR
- Update of St Edmunds Game & Project's Software to Gita ma'am by PMR including
search_lib
pygetpapers
and their working - Review of
search_lib
with multilingual (English, Hindi, Urdu) with the different DictionaryActivity
,Plant_Part
,Plant_Compound
,Plant_genus
- Review of
pygetpapers
by Ayush including latest version, debug log-level, supplementary files, CSV file contain title & full column as well. - Every intern's Stand up.
- Debugging the
Search_lib
for the command line so every body test on their system- Query:
python search_lib.py --dict --sect --proj
- Example:
python search_lib.py --dict country --sect introduction method --proj oil186
- Query:
-
--help
will give you the following output to understand the query structure
C:\Users\DELL\Radhu\openDiagram\physchem\python>python search_lib.py --help
running search main
usage: search_lib.py [-h] [--dict DICT [DICT ...]] [--sect SECT [SECT ...]] [--proj PROJ [PROJ ...]] [--patt PATT [PATT ...]]
Search sections with dictionaries and patterns
optional arguments:
-h, --help show this help message and exit
--dict DICT [DICT ...]
dictionaries to search with (lookup table from JSON (NYI); empty gives list
--sect SECT [SECT ...]
sections to search; empty gives all (Not yet tested
--proj PROJ [PROJ ...]
projects to search; empty will exit
--patt PATT [PATT ...]
patterns to search with; regex may need quoting
- Dictionary
Activity
,Plant_Parts
,Plant_compound
,country
work excellently with thesearch_lib
- everyone need to do Alpha testing of
pygetpapers
and make a report on wikipage - everyone need to do
search_lib
with their dictionary on own system
- UTF-8 is a variable-width character encoding used for communication discussed by PMR.
- Every intern's Stand up
- Feedback of
pygetpapers
by Ayush based on Alpha Test Reports and continue worked on--restart
and--update
for better improvement - We will introduced Tutorials Documentation for
pygetpapers
so people can understand well!(Volunteers Intern's worked on this with Ayush) - Review of
Search_lib.py
with different dictionary and demo set.- The scheme is:
search SECTIONS in PROJECTS with (DICTIONARIES and/or PATTERNS) with (DISPLAY and/or ANALYSIS) options
-
python search_lib.py --demo
gives different project and we have to choose from'ethics', 'luke', 'plant_parts', 'worcester', 'word'
- Alpha code for searching document corpus SEARCH TUTORIAL
- The scheme is:
- PMR explain file manager of
search_lib
, Standard-graph(matplotlib, seaborn) for graphical visualization - PMR explain supervised learning concepts in our project, powerful way of classifying section
- dictionary review
activity
,plant
,plant_part
,plant_compound
disease
- Everyone make sure they have a dictionary that works and has standard attributes
name
term
-
wikidataID
(if known) -
wikipediaPage
(if known) -
description
(EN)
- Everyone make sure they have a dictionary that works and has standard attributes
- Every dictionary should have a name which is
lowercase_underscore
anda title which contains this value
- Everyone make sure they have only ONE top level *.xml file for their dictionary .
- This should have a name with is LOWERCASE (and optional UNDERSCORES) ONLY
- It should be the same as the title attribute in the dictionary
- We are starting to come up with a unified Dictionary Naming Scheme
- Every dictionary should have a name which is
lowercase_underscore
anda title which contains this value
- Review the dictionary
activity
,plant
,invasive_plant
,plant_compound
- We make sure that dictionary works and has standard attributes
name
term
wikidataID
wikipediaPage
description
- and children:
-
<description xml:lang...>
(optional) -
<synonym>
(optional) <synonym xml:lang...>
-
<related ...>
(optional)
-
- We make sure that dictionary works and has standard attributes
-Every dictionary should also have a minicorpus which contains content enriched in its terms. Please check that your minicorpus works with your dictionary. -Talha: create a document and maintain the current record of each dictionary including minicorpus, file name, location etc.