Importing of BibDesk Groups and Linked Files #10968

Frequinzy · 2024-03-03T22:17:32Z

Added functionality in the BibtexParser to parse BibDesk Static Groups and convert them to Explicit Groups in JabRef.
Also added functionality for the BibtexParser to parse Linked Files in BibDesk and convert then to Linked FIles in JabRef.
Note: gets the relative path from the .bib file to the linked file. So if the .bib is saved in another location the link might break.

We have looked into how to implement Smart and Mixed Groups from BibDesk, but it proved difficult since they don't have any clear equivalent in JabRef. Out primary idea is that we could at least parse the Smart Group and then create an Explicit Group based on the information. This is what the added tests for Smart and Mixed Groups test for, even though that functionality isn't implemented yet.

Closes #10381

Mandatory checks

Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

* Add test to check parsing of BibDesk Static Groups * Change isExpanded attribute to false in expected groups * remove extra blank line

* Attempts at plist * Now parses bdsk-file and shows it as a file in JabRef * Add test for parsing a bdsk-file field * Fix formatting * Add dd-plist library to documentation --------- Co-authored-by: Tian0602 <[email protected]>

* Creates an empty ExplicitGroup from BibDesk comment * Adds citations to new groups modifies group creations to support multiple groups in the same BibDeskFile * Fix requested changes Refactor imports since they did not match with main Add safety check in addBibDeskGroupEntriesToJabRefGroups --------- Co-authored-by: Filippa Nilsson <[email protected]>

Co-authored-by: Filippa Nilsson <[email protected]>

external-libraries.md

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

Siedlerchr

Thank you very much! In general this already looks good, I only have a few smaller comments

Siedlerchr · 2024-03-03T22:34:06Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

+                        LinkedFile file = new LinkedFile("", path, "");
+                        entry.addFile(file);
+                    } catch (Exception e) {
+                        throw new IOException();


Add a message to the exception e.

calixtus · 2024-03-03T22:43:43Z

external-libraries.md

+```yaml
+Id:      com.googlecode.javaewah:JavaEWAH
+Project: JavaEWAH
+URL:     https://github.com/lemire/javaewah
+License: Apache-2.0
+```
+
+```yaml
+Id:      com.googlecode.plist:dd-plist
+Project: com.dd.plist - A Java library for working with property lists
+URL:     https://github.com/3breadt/dd-plist
+License: MIT
+```
+


Do not edit external-libraries by hand. It will be edited when releasing a new version. There are too many possible errors. Please revert the changes to this file.

Na actually it looks good so far

The concrete change is OK for me. However, the changes below are not.

commons-collections:commons-collections:3.2.2 is added, but no added entry in external-libraries.md. Thus, either all or none.

So lets keep just the license and revert the other changes. Have to be done anyway on next release. That what I call in german "verschwendete Liebesmüh".

calixtus · 2024-03-03T22:44:19Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

@@ -318,7 +337,7 @@ private void parseJabRefComment(Map<String, String> meta) {
                }
            }
        } else if (comment.substring(0, Math.min(comment.length(), MetaData.ENTRYTYPE_FLAG.length()))
-                          .equals(MetaData.ENTRYTYPE_FLAG)) {
+                .equals(MetaData.ENTRYTYPE_FLAG)) {


please do not change the formatted indentation.

koppor

Thank you for working on this. The code works, but we also have "good code qualtiy" in "definition-of-done", thus some comments in that direction.

koppor · 2024-03-04T10:30:02Z

external-libraries.md

+```yaml
+Id:      com.googlecode.javaewah:JavaEWAH
+Project: JavaEWAH
+URL:     https://github.com/lemire/javaewah
+License: Apache-2.0
+```
+
+```yaml
+Id:      com.googlecode.plist:dd-plist
+Project: com.dd.plist - A Java library for working with property lists
+URL:     https://github.com/3breadt/dd-plist
+License: MIT
+```
+


The concrete change is OK for me. However, the changes below are not.

commons-collections:commons-collections:3.2.2 is added, but no added entry in external-libraries.md. Thus, either all or none.

koppor · 2024-03-04T10:31:04Z

licenses/com.googlecode.plist_ddplist.txt

@@ -0,0 +1,20 @@
+dd-plist - An open source library to parse and generate property lists


Haha, I totally forgot about the licenses sub folder... Lat change: 9 years ago...

OK to add something there... We need to cleanup at some stage.

SBOM generation etc.

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

koppor · 2024-03-04T10:38:18Z

src/test/java/org/jabref/logic/importer/fileformat/BibtexParserTest.java

+        BibDatabase database = result.getDatabase();
+        BibEntry entry = database.getEntryByCitationKey("Kovakkuni:2023aa").get();
+        String link = entry.getFiles().getFirst().getLink();
+        assertEquals(link, "../../Downloads/2309.06478.pdf");


It's hard to see the effects of the import. Is bdsk-file-1 be kept?

Maybe, you can test with

BibEntry expected = new BibEntry(StandardType.Article) .withField(...) .withField(...) ... assertEquals(List.of(expected), result.getDatabase().getEntries());

Adjust groups serializing

* upstream/main: (36 commits) chore: remove repetitive words (JabRef#11015) Fix test names (JabRef#11014) Remove obsolete "Comments" tab configuration (JabRef#11011) Fix "Other fields" tab respecting custom tabs (JabRef#11012) [WIP] Extract PDF References (JabRef#10437) Fixed jump to entry from crossref (JabRef#11009) fix suggestion provider for crossref field (JabRef#10962) Use SequencedSet for required and optional fields (JabRef#11007) Bump io.github.classgraph:classgraph from 4.8.165 to 4.8.168 (JabRef#11005) Bump org.glassfish.hk2:hk2-api from 3.0.6 to 3.1.0 (JabRef#11006) Bump org.apache.logging.log4j:log4j-to-slf4j from 2.23.0 to 2.23.1 (JabRef#11003) Bump org.javamodularity.moduleplugin from 1.8.14 to 1.8.15 (JabRef#11002) Bump jakarta.xml.bind:jakarta.xml.bind-api from 4.0.1 to 4.0.2 (JabRef#11004) Bump softprops/action-gh-release from 1 to 2 (JabRef#11000) Bump gittools/actions from 0.13.2 to 0.13.4 (JabRef#11001) Update custom-svg-icons.md (JabRef#10999) Update Texworks icon (JabRef#10998) Use tags editor for auto completion preferences (JabRef#10990) Enable auto merge of CHANGELOG.md (JabRef#10986) Enhance DOI parser to deal with special characters (JabRef#10989) ... # Conflicts: # build.gradle

Siedlerchr · 2024-03-14T07:50:43Z

TODO: Just a quick thougt I had this morning, we need to not only set Groups but we need to mege the existing group tree from metadata. Because BIbDesk is not an importer it's still a bib file

Add new BibDesk group Fix IOB for change listeing

src/main/java/org/jabref/logic/bibtex/comparator/BibDatabaseDiff.java

* upstream/main: Convert RemoveBracesFormatterTest to @ParameterizedTest (JabRef#11033) Update teaching.md Remove non-existing recipe (JabRef#11029) Update CSL styles (JabRef#11031) Clean up defintions of entry types (JabRef#11013) Fix log file path on Windows (JabRef#11028) Change to rolling logs (JabRef#11023)

* upstream/main: (26 commits) Speed up failure reporting (#11030) Importing of BibDesk Groups and Linked Files (#10968) Convert RemoveBracesFormatterTest to @ParameterizedTest (#11033) Update teaching.md Remove non-existing recipe (#11029) Update CSL styles (#11031) Clean up defintions of entry types (#11013) Fix log file path on Windows (#11028) Change to rolling logs (#11023) chore: remove repetitive words (#11015) Fix test names (#11014) Remove obsolete "Comments" tab configuration (#11011) Fix "Other fields" tab respecting custom tabs (#11012) [WIP] Extract PDF References (#10437) Fixed jump to entry from crossref (#11009) fix suggestion provider for crossref field (#10962) Use SequencedSet for required and optional fields (#11007) Bump io.github.classgraph:classgraph from 4.8.165 to 4.8.168 (#11005) Bump org.glassfish.hk2:hk2-api from 3.0.6 to 3.1.0 (#11006) Bump org.apache.logging.log4j:log4j-to-slf4j from 2.23.0 to 2.23.1 (#11003) ... # Conflicts: # src/main/resources/csl-styles

…link * upstream/main: Bump org.apache.pdfbox:fontbox from 3.0.1 to 3.0.2 (JabRef#11042) Bump com.dlsc.gemsfx:gemsfx from 2.2.0 to 2.4.0 (JabRef#11044) Bump org.apache.pdfbox:xmpbox from 3.0.1 to 3.0.2 (JabRef#11041) Bump com.googlecode.plist:dd-plist from 1.23 to 1.28 (JabRef#11040) Bump gittools/actions from 0.13.4 to 1.1.1 (JabRef#11039) Change copy-paste function to handle string constants (follow up PR) (JabRef#11037) Fixes Zotero file handling for absolute paths (JabRef#11038) Speed up failure reporting (JabRef#11030) Importing of BibDesk Groups and Linked Files (JabRef#10968) Convert RemoveBracesFormatterTest to @ParameterizedTest (JabRef#11033) Update teaching.md Remove non-existing recipe (JabRef#11029) Update CSL styles (JabRef#11031) Clean up defintions of entry types (JabRef#11013) Fix log file path on Windows (JabRef#11028) Change to rolling logs (JabRef#11023) chore: remove repetitive words (JabRef#11015) Fix test names (JabRef#11014)

* upstream/main: (40 commits) Improve citation relations (JabRef#11016) Keep enclosing braces of authors (JabRef#11034) Bump org.apache.pdfbox:fontbox from 3.0.1 to 3.0.2 (JabRef#11042) Bump com.dlsc.gemsfx:gemsfx from 2.2.0 to 2.4.0 (JabRef#11044) Bump org.apache.pdfbox:xmpbox from 3.0.1 to 3.0.2 (JabRef#11041) Bump com.googlecode.plist:dd-plist from 1.23 to 1.28 (JabRef#11040) Bump gittools/actions from 0.13.4 to 1.1.1 (JabRef#11039) Change copy-paste function to handle string constants (follow up PR) (JabRef#11037) Fixes Zotero file handling for absolute paths (JabRef#11038) Speed up failure reporting (JabRef#11030) Importing of BibDesk Groups and Linked Files (JabRef#10968) Convert RemoveBracesFormatterTest to @ParameterizedTest (JabRef#11033) Update teaching.md Remove non-existing recipe (JabRef#11029) Update CSL styles (JabRef#11031) Clean up defintions of entry types (JabRef#11013) Fix log file path on Windows (JabRef#11028) Change to rolling logs (JabRef#11023) chore: remove repetitive words (JabRef#11015) Fix test names (JabRef#11014) ...

* issue #10993 - feat: added ability to parse preferred-citation field to CffImporter * issue #10993 - feat: added all fields of JabRef/CITATION.cff to CffImporter * issue #10993 - feat: rewrote CffExporter to parse Software, Dataset types and authors names correctly * issue #10993 - feat: added keywords and unknown fields support * issue #10993 - feat: added round-trip test * issue #10993 - doc: updated CHANGELOG.md * Convert RemoveBracesFormatterTest to @ParameterizedTest (#11033) * Convert to @ParameterizedTest * Convert to csvsource --------- Co-authored-by: Carl Christian Snethlage <[email protected]> * Importing of BibDesk Groups and Linked Files (#10968) * Add test to check parsing of BibDesk Static Groups * Add test to check parsing of BibDesk Static Groups * Change isExpanded attribute to false in expected groups * remove extra blank line * Add tests to check parsing of BibDesk Smart and mixed groups * Add parsing of BibDesk Files * Attempts at plist * Now parses bdsk-file and shows it as a file in JabRef * Add test for parsing a bdsk-file field * Fix formatting * Add dd-plist library to documentation --------- Co-authored-by: Tian0602 <[email protected]> * Add creation of static JabRef group from a BibDesk file * Creates an empty ExplicitGroup from BibDesk comment * Adds citations to new groups modifies group creations to support multiple groups in the same BibDeskFile * Fix requested changes Refactor imports since they did not match with main Add safety check in addBibDeskGroupEntriesToJabRefGroups --------- Co-authored-by: Filippa Nilsson <[email protected]> * Refactor newline to match main branch Co-authored-by: Filippa Nilsson <[email protected]> * Add changes to CHANGELOG.md * Reformat indentation to match previous * Revert external libraries Adjust groups serializing * checkstyle and optional magic * fix * fix tests * fix * fix dangling do * better group tree metadata setting * merge group trees, prevent duplicate group assignment in entry Add new BibDesk group Fix IOB for change listeing * fix tests, and extract constant * return early * fixtest and checkstyle --------- Co-authored-by: Anna Maartensson <[email protected]> Co-authored-by: Tian0602 <[email protected]> Co-authored-by: LottaJohnsson <[email protected]> Co-authored-by: Filippa Nilsson <[email protected]> Co-authored-by: Filippa Nilsson <[email protected]> Co-authored-by: Oliver Kopp <[email protected]> Co-authored-by: Siedlerchr <[email protected]> * Speed up failure reporting (#11030) * Fixes Zotero file handling for absolute paths (#11038) * Fixes Zotero file handling for absolute paths Fixes #10959 * checkstyle mimiimm * fix changelog * cannot fix * Change copy-paste function to handle string constants (follow up PR) (#11037) * [Copy] Include string constants in copy (#11) Signed-off-by: Anders Blomqvist <[email protected]> * [Copy] New method for serializing string constants (#12) Signed-off-by: Anders Blomqvist <[email protected]> * Add a sanity check for null for clipboard content Currenlty, the clipboard content can be null since the database does not seem to be updating. This is a sanity check to prevent the program from adding null to the clipboard. Link to DD2480-Group1#13 * [Fix] Add parsed serilization when save settings When loading from existing files or libraries, the parser will set the serilization of the string constant to the correct value. However, when editing via the GUI, the serilization was not set and a new string constant list will be created without the serilization. This result in the serilization being null and when copying with the clipboard. Link to DD2480-Group1#13 * feat: import string constants when pasting #9 Add functionality to import string constants in the paste function Should add functionality to handle colliding string constants. Should also check that the constants are valid using the ConstantsItemModel class. * feat: Add string constant validity checker and dialog messages #9 Check that a pasted string constant is valid using the ConstantsItemModel class. Add diagnostic messages notifying users when adding a string constant fails while pasting. * [Copy] Copy referenced constant strings to clipboard (#16) * feat: Add parsed serialized string when cloning * feat: Add sanity check for null in ClipBoardManager * closes #15 * feat: new unit tests Add 4 new unit tests, testing the new features added for issue-10872. Specifically the tests are for the `storeSettings` method in the ConstantsPropertiesViewModel.java, and `setContent` in the ClipBaordManager.java. Closes #6 * Update CHANGELOG with copy and paste function * Fix Checkstyle failing by reformat the code * Fix OpenRewrite failing by running rewriteRun * Refactor by extract methods in setContent * collet failures * changelog and use os.newline * checkstyle * use real bibentrytypes manager * Fix CHANGELOG.md * Swap if branches * Code cleanup * Use List for getUsedStringValues * Fix submodule * Collection is better * Fix csl-styles * Remove empty line * Group BibTeX string l10n together --------- Signed-off-by: Anders Blomqvist <[email protected]> Co-authored-by: Anders Blomqvist <[email protected]> Co-authored-by: ZOU Hetai <[email protected]> Co-authored-by: Hannes Stig <[email protected]> Co-authored-by: Elliot <[email protected]> Co-authored-by: Oliver Kopp <[email protected]> * Bump gittools/actions from 0.13.4 to 1.1.1 (#11039) Bumps [gittools/actions](https://github.com/gittools/actions) from 0.13.4 to 1.1.1. - [Release notes](https://github.com/gittools/actions/releases) - [Commits](GitTools/actions@v0.13.4...v1.1.1) --- updated-dependencies: - dependency-name: gittools/actions dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.googlecode.plist:dd-plist from 1.23 to 1.28 (#11040) Bumps [com.googlecode.plist:dd-plist](https://github.com/3breadt/dd-plist) from 1.23 to 1.28. - [Release notes](https://github.com/3breadt/dd-plist/releases) - [Commits](3breadt/dd-plist@dd-plist-1.23...v1.28.0) --- updated-dependencies: - dependency-name: com.googlecode.plist:dd-plist dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump org.apache.pdfbox:xmpbox from 3.0.1 to 3.0.2 (#11041) Bumps org.apache.pdfbox:xmpbox from 3.0.1 to 3.0.2. --- updated-dependencies: - dependency-name: org.apache.pdfbox:xmpbox dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump com.dlsc.gemsfx:gemsfx from 2.2.0 to 2.4.0 (#11044) Bumps [com.dlsc.gemsfx:gemsfx](https://github.com/dlsc-software-consulting-gmbh/GemsFX) from 2.2.0 to 2.4.0. - [Release notes](https://github.com/dlsc-software-consulting-gmbh/GemsFX/releases) - [Changelog](https://github.com/dlsc-software-consulting-gmbh/GemsFX/blob/master/CHANGELOG.md) - [Commits](dlsc-software-consulting-gmbh/GemsFX@v2.2.0...v2.4.0) --- updated-dependencies: - dependency-name: com.dlsc.gemsfx:gemsfx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump org.apache.pdfbox:fontbox from 3.0.1 to 3.0.2 (#11042) Bumps org.apache.pdfbox:fontbox from 3.0.1 to 3.0.2. --- updated-dependencies: - dependency-name: org.apache.pdfbox:fontbox dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Keep enclosing braces of authors (#11034) * Add test cases * Add test cases * Keep braces for last part * Refine method description * Adapt test to new braces keeping * Add CHANGELOG.md entry * Adapt tests * More edge cases * Minor code beautification * Simplify code * Fix braces removing * Extract static fields, refactor code * Fix removal of {} for export * Re-add Objects.requireNonNull * Fix typo * Re-add NPE throwing * Rename to modern terms * Consistent initialization * Improve citation relations (#11016) * Collect DOI and publication type from semantich scholar to be able to expand the information of the new entries later by search through DOI * Include abstract in the request. This lets the GUI show the abstract since that was implemented already. Refactor api request string since most of it is shared * Add button to open the relation paper's DOI URL. Fix DOI for some ArXiv entries. * Don't show the open link button if there is no link to open. * Make field value null error a bit more useful * Include SemanticScholar url in the request and use it as the URL field. * Add changes to changelog * Change tooltip text to an existing, more informative one * Run rewriter to fix pull request * improve url optional handling --------- Co-authored-by: Siedlerchr <[email protected]> * issue #10993 - doc: updated CHANGELOG.md * fix: fixed unit tests not passing due to name changes in Author interface (#10995) * feat: changed CFFExporter to use YAML library snakeyaml instead (#10995) * feat: added support for references and ALL possible CFF fields in importer (#10995) * fix: added requested changes (#10995) + updated CHANGELOG.md + removed useless comments + refactored both CffImporter and CffExporter to use more specific methods + used a BiMap to avoid repeating mappings between CffImporter and CffExporter + copied entryMap in exporter to avoid side-effects * fix: task rewriteDryRun fixed to pass by removing test in BibEntryTest * refactor: deleted useless methods in CffImporter (#10995) * doc: added decision MADR document for cff export (#10995) * feat: add a cites or related relationship between imported entries in CffImporter (#10995) * doc: updated MADR decision document for cff export to pass markdownlint (#10995) * fix: fixed round-trip test to use mock citatioKeyPatternPreferences correctly (#10995) * fix: fixed MADR document for CFF export decision to pass Jekyll CI check (#10995) * fix: fixed requested changes (#10995) + fixed typo in CHANGELOG.md + tested multiline abstract in CFFImporter * feat: finished CFFExporter logic and crafted working round-trip test (#10995) * fix: fixed typos in MADR decision doc for CFF export and refactore ImportFormatReader signature (#10995) * Some code beautification * Use existing method getEntryLinkList * Use getEntryLinkList * Use JabRef's Date class for parsing --------- Signed-off-by: Anders Blomqvist <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Oliver Kopp <[email protected]> Co-authored-by: Carl Christian Snethlage <[email protected]> Co-authored-by: Emil Hultcrantz <[email protected]> Co-authored-by: Anna Maartensson <[email protected]> Co-authored-by: Tian0602 <[email protected]> Co-authored-by: LottaJohnsson <[email protected]> Co-authored-by: Filippa Nilsson <[email protected]> Co-authored-by: Filippa Nilsson <[email protected]> Co-authored-by: Siedlerchr <[email protected]> Co-authored-by: Anders Blomqvist <[email protected]> Co-authored-by: ZOU Hetai <[email protected]> Co-authored-by: Hannes Stig <[email protected]> Co-authored-by: Elliot <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Roc <[email protected]>

annamaartensson and others added 6 commits March 3, 2024 23:07

Add test to check parsing of BibDesk Static Groups

3a911f7

* Add test to check parsing of BibDesk Static Groups * Change isExpanded attribute to false in expected groups * remove extra blank line

Add tests to check parsing of BibDesk Smart and mixed groups

835c3fd

Add parsing of BibDesk Files

b04bfe9

* Attempts at plist * Now parses bdsk-file and shows it as a file in JabRef * Add test for parsing a bdsk-file field * Fix formatting * Add dd-plist library to documentation --------- Co-authored-by: Tian0602 <[email protected]>

Refactor newline to match main branch

d34c949

Co-authored-by: Filippa Nilsson <[email protected]>

Add changes to CHANGELOG.md

0cccf88