Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search for an entry without triggering group or other fields #7996

Closed
ThiloteE opened this issue Aug 16, 2021 · 5 comments · Fixed by #11542
Closed

Search for an entry without triggering group or other fields #7996

ThiloteE opened this issue Aug 16, 2021 · 5 comments · Fixed by #11542

Comments

@ThiloteE
Copy link
Member

ThiloteE commented Aug 16, 2021

JabRef 5.4--2021-08-15--96061b7
Windows 10 10.0 amd64
Java 16.0.2
JavaFX 16+8

Problem

  • when i search for an entry that is within a group and the groupname is similar to what i search for, the group field triggers and thereby unfortunately shows all entries within that group, even though there are many entries that are not relevant to my search.
  • the problem intensifies when having hundreds of groups and thousands of entries
  • Search commands such as "not group and title" do not work

Example
In the following example, the entry "Bordo" is not wanted
bug 5 4 search for entries without triggering group

Workaround
In the following screenshot i use author=böhme as a workaround to exclude other entries.
bug 5 4 search for entries without triggering group  solution author=xxx

Why post this on github, even though there is a workaround?

  • If there are 10 fields (example for a field: author, range of date, title, group, etc.) and one wants to have a search with 8 out of 10 included, but 2 fields excluded, then this will lead to a long chain of search commands.
  • Typing long chains of search commands takes a long time and is prone to errors
  • Typing long chains of search commands just to exclude ONE single field of entry (in this case 'groups') seems not very efficient.
  • Having to use search commands is not very intuitive for new users of JabRef. I only found out they existed after finding this website: https://docs.jabref.org/finding-sorting-and-cleaning-entries/search, not from within JabRef. Only later i noticed that there is a short explanation when hovering over search for a little while.

Solution

  • Increase visibility of search commands
  • Create new functionality and/or extend existing functionality that allows to exclude certain fields (e.g. Groups) from a search.

i am sure there are many ways to Rome.

Additional Notes

To be clear: i do NOT argue for the removal of to search all fields. (= search everywhere) I think it is a very good start for a search. Also, i do NOT argue for the removal of being able to search for terms that are located within the group field.

@ThiloteE
Copy link
Member Author

suggested tags:

  • enhancement
  • search
  • groups

@ThiloteE
Copy link
Member Author

thank you :) That came as fast as lightning xD

@ThiloteE
Copy link
Member Author

ThiloteE commented Nov 23, 2021

JabRef 5.4--2021-11-15--9955a33
Windows 10 10.0 amd64
Java 16.0.2
JavaFX 17.0.1+1

I checked again today if this works. To make it more clear, why this issue is relevant and not just a hypothetical theoretical plaything, i provide a real world example:

What i want:

  • I want to create one big library(-file) for all my projects, therefore i create groups for each single project. (I know we now have global search again, but still...)
  • I wanted to search within a group with many entries for the author "Keynes" or any entries that mention Keynes in any other field EXCEPT GROUPS, because the groupname is related to Keynes.

What happens:

  • ALL entries within that group are displayed as search result.
    image

  • this stands at odds with more stringent search results. If i remove all the entries from the group, i only find 5 entries that have bibliographic metadata that is related to Keynes in my library:
    image
    Finding only these 5 entries is what is desired.

  • when i attach all the entries to the group again and try to search within this group, but try to exclude the group from being searched with the syntax @koppor used in Fix search: NOT binds more than AND #8241, then it results in NO entries being displayed at all:
    image

All this becomes even messier when entries are part of more than one group.

This issue blocks my aspiration of creating one big library file.

@ThiloteE
Copy link
Member Author

related discussion in the forum: https://discourse.jabref.org/t/can-i-have-mutually-exclusive-groups/3015

@ThiloteE
Copy link
Member Author

ThiloteE commented May 25, 2022

Another workaround via RegEx search:

  1. Have a group called Colistra.

  2. Have two entries:

    @Article{test1,
      author       = {Rita Colistra},
    

    and

    @Article{test2,
      groups = {Colistra},
    }
    
  3. Activate RegEx search

  4. Enter this line into search bar: Colistra and groups != Colistra

Result:

only the entry test1 will show, which is exactly what I want.

Something similar should work out of the box and not just via RegEx.

Additional Info:

For anybody who is willing to work on this:

JabRef maintainers are currently in the middle of adopting Lucene as search syntax, so this is relevant, but adopting Lucene is the main issue. See #8206

@LoayGhreeb LoayGhreeb mentioned this issue Sep 2, 2024
6 tasks
github-merge-queue bot pushed a commit that referenced this issue Sep 5, 2024
* Use pattern matching for cast

Co-authored-by: Christoph <[email protected]>

* Fix pattern matching

* Fix merge

* Speed up switches between sorting/filtering modes

* Fixed merge errors

* Fixed small issues

* Removed obsolete tests, fixed some tests

* Fixed merge error in CHANGELOG.md

* Fixed checkstyle

* Fixed more tests

* Removed obsolete tests

* Fixes "Fixed merge error in CHANGELOG.md" by removing duplicate entries

This reverts commit 536ecfa.

* WiP on tests

* Checkstyle

* Checkstyle

* Update Java version

* Refine logging

* Fix compile error

* Add LuceneTest

* Update CHANGELOG.md

* Move search classes to pdf package

* Move search classes to search package

* rewriteRun

* Remove bibEntry from DocumentReader

* Rewrite LuceneIndexer

* Remove IndexingTaskManager

* Separate Bib fields index and LinkedFiles index

* Fix null LuceneManager in ExternalFilesEntryLinker

* Save as action

* Clear linkedFiles indexer when fullText indexing is disabled in preferences

* remove comments

* get indexed files on update

* Add LUCENE_MANAGERS map for accessing managers by databaseContext.getUid

* Move LuceneManager from search.indexing package to search

* Fix wrong order for import

* Move SearchQuery to model package

* Fix issue with opening multiple unsaved libraries

* Pass LuceneManager down to the entry editor

* Improve searching performance

* Change SearchFieldConstants to enum

* More performance improvements for searching

- Read document only one time
- getHighlighterFragments only when the search results tab is opened

* Update FulltextSearchResultsTab.java

* Fix group union, intersection

* Fix backgroundtask

* Fix subscriptions

* Remove lastSearchQueryLogic

* Fix possible NPE

* Fix searchTask check

* Remove sort by score flag

* Fix score column sorting

* Fix modifier buttons listener

* Add search rank column

In floating mode entries will be ranked and sorted by it.
Rank: (1= entry matches group and search, 2= matches group but not search, 3= matches search but not group, 4= matches nothing)

* hide search rank column from preferences

* Add search_rank column to sort order by default

* Update CHANGELOG.md

* fix typo

* Change the order of the rank

1= entry matches group and search, 2= matches search but not group, 3= matches group but not search, 4= matches nothing

* Use NGramAnalyzer for indexing

* Resolve conflicts

* update search matches with lucene

* PreviewViewer highlighting with Lucene

* Delete IndexingTaskManager.java

* SourceTab highlighting with Lucene

* Fix non-ASCII characters

* Extract query terms from search query

* Highlight regex queries

* return js highlight function

* Fix invalid search query throw exception

* Refactor Lucene indexer classes

* Refactor linked files indexer

* Update search matches when entries are added or updated

* Remove preferences from ActionHelper

* checkstyle

* comment out search tests

* OpenRewrite

* Fix Groups Parser/Serializer

* Localization

* Search groups

* Release `IndexSearcher` after completing search task

* Checkstyle

* Correct typo

* Remove GroupSearchQuery

* Remove EventBus from LuceneManager and use BibDatabase eventBus

* Fix number of matched entries in groups

* Fix search groups

* Localization

* Remove bib fields highlighter

* Pass LuceneManager to search groups

* Fix performance issues by caching matched entries

* Update GroupDialogViewModelTest.java

* Update main table matches

* Fix groups icon

* Restore Search.g4 and GrammarBasedSearchRule

* First version of search group migration

Co-authored-by: Loay Ghreeb <[email protected]>

* Add groups field to the index

* Remove search rules

* Localization

* Add test cases

* Fix names

Co-authored-by: Loay Ghreeb <[email protected]>

* Add some more functionality

Co-authored-by: Loay Ghreeb <[email protected]>

* Always add "all" prefix

Co-authored-by: Loay Ghreeb <[email protected]>

* Add comment for alternative implementation

Co-authored-by: Loay Ghreeb <[email protected]>

* Mark library tab changed after migration

Co-authored-by: Loay Ghreeb <[email protected]>

* Add another test for regular expression

Co-authored-by: Loay Ghreeb <[email protected]>

* Small fixes

* Fix markBaseChanged

* Fix adding new entries did not update MatchCategory

* Fix searching for Non-ASCII characters

* Fix escaping special characters

Use WhitespaceTokenizer instead of StandardTokenizer
https://stackoverflow.com/a/6119584/21694752

* Fix tests

Co-Authored-By: Oliver Kopp <[email protected]>

* Add first draft of LatexToUnicodeFoldingFilter

Co-authored-by: Loay Ghreeb <[email protected]>

* Fix LatexToUnicodeFoldingFilter

Co-Authored-By: Oliver Kopp <[email protected]>

* Remove LatexToUnicode from SearchQuery

* Localization

* AllowedToUseLogic

* Update CHANGELOG.md

* Use sentence case for search result heading

* Add CHANGELOG for change in JabRefFrameViewModel

* Add more changes to CHANGELOG.md

* Add ADR-0038

* Rename "SCORE" to "MATCH_SCORE"

* Add link to ADR-0038

* Add another CHANGELOG.md entry

* Add CHANGELOG.md entry

* Revert change of filename

* Add JavaDoc comment

* Trying to find better names

* Discard changes to src/main/resources/tinylog.properties

* Remove commented out code

* Remove obsolete testing class

* Remove obsolete test

* Discard changes to src/test/resources/tinylog-test.properties

* Remove completely disabled code

* Rename "all" to "any"

* Catch thrown exception

Invalid regex queries throws an exception

* Remove groups field from the default field

#7996

* Remove SearchGroupsListener

* Update Benchmarks.java

* Update module-info.java

* Fixes from code review on LibraryTab

* Remove regex button from search bar

* Use BibEntry.getId instead of System.identityHashCode

* Add BibEntry index map

* Readd option

* Add `@ADR` annotation

* Add some comment

* One more annotation

* Add CHANGELOG.md entry

* One more annotation

* Add CHANGELOG.md entry

* Revert "Add BibEntry index map"

This reverts commit 27ed105.

* Use binary search to find the index of the entry

* openrewrite

* Tests for LinkedFilesIndexer

* Fix DatabaseSearcher

* LocalizationConsistencyTest

* DatabaseSearcherWithBibFilesTest

* Fix typo in CHANGELOG.md

* Fix typo

* Use parameterized test for DatabaseSearcherTest

* Fix DatabaseSearcherWithBibFiles tests

* Fix exportMatches test

* Remove regex check box from search groups dialog

* JavaDoc

* Fix SearchGroups test

* Remove closeAndWait methods and use CurrentThreadTaskExecutor

* Fix architecture test

* Allow to use logic

* Add debug logging for search

* Add more logging

* Assert with containsInAnyOrder

* Fix DatabaseSearcher test

* Global search dialog

* Rename method

* Improve code quality

- Maintain a map of BibEntryId to BibEntry.
- Store search results within SearchQuery instead of using the map in StateManager.
- Remove LuceneManager from SearchGroups.
- Use a different Analyzer for PDFs.

* Use non-static preferences variables

* Update CHANGELOG.md

* Delete SearchGroupTest.java

* fix typo

* fix indentation

* Update matchedEntries on the UI thread

matchedEntries should be updated on the UI thread because the size binding of matchedEntries will be reflected in the UI.

* Discard changes to src/main/java/org/jabref/gui/importer/actions/GUIPostOpenAction.java

* Fix LoayGhreeb#12

* Sync search flags between search bar and global search bar

* Move VERSION_6_0_ALPHA const to SearchGroupsMigrationAction

* Refactor LuceneSearcher

* Use linked files analyzer for highlighting full-text results

* Fix line break

* Fix tests

* Use EnglishAnalyzer for indexing/searching linked files

https://github.com/apache/lucene/blob/68cc8734ca28a9db800e4192a636d3b490cfd41a/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishAnalyzer.java#L101-L110

* Ask to wait for linked files indexing on shutdown

When closing JabRef, only ask users to wait for the linked files indexer to finish. The bib fields indexer is recalculated on startup, so it doesn't need to be completed before shutdown.

* Use EdgeNGram instead of NGram

* Return comment

* Update CHANGELOG.md

---------

Co-authored-by: Benedikt Tutzer <[email protected]>
Co-authored-by: Christoph <[email protected]>
Co-authored-by: Carl Christian Snethlage <[email protected]>
Co-authored-by: Oliver Kopp <[email protected]>
LoayGhreeb added a commit that referenced this issue Oct 4, 2024
github-merge-queue bot pushed a commit that referenced this issue Oct 23, 2024
* Convert to record

* Add embedded-postgres dependency

* Setup embedded postgre server

* Indexing entries with Postgres

* Remove Lucene bib fields indexer

* Use PreparedStatement to fix escaping characters

* Rename LuceneManager to IndexManager

* Begin to implement "new" search syntax

Co-authored-by: Loay Ghreeb <[email protected]>
Co-authored-by: Carl Christian Snethlage <[email protected]>

* Some new operators

* Change log level of EmbeddedPostgres

* Handle search flags

* Fix default field

* Fix handling of anyfield (and add "any" as alias)

* Openrewrite...

* More test cases

* Remove non-covered libraries

* Use LIKE syntax as default instead of regex

Disabled exact match

* Update module-info.java

* Create "query" package

* Postgres searcher

* Return back the exact match operator

* checkstyle

* Add link

* WIP

* Fix compilation

* WIP

* Intermediate result

Co-authored-by: Loay Ghreeb <[email protected]>

* Query should be OK

Co-authored-by: Loay Ghreeb <[email protected]>

* Indexing of split values

Co-authored-by: Loay Ghreeb <[email protected]>

* Fix tests compile

* Use first name Last name for authors

* Refactor SQL query visitor

* Adapt tests

* Use join with EXACT_MATCH only

* Update to Postgres 17

* Attempt to use sub-queries with CTEs

* Fix CTEs sub-queries and grouping

TODO: EXCAT_MATCH to search in split table

* group matches by entry_id

* Use NOT IN for negation queries

* Fix unary NOT operator

* Use split values table for EXACT_MATCH queries

* Prepare for linked files index (full-text)

* Prepare linked files tables

* Fix searching

* Use multi column index

* fix merged module issues

* Fix update event

* Remove postgres linked files indexer

* Remove and insert field on update event

* Remove search score column

* Update search groups matches

* Remove search_score from tale preferences

* Migrate search groups flags to new syntax

* Localization

* Fix dialog message

* Ignores groups field from default searches

Fixes #7996

* Use TYPE_HEADER field for entrytype

* Search to Lucene query for linked files searching

* Marge linked files and bib fields results

* Searching in background task

* Fix search to SQL tests

* Localization test

* Fix DatabaseSearcherTest

* Fix DatabaseSearcherWithBibFilesTest

* Fix exportMatches test

* Update src/main/java/org/jabref/model/entry/BibEntry.java

Co-authored-by: Oliver Kopp <[email protected]>

* Add SINGLE_ENTRY_LINK to latex field

* Remove changelog entries

* Remove groups migration from localization

* Extract search terms from query (ignore negated terms)

* Fix architecture test

* Highlight Preview viewer with Postgres regexp_replace

* OpenRewrite

* Remove onRunning

* Set search query listener in the constructor

* Fix preview tab scrolling

* Use prepared statement to fix escaping

* Use prepared statement for sql query

* Store the start and end positions for every field

* WIP highlight source tab

* Fix source tab highlighting

* Return regex, case-sensitive flags back to the search bar

* Use for search bar flags for unfielded terms

* Skip migrations for unfieleded terms

* Return regex, case-sensitive CheckBox to search groups dialog

* Apply suggestions from code review

Co-authored-by: Oliver Kopp <[email protected]>

* Update JabRef_en.properties

* Fix search grammar to support special chars

* Create SearchQueryTest.java

* Adapt SQL visitor with new grammar

* Allow to use quotes without escaping

* escape SQL wildcard chars

* Reorder methods

* Adapt SearchFlagsToExpressionVisitor

* Adapt SearchToLuceneVisitor

* Adapt SearchQueryExtractorVisitor

* Fix tests

* Fix DatabaseSearcherTest

* Fix search terms patten for highlighting

* Highlight source tab field by field according to the search query

* Apply suggestions from code review

* Update src/main/java/org/jabref/gui/importer/actions/SearchGroupsMigrationAction.java

Co-authored-by: Oliver Kopp <[email protected]>

* Set default operator to AND

* remove debug

---------

Co-authored-by: Oliver Kopp <[email protected]>
Co-authored-by: Carl Christian Snethlage <[email protected]>
Co-authored-by: Siedlerchr <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

Successfully merging a pull request may close this issue.

3 participants