[Bug] _onApiGetAnkiNoteInfo is slow, causing the "View added note" (duplicate note check) in the yomichan popup to be slow #2184
Comments
|
I did some extra testing, and I found something very strange. I edited anki-connect's def isNoteDuplicateOrEmptyInScope(
self,
note,
deck,
collection,
duplicateScope,
duplicateScopeDeckName,
duplicateScopeCheckChildren,
duplicateScopeCheckAllModels
):
return 2
# Returns: 1 if first is empty, 2 if first is a duplicate, 0 otherwise.
# note.dupeOrEmpty returns if a note is a global duplicate with the specific model.
# This is used as the default check, and the rest of this function is manually
# checking if the note is a duplicate with additional options.
if duplicateScope != 'deck' and not duplicateScopeCheckAllModels:
return note.dupeOrEmpty() or 0where the function immediately returns that every card is a duplicate (no database queries) and it's even slower. Scanning a word like すき焼き (has 33 entries with my 32 enabled dictionaries) taking about ~8 seconds to return even though there shouldn't be any database queries. And strangely, it even freezes my anki during these 8 seconds. I try to move the anki window around but it freezes until it resolves the Making the function I can only assume that this lag I'm experiencing is because we're throwing exceptions when calling duplicateOrEmpty = self.isNoteDuplicateOrEmptyInScope(
ankiNote,
deck,
collection,
duplicateScope,
duplicateScopeDeckName,
duplicateScopeCheckChildren,
duplicateScopeCheckAllModels
)
if duplicateOrEmpty == 1:
raise Exception('cannot create note because it is empty')
elif duplicateOrEmpty == 2:
if allowDuplicate:
return ankiNote
raise Exception('cannot create note because it is a duplicate')
elif duplicateOrEmpty == 0:
return ankiNote
else:
raise Exception('cannot create note for unknown reason')Anki hanging and freezing for ~8 seconds when hovering over a word for checking duplicates is really concerning though, is anki-connect blocking and running on the main thread? Or is it running on a separate thread? |
|
I've done some more testing, I think the lag isn't coming from the In def isNoteDuplicateOrEmptyInScope(
self,
note,
deck,
collection,
duplicateScope,
duplicateScopeDeckName,
duplicateScopeCheckChildren,
duplicateScopeCheckAllModels
):
return 2
# Returns: 1 if first is empty, 2 if first is a duplicate, 0 otherwise.
...so that everything is a duplicate, and I also added some logs in the def createNote(self, note):
collection = self.collection()
...
print("\n-------------")
print("before duplicateOrEmpty")
duplicateOrEmpty = self.isNoteDuplicateOrEmptyInScope(
ankiNote,
deck,
collection,
duplicateScope,
duplicateScopeDeckName,
duplicateScopeCheckChildren,
duplicateScopeCheckAllModels
)
print("after duplicateOrEmpty")
...And also in the @util.api()
def findNotes(self, query=None):
if query is None:
return []
print("Calling findNotes with query: ", query)
return list(map(int, self.collection().findNotes(query)))I ran The I'm not sure what these queries actually are, it seems like Powershell didn't properly render it, but it calls |
|
I wrote a small script testing while true
do
for i in $(seq 1 10)
do
curl localhost:8765/findNotes -X "POST" -d '{"action":"findNotes","version":6,"params":{"query":"deck:current"}}'
done
sleep 1
doneThis sends 10 I found that we call But for some reason this lags, and it's only doing |
kuroahna
changed the title
|
Since I suspect this might have something to do with the large amount of cards you have, sharing an export of your Anki collection might be useful. Sharing your Yomichan settings file would be useful also, then I can try to reproduce. Related issue from a while ago: #993 |
Anki Version 2.1.53 (96bacf79) Yomichan version: 22.4.4.0 Oh, I didn't know about that issue, this is definitely related |
|
Testing with your setup, I do see a bit of delay, but that all seems to be coming from the AnkiConnect side. Nothing looks out of the ordinary from findNoteIds duration measured from Anki: |
|
On Chrome, I edited async _onApiGetAnkiNoteInfo({notes, fetchAdditionalInfo}) {
const t1 = performance.now();
const results = [];
const cannotAdd = [];
const canAddArray = await this._anki.canAddNotes(notes);
for (let i = 0; i < notes.length; ++i) {
const note = notes[i];
let canAdd = canAddArray[i];
const valid = AnkiUtil.isNoteDataValid(note);
if (!valid) { canAdd = false; }
const info = {canAdd, valid, noteIds: null};
results.push(info);
if (!canAdd && valid) {
cannotAdd.push({note, info});
}
}
if (cannotAdd.length > 0) {
const cannotAddNotes = cannotAdd.map(({note}) => note);
const noteIdsArray = await this._anki.findNoteIds(cannotAddNotes);
for (let i = 0, ii = Math.min(cannotAdd.length, noteIdsArray.length); i < ii; ++i) {
const noteIds = noteIdsArray[i];
if (noteIds.length > 0) {
cannotAdd[i].info.noteIds = noteIds;
if (fetchAdditionalInfo) {
cannotAdd[i].info.noteInfos = await this._anki.notesInfo(noteIds);
}
}
}
}
const t2 = performance.now();
console.log(`getAnkiNoteInfo: ${t2 - t1}`)
return results;
}Scanning the following words, I get:
Now, if in anki connect in def isNoteDuplicateOrEmptyInScope(
self,
note,
deck,
collection,
duplicateScope,
duplicateScopeDeckName,
duplicateScopeCheckChildren,
duplicateScopeCheckAllModels
):
# Returns: 1 if first is empty, 2 if first is a duplicate, 0 otherwise.
return 2
...Scanning the same words above, I get:
すき焼き takes about 7.1 seconds to finish. Adding some more logs in anki connect, if we edit @util.api()
def findNotes(self, query=None):
if query is None:
return []
print("findNotes called with query: ", query)
return list(map(int, self.collection().findNotes(query)))And open up The |
|
Modify your AnkiConnect diff --git a/plugin/__init__.py b/plugin/__init__.py
index e51bbeb..f89faa8 100644
--- a/plugin/__init__.py
+++ b/plugin/__init__.py
@@ -83,7 +83,8 @@ class AnkiConnect:
def logEvent(self, name, data):
if self.log is not None:
- self.log.write('[{}]\n'.format(name))
+ t = time.perf_counter()
+ self.log.write('[{0}] @ {1:0.4f}\n'.format(name, t))
json.dump(data, self.log, indent=4, sort_keys=True)
self.log.write('\n\n')
self.log.flush()Then, check the log file to compare how long it takes from [request] to [response]. Your 7.1seconds for 64 findNotes requests isn't too outlandish, as it's only 7.1/64=~0.1109375 seconds per lookup. This is a bit slower than what I'm seeing though, so I'm not sure what accounts for that. What are the specs of the device you are using? |
|
Editing def isNoteDuplicateOrEmptyInScope(
self,
note,
deck,
collection,
duplicateScope,
duplicateScopeDeckName,
duplicateScopeCheckChildren,
duplicateScopeCheckAllModels
):
# Returns: 1 if first is empty, 2 if first is a duplicate, 0 otherwise.
return 2
...and diff --git a/plugin/__init__.py b/plugin/__init__.py
index e51bbeb..f89faa8 100644
--- a/plugin/__init__.py
+++ b/plugin/__init__.py
@@ -83,7 +83,8 @@ class AnkiConnect:
def logEvent(self, name, data):
if self.log is not None:
- self.log.write('[{}]\n'.format(name))
+ t = time.perf_counter()
+ self.log.write('[{0}] @ {1:0.4f}\n'.format(name, t))
json.dump(data, self.log, indent=4, sort_keys=True)
self.log.write('\n\n')
self.log.flush()and editing Yeah, it seems to be about ~0.11 seconds per lookup The specs of my PC is pretty old now, but it should be plenty fast: According to dxdiag: I wrote a bash script that queries and ran And looking at the logs as well: logsDeckCurrent.zip 8.5991 - 4.4325 = 4.1666 seconds for it to finish querying Would it be possible to somehow make EDIT: Or, would it somehow be possible to make it so that instead of doing 66 EDIT 2: I made another script which queries #!/bin/bash
for i in $(seq 1 66)
do
curl localhost:8765/findNotes -X "POST" -d '{"action":"findNotes","version":6,"params":{"query":"deck:current"}}' &
done
waitAnd running it: This takes about 0.383 seconds, that's almost 11 times faster |
|
I have a suggestion that could speed this up too. How about only checking the first 2-3 terms which would load nearly instantly, and then only checking anything else if the user scrolls down in the Yomichan window? You could put a loading GIF on each entry in place of the "View added note" button while waiting for a response from Anki. I don't see the need to check every single term every time we open the window when most of the time, most users are not going to even look at them. |
|
Yeah, I think that would make Yomichan feel a bit more snappier. Instead of Would this be a potential solution? Also, unfortunately it seems impossible for anki-connect to be multi threaded due to Python's Global Interpreter Lock, and Anki/sqlite not being thread safe (although, I feel like |
It's certainly a potential solution, but it does add complexity and introduce other latency. The Anki connection is run on the background page, so requests have to be proxied through that. Additionally, every AnkiConnect HTTP request incurs additional delay. So overall, it will likely take longer to query every note status. This may be offset by the "perceived" speed improvement of the first few entries appearing more quickly, so the added complexity is probably a larger issue. |
|
Yeah the added complexity doesn't sound too good. I do notice however that anki-connect has the option I was playing around in anki-connect and trying to build an internal cache of all the I was also trying to use python's multiprocessing module to get around the Global Interpreter Lock issue, but it seems like it's not possible either since it tries to open multiple instances of Anki which doesn't work since you can only have 1 instance of Anki open. It sounds like the solution here is for anki-connect to be multi-threaded, but that's impossible due to Python's GIL limitation. Not sure what alternatives there are other than to make a fork of Anki, having the multithreaded web server built directly inside Anki via Rust. Also, I was testing out your PR, this is pretty nice, retesting some of the searches like すき焼き and modifying anki connect's |
|
I realized I didn't have to call I only made it so that it works for the Yomichan settings
since those are my settings I use for Yomichan, but it could be improved to support other settings as well. I'm not sure if we want to close this issue now? I don't know if we really want to push my fork to the main anki-connect repo as it may not be desirable for other users to have this internal cache and take extra memory to use the addon, and my code is pretty ugly (something I hacked together quickly), and this also means anki-connect's cache depends on Yomichan implementation for the queries which won't work for users who aren't using Yomichan, but I'm satisfied with my own solution to the problem. |
Perhaps this should be raised in
anki-connectinstead of Yomichan, but if I understand it properly, Yomichan calls anki-connect's APIs to do the duplicate card checking And we do database queries such asselect id from notes where csum=? and mid=?select id from notes where csum=? and id!=?select did from cards where nid=?select id, field_at_index(flds, 0) from notes where csum=? and mid=?if duplicateScope != 'deck' and not duplicateScopeCheckAllModelswhich callsis_duplicate()and does the database queryIt would be nice if we cached all the cards in the anki collection that we know exists and avoid having to query the database to see if it's a duplicate card or not.
Would it be as simple as going through all the notes in the anki collection on
anki-connectstartup, caching the note's first field checksum into a Set, and then when we want to check if a new note is a duplicate or not, we get the checksum of the new note's first field and check if it is in the Set? This will probably slow anki on startup since it has to build the cache, but perhaps this could be an optional option/flag? Also not sure how much memory this will take, but my computer has 16 GB of RAM which should be plenty, and I feel like most computers nowadays ship with 8 GB at leastI heavily use the duplicate card check feature, and when hovering some words/phrases like 今日 or 大体の, it takes about ~3 seconds to show that it is a duplicate in my anki collection. But for other words like 食べる, it takes about ~1 second. I have tried this on both firefox and chrome, and they're both the same speed.
I have ~17.5k cards created so far, and I have 34 dictionaries installed, so Yomichan feels somewhat sluggish.
I can share my anki collection if needed.
EDIT:
Yomichan version: 22.4.4.0
Chrome: 102.0.5005.115 (Official Build) (64-bit)
Firefox Developer Edition: 102.0b9 (64-bit)
The text was updated successfully, but these errors were encountered: