[Feature Request] Improve the S increase formula for same-day reviews

[Expertium]

Conversation on Discord

this won't be much help, but can u start an issue in the repo so that at least there's a place tracking this? someone might help in the future

#707 (comment)

Yes, this is one of the three edge cases where this formula doesn't work well.

1. If the user had one or two learning steps, but then switched to something like 30s 2m 5m 15m 30m 1h 2h 4h 6h 8h, then his stability will be overestimated.
2. If the user uses a filtered deck to do an unlimited number of same-day reviews (your case).
3. If the user is in a Good - Again - Good - Again loop (during the same day), stability will either explode or fall to near 0. This is possible if his learning steps are, for example, 10m 30m.

There is no solution as of today, so all 3 of these will remain troublesome for the foreseeable future.

@brishtibheja

[brishtibheja]

I'll explain my problem which I think has some solution.

For this one particular card, it graduates from learn state with an interval of 4d. Then after 4 days, it's failed which decreases the S but after going through the 1m 10m steps, the S ends up higher than what it was previously.

Now, that card gets a 9d interval and I fail it again. This card then graduates with a 14d interval, and I think I'll fail it again and it'll still keep increasing.

---

What about capping the stability such that same-day reviews cannot make S higher than what it was before entering learn state?

[L-M-Sherlock]

@brishtibheja what's your FSRS params?

[brishtibheja]

0.5449, 0.9534, 2.2070, 3.8619, 7.3596, 0.5176, 1.7702, 0.0131, 1.3728, 0.4685, 0.8708, 2.0235, 0.0497, 0.3956, 2.2553, 0.9520, 2.5297, 0.5325, 0.7536

[L-M-Sherlock]

According to the previewer, the PLS is not greater than the stability before entering relearning state:

[brishtibheja]

Not completely sure how will I get increased ivls then.

Here's the file:
選択中のノート.zip

(rename to .apkg)

[L-M-Sherlock]

Emmm. This problem is caused by two factors:

1. The PLS seems too large (1.22⁩ -> 1.19, 3.15⁩ -> 1.92⁩⁩, 4.27⁩ -> 2.55⁩).
2. The stability increases too fast during the same-day reviews.

Solution:

I haven't figured it out.

[L-M-Sherlock]

@brishtibheja, could you send your collection file to me? I can make some analyses about the PLS based on that.

[brishtibheja]

FYI I have tried Sherlock's recommendation and changed relearning step to 8m from 1m 10m but I still observe cards getting higher stability than they entered learn state with. Though it's less aggressive than before with this.

I want to ask if some sort of capping the stability wouldn't work in this case?

[L-M-Sherlock]

but I still observe cards getting higher stability than they entered learn state with.

What about setting the last parameter to zero?

[Expertium]

I have a good idea, but it requires adding another thingy to the memory state - the number of same-day reviews - and more parameters. I haven't benchmarked it yet.
Also, if Jarrett is unwilling to release FSRS-5.X, then it's moot anyway.

[L-M-Sherlock]

Another case:

• A had no relearning steps: again interval was 6 days, and the subsequent interval 13 days (rated easy)

• B had a 15 minute relearning step: subsequent interval was 23 days when adjusted to the same parameters (rated good)

This seems to follow a larger pattern and I'm confused as to why it is happening. Intuition would suggest that recalling a card after the more aggressive interval FSRS provides would result in a larger subsequent interval but this has not been my observation.

In summary: Rating Good in the same day has a higher SInc than rating Good in the next day. It violates the spacing effect.

source: https://www.reddit.com/r/Anki/comments/1h9g1n7/comment/m10vqz0/

[Expertium]

I actually think that this is plausible. What I mean is that if someone from 2100 handed us the code for the world's most advanced spaced repetition algorithm that models short-term and long-term memory differently, it would behave this way.
If we use the same R-dependent model for all reviews, then yes, this shouldn't happen. But I think short-term SInc doesn't depend on R.
Again, I haven't benchmark it yet. First, I'll work on FSRS-5-secs, and then, if my new formula for short-term SInc works in FSRS-5-secs, I'll adapt it to FSRS-5 and benchmark it.
Note that it will require new parameters, so if we decide to use it, you will have to release a new version of FSRS.

[L-M-Sherlock]

@dae, I saw that you're reviewing ankitects/anki#3707. As you seen, I added a new arg to compute_parameters. I plan to add a switch in the Advance section of the deck option page to allow the user to control it. What do you think of? The switch could prevent FSRS update S during same-day reviews.

[Expertium]

I'm not dae, lol, but I want to share my opinion.
I think it's a bad idea. Most people have no idea how FSRS works, and even noticing that they have a problem with S because of same-day reviews requires prior knowledge. It's safe to say that most users will have no idea what "prevent FSRS from updating S during same-day reviews" means, or when enabling/disabling it is beneficial.
We could rename it to "Let FSRS use same-day review data", but I'm afraid that even that is too technical, and users won't know when and why to enable/disable it.
@brishtibheja @user1823 @Danika-Dakika your opinions are also welcome

[L-M-Sherlock]

Fine. Maybe just allow users like @brishtibheja could set it via debug console?

[brishtibheja]

I don't use desktop Anki (AnkiDroid).

[dae]

@L-M-Sherlock I haven't had time to closely follow FSRS development recently, so I don't know enough yet to give you an answer. Can you briefly summarise why a user would want one way vs the other, and why we need to offer both?

[L-M-Sherlock]

Can you briefly summarise why a user would want one way vs the other, and why we need to offer both?

If the user has two or more relearning steps, the stability on the same day may increase to exceed the stability prior to when the user pressed 'again'. The option could prevent FSRS from updating the stability during the same day reviews.

[brishtibheja]

To add to this: This may also happen with just one learning step.

The issue can be seen in the image here: #708 (comment)

The user fails something, relearns it, the new interval is higher than before so they fail it again, relearn it, and the cycle repeats. I had cards with a month's interval that I consistently failed everytime they came up for review.

[L-M-Sherlock]

To add to this: This may also happen with just one learning step.

It has been fixed in open-spaced-repetition/fsrs-optimizer#150 and open-spaced-repetition/fsrs-rs#257

[dae]

Why do we need to make it configurable? Are there times when the current behavior makes more sense?

[L-M-Sherlock]

Why do we need to make it configurable? Are there times when the current behavior makes more sense?

The current behavior is counterintuitive. For example, the last interval is 10 days and I press again on this card today, the next interval should be shorter than 10 days intuitively. But with current implementation, if the user repeatedly press 'good' for the same card because they have a lot of relearning steps, the interval exceeding 10 days when the card exits the relearning state.

[dae]

I plan to add a switch in the Advance section of the deck option page to allow the user to control it.

That seems to indicate you want to give users the choice. Why do you want to give them a choice if you think the current behaviour is bad? Why don't we just fix it without any option in the advanced section?

[L-M-Sherlock]

Why don't we just fix it without any option in the advanced section?

I have another solution:

1. read the number of relearning steps and the optimized parameters from the deck option.
2. simulate the relearning case and check whether the weird behavior would happen.
3. If the answer is yes, automatically set the option enable_short_term to false.
4. re-optimize the parameters

It could fix the problem without any option the advance section. But the workflow is likely to confuse users: why the optimization runs twice?

[user1823]

Why do you want to give them a choice if you think the current behaviour is bad?

The current behaviour is bad for some people (especially those having multiple same-day relearning steps).

For most people, using same-day reviews improves RMSE (and that's the main change from FSRS-4.5 to FSRS-5).

[brishtibheja]

It could fix the problem without any option the advance section. But the workflow is likely to confuse users: why the optimization runs twice?

(just as FYI) I also talked about storing the stabilities and not let S increase past the previous S.

[dae]

If it's not possible to make the algorithm robust against the outliers such as this affected user, then another toggle might be the only practical option if this issue is common enough.

[Expertium]

@dae the issue is that we can't explain it to a new user, or even to an average user, without getting very technical. It would have a name like "Don't update memory stability during same-day reviews" or "Freeze stability during same-day reviews" or something like that. To the majority of users it may as well be in Chinese.
In my opinion, for a feature to be useful, users not only need to understand what it does, but also why they would want/need that. And I'm afraid that in this case, users won't understand neither what it does nor why it's important.

@L-M-Sherlock running the optimization twice doesn't sound so bad, compared to the alternative of adding yet another setting that makes FSRS and Anki even more complicated.

[emiham]

why the optimization runs twice?

Does it have to show it's running twice? I think most users would accept that sometimes it takes longer, without thinking something might be wrong.

[dae]

"The average user wouldn't understand what it does" is not a convincing argument for an option that would be placed in the advanced section. It's not intended for them, and they don't need to understand it.

User ability is on a spectrum. Sometimes we can make changes that benefit the entire spectrum; other times improvements on one end mean regressions for the other end. Optimising twice imposes a 100% speed penalty on the entire userbase for the benefit of a few.

[user1823]

@dae, I agree with most of your above comment, but

Optimising twice imposes a 100% speed penalty on the entire userbase

If this is implemented as Jarrett proposed, optimization will occur twice only for the affected users. Simulation will be run for all users though, but that shouldn't consume much system resources.

An important point is that this issue is not easy to identify. So, automatic detection will ensure that most people having the issue will get the benefits of the workaround.

[dae]

If it can be done in such a way that the extra computation is only done for users who would benefit from this change, then I agree that would be the preferable option.