iOS Beta, version 2024.25+

[biblicabeebli]

2024.45 is live on Testflight and contains a breaking change! Old versions of the app will get stuck in a resend loop. Please upgrade.

• This build has had virtually all code libraries updated due to cascading requirements changes, random things may have changed. We need wide-spectrum testing of all features.
• Please include weekly surveys in your test studies.
  • They don't need to test anything in the survey, there is an issue where a participant could fail to correctly generate new "Scheduled Events" affecting weekly surveys.
• Please report graphical glitches and other visually-observable changes.
  • With screenshots!
• Please attempt installing this version of the app over an existing Beiwe app installation from the app store to ensure we get decent coverage.
• There is a known issue (purely on the backend) where a push notification resend could get stuck in an endless resurrection loop causing infinite resends.
  • There is a possible fix deployed to the staging server - this issue is entirely resolvable on the backend.

these issues were identified, resolved by library rollback

we can resolve these in a future release of an update to researchkit

• Two survey navigation bugs have been identified:
  • one is a crash involving navigation with the back button.
    • Due to updated version of ResearchKit (2.1.0) - App crash while hitting the back button #71
  • one is a new and exciting way of triggering a scenario where the user can no longer move through a survey because the next and skip buttons disappear.
    • Due to updated version of ResearchKit (2.1.0) - Survey interaction that results in failure to render the Next and Submit buttons #72

Notes so far:

• black/white bar during registration.
• see There is no recovery mechanism for missed notifications on the iOS App, particularly multiple notifications when the app is closed. #59 for discussion
• see Investigation of missing notifications, particularly on iOS because it does not have a fallback beiwe-backend#402 for backend investigation.
• registration-time crash on ios 18? The app crashed on iOS 18 if installed through TestFlight (and presumably the App Store, as they are very similar). Underlying cause was some new compiler/os-level security protection that broke a use of a mutable pointer inside the encryption code.
• • Squashed bug: app badge becomes desynced from number of surveys to take #47
• • The UI libraries were revved, and believe-it-or-not you are probably a better judge of whether something has changed
• (The white/black bar the registration page has been fixed as of 2024.44.)

[biblicabeebli]

And let's add to the testing list - check resend behavior (and app behavior) if you clear the notfication and then proceed with various actions (opening the app, dismissing the app from the app switcher, reboot the phone...)

[biblicabeebli]

I have not gotten to this yet.

However, I did notice there is some strange scheduling with the notifications. We had survey's scheduled to go out at 10:15AM and 11AM, and I received notifications at: 10:18, 10:54, 11, 11:30, 11:36, 12, and 12:06 for at least 3 days. There seems to be a 3 minute offset in the beginning for the first survey, then surveys come in at 36/30 minute increments.

I'll work on that, I forgot about that, it's usually I forgot to zero some like a datetime microseconds value to a time comparison gets weird.

[biblicabeebli]

App Build 2024.45 contains a breaking change. Older versions of the app will (I think) be stuck in an endless loop of resends because they no longer understand the format of the data the backend sends them.

Apologies, there was an oversight on my part for the last detail to get all the data where it needs to be for the app to report notification uuids.

[MMel099]

new issue has been created, will reference this as original notification, then hide

• Due to updated version of ResearchKit (2.1.0) - Survey interaction that results in failure to render the Next and Submit buttons #72

(editing to stick the screenshots here inside detail tags for usability, but thanks for sizing them)

Another interesting issue has come out of Yale's beta testing. They had an instance where the survey came in and was available but once opened, there was no "next" button to proceed to the next question. Attaching a screenshot with the missing "next" button below.

iOS: iOS 18.1
Phone: 15

click for pic

Survey:

click for pic

Separately, the Yale study has seen some instances where participants do not get surveys including manual pushes from the dashboard. I am wondering if you have any sense as to whether resends would resolve this?

Thank you!

[biblicabeebli]

We need to identify if this bug:

click for pic

is actually this issue:

• If you clear out the answers for a required question (this cannot be done with all question types, like radio buttons and date/time) and then go back 1 question to another [answered] required question, that previous question will be displayed without a next button until you refresh the answer by interacting with the question.

(this is inside a collection of minor ui weirdnesses, which are collected in issue # 199 over on an old tracking issue in our internal tracking repo. (We should break those out into tracking issues in their respective repo)

hiding all other commentary here unless it becomes relevant again.

OK - adding this comment after the fact and after like an hour - this might be a known issue - (but I'm done for the day):

• during development of date/time questions, when interacting with the conditional question display logic... which was a few years ago now...
• I discovered an issue where these buttons would [need to?] disappear when navigating backwards from ....
  • I think it was when the next a question that was currently displayed had conditional logic that depended on the previous question that you were going back to...
  • and/or/but maybe it was also... when .... going backwards skipped over a bunch of questions that were never displayed?
    • because the app state only considers the current question and doesn't know which question in the list it is jumping to, because they may themselves have conditional logic.
    • ooorrrr possibly even more obscure, like ... it doesn't know until "too late" (After the BACK button get's pressed)
      • and then isn't "given the chance" to check which question will be displayed as a result of going backwards
      • so from the code's perspective the app "suddenly renders the question it has jumped to + those buttons without telling it what its doing"
      • so in order to correctly display/not-display a pressable SKIP or NEXT button on a unknown-but-potentially-required question.... it is forced to err towards hiding them
        • (and then it finally learns which question is currently displayed only after the user interacts with it, and that is in particular bad UX with Date questions because the "autofill" weirdly.)
• Yeeah. It was something super obscure like that.
• BUUUUUUUUT I can't remember if it was ios or android. 🫠 ....
  • I know I posted it somewhere, possibly in our internal work issues, but could be on the android or ios... ug....
  • @MMel099 @jprince127 @hydawo I could use help tracking this one down.
    • the ios app store release notes say Jul 21, 2023 was when we released date/time questions for iOS
    • if this can be reproduced and shown that the app correctly displays the next buttons after interacting with it then this is user error, not release blocker.

---

• Is it possible that this is a required question, and they need to change the value in order for the app to count it as selected and display the buttons?
  • iirc date/time questions cannot be set as required in the web interface due to issues like this across both android and ios.
    • but editing the question JSON directly can create such a question (probably)
    • or maybe if you just set a question as required and then changing the question type results in a required date/time - if so that is a bug on the backend, and.... that's fun but not a release blocker.

---

@MMel099 I need the survey that has that question, and I guess I need the participant ID. (email a link to their origin post, I might need to talk with their tester.)

Assuming it's not the question above -- What happens when that survey on that device gets loaded now? is the user blocked entirely from ever finishing that survey?

• If there is a tiny random chance where sometimes the survey transiently the doesn't show the button for a question, or if a back-forward can cause this, then we should proceed with the release and debug this later.
• Buuuuuut if we can reproduce it ~easily then we need to stress test and it is going to be awful.

---

The rest of this is stream of consciousness brainstorming on how to even do anything for this one, no need to respond to anything below:

• If this is an app-side code bug in the button/question rendering component of researchkit then...

  • it will be reproduceable regardless of the device.
  • and it will happen randomly or seemingly randomly.
  • usually the weird stuff happens under ... "high device load" conditions.
    • (I guess I can run a memory pressure test? this requires xcode)
    • I can check version updates to researchkit, the newest version cuts out a feature we use on the consent screen and I would have to write that entire flow from scratch
      • that would be a disaster, I don't think that would be a 2024 thing.

• if this app installation becomes perma-stuck with no rendered button OR This goes away only after receiving a notification for that same survey again ....

  • then it is a bug where the specific local copy of the survey in live memory is.... somehow at fault.
  • ALSO if this happens but then goes away after dismissing the app from the task switcher and reopening, then it ... involves the local database too?

• If this happens on only that question type.... ok we narrowed down the bug, cool.

• if this happens only on required questions -

• if this can be caught on an always-available or trigger-on-first-download survey, and the behavior is different ... that ... would be interesting, but not unexpected.

• The only way to test the entire render process for fully disappearing and reloading the survey is by sending a push notification. Maybe.

  • TODO: when a survey notification is received for closed, partially-completed, survey - does the survey progress reset?
    • if yes, then receiving a survey notification probably causes a full survey reload.
    • if not, then it just marks the survey as visible..... - I would have to read a bunch of code.

[hydawo]

Hi @biblicabeebli and @MMel099 here is some information on my beta testing ID that had multiple surveys sent/completed on 12/3/24

m1gz9ib3 in the Hassan_Beta_Test_2_13_23 on Staging

I took surveys normally for quite a while (from registration to 11/25/24). I stopped for ~1 week and resumed surveys on 12/2/24. On 12/3/24, I received the daily Diary survey 3 times. I was able to take it each time and shortly thereafter (within an hour) another copy of the survey would be received and I would take it.

(Eli: This zombie resend issue should be resolved now.)

I've confirmed in the output data that there are 3 answer files and 3 timings files uploaded on 12/3/24.

I've now moved to another beta testing ID in the Yale beta study on staging, so this ID is no longer active. Let me know if I can provide more details. Thank you!

[hydawo]

@MMel099 @jprince127 @QianJingyun make sure you are running the latest TestFlight build (2024.45) and registered to the Yale Beta Study

Here are 2 tests we need to perform this week; we'll discuss further in our RA meeting:

Test 1: Skip a day of surveys → wait for next day’s survey → take next day’s survey and monitor if another survey is deployed to you (before next scheduled time)

Disregard this test for now, we may come back to it. Yale reported one user missing a notification, sending a manual push, taking the survey, and receiving the survey again a few hours later (resend kicked in and ignored the manual push).
Test 2: Receive a survey → wait 1 hour → manually resend survey to yourself → check how many surveys you have available → take them (hopefully only 1 copy) → monitor if another survey is deployed to you (before next scheduled time)

[hydawo]

To test this, let's do the following:

Test 1: In the Diary survey, answer question 1, click next and move to question 2 (do not answer question 2), go back to question 1, is Next button present?

Test 2: In the Diary survey, answer question 1, click next and move to question 2, answer question 2, go back to question 1, is Next button present?

We need to identify if this bug: is actually this issue:

• If you clear out the answers for a required question (this cannot be done with all question types, like radio buttons and date/time) and then go back 1 question to another [answered] required question, that previous question will be displayed without a next button until you refresh the answer by interacting with the question.

(this is inside a collection of minor ui weirdnesses, which are collected in issue # 199 over on an old tracking issue in our internal tracking repo. (We should break those out into tracking issues in their respective repo)

[biblicabeebli]

I've got a possible fix for the zombie survey notifications deployed to staging.

[jprince127]

Hello!

Here are the results of my testing:

Test 1: Skip a day of surveys → wait for next day’s survey → take next day’s survey and monitor if another survey is deployed to you (before next scheduled time)

I skipped the survey on Wednesday (12/11) and waited for the next day's survey and there was not another survey that was deployed to me before the next scheduled time

Test 1: In the Diary survey, answer question 1, click next and move to question 2 (do not answer question 2), go back to question 1, is Next button present?

Yes, the next button is present

Test 2: In the Diary survey, answer question 1, click next and move to question 2, answer question 2, go back to question 1, is Next button present?

Yes, the next button is present

However, I did manage to trigger a survey answer bug, where I answered question 1, then answered question 2, progressed to the third question, then went back to the first question, and hit cancel. Then, when I reopened the survey, there was no longer the back button, and also on the THIRD question there was no longer an option to progress past it. Attached is a screenshot (no "Next" or "Back" button)

[QianJingyun]

Hello!

Here are the results of my testing:

Test 1: Skip a day of surveys → wait for next day’s survey → take next day’s survey and monitor if another survey is deployed to you (before next scheduled time)

I skipped the 'Diary' survey on Wednesday (12/11) and both the 'Diary' and 'Audio Survey - The Caterpillar Passage' survey on Thursday (12/12) between 9:15 AM and 9:21 AM. There was not another survey that was deployed to me, at least not until (12/13 3:00 AM).

Test 1: In the Diary survey, answer question 1, click next and move to question 2 (do not answer question 2), go back to question 1, is Next button present?

Yes, the next button is present.

Test 2: In the Diary survey, answer question 1, click next and move to question 2, answer question 2, go back to question 1, is Next button present?

Yes, the next button is present.

I received notification ' Beiwe may not be running correctly, please open the Beiwe app.' This is probably because my phone closed the app back stage. But this did not affect me receiving notification nor finishing the survey.

[MMel099]

Testing results:

Test 1: Skip a day of surveys and observe for additional deployments the next day.

I skipped the Diary survey last Wednesday, Dec 11. On Thursday, I got the 9am deployment which I took. Later that day, I received an audio survey (as expected based on the schedule) and no further redeployments of the diary.

Test 2 and 3: Test whether clicking 'back' button affects UI bug related to the 'back' and 'next' buttons. For test 2 do not answer Q2, for test 3 answer it.

For both of these tests, I could not recreate the UI bug. However, I was able to recreate it yesterday when I exit out of the survey and reopen it. Yesterday (Sunday, Dec 15), I started the survey, completed the first question, went on to the second question and then exited the survey. After reopening it, the 'next' button was still present for first six questions but the 'back' button disappeared. After Q6, the 'next' button was also gone. I had to leave this survey as 'incomplete' because I could not advance within it. Early this morning, I checked in on Beiwe and the status of the survey was 'new' (which I would not expect because there should not have been any pushes yet). I was able to complete the survey. After getting the 9am notification, I once again attempted to recreate the UI bug. I followed the same steps (exiting out of the survey) and got the same results, except this time the 'next' button was missing by Q3.

[biblicabeebli]

Development of this version of the Beiwe app is completed, the app has been released (rolling release, over the next 7 days) as build 2024.48. If you encounter bugs referenced here on the released app please create a new issue referencing this issue's appropriate comment..

Thank you everyone !