Android Data Upload

File creation frequency and data upload frequency

The App Settings page for each study has two checkboxes titled "Create new data files frequency" and "Upload data files frequency."

• Create new data files frequency: most of the passive data streams (GPS, accelerometer, power state, phone calls, text messages, Bluetooth, and the debug log) are written to open log files, and this setting controls how frequently the current log files are closed and new log files are created.

  This setting was more important when Beiwe was first created, because back then, researchers would download the raw data files created by the phone, so this setting controlled the duration of the log files that the researchers saw. But now that the backend repackages raw data files into hour-long chunks for researchers to download, this setting doesn't affect the data the researchers see.

  It's probably best to set this to some value between 5 minutes (300s) and 1 hour (3600s). Below 5 minutes, there would be so many files created that it may slow down the data upload. Above 1 hour, the delay in uploading data starts to become significant, because data files only get uploaded after they're closed.

  This setting doesn't affect audio recordings, survey answers, or survey timings files, because those are created when the participant starts an audio recording or survey, and they're closed when a participant ends the audio recording or survey. This setting also doesn't affect the WiFi log, because the WiFi logger just creates a new WiFi log file every time Beiwe runs a WiFi scan; so the separate setting "WiFi log frequency" controls how often new WiFi log files are created.

• Upload data files frequency controls how often Beiwe tries to upload completed data files. Beiwe basically has a recurring timer that runs on this frequency, and every time the timer goes off, it checks to see if there are completed data files, and if there are, it uploads them. If this timer goes off while an accelerometer log file is open and being written, that accelerometer log file doesn't get uploaded; likewise, if the timer goes off during a survey, the survey timings file doesn't get uploaded- both will wait until the next time the upload timer goes off.

Do not set any frequencies to 0, to a negative number, or to anything less than 1 second. Beiwe uses the frequency number to tell how far in the future to run the next operation. If you set upload data files to anything less than 1 second, Beiwe may set the next upload for a time that has already passed, in which case Beiwe will never upload data files.

Example

Let's say the create new files frequency is set to 10 minutes, and the upload frequency is set to 5 minutes. Here's an example of how this would look in a study that uploads only over WiFi (the default behavior). We'll only discuss two types of data files, accelerometer and survey timings:

• At 0 minutes, Beiwe opens a new accelerometer data file ("A1") and starts writing to it.

• At 5 minutes, Beiwe checks for data files to upload, but it doesn't find any because accelerometer data file A1 is still open.

• At 10 minutes, Beiwe closes accelerometer data file A1 after writing 10 minutes of data to it, and opens a new accelerometer file, A2. Beiwe checks if there are data files to upload, finds A1 and uploads it, and doesn't upload A2 because A2 is still being written.

• At 12 minutes, the participant starts a survey, which creates survey timings file "S1" to track how long the participant takes to answer each question.

• At 15 minutes, Beiwe checks if there are data files to upload, and doesn't find any: A2 is still being written, and S1 is still being written because the survey is still open.

• At 18 minutes, the participant finishes and submits the survey, which closes survey timings file S1. For some unrelated reason, the phone loses its WiFi connection.

• At 20 minutes, Beiwe closes accelerometer file A2 and opens a new accelerometer file, A3. Beiwe checks for data files to upload, and finds A2 and S1, but because the phone isn't connected to WiFi, Beiwe doesn't upload them.

• At 23 minutes, the phone connects to WiFi again.

• At 25 minutes, Beiwe checks for files to upload, finds A2 and S1, and uploads them over WiFi.

Allowing data upload over a cellular data plan

On the App Settings page for each study, there's a checkbox titled "Allow data upload over a cellular data plan."

• The default behavior (box unchecked) is that the app will upload data only when the phone is connected to WiFi. This is the default option because the data files (especially accelerometer data files) can be very large, and we're concerned about overusing a study participant's cellphone plan and incurring extra costs for them.

• If the box is checked, the app will upload data when the phone has an internet connection, regardless of whether that connection is WiFi or cellular data. A phone only has one type of internet connection at a time, and phone behavior (set by phone manufacturers) is usually to prioritize connecting to a known WiFi network over connecting to a cellular network. So with "allow data upload over a cellular data plan" checked, Beiwe will upload over WiFi if WiFi is available, and upload over cellular data if WiFi is not available.

The advantage of allowing data upload over a cellular network is that the participant's data are sometimes uploaded faster. Some participants only have their phones connect to WiFi at home, which is why we see a large spike in data upload volume around 6pm on weekdays, when many people get home from work. However, because of the extra data costs that Beiwe could incur for the participant, we recommend only enabling upload over cellular data for studies that either:

1. Do not record accelerometer data (since that's most of the data volume), or

2. Involve the study paying for the participant's data plan.

This checkbox only affects uploading data files; it doesn't affect other internet operations, including registering a phone for a new participant, resetting a password, or the participant viewing their own data. Those functions involve a negligible amount of data transfer, so those are always allowed to happen over WiFi or cellular data, regardless of whether the checkbox is checked.