better error message when a batch is too large

closes #400

NEWS.md

@@ -8,6 +8,7 @@ Upcoming Versions
 * `redcap_read()` and `redcap_read_oneshot()` accept a new `locale` parameter that specifies date, time, and number formats, like using a comma as the decimal separator.  It is a [`readr::locale`](https://readr.tidyverse.org/reference/locale.html) object.  (#377, suggested by @joundso)
 * New `redcap_instruments()` function exports a list of the data collection instruments for a project.  (#381, @vcastro)
 * New `redcap_event_instruments()` function exports the instrument-event mappings for a project (i.e., how the data collection instruments are designated for certain events in a longitudinal project)..  (#381, @vcastro)
+* New detection when REDCap has trouble with a large request and drops records. (#400 w/ @TimMonahan)
 
 ### Minor Enhancements
 
@@ -20,7 +21,7 @@ Upcoming Versions
 * Modify  `redcap_users_export()` (which calls REDCap's user export).  The API dropped the `data_export` variable and added the `forms_export` variable. (#396)
 * For `redcap_read_oneshot_eav()`: if the project isn't longitudinal, a dummy value for `event_id` is used internally (#396)
 * For the testing server & projects, the http errors are a little different, so the testing code was adjusted (#396)
-* Set `httr::user_agent`, following the advice of its vignette (#397)
+* Set `httr::user_agent`, following the advice of httr's vignette (#397)
 
 ### Test Suite 
 

R/redcap-read.R

@@ -400,8 +400,21 @@ redcap_read <- function(
   ids_missing_rows  <- setdiff(unique_ids, unique_ids_actual)
 
   if (0L < length(ids_missing_rows)) {
+    message_template <-
+      paste0(
+        "There are %i subject(s) that are missing rows in the returned dataset. ",
+        "REDCap's PHP code is likely trying to process too much text in one bite.\n\n",
+        "Common solutions this problem are:\n",
+        "  - specifying only the records you need (w/ `records`)\n",
+        "  - specifying only the fields you need (w/ `fields`)\n",
+        "  - specifying only the forms you need (w/ `forms`)\n",
+        "  - specifying a subset w/ `filter_logic`\n",
+        "  - reduce `batch_size`\n\n",
+        "The missing ids are:\n",
+        "%s."
+      )
     stop(sprintf(
-      "There are %i subject(s) that are missing rows in the final dataset.\nCheck for funny values that could trip up REDCap's PHP code:\n%s.",
+      message_template,
       length(ids_missing_rows),
       paste(ids_missing_rows, collapse=",")
     ))