-
Notifications
You must be signed in to change notification settings - Fork 45
Tracing popHealth Calculations
popHealth allows you to turn on tracing (logging) for its calculation engine. While the output requires some deciphering, it's a helpful way to understand why a patient is or is not being included in a measure population.
WARNING: The tracing functionality will cause popHealth to run significantly slower. You should not use this in a production setting. If you need to use it to troubleshoot a production setup, be sure to revert your changes when done.
To enable logging, you first need to turn this on in the popHealth configuration. In your popHealth instance, edit config/popHealth.yml and set the following two entries to "true"
enable_map_reduce_rationale: true
enable_map_reduce_logging: true
It may be easier to modify this for the particular environment (e.g., development, staging) that you're running in. For example, you can override the default setting just for the development environment as follows:
development:
<<: *defaults
enable_map_reduce_rationale: true
enable_map_reduce_logging: true
Note that the additional logs generated by making the above settings change are all stored in the MongoDB. Therefore, to view the logs, you will need a MongoDB database viewer such as Robo 3T - https://robomongo.org/download.
By default, the rationale logging in popHealth will not log the criteria for the initial patient population (IPP). If you wish to enable this, you will need to make a change within MongoDB.
If you are using a client, such as Robomongo*, you can edit the map_reduce_utils script directly from the client.
*NOTE - As of spring 2018, Robo 3T has a known issue where editing scripts does not save. This does work in the older Robomongo client. If you only have access to Robo 3T, please follow the instructions for making the changes from the command line
Edit the map_reduce_utils script (located under the Functions section of the database) that MongoDB has loaded. By default it should have this on lines 10-12:
root.emitResult = root.emitResult || function(value) {
if (value['IPP'] > 0) emit(ObjectId(), value);
}
The change you will make is simply to remove the check for the IPP being >0.
root.emitResult = root.emitResult || function(value) {
emit(ObjectId(), value);
}
You will first need to get a copy of the map_reduce_utils.js file. If you have downloaded and installed a measure bundle, you can unpack the measure bundle ZIP, and find the functions under library_functions/. As shown in the previous section, you will want to remove the "if" statement that checks that the IPP criterion is met. Once you have saved map_reduce_utils.js, go to the command line and run the following to update the function in the database.
This assumes that you are updating a database called pophealth-developoment, and that the map_reduce_utils.js file is located at ~/popHealth/bundles/library_functions/map_reduce_utils.js.
mongo pophealth-development --eval "db.system.js.remove({_id : 'map_reduce_utils'}); db.system.js.insert({_id: 'map_reduce_utils', value: Code('function() {\n' + cat('~/popHealth/bundles/library_functions/map_reduce_utils.js') + '\n }')})";
Restart your delayed_job processor and the popHealth web application. This will ensure the configuration changes are picked up by both parts of popHealth. Next, clear out any existing calculations (Menu -> Patients -> click on "Delete Caches"). This will ensure popHealth is actually recalculating all of the measures and including the tracing information.
Navigate to the practice that you wish to run the calculations for. As previously noted, calculations with logging enabled will take longer to complete. When the results display, you're ready to begin your analysis.
Within your Mongo database, look for a collection named "patient_cache". For each document in the collection, underneath "value" there should be an array of strings named "logger". Each line in the "logger" array will show you a particular step from the measure logic that was executed, and if the patient matched that particular criteria or not.
The first thing you may notice is that the names used for the different parts of the logic are not very intuitive. These are system-generated from the HQMF document that defines the measure logic, and so if you're not sure what a particular block or element represents, you will need to look at the original measure logic.
Here's an example output for a patient that does not match the initial patient population (IPP):
[
" IPP:",
" called allTrue(3,[object Object],,function () {\n var args = [func];\n push.apply(args, arguments);\n return wrapper.apply(this, args);\n },function () {\n var args = [func];\n push.apply(args, arguments);\n return wrapper.apply(this, args);\n }):",
" During_50C7634A_3BDF_4DC6_B589_7EFBC3A41AEF_F427B118_77F0_4D13_A5C5_A8762F733EBB:",
" called patient.allProcedures():",
" called patient.procedures():",
" patient.procedures() -> 0 entries",
" called patient.immunizations():",
" patient.immunizations() -> 0 entries",
" called patient.medications():",
" patient.medications() -> 0 entries",
" patient.allProcedures() -> 0 entries",
" matched -> 0 entries",
" MeasurePeriod:",
" MeasurePeriod -> true",
" During_50C7634A_3BDF_4DC6_B589_7EFBC3A41AEF_F427B118_77F0_4D13_A5C5_A8762F733EBB -> false",
" During_2DCD9F36_5921_479E_AF70_F275727A434B_BB7AA963_EB69_46B5_BBE1_F86CC145535F:",
" PreventiveCare_InitialOfficeVisit0to17_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6261:",
" called patient.encounters():",
" patient.encounters() -> 2 entries",
" matched -> 0 entries",
" PreventiveCare_InitialOfficeVisit0to17_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6261 -> false",
" PreventiveCare_EstablishedOfficeVisit0to17_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6262:",
" matched -> 0 entries",
" PreventiveCare_EstablishedOfficeVisit0to17_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6262 -> false",
" PreventiveCareServices_EstablishedOfficeVisit18andUp_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6265:",
" matched -> 0 entries",
" PreventiveCareServices_EstablishedOfficeVisit18andUp_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6265 -> false",
" PreventiveCareServices_InitialOfficeVisit18andUp_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6264:",
" matched -> 0 entries",
" PreventiveCareServices_InitialOfficeVisit18andUp_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6264 -> false",
" OfficeVisit_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6260:",
" matched -> 2 entries",
" OfficeVisit_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6260 -> true",
" Specific context",
" [\"OccurrenceA_Referral_InterventionPerformed_40280381_3d61_56a7_013e_7aa509fd625d_source\"]",
" [\"*\"]",
" ------",
" Face_to_FaceInteraction_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6263:",
" matched -> 0 entries",
" Face_to_FaceInteraction_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6263 -> false",
" OphthalmologicalServices_EncounterPerformed_ecbd6358_a35c_4cac_b0fb_0cc958d11e47:",
" matched -> 0 entries",
" OphthalmologicalServices_EncounterPerformed_ecbd6358_a35c_4cac_b0fb_0cc958d11e47 -> false",
" MeasurePeriod:",
" MeasurePeriod -> true",
" During_2DCD9F36_5921_479E_AF70_F275727A434B_BB7AA963_EB69_46B5_BBE1_F86CC145535F -> true",
" Specific context",
" [\"OccurrenceA_Referral_InterventionPerformed_40280381_3d61_56a7_013e_7aa509fd625d_source\"]",
" [\"*\"]",
" ------",
" Intersecting (2):",
" [\"OccurrenceA_Referral_InterventionPerformed_40280381_3d61_56a7_013e_7aa509fd625d_source\"]",
" [\"*\"]",
" Intersected result:",
" allTrue -> false",
" called allTrue(IPP,[object Object],,false):",
" Intersecting (1):",
" Intersected result:",
" allTrue -> false",
" IPP -> false",
" VARIABLES:",
" VARIABLES -> false"
]
Walking through this, we can start to decipher what's going on. The call to called allTrue(... is applying logic where all of the criteria must be met. The first block is named During_50C7634A_3BDF_4DC6_B589_7EFBC3A41AEF_F427B118_77F0_4D13_A5C5_A8762F733EBB, and if we go to the end of that block (at the end of the name) you'll see -> false",. This means that this block did NOT pass. We know right away then that this criteria block failed, so that's why the patient isn't in the IPP. We can look inside the block (the lines indented from the block's name) to see that it's trying to find procedures, immunizations and procedures and finding 0 entries. That's probably why it's failing. As for what it's exactly looking for (Which procedures? Which immunizations?), that is not included in the log so we will need to go to the measure logic to get the details.
We can follow our curiosity and looking at the next block, we do see that the patient passes that logic During_2DCD9F36_5921_479E_AF70_F275727A434B_BB7AA963_EB69_46B5_BBE1_F86CC145535F -> true", Following it down towards the end, we can also see for the IPP population that the patient was excluded IPP -> false.
It may be easier sometimes to read these logs in reverse. Find the population that you're trying to troubleshoot (e.g., IPP, DENOM), and then start finding blocks that are false where you need them to be true. Remember the measure logic too - not all blocks have to be true, like if there are multiple within an OR condition. Refer back to your measure logic if you're not sure.
The logger field shows the path of the business rules used to assign a patient to one of the components of the quality measure, i.e., IPP, NUM, DENOM, etc. However, to find out the specific data element within the patient record used in the measure logic to assign a patient to component such as NUM or DENOM, the rationale field provides the conditions, i.e., so called "precondition_#" and the data elements, i.e., object id used in the measure logic. For example, in the above logger example, the OfficeVisit_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6260 element would be described in the rationale field by corresponding JSON element within the patient record such as the code used, i.e., SNOMED, CPT, etc.
To fully understand the meaning of precondition_#, one need to examine the measure logic. You can view the measure logic by opening the specific measure document under the measures collection. From the map_fn, one can search for the "preconditions", which will lead to the precondition # used in the rationale field. The precondition # will also specify the matched condition which are defined in the later part of the document such as OfficeVisit_EncounterPerformed_40280381_3d61_56a7_013e_7aa509fe6260. The definition of the condition will include the valueset and whether the status and negation are true or false. These information will help you determine for example why a particular encounter within the patient did not match the condition needed for the denominator, i.e., the snomed code used is not in the latest valueset published by AHRQ.