[SPIKE] Missing Multi-specialty PPMS Provider

[mmiddaugh]

SNOW ticket ID: INC33413316

Describe the defect

An active multi-specialty community provider in New Jersey is not included in Facility Locator search results as expected.

VIRTUA WILLINGBORO HOSPITAL INC
239 HURFFVILLE CROSSKEYS RD STE 340 SEWELL NJ 08080
Service type: Multi-Specialty
NPI 1053316844

To Reproduce

Steps to reproduce the behavior:

1. Go to the Facility Locator
2. Enter 80406 as the geographic search parameter, Community providers (in VA's network) as the Facility type, and Multi-Specialty as the Service type
3. Click search
4. Review search results to observe the provider and address named above are not included in the list.

Expected behavior

Active providers should be included in search results for matching location and specialty.

Screenshots

Screenshot provided by PPMS team showing the details for the missing provider

Facility Locator search results

Additional context

The NPI is associated with the Virtua provider on 218A Sunset shown on the top line of the screenshot from PPMS. It is returned to a search for General Acute Care Hospitals in the area. I’m wondering if the missing multispecialty location is somehow being removed in a deduplication process.

ACs

• Root cause and scope of impact are documented
• Review is scheduled to articulate next steps

[xiongjaneg]

Questions:

• This location has a duplicate ID as another. Is that affecting the display of this one?
• Is this affected by the Zoom not showing additional locations?

[xiongjaneg]

Michelle will do some more exploration on this.

[jilladams]

Potentially related to PPMS pagination (#17646, #17649 ) problems, where we can't get multiple pages of data from PPMS currently.

[mmiddaugh]

Under Non-quarterly prio injections for Sprint 3, I see the suggestion to do #17646 and #17649 before #17547 but I'd suggest starting with a query to confirm if the provider is in the data set rec'd in response. This would allow us to give feedback to the PPMS team if that's the issue. I've located the provider on a completely separate map and then zoomed into the same area on the FL and the provider is not in the results list. I'm not sure pagination is in play here.

[davidmpickett]

Confirming what @mmiddaugh said above, here are screenshots of all 8 results I got when zoomed in to 239 HURFFVILLE CROSSKEYS RD. Since there are only 8 results and the pagination issue only occurs when there are 10 or more results, this is not pagination related.

Notice that there are other results from 239 HURFFVILLE CROSSKEYS RD in results. But none of them have a STE number. Is that possibly part of the issue?

[maxx1128]

I looked at the query results at this prod API endpoint that's right at the missing provider's address, with a radius of 25 and showing up to 5000 results. Seven results are at that address, and none are the missing provider. This leads me to think there are two possibilities as to why this provider isn't appearing.

1. Some additional, unknown parameter(s) may be filtering it out.
2. PPMS has its details but is somehow still excluding the provider.

I double-checked the parameters and didn't find anything that could be accidentally filtering it out, so I'm inclined to think it's #2.

{
  "id": "fd0aea9064f87e0a3965214d057fa348c485f80c8b848afd0ce367dfd32b121e",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE CROSSKEYS RD STE 480",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080-4009"
    },
    "caresitePhone": "856-424-3311",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "CLINICAL HEALTHCARE ASSOCIATES OF NJ",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1669485363"
  }
}, {
  "id": "aba0e7b83cccd0679f803c5ee1763cf957fd7c2cdd2e8467e9b4f22fa5ba8a2c",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE RD STE 470B",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080"
    },
    "caresitePhone": "856-355-7130",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "VIRTUA PAIN SPINE MARLTON RTE 73",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1649226515"
  }
}, {
  "id": "6d2537824251e12ebdba6919f5ea4b32954137d0313fc1566a1f28d08f95c1ef",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE CROSSKEYS RD",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080-4002"
    },
    "caresitePhone": "856-582-2000",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "CARDIOVASCULAR ASCS OF DELAWARE VALLEY",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1366495632"
  }
}, {
  "id": "f46815bca9bdd6b55397e30c46de77e1ea6e9f79dc611be2225823823f2324b3",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE CROSSKEYS RD STE 470A",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080-4009"
    },
    "caresitePhone": "856-983-4263",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "VIRTUA MEDICAL GROUP",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1336560432"
  }
}, {
  "id": "f6a5d3c32e1769e4a2b109f25e0da4922e50fc4e015a791188833b588a3f53e4",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE CROSSKEYS RD STE 340",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080-4007"
    },
    "caresitePhone": "856-265-0500",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "RANCOCAS HOSPITALPROF FEES",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1053316844"
  }
}, {
  "id": "95460cc93ea7268ddfc5f5b4856256bb757d2c7c28bfa5eca7bb220c102c5767",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE CROSSKEYS RD #106",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080-4002"
    },
    "caresitePhone": "856-341-8200",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "VIRTUA PULM & SLEEP MED-WASHINGTON TWP",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1649226515"
  }
}, {
  "id": "121c497907a45b3b01e5b5b9b78561dad8aa3e132080922778c278f0fd9b1bda",
  "type": "provider",
  "attributes": {
    "accNewPatients": "false",
    "address": {
      "street": "239 HURFFVILLE CROSSKEYS RD STE 190",
      "city": "SEWELL",
      "state": "NJ",
      "zip": "08080-4005"
    },
    "caresitePhone": "856-341-8400",
    "email": null,
    "fax": null,
    "gender": "NotSpecified",
    "lat": 39.76309124,
    "long": -75.07579589,
    "name": "UNIVERSITY OF PENN MEDICAL GROUP",
    "phone": null,
    "posCodes": null,
    "prefContact": null,
    "trainings": [],
    "uniqueId": "1003923707"
  }
}

[davidmpickett]

@maxx1128 What's also weird to me is that your API query results for 239 Hurffville are not the same as the ones my FL search puled up

[davidmpickett]

@maxx1128 Possible that FL results are actually combining two separate records? Take Virtua Primary Care for example.

• Facility Locator Result B has Virtua Primary Care - Voorhees at 2225 Ev at 239 Hurffville
• Google Search returns a Virtua Primary Care - Voorhees at 2225 Evesham Road located at 2225 Evesham Road
• Virtua website has these two locations listed right next to each other

[jilladams]

We think the next step is to reach out to PPMS to confirm what API query we need to send, to return this data. We have an email address we can reach out to that may or may not get a response. We likely should plan to attend the next office hours with our details, to get help.

@maxx1128 to start the reach out (cc Jill, Michael, Eli), and plan to attend 4/30 office hours.

[davidmpickett]

Another report that may or may not be related:

Navy Yard Dental (EIN 81-2542887 NPI 1669801569) is a Community Care Network provider but the dental office does not come up on the https://www.va.gov/find-locations/ search function

Good morning. That NPI directs to a specific provider, Chad Jones, who works at Navy Yard Dental, 55 M South East, Suite 103, Washington, DC.

Angel Tucker is another dentist who works at Navy Yard Dental

Locator query link: https://www.va.gov/find-locations/?address=Washington%2C%20District%20of%20Columbia%2020003%2C%20United%20States&bounds%5B%5D%5B%5D=-77.747625&bounds%5B%5D%5B%5D=38.135585&bounds%5B%5D%5B%5D=-76.247625&bounds%5B%5D%5B%5D=39.635585&context=Washington%2C%20District%20of%20Columbia%2020003%2C%20United%20States&facilityType=provider&latitude=38.885585&longitude=-76.997625&page=1&radius=10&serviceType=1223G0001X&bounds%5B%5D=-77.747625&bounds%5B%5D=38.135585&bounds%5B%5D=-76.247625&bounds%5B%5D=39.635585

[jilladams]

@maxx1128 could you take a look at Dave's comment above, and make note of this for your visit to PPMS office hours tomorrow?

[jilladams]

Eli tested the Dental example, against the PPMS endpoint (on CAG). The facility doesn't come up. Eli shared that info with Max. @maxx1128 will add the Dental example to the PPMS email thread.

NPI 1 = Individual providers
NPI 2 = Facilities

Might be that PPMS changed something related to NPI 2 and we haven't caught up with it, in our implementation against their endpoint(s). We use the Facility Service endpoint, so: that's weird.

[jilladams]

From planning:

• We expect that once PPMS provides the query that will successfully return these facilities, we'll need to assess what we're doing in our API calls that's missing the data, and figure out how to revise / combine, etc. our API calls in order to get the data back.
• Because we don't know what's wrong, we won't know how to fix it til we do. Thus: we've framed this as a SPIKE to figure that out, once we hear back. We understand that this is an urgent issue, so once we have a path forward, we may inject the ticket to fix it.
• There's an opportunity for Michael to get engaged with PPMS to try and build relationship / break down the comms wall.

[maxx1128]

Still no response from Anupam about the data as of 5/2.

[jilladams]

Assignment pivot: https://dsva.slack.com/archives/C0FQSS30V/p1715208560396599 🛋️

[eselkin]

I discovered that in a direct production PPMS query with the address set to the desired 239 HURFFVILLE CROSSKEYS RD address, they produce the desired result in the 208th and 209th positions. Currently vets-api pagination is broken with CCP results, showing only the first 7-10. Despite positioning the desired box exactly over the lat/long coordinates we cannot get the result in the first 10. To ever be able to get these results to show up on FL we need to get PPMS to present the result earlier (top 10) or fix pagination (so that on page 21 we see the results) or both.

[davidmpickett]

From Scrum this morning, @jilladams @Agile6MSkinner and I said that we are comfortable considering this spike closed as it has identified the specific reason the provider is not showing up and we know which tickets to prioritize to resolve it.

Want to let @mmiddaugh have final review before closing

[davidmpickett]

Candidates for Sprint 4 to resolve root cause

• [BLOCKED] [CCP & Facility Locator] (#1) Confirm location of lost pagination data for FacilityServiceLocator in Vets-api with rails terminal access #17646
• [BLOCKED] [CCP] (#2) Modify vets-api to handle pagination from CCP PPMS and verify that vets-website facility locator is resolving pagination correctly  #17649

[mmiddaugh]

I'm comfortable closing this issue as the root caused has been identified, thank you.

[jilladams]

Based on office hours today, we went ahead and filed a SNOW ticket with PPMS to track this question around why this provider returns 208th in the list:

INC33413316 - https://yourit.va.gov/va?id=ticket&is_new_order=true&table=incident&sys_id=f0ab1d5687260e54e3ff84450cbb3570

[jilladams]

From call today with John Bryant at PPMS:

PPMS is an API fronting a data warehouse. Their API is only returning a dump of data based on address query, and not doing anything nuanced when it returns data. Any sort / filtering behavior must happen on the "client" side receiving that data back from PPMS. PPMS returns data sorted by distance.

When we put in this missing provider address, the PPMS response is coding that address as 4miles away from the center of the target, which is contributing to this result showing up 208th in the list.

John requested a time we could demonstrate the issue live. We will again on Mon 6/24 at 10a PT / 1p ET.

(cc @Agile6MSkinner for awareness.)

[jilladams]

(I'm keeping notes here to reflect that we're still working with PPMS until we know enough to cut a "next steps" ticket if we are going to need to do things on our end. So far that's not clear.)

[jilladams]

The zoomed-est version that should return this provider: https://www.va.gov/find-locations/?page=1&address=Sewell%2C%20New%20Jersey%2008080%2C%20United%20States&facilityType=provider&serviceType=193200000X&latitude=39.725448779485816&longitude=-75.04365344515719&radius=0&bounds%5B%5D=-75.05279840463538&bounds%5B%5D=39.71831367598011&bounds%5B%5D=-75.03450848567562&bounds%5B%5D=39.732583144717125&context=Sewell%2C%20New%20Jersey%2008080%2C%20United%20States&searchString=Sewell%2C%20New%20Jersey%2008080%2C%20United%20States

[jilladams]

@mmiddaugh FYI: Eli and I did a screenshare with John Bryant (PPMS Support) today. We found a couple of things:

1. This particular provider shows in the PPMS Provider Locator as "Mapped location not validated". He needs to get that validated with his engineering team, but if so: the provider is responsible for reaching out to Trinet and Optum to get that address validated. We could let the provider know, and suggest they proactively reach out. (IT's also possible that once it's validated, the Data team will reach out to Optum/Trinet.)
2. There are other results, further from the original query address / geobox, but show up in the PPMS provider locator as 0.0 distance, as the closest. There are results from the same address that show as 3.9 miles away. That distance sort is provided by the Locator so if their distance is off, our results will also be off. (Providing a sort mechanism to the API is a major effort for PPMS, and probably a non-starter.)

John will follow up on those on the SNOW ticket, and we can follow up in office hours as needed.

For future: our team can request access to the PPMS Provider Locator, if we do a training. Steps to do that:

1. HOW TO GET TO THE PPMS ACCESS REQUEST FORM 1.pdf
2. HOW TO SUBMIT A PPMS ACCESS REQUEST 2.pdf
3. HOW TO ADD AN ATTACHMENT TO A REQUEST.pdf
4. How to Save PPMS Training Certificates from TMS.pdf
5. Active Station Numbers.pdf

[jilladams]

From office hours: PPMS determined that the Address validation is not affecting sort behavior. We will find time to talk to John Bryant about this to better understand the findings.

PPMS stated we'll need to file a change request to deal with filtering / sorting, and will provide Product contacts in order to file a change request.

[jilladams]

From PPMS:

For the NPIs, the TPAs need to validate the information in our system.
As for the change request, I spoke with several team members. The API already does that function, but our system has a bug that shows invalid information regarding distance. We are advocating for this bug to be a priority in the upcoming PI.

Michelle to follow up with the NPI folks who can reach out to the TPA for the validation.

[jilladams]

PPMS is tracking a JIRA ticket to resolve the sort order. They are migrating some infrastructure to Azure in order to consider improvements to their provider locator, and this issue may or may not move before the infrastructure move / planning for that initiative. It will definitely not get fixed before their upcoming PI Planning on 9/18. The Azure move will be a "several PI" process, and each PI lasts 6 sprints/12 weeks.

Their JIRA ticket is: "PPMS-5534: Distances are zero in Error."

If we ever want to check on progress, we can log into the bi-weekly PPMS call and ask about it.

PPMS did suggest that if we don't want to wait for their work to ship, we could prefilter our responses to only display results that exactly match the requested address. This feels like a no-op given the map drag functionality we want to support in the UI, etc. but noting it for due diligence purposes. Oneil Lespierre is an engineer on PPMS who said we can reach out for support / questions on the API return data if we opt to go this route.