clinicaltrials.gov website changes

[agitter]

ClinicalTrials.gov updated their website (announcement). This will likely affect clinicaltrials.gov.js. For instance, the classic version of the site now has URL patterns like https://classic.clinicaltrials.gov/ct2/show/NCT04292899, and the translator target is "target": "^https://(www\\.)?clinicaltrials\\.gov/ct2/(show|results\\?)". The new version of the site may store metadata differently or different metadata altogether.

Here's an example of the new and classic sites:

• https://classic.clinicaltrials.gov/ct2/show/NCT04292899
• https://www.clinicaltrials.gov/study/NCT04292899

[zoe-translates]

The current implementation uses the API for most of the work, and we should migrate to the new API too.

@adam3smith, may I have your opinion about migration? Based on your changes to the MathSciNet translator, am I correct to say that we want to move the existing translator script (without changing the UUID) to a new name like clinicaltrials.gov (Legacy).js, and create a new translator (with new UUID) that takes over the name clinicaltrials.gov.js?

[dstillman]

Is there any reason to support the classic domain?

We usually just update the translator for the current version of the site.

[zoe-translates]

From the site: https://www.clinicaltrials.gov/data-about-studies/learn-about-api
(emphasis mine)

This is a draft version of the new API
Authentication is not required to access the API at this time but will be required in future.
The classic ClinicalTrials.gov API will remain available for some time.

And from https://www.clinicaltrials.gov/data-about-studies/api-migration

The legacy API will remain available for the foreseeable future, and users will be notified in advance of any changes to that status.

It seems that for API access, the classic domain will be supported "for some time". Meanwhile, the default user-facing web interface switched to the new design, while the old design (for which the translator was written) is being temporarily maintained at the classic domain.

I think I'm going to do the following:

• We use URL and on-page design elements for detection, so this definitely needs fixing. But my feeling is that it shouldn't require a split into two translators, because we only use these elements sparingly. The actual item metadata are obtained from the API.
• For API use, we don't have to do much for now; perhaps just change the domain in order to save a redirect for each call.
• Meanwhile, it's advisable to prepare for the new API. The future requirements of authentication is a bit concerning but we don't have much concrete info about that atm.