Add URL Details endpoint to REST API to allow retrieval of info about a remote URL #18042

getdave · 2019-10-21T09:20:18Z

Adds a new REST API endpoint to grab details from a external URL. So now if I want to get the contents of the title tag on https://wordpress.org I can send a request to this API endpoint and it will return the value in the response for use in the Gutenberg UI.

Why might this be useful? See #19387.

~~As part of #17846 we need to display the contents of the <title> tag from the url the user has typed into the input field.~~ This has now long since shipped. Nonetheless, it might be a nice enhancement to be able to show details about the remote URL that has been selected.

This PR addresses this by adding a new experimental custom Gutenberg REST API endpoint. This endpoint:

makes a request for a remote URL.
~~parses the response in a DOMDocument~~ we should investigate this in future (see here)
retrieves the title tag contents.

To prevent abuse, the endpoint is authenticated and requires the user to have the edit_posts capability to be able to make a request.

How has this been tested?

Manual testing (see below)
Automated tests

Automated Testing Instructions

npm run test-php should run the unit tests which reside in phpunit/class-wp-rest-url-details-controller-test.php.

Manual Testing Instructions

Check out this PR.
Open the Gutenberg local testing env at http://localhost:8889/wp-admin/ and login.
Create a new Post.
Choose one of the following options:

Option 1

Open devtools and in the console enter:

wp.apiFetch( { path: '/__experimental/url-details/?url=https://wordpress.org' } ).then( data => {
    console.log( data );
} );

You'll see the response returned there.

Option 2

Open devtools and in the console enter wpApiSettings.nonce. You should see a valid nonce returned as a string.
Copy the nonce.
In your browser goto: http://localhost:8889/wp-json/__experimental/url-details/?url=https://wordpress.org&_wpnonce=%YOUR_NONCE_HERE% - be sure to replace the nonce value with that copied from the previous step.
You should see a valid REST API response containing the contents of the title tag from wordpress.org (ie: "Blog Tool, Publishing Platform, and CMS \u2014 WordPress.org").

Types of changes

New feature (non-breaking change which adds functionality).

Checklist:

My code is tested.
My code follows the WordPress code style.
My code follows the accessibility standards.
My code has proper inline documentation.
I've included developer documentation if appropriate.

lib/class-wp-rest-url-details-controller.php

getdave

@obenland Thanks for the changes! Really appreciate you taking this on and making it much better.

Left some small comments. Interested in what you think.

We need to wrangle some more feedback from the REST API group in Core.

We also need a rebase to make the playground pass (it's happened on all PRs recently).

Lastly, what do you think about this comment on the Issue?

lib/class-wp-rest-url-details-controller.php

TimothyBJacobs · 2019-11-08T22:10:41Z

Has @mnelson4's comment been addressed?

getdave · 2019-11-18T15:14:16Z

@kadamwhite We'd really appreciate your expertise and input on this if at all possible.

getdave · 2019-12-03T07:22:21Z

@obenland Shall we try and move this forward in our next 20% time? What should we do to prepare for that so we can make progress at that time?

obenland · 2019-12-03T20:01:00Z

Sure. I think double-checking #18042 (comment) would be the next step

getdave · 2020-01-02T14:45:47Z

Sure. I think double-checking #18042 (comment) would be the next step

I've addressed the issue raised here.

getdave · 2020-01-16T10:27:38Z

Waiting on further feedback here. The outstanding issue blocking this from merging is the concern about exposing data about websites behind a firewall.

Oh...and it looks like we have some PHPUnit test warnings @obenland.

mnelson4 · 2020-01-21T06:19:55Z

From what I could test, this doesn't expose data local data. Although I don't have a network behind a firewall etc, so I can't really comment about that.

I tested this out and it worked fine. It did seem a bit odd the response wasn't actually JSON, just a string of the site's title. (I suppose returning JSON leaves the door open to returning other metadata about the site.) I wasn't aware of anywhere else in the REST API that just does that, but it's not the end of the world.

…ter args

Addresses #18042 (comment)

See #18042 (comment)

Addresses #18042 (comment)

getdave · 2021-02-01T20:47:05Z

Merged 🎉

I will raise followups to add extra functionality.

Thanks to everyone for your help in getting this over the line (particularly @TimothyBJacobs and @swissspidy).

obenland · 2021-02-01T20:59:49Z

Congrats @getdave! Nice work

@access

… a remote URL (#18042) * Scaffold out basic new endpoint * Implement basic retrival of title tag from remote url * Adds validation, sanitization and permissions checks. * i18n fixes and docblocks * Adds caching of remote request * Update with feedback * Tie up loose ends * Remove unneed replacements * Account for Custom Post Types in permissions check Addresses #18042 (comment) * Improve error handling * Improve args checking * Refactor to enable query of more data in future. * Use md5 Addresses #18042 (comment) * Extract utility functions * Fix lint errors * Add unit test scaffold * Attempt to mock HTTP requests for testing. Needs work. * Use DIR convention * Allow opt out of cache functionality via filter * Force tests to opt out of cache. Fix broken tests. * Adjust 404 case to use standard response code * Add dedicated methods to mock success/failure HTTP responses * Add test for empty body in response. This should trigger a failed response because we can’t read details from a URL which doesn’t provide a HTTP body. * Add ability to filter request args. Update tests to account. * Document cache filter * Iniital attempt at adding schema * Add test for unautenticated user. * Adjust schema to use WP convention * Adjust schema defaults * Add endpoint args schema test * Removes filter to bypass cache. This already exists in core as https://core.trac.wordpress.org/browser/tags/5.6/src/wp-includes/option.php#L785 * Use existing transient filter to test cache and bypass for majority of tests Fixes #18042 (comment) * Use default timeout. Addresses #18042 (comment) * Removes unnecessary default from schema Addresses #18042 (comment) * Use predefine constants for response size Resolves #18042 (comment) * Rearrange test code order * Adds ability to filter cache expiration Addresses #18042 (review) * Allow filtering the data retrived for a given URL Addresses #18042 (comment) * Remove @access comments Addresses #18042 (comment) * Prefer custom error message over get_status_header_desc Addresses #18042 (comment) * Utilise add_additional_fields_schema Addresses #18042 (comment) * Utilise add_additional_fields_to_object Addresses #18042 (comment) * Enable filtering of response object Filter on both cached and uncached response. * Add test to assert on ability to filter uncached responses * Rename url placeholder literal * Ensure accurately testing for correctly passed cached variable in filter args * Update to test filtering of both cached and uncached responses * Fixing linting * Use array offset syntax prefered by Core Addresses #18042 (comment) * Remove from cache and update filter See #18042 (comment) * Update filter name to align with standards Addresses #18042 (comment) * Update response code to align with Core standards Addresses #18042 (comment) * Update cache ttl in docs to use variable Addresses #18042 (comment) * Use comment to reference canonical version of filter docs Addresses #18042 (comment) * Update to cache entire HTTP response body and remove additional filter Addresses #18042 (comment) * Tweak code comments * Remove need to encode/decode cache value Co-authored-by: Konstantin Obenland <[email protected]> Add Bottom Sheet Select Control componet Add Bottom Sheet Select Control componet Remove file

getdave · 2021-02-05T16:37:12Z

@TimothyBJacobs At some point if this endpoint is accepted would I need to follow up with this in WordPess Core?

TimothyBJacobs · 2021-02-05T19:31:10Z

Whenever the JS feature that uses this feature is slated to be included in Core, we'll create a Core ticket for merging the URL details endpoint. I can take care of merging it into Core.

getdave · 2021-02-06T15:25:13Z

Whenever the JS feature that uses this feature is slated to be included in Core, we'll create a Core ticket for merging the URL details endpoint. I can take care of merging it into Core.

That's great thanks. Any chance I could get involved there? Only Ive not committed to WP Core and this seems like a good opportunity.

TimothyBJacobs · 2021-02-07T01:19:35Z

That's great thanks. Any chance I could get involved there? Only Ive not committed to WP Core and this seems like a good opportunity.

Yeah, you can absolutely take the lead on that!

getdave · 2021-04-22T12:09:54Z

Here is a follow up #28791

getdave requested review from noisysocks, obenland, retrofox, frontdevde and marekhrabe October 21, 2019 09:22

getdave self-assigned this Oct 21, 2019

getdave added the REST API Interaction Related to REST API label Oct 21, 2019

getdave mentioned this pull request Oct 21, 2019

REST API endpoint to retrieve information about a remote URL. #18047

Closed

getdave added the Needs Technical Feedback Needs testing from a developer perspective. label Oct 21, 2019

obenland requested changes Oct 21, 2019

View reviewed changes

xknown reviewed Oct 25, 2019

View reviewed changes

lib/class-wp-rest-url-details-controller.php Outdated Show resolved Hide resolved

lib/class-wp-rest-url-details-controller.php Outdated Show resolved Hide resolved

obenland self-requested a review October 31, 2019 22:10

obenland marked this pull request as ready for review October 31, 2019 22:10

obenland requested a review from TimothyBJacobs as a code owner October 31, 2019 22:10

obenland approved these changes Oct 31, 2019

View reviewed changes

obenland self-requested a review October 31, 2019 22:11

getdave commented Nov 1, 2019

View reviewed changes

obenland force-pushed the try/add-url-preview-endpoint-to-rest-api branch from 2722e34 to 4c330d2 Compare November 1, 2019 22:10

getdave force-pushed the try/add-url-preview-endpoint-to-rest-api branch from 4c330d2 to a5d9c25 Compare January 2, 2020 14:46

getdave mentioned this pull request Jan 2, 2020

Add URL preview to Link UI #19387

Closed

6 tasks

getdave requested review from draganescu and aduth January 16, 2020 10:26

aduth mentioned this pull request Feb 3, 2020

How we can approach the introduction of a new link component within Gutenberg #18061

Closed

getdave added 12 commits February 1, 2021 19:00

Ensure accurately testing for correctly passed cached variable in fil…

5f4cc4e

…ter args

Update to test filtering of both cached and uncached responses

3d8becc

Fixing linting

fd44a72

Use array offset syntax prefered by Core

5d9f314

Addresses #18042 (comment)

Remove from cache and update filter

3bcd206

See #18042 (comment)

Update filter name to align with standards

802779f

Addresses #18042 (comment)

Update response code to align with Core standards

af45585

Addresses #18042 (comment)

Update cache ttl in docs to use variable

99f30f6

Addresses #18042 (comment)

Use comment to reference canonical version of filter docs

27de858

Addresses #18042 (comment)

Update to cache entire HTTP response body and remove additional filter

12f9724

Addresses #18042 (comment)

Tweak code comments

5947a94

Remove need to encode/decode cache value

5967e70

getdave force-pushed the try/add-url-preview-endpoint-to-rest-api branch from 3067b99 to 5967e70 Compare February 1, 2021 19:04

getdave merged commit 74ec7d0 into master Feb 1, 2021

getdave deleted the try/add-url-preview-endpoint-to-rest-api branch February 1, 2021 20:45

github-actions bot added this to the Gutenberg 10.0 milestone Feb 1, 2021

getdave mentioned this pull request Feb 5, 2021

Improve the parsing of remote url details in the URL Details endpoint to REST API #28791

Closed

6 tasks

getdave mentioned this pull request Apr 22, 2021

Add experimental util to allow fetch remote url data from REST API #31085

Merged

7 tasks

This was referenced May 4, 2021

Enhance link control UI with rich URL previews #31464

Merged

Show "rich" meta data about remote URL when adding URLs as links #31466

Closed

getdave mentioned this pull request May 12, 2021

Improve parsing and retrieve additional data in REST url-details endpoint #31763

Merged

5 tasks

getdave mentioned this pull request Oct 19, 2021

Handle merging of URL Details REST API endpoint into Core for WP 5.9 #35756

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add URL Details endpoint to REST API to allow retrieval of info about a remote URL #18042

Add URL Details endpoint to REST API to allow retrieval of info about a remote URL #18042

getdave commented Oct 21, 2019 •

edited

Loading

getdave left a comment •

edited

Loading

TimothyBJacobs commented Nov 8, 2019

getdave commented Nov 18, 2019

getdave commented Dec 3, 2019

obenland commented Dec 3, 2019

getdave commented Jan 2, 2020

getdave commented Jan 16, 2020 •

edited

Loading

mnelson4 commented Jan 21, 2020

getdave commented Feb 1, 2021

obenland commented Feb 1, 2021

getdave commented Feb 5, 2021

TimothyBJacobs commented Feb 5, 2021

getdave commented Feb 6, 2021

TimothyBJacobs commented Feb 7, 2021

getdave commented Apr 22, 2021

Add URL Details endpoint to REST API to allow retrieval of info about a remote URL #18042

Add URL Details endpoint to REST API to allow retrieval of info about a remote URL #18042

Conversation

getdave commented Oct 21, 2019 • edited Loading

How has this been tested?

Automated Testing Instructions

Manual Testing Instructions

Option 1

Option 2

Types of changes

Checklist:

getdave left a comment • edited Loading

Choose a reason for hiding this comment

TimothyBJacobs commented Nov 8, 2019

getdave commented Nov 18, 2019

getdave commented Dec 3, 2019

obenland commented Dec 3, 2019

getdave commented Jan 2, 2020

getdave commented Jan 16, 2020 • edited Loading

mnelson4 commented Jan 21, 2020

getdave commented Feb 1, 2021

obenland commented Feb 1, 2021

getdave commented Feb 5, 2021

TimothyBJacobs commented Feb 5, 2021

getdave commented Feb 6, 2021

TimothyBJacobs commented Feb 7, 2021

getdave commented Apr 22, 2021

getdave commented Oct 21, 2019 •

edited

Loading

getdave left a comment •

edited

Loading

getdave commented Jan 16, 2020 •

edited

Loading