URL saved in database without %20 characters

[nikhiltalati]

THe original URL saved in database is stripped of spaces and on opening Short URL we get 404 error.

[ozh]

Cannot reproduce.

• Go to https://ozh.org/temp/hello spaces.txt
• Copy paste the URL in your YOURLS interface
• it gets shortened and %20 are preserved, example https://yourls.org/fm

[dgw]

This sounds like you are shortening links via the API, which adds another layer of required encoding if you're sending the long URL as a GET parameter. Try sending https://ozh.org/temp/hello%2520spaces.txt as the URL to shorten.

If your input needs to be percent-escaped, you have to escape it again to send it through the query parameters. Chances are that whatever library you're using to call the YOURLS API has an option to handle urlencoding for you, e.g. the requests library for Python automatically handles encoding for a dictionary of parameters.

[counterpoint]

Thanks. Although I have tried using the API, the tests I refer to above were run simply by entering the request to a browser. Attempting to use the API, I run into a range of problems. The URL starts out with the query string encoded. Submitting it to the API as is, the result is all the spaces are lost. Simply URL encoding the whole URL results in things like the "::/" getting encoded which then doesn't seem to work at all. Splitting up the URL and encoding (again) the query string stores the double encoded URL, which doesn't seem to work either.

I tried submitting your suggested URL via the API using the following code (YOURLS_URL is defined as https://yourls.example.com/yourls-api.php):

$longurl = "https://ozh.org/temp/hello%2520spaces.txt"; $postparms = [ 'signature' => '????????', 'action' => 'shorturl', 'url' => $longurl, 'format' => 'json' ]; foreach ($postparms as $key=>$value) $query[] = "$key=$value"; $bitlycurl = curl_init(); curl_setopt($bitlycurl, CURLOPT_URL, YOURLS_URL.'?'.implode('&',$query)); curl_setopt($bitlycurl, CURLOPT_RETURNTRANSFER, true); curl_setopt($bitlycurl, CURLOPT_HEADER, false); curl_setopt($bitlycurl, CURLOPT_POSTFIELDS, $postparms); $response = curl_exec($bitlycurl); $info = json_decode($response); var_dump($info);
(Sorry I can't see how to get code to honour newlines). The response is:

object(stdClass)#2 (8) { ["status"]=> string(7) "success" ["code"]=> string(0) "" ["message"]=> string(59) "https://ozh.org/temp/hello%2520spaces.txt added to database" ["errorCode"]=> string(0) "" ["statusCode"]=> int(200) ["url"]=> object(stdClass)#3 (5) { ["keyword"]=> string(2) "43" ["url"]=> string(41) "https://ozh.org/temp/hello%2520spaces.txt" ["title"]=> string(13) "404 Not Found" ["date"]=> string(19) "2024-06-19 11:00:43" ["ip"]=> string(10) "37.9.57.52" } ["title"]=> string(13) "404 Not Found" ["shorturl"]=> string(45) "https://yourls.example.com/43" }
which is not really a success, and when clicking on the entry in the admin interface, it goes to the browser as "https://ozh.org/temp/hello%2520spaces.txt" and is not found.

[dgw]

Am I reading that code snippet right? It looks like you add the same parameters twice, in a query string and as POST data. 😕

At any rate, I tested the original sample link with %20 on my (company's) own YOURLS instance and it worked fine, so like @ozh I can't reproduce the original issue with shortening through the web UI. It may be a server configuration difference.

[counterpoint]

Thanks - that helped. You're correct, although I removed part of the curl settings for POST, part was mistakenly left behind. Your example now works.

But it seems a pretty laborious process to make this work in general. Is there a simpler way? I had to code the following to get a URL that would work:

$parsedURL = parse_url($longurl); $pathparts = explode('/', $parsedURL['path']); foreach ($pathparts as $i=>$part) $pathparts[$i] = rawurlencode(rawurlencode($part)); $path = implode('/', $pathparts);
and then call YOURLS with the url parameter set to:

'url' => $parsedURL['scheme'].'://'.$parsedURL['host'].$path,

It would be nicer if YOURLS could handle the original URL! (Hate to say it, but bitly does - but has other drawbacks).

[dgw]

I'd say that's a lot of extra work, indeed. The following API call works just fine on my (company's) instance of 1.9.2:

https://our.shorten.er/yourls-api.php?signature=mysigstring&action=shorturl&url=https%3A%2F%2Fozh.org%2Ftemp%2Fhello%2520spaces.txt

It isn't a problem to encode the whole url parameter in a GET request, assuming you start with a correctly-encoded link. While it might be "nice" if YOURLS could handle an original URL containing an actual space, that's not a valid URL and (speaking for myself, not necessarily the project) it isn't worth introducing the extra fragility into the codebase to handle invalid edge cases.

[counterpoint]

Well, I think I'm reaching a final solution now, thanks. It's a bit unkind to call it an "invalid edge case". The URLs I am starting out with are simply Linux file paths (which quite legally include blanks). Bitly's API handles them happily, but I accept your comment that it isn't worth putting extra functionality into YOURLS.

There is a somewhat simpler alternative to my earlier code, but the first step needed is to turn the Linux file path into a valid URL path. That can be achieved in PHP with:

function standardURL ($longurl) { $parsedURL = parse_url($longurl); $path = str_replace('%2F', '/', rawurlencode($parsedURL['path'])); return $parsedURL['scheme'].'://'.$parsedURL['host'].$path; }

That amount of complexity seems unavoidable. Once that has been achieved, then it is possible to rawurlencode the entire URL to pass to YOURLS. What isn't workable is to do what was implied in an earlier comment somewhere to "double" urlencode.

(PHP's parse_url is capable of returning more than just scheme, host and path, but the other items do not arise when the parameter is a Linux file path)

A further simplification can be achieved by using PHP's http_build_query function, which will implicitly urlencode the URL:

$queryparms = [ 'signature' => '963100bfb5', 'action' => 'shorturl', 'url' => standardURL($longurl), 'format' => 'json' ]; curl_setopt($bitlycurl, CURLOPT_URL, YOURLS_URL.'?'.http_build_query($queryparms));

[dgw]

It's a bit unkind to call it an "invalid edge case". The URLs I am starting out with are simply Linux file paths (which quite legally include blanks).

I don't think it's that unkind. It might be legal in a Linux filesystem, but an unencoded space character appearing in a URL is invalid. It must be encoded, as otherwise whitespace is treated as a delimiter from context. That is, normally, if https://ozh.org/temp/hello spaces.txt appears in text, https://ozh.org/temp/hello is all you will get by looking for URLs in the text and spaces.txt will be ignored.*

* — Well, it will be ignored unless one uses a bare-URL detector that doesn't require a scheme, and someone decides that we should have a .txt TLD to go with .zip and .mov.

It sounds like ultimately, the "pain point" in your use case is that you're not starting with well-formed URLs. If you are building your own URLs from the filesystem, yes, you will need to make sure that the components of the resulting URL are correctly encoded. And the good news is that you're arriving at a solution to do just that.

Typically I would suggest that it could be easier to implement a YOURLS plugin that deals with the encoding server-side in case a space slips through, as you say Bitly does, but I don't see any place to hook in a plugin before that part of the sanitization routine. Many parts of YOURLS allow plugins to override or skip over them, but the URL sanitization code is a bit lax in that area. Such hooks could be a worthwhile feature request to consider, if you wanted to open it as a new issue.

[counterpoint]

Well, yes, although to be pernickety two things work aagainst links being a problem in text. One is that the whole point of the exercise is to only use the shortened URLs in text. The other is that if you copy and paste a link that is a simple file path from a browser, the browser obligingly encodes it. We never had a problem in that respect - it was more that the links were very long and looked horrid because of the URL encoding.

I'm not bothered about a plugin now that I have enough information to get a solution. There has to be code to create the web site from the Linux file system. Adding calls to functions that shorten or standardise links isn't a concern. And I don't really want any more work than necessary! Bitly just accepts spaces in its API, but then again Bitly has other drawbacks (notably the inability to edit links on the free plan), so the comparison isn't really important.

There could be a case for adding something to the documentation to help people like me!

[dgw]

There could be a case for adding something to the documentation to help people like me!

Where would you, as a user, expect to find a hint at this need for encoding? On the API page?
https://yourls.org/docs/guide/advanced/passwordless-api

It's also important to have correctly-encoded spaces when shortening through the admin interface, though. Maybe it's more of a Troubleshooting topic, and belongs on the Frequent Issues page? https://yourls.org/docs/guide/troubleshooting/frequent-issues

We're happy to take documentation contributions over at https://github.com/YOURLS/website if you already have a sense of how it could be presented in YOURLS' docs. You or I can open an issue at the very least, if you aren't ready to contribute a full-on patch.

[counterpoint]

I should add that I'm grateful to you and your colleagues who work on YOURLS. I'm a strong supporter of open source (and create some). The final result is now looking good, thanks.