[Bug] Extracting substring to token value doesn't reference tag properly

[ronkot]

I'd need to chain certain requests. In the middle of the chain I'd need to parse a substring from a response JSON field and use that value in the next step. I couldn't find out way to do this. Is it possible somehow?

Example:

Response body

{
  "url": "https://redirect.to?code=abc123"
}

And I'd like to extract the code query param value from the $.url field.

[welcome]

👋 Thanks for opening your first issue! If you're reporting a 🐞 bug, please make sure
you include steps to reproduce it. If you're requesting a feature 🎁, please provide real
use cases that would benefit. 👪

To help make this a smooth process, please be sure you have first read the
contributing guidelines.

[gschier]

You'll have to store the response value in a temporary environment variable first.

For example:

{
  "url": "<RESPONSE TAG>",
  "code": "{{ url | replace('https://redirect.to?code=', '') }}"
}

Is this for OAuth 2.0? If so, have you tried using Insomnia's built-in OAuth 2.0 support?

[ronkot]

Thanks @gschier, nice idea 👍 I tried this approach but unfortunately it doesn't work fully. When I create the temporary code environment variable in Manage Environments, the extracted code value seems ok. But when I use the code environment value in a request, the value that gets injected is the original url (not the extracted code). I try to play around more to see if it's just PEBCAK.

Is this for OAuth 2.0? If so, have you tried using Insomnia's built-in OAuth 2.0 support?

Yes it is. I tried also the built-in OAuth 2.0 flow, but unfortunately it seems that our OAuth flow does not fully follow the standard (shame). Hopefully we'll get that fixed in the future!

[ronkot]

I tried to debug my configuration but couldn't make it work. Tested that when I change the url environment variable to be a string constant everything works as expected. But when I use the response tag method you suggested the code environment value that is used in request is plain url value. Looks like the the replace filters are not run for response tags.

Could this be a problem in insomnia?

[gschier]

Okay, I just reproduce this. Posting a reproducible export here for reference:

[develohpanda]

I spent some time investigating this one, and here are my findings. There isn't an issue with how a variable is resolved for a request, there is a more fundamental issue with resolving variables, because I managed to reproduce this bug within the environment variable editor.

If the piped variable is coming from a block using insomnia-plugin-response, we see this behavior

{
  "url": "{% response 'body', ... %}", // returns https://redirect.to?code=123
  "code": "{{ url | replace('https://redirect.to?code=', '') }}" // returns 123
  "use-code": "{{ code }}" // returns https://redirect.to?code=123
}

And if the piped variable is coming from a regular string, we see this behavior

{
  "url": "https://redirect.to?code=123", // returns https://redirect.to?code=123
  "code": "{{ url | replace('https://redirect.to?code=', '') }}" // returns 123
  "use-code": "{{ code }}" // returns 123
}

It seems there is an issue with nunjucks, when nesting variable <- variable <- block, while variable <- variable <- variable works.

I can see some interesting behavior inside the templating.render, when we call the nunjucks renderString method, but I'm going in circles at this point and need to bounce some ideas I think.

insomnia/packages/insomnia-app/app/templating/index.js

Lines 42 to 46 in e485d55

	const nj = await getNunjucks(renderMode);
	nj.renderString(text, context, (err, result) => {
	if (err) {
	const sanitizedMsg = err.message

[gschier]

Thanks for taking a look @develohpanda! Seeing your example environments made me remember that this is in the codebase:

insomnia/packages/insomnia-app/app/common/render.js

Lines 431 to 447 in f920ba1

	/**
	* Sort the keys that may have Nunjucks last, so that other keys get
	* defined first. Very important if env variables defined in same obj
	* (eg. {"foo": "{{ bar }}", "bar": "Hello World!"})
	*
	* @param v
	* @returns {number}
	*/
	function _nunjucksSortValue(v) {
	if (v && v.match && v.match(/({%)/)) {
	return 3;
	} else if (v && v.match && v.match(/({{)/)) {
	return 2;
	} else {
	return 1;
	}
	}

Basically, it ensures that variables are rendered before template tags because I figured the most common use case would be to pass a variable into a tag instead of the other way around. In hindsight, my assumption might have been backwards. Either way, sorting either breaks one use case or the other.

But I just had a thought. Now that we have merged your PR to preserve sort order of environment properties, we probably don't need to sort anymore. We should be able to rely on the user to define variables in preferred render order (top-to-bottom).

[develohpanda]

Thanks @gschier, that was the pointer I needed!

Either way, sorting either breaks one use case or the other.

I'm concerned about changing the sorting because this is likely to introduce breaking changes, but it is also a fairly specific use case.

I've written a breaking test for this and updated the logic to validate it fixes the behavior, which it does, to an extent. Now, both tag ({%) and variable ({{) nunjucks are given the same priority, which means if they are run in the user-defined order, they work properly.

However... the PR around preserving sort-order only kicks in while rendering the keys, and doesn't impact the order in which keys are processed through the rest of the code-base. That object loaded from the db stays as-is.

While debugging, I set the keys in a particular order, but they are being processed in alphabetical order:

But if I rename the keys to ensure they are processed in the correct order, things work.

The direction I will go to fixing this up will be to re-order the environment.data object, but where this re-order happens is TBC. I'm thinking that we re-order immediately after loading from the database, but this could introduce regressions and I'm leaning towards ordering only when the render context is being created. I will be publishing a new version of json-order to achieve this.

Let me know if you have any thoughts, specifically around introducing a potentially breaking change! 😄