invalid characters are added after replace token task

[ramkark]

section name="dataConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Data.Configuratiÿÿÿÿÿÿon.DatabaseSettings, Microsoft.Practices.EnterpriseLibrary.Data" requirePermission="true"

ÿÿÿÿÿÿ characters are added in the above string when i try to use the Replace token task in VSTS.

in my Web.config, the below lines are getting affected after the task.

"

"

[qetza]

Hi @ramkark,

Could you share the detailed logs of the task? (use the new verbosity parameter)

[ramkark]

2019-03-14T20:52:52.1727722Z ##[section]Starting: Replace tokens in Web.ENV.config for WebSite
2019-03-14T20:52:52.1727722Z ==============================================================================
2019-03-14T20:52:52.1727722Z Task : Replace Tokens
2019-03-14T20:52:52.1727722Z Description : Replace tokens in files
2019-03-14T20:52:52.1727722Z Version : 3.1.0
2019-03-14T20:52:52.1727722Z Author : Guillaume Rouchon
2019-03-14T20:52:52.1727722Z Help : v3.1.0 - More Information
2019-03-14T20:52:52.1727722Z ==============================================================================
2019-03-14T20:52:52.3759002Z pattern: ((?:(?!).)*)__
2019-03-14T20:52:52.3915237Z replacing tokens in: XXXXXXXXXXXXXXXXXXXXXX\Web.config
2019-03-14T20:52:52.3915237Z using encoding: utf-8
2019-03-14T20:52:52.4071488Z WEBSERVICESERVERPATH: XXXXXXXXXXXXXXXXXXXXXXXXXXX
2019-03-14T20:52:52.4071488Z ##[section]Finishing: Replace tokens in Web.ENV.config for WebSite

[qetza]

Hi @ramkark,
Would it be possible to send your original file (without sensitive data) so that i can do some testing?

[mikeruhl]

Hey, we're getting the same issue. Here's a snippet of a bundled js file with react-bootstrap getting the pagination.js characters replaced:
Before:

m.a.createElement("span",{"aria-label":"First"},!0===c?"«":c)),f&&m.a.createElement(x.a,a()({},O,{eventKey:e-1,disabled:1===e}),m.a.createElement("span",{"aria-label":"Previous"},!0===f?"‹":f)),this.renderPageButtons(e,r,i,u,s,O),p&&m.a.createElement(x.a,a()({},O,{eventKey:e+1,disabled:e>=r}),m.a.createElement("span",{"aria-label":"Next"},!0===p?"›":p)),l&&m.a.createElement(x.a,a()({},O,{eventKey:r,disabled:e>=r}),m.a.createElement("span",{"aria-label":"Last"},!0===l?"»":l)))},e}(m.a.Component);

After:

m.a.createElement("span",{"aria-label":"First"},!0===c?"��":c)),f&&m.a.createElement(x.a,a()({},O,{eventKey:e-1,disabled:1===e}),m.a.createElement("span",{"aria-label":"Previous"},!0===f?"���":f)),this.renderPageButtons(e,r,i,u,s,O),p&&m.a.createElement(x.a,a()({},O,{eventKey:e+1,disabled:e>=r}),m.a.createElement("span",{"aria-label":"Next"},!0===p?"���":p)),l&&m.a.createElement(x.a,a()({},O,{eventKey:r,disabled:e>=r}),m.a.createElement("span",{"aria-label":"Last"},!0===l?"��":l)))},e}(m.a.Component);

So if you want to do some testing, you could use the react-bootstrap npm package, it has pagination.js which is causing us the most relevant issues.

Here's our verbose logs:

2019-03-20T15:32:39.2230344Z pattern: #\{((?:(?!\}#).)*)\}#
2019-03-20T15:32:45.5068304Z replacing tokens in: E:\[xxx]\PackageTmp\Scripts\_react-bundles\vendor.bundle.js
2019-03-20T15:32:45.5068629Z BOM no found: default to ascii.
2019-03-20T15:32:45.5068945Z using encoding: ascii

So actually, it looks like it defaults to ascii which is exactly why we're having issues. my vote would be for Auto to default to utf-8. You could also look into implementing something like python's chardet to determine the pageset being used. I used it in a project and it works great.

And after changing my encoding from auto to utf-8 it looks like it's back to normal. Glad I started typing this so I could solve my own problem! Perhaps @ramkark 's is in the wrong encoding? CP1252 maybe? That was my original issue.

[qetza]

Hi @mikeruhl,
Thanks for the feedback, i'll look at the jschardet and see if i can integrate it to change my simple charset detector (based on BOM).
To avoid any regression i cannot change the default auto value. Maybe when i release a new major version and if the jschardet doesn't work i'll introduce this breaking change.

[qetza]

@mikeruhl do you have an account i could share a dev version of the task to test the new encoding detection? (this task can be installed side by side with the public one)

[ramkark]

Thank you @mikeruhl and @qetza.

My Files encoding is 'auto', i dont know theexact encoding used.

In my case, I deleted and retyped (typing instead of copy/paste from source) the line were i get invalid character in web.config file and the issue is not occurring anymore.

[mikeruhl]

@mikeruhl do you have an account i could share a dev version of the task to test the new encoding detection? (this task can be installed side by side with the public one)

@qetza can you email me at [email protected] and I will share my work email with you? I would like to associate it with that since I use that far more but do not want to publicly share it, I get enough spam ;)

Thanks for the quick turnaround.

[qetza]

The new version 3.2.0 is now published and uses jschardet for detecting encoding which should fix your issues.
Feel free to reopen the issue if you still have issues.