Windows: Invalid UTF-8 stream when using Pandoc as a filter

[georgejean]

Hi,

I'm using Pandoc 1.17.2 on Windows 10.

When using pandoc as a filter (giving a string with accent as input)

pandoc
# Gérard
^Z

it displays

pandoc.exe: Cannot decode byte '\x82': Data.Text.Internal.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream

I get the expected output on the terminal if I put the text in a UTF-8 encoded file essai.txt and use in a Powershell console

$outputencoding = [ System.Text.Encoding]::UTF8
[Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8
get-content essai.txt | pandoc

I've made some attempts. For example, inserting

$a = [ System.Text.Encoding]::UTF8
$outputencoding = $a
[Console]::OutputEncoding = $a
[Console]::IntputEncoding = $a

before

pandoc
# Gérard

displays a blank output when we finish the first line by pressing the key enter.

[jgm]

Can someone else who uses pandoc on Windows try to confirm
this? I don't have a Windows setup handy.

[sergiocorreia]

I can reproduce it, but TBH there are workarounds:

C:\Users\Sergio>C:\Users\Sergio>echo é | pandoc
pandoc: Cannot decode byte '\x82': Data.Text.Internal.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream

C:\Users\Sergio>chcp 65001

C:\Users\Sergio>echo é | pandoc
<p>é</p>

See this thread for more details.

Now, I wouldn't be surprised if there are bugs in citeproc or other programs that eat from stdin, as early on I also had some troubles with this and panflute (workaround). It seems that you need to specify you want to use a utf8 encoding, or else some other encoding might be used, which could corrupt the input.

[georgejean]

Ok, thanks for your responses and for the tip. This is not a big problem.

Just a remark: In a powershell console, chcp 65001 is not enough.
It must be replaced by

$outputencoding=[console]::outputencoding=[text.encoding]::utf8

and allows to do such things:

"Gérard"|pandoc

<p>Gérard</p>

or

$a, $b, $c = "Manger équilibré", "pâtes", "râpé"
@"
# $a
## Les $b
Faites cuire 8 minutes les $b dans deux volumes d'eau. Servez les $b avec du fromage *$c*   
"@|pandoc

<h1 id="manger-équilibré">Manger équilibré</h1>
<h2 id="les-pâtes">Les pâtes</h2>
<p>Faites cuire 8 minutes les pâtes dans deux volumes d'eau. Servez les pâtes avec du fromage <em>râpé</em></p>

[jgm]

Nothing to change in pandoc, correct?

[georgejean]

I'm not able to answer this question.
In a nutshell:
I can't test the Step 4: Using pandoc as a filter of getting-started on Windows because it doesn't work when strings contain accents.
I've not found a solution to this problem (I don't know how if there is a correct encoding's configuration so that it works. Therefore I don't know if the problem comes from Pandoc) but there are easy workarounds.

[jgm]

It seems to me that you have the solution above: if you set the console encoding to UTF-8 using

$outputencoding=[console]::outputencoding=[text.encoding]::utf8

then it works, correct? Pandoc documents that it expects all input, and gives all output, in UTF-8. And it seems that the issue was that your console was not set up by default this way.

But perhaps I've misunderstood, because you say "I've not found a solution."

[georgejean]

In a powershell console,

$outputencoding=[console]::outputencoding=[text.encoding]::utf8
"é"|pandoc

works fine but

$outputencoding=[console]::outputencoding=[text.encoding]::utf8
pandoc
é

outputs (after ctrl+z and Enter)

pandoc.exe: Cannot decode byte '\x82': Data.Text.Internal.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream

In the standard command prompt

chcp 65001
echo é | pandoc

works, but

chcp 65001
pandoc
é

doesn't work.
There must be a way to properly configure the encoding, but I've not found yet.

[sergiocorreia]

I see your point:

>chcp
Active code page: 437

>chcp 65001
Active code page: 65001

C:\Users\Sergio>pandoc
a
é
<p>a</p>

In this case, pandoc exits silently after reading a non-ascii character

PS: These one liners sets cmd and PowerShell to use UTF8 by default, in case it's useful

[jgm]

I understand the issue now, but I can't see anything in
pandoc that is wrong. I will have to do some experiments
in Windows to see better what is going on.

[georgejean]

Thanks for your replies.
Other experiments:

1

[console]::inputencoding=[text.encoding]::ascii
$outputencoding=[console]::outputencoding=[text.encoding]::utf8
pandoc
a

é

output:

<p>a</p>
<p>e</p>

2 (doesn't work)

$outputencoding=[console]::outputencoding=[console]::inputencoding=[text.encoding]::utf8
pandoc
a
é

output: (without Ctrl-Z)

<p>a</p>

[mb21]

Closing this, since it looks like an issue with the windows console, not pandoc.