Character code errors in multi-byte character environments

[n-etupirka]

In Japan, multi-byte characters are used.
If the output of a command executed by the shell command contains multi-byte characters, an error will result.

Prerequisite

• I have read the README
• I have search the opened & closed issues
• I have search the WIKI and its FAQ page

Environment Data

• Merlin Server Version: 2.0.0
• Merlin Server Build: e80c47a
• Merlin Client Version: 1.1.0
• Merlin Client Build: 5f64e840194b220b4c49ad0560dd7128f51dc0fa
• Merlin Agent Version: 2.0.0
• Merlin Agent Build: 8a421ca159aab4667e7f332c9ad157820faeeed7
• Server Operating System: Ubuntu 22.04.3
• Victim Operating System: Windows 11

Expected Behavior

The output of the command is displayed on the server.

Actual Behavior

An error is displayed.

[!] 2023-11-30T07:31:11Z there was an error receiving a message from the server stream: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8

Steps to Reproduce Behavior

1. PS C:\> .\merlinAgent-Windows-x64-Debug.exe -proto http -psk ***** -sleep 5s -url http://*.*.*.*:80 -v
2. Merlin>> interact <AGENT GUID>
3. Merlin>> shell whoami //success (only in English)
4. Merlin>> shell whoami /priv //error (Japanese is included)

Misc Information

• Server Language: English
• Victim Language: Japanese

[Ne0nd0g]

@n-etupirka thanks for reporting this. I'll take a look and see what I need to do to resolve the problem.

[Ne0nd0g]

@n-etupirka Can you share what the output of whoami /priv looks like on the endpoint? I changed my Windows 11 VM's primary language to Japanese and see this:

[+] 2023-12-05T13:32:47Z Created C:\WINDOWS\system32\cmd.exe process with an ID of 2380

PRIVILEGES INFORMATION
----------------------

Privilege Name                Description                          State   
============================= ==================================== ========
SeShutdownPrivilege           Shut down the system                 Disabled
SeChangeNotifyPrivilege       Bypass traverse checking             Enabled 
SeUndockPrivilege             Remove computer from docking station Disabled
SeIncreaseWorkingSetPrivilege Increase a process working set       Disabled
SeTimeZonePrivilege           Change the time zone                 Disabled

[Ne0nd0g]

It looks like I'm able to send Japanese characters back and forth without problem

Merlin[agent][fd05e3d3-bbb3-4697-b766-66befed0825b]» ls C:\\Users\\Rick\\Downloads
Merlin[agent][fd05e3d3-bbb3-4697-b766-66befed0825b]»  
[-] 2023-12-05T13:38:42Z Created job fcDFUpydUL for agent fd05e3d3-bbb3-4697-b766-66befed0825b at 2023-12-05T13:38:42Z

[-] 2023-12-05T13:38:58Z Results of job fcDFUpydUL for agent fd05e3d3-bbb3-4697-b766-66befed0825b at 2023-12-05T13:38:58Z

[+] 2023-12-05T13:38:58Z Directory listing for: C:\Users\Rick\Downloads

-rw-rw-rw-      2023-12-01 09:23:24     282     desktop.ini
-rw-rw-rw-      2023-12-05 08:35:45     6       テスト.txt

[n-etupirka]

@Ne0nd0g The result of whoami /priv in the Japanese environment Windows 11 is output like this.

C:\Users\pentester>whoami /priv

PRIVILEGES INFORMATION
----------------------

特権名                        説明                                            状態
============================= =============================================== ====
SeShutdownPrivilege           システムのシャットダウン                        無効
SeChangeNotifyPrivilege       走査チェックのバイパス                          有効
SeUndockPrivilege             ドッキング ステーションからコンピューターを削除 無効
SeIncreaseWorkingSetPrivilege プロセス ワーキング セットの増加                無効
SeTimeZonePrivilege           タイム ゾーンの変更                             無効

This may be accomplished by setting not only the primary language to Japanese, but also the system locale to Japanese.
(Settings -> Time & language -> Language & region -> Administrative language settings -> Change system locale... -> Japanese)

[n-etupirka]

@Ne0nd0g The ls command, which is built into Merlin, did not cause the error. (The second image is Agent's debug log.)

However, when I run dir from a shell command, I get an error.

This is the correct output of the dir command at the endpoint.

C:\Users\pentester>dir C:\Users\pentester\Downloads
 ドライブ C のボリューム ラベルがありません。
 ボリューム シリアル番号は 1A28-6E18 です

 C:\Users\pentester\Downloads のディレクトリ

2023/12/06  15:27    <DIR>          .
2023/12/06  14:11    <DIR>          ..
2023/12/06  13:52                 9 テスト.txt
               1 個のファイル                   9 バイト
               2 個のディレクトリ  107,650,621,440 バイトの空き領域

[Ne0nd0g]

I just wanted to follow up and say that I'm able to duplicate the problem and I'm working on a fix.

[Ne0nd0g]

OK, I have a fix now. It's brittle, but it works. I check to make sure the output is UTF-8. If it isn't, I check the active code page. In this case, the code page is 932 for ShifJIS. I'm then able to decode the data to that encoding format. It is brittle because I've only handled decoding for UTF-8 and ShiftJIS. Anything else will just replace all invalid UTF-8 characters with this character: �. If others need their handling included for a different format, I will add it at that time.

The fix for handling ShiftJS is in the https://github.com/Ne0nd0g/merlin-agent repository. I still need to fix the fact that the gRPC stream broke and did not fix itself.

[Ne0nd0g]

@n-etupirka Are you comfortable building the Merlin Agent from source code using the dev branch to see if everything is working as expected? https://github.com/Ne0nd0g/merlin-agent/tree/dev

[n-etupirka]

@Ne0nd0g I have confirmed that everything is working as expected in the Japanese environment!

Thank you for providing a great tool and great response.

Incidentally, are there any plans to fix the bug that breaks gRPC streams in languages other than UTF-8 and ShiftJIS?

[Ne0nd0g]

Yes, I do plan to fix the bug that broke gRPC. There are two different problems. One is that the gRPC stream received non UTF-8 text. On the Agent, I now detect non UTF-8 and convert it to UTF-8. The second is that the gRPC stream didn't recover. I'll work on that next.

[Ne0nd0g]

I'm going to use this issue to fix the gRPC error too.

[Ne0nd0g]

Updated merlin-cli to try and recover broken gRPC message stream identified in this issue Ne0nd0g/merlin-cli@e4ef7b9

[n-etupirka]

I tried it in the following version, and confirmed that even in a non UTF-8 environment, no errors occurred and it was replaced with "�".

• Merlin Server Version: 2.1.0, Build: nonRelease (dev branch)
• Merlin Clinet Version: 1.1.1, Build: nonRelease
• Merlin Agent Version: 2.2.0, Build: nonRelease

Merlin[agent][5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313]»
[-] 2023-12-15T07:58:13Z Created job uawOwEVDRd for agent 5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313 at 2023-12-15T07:58:13Z

[-] 2023-12-15T07:58:23Z Results of job uawOwEVDRd for agent 5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313 at 2023-12-15T07:58:23Z

[+] 2023-12-15T07:58:23Z Created C:\Windows\system32\cmd.exe process with an ID of 10716

***The output was not valid UTF-8 and there isn't a configured decoder for code page 936***


�Ȩ�Ϣ
----------------------

�Ȩ�                        �                 ״̬
============================= ==================== ======
SeShutdownPrivilege           �ر�ϵͳ             �ѽ�
SeChangeNotifyPrivilege       �ƹ�         �
SeUndockPrivilege             �չ�ȡ�¼� �ѽ�
SeIncreaseWorkingSetPrivilege �ӽ�̹�       �ѽ�
SeTimeZonePrivilege           �ʱ�             �ѽ�

Merlin[agent][5986e2d7-a3f3-4eb8-9f3d-0a5fbb4d3313]»

[Ne0nd0g]

Awesome, thank you for trying it out. I added support for simplified and traditional Chinese as well as Korean in this commit if you want to try that out as well Ne0nd0g/merlin-agent@4c60e45