Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable mapping replaces non-ascii characters with '?' in string values #8579

Closed
lenaschoenburg opened this issue Jan 12, 2022 · 1 comment · Fixed by #8580
Closed

Variable mapping replaces non-ascii characters with '?' in string values #8579

lenaschoenburg opened this issue Jan 12, 2022 · 1 comment · Fixed by #8580
Labels
kind/bug Categorizes an issue or PR as a bug version:1.3.1 Marks an issue as being completely or in parts released in 1.3.1

Comments

@lenaschoenburg
Copy link
Member

lenaschoenburg commented Jan 12, 2022

Describe the bug

When using a simple variable mapping, non-ascii characters (eg. umlaute such as "ü") are replaced with '?'.

To Reproduce

  1. Deploy this model: non-ascii-mapping.bpmn
  2. Start a process instance with a source variable: {"source": "äüö"}
  3. After the mapping the variables will be {"source": "äüö", "target": "???"}

This is not reproducible with 1.2.9.

Expected behavior
Non-ascii characters are supported.

@lenaschoenburg lenaschoenburg added kind/bug Categorizes an issue or PR as a bug 1.3.0 labels Jan 12, 2022
@Zelldon
Copy link
Member

Zelldon commented Jan 12, 2022

Based on the information here https://stackoverflow.com/a/23932541/2165134 we were able to track the bug down to encoding issues on the local env.

With 1.2.9 we have set the encoding:

root@88ceae3dd2cb:/usr/local/zeebe# locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=

With 1.3.0 not (reproduced also by @oleschoenburg )
1.3.0:

root@65cdf6b25ed0:/usr/local/zeebe# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

We have switched to a new docker image with 1.3.0 (eclipse java image based on ubuntu)

ghost pushed a commit that referenced this issue Jan 13, 2022
8577: deps(maven): bump protobuf-bom from 3.19.2 to 3.19.3 r=Zelldon a=dependabot[bot]

Bumps [protobuf-bom](https://github.com/protocolbuffers/protobuf) from 3.19.2 to 3.19.3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/protocolbuffers/protobuf/releases">protobuf-bom's releases</a>.</em></p>
<blockquote>
<h2>Protocol Buffers v3.19.3</h2>
<h1>Python</h1>
<ul>
<li>Fix missing Windows wheel for Python 3.10 on PyPI</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/cc7b1b53234cd7a8f50d90ac3933b240dcf4cd97"><code>cc7b1b5</code></a> Update protobuf version</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/ee432e55b808a57e44115178e41cd5ceae91b205"><code>ee432e5</code></a> Update CHANGES.txt for 3.19.3</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/203c593b9681d98e7e05f3d8f862aa183f5d9c91"><code>203c593</code></a> Fix build error with Python 3.10 build on Windows</li>
<li><a href="https://github.com/protocolbuffers/protobuf/commit/922a7ec18926aa2612d14a2b4abb8689477e97a4"><code>922a7ec</code></a> Fix Python 3.10 C++ tests (<a href="https://github-redirect.dependabot.com/protocolbuffers/protobuf/issues/9128">#9128</a>)</li>
<li>See full diff in <a href="https://github.com/protocolbuffers/protobuf/compare/v3.19.2...v3.19.3">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.google.protobuf:protobuf-bom&package-manager=maven&previous-version=3.19.2&new-version=3.19.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

8580: fix: set default encoding to UTF-8 r=oleschoenburg a=oleschoenburg

## Description

* Sets the default encoding to UTF-8
* Switches the prod base image to `eclipse-temurin:17-jre-focal` which sets a system locale


## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #8579



Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ole Schönburg <[email protected]>
ghost pushed a commit that referenced this issue Jan 13, 2022
8580: fix: set default encoding to UTF-8 r=oleschoenburg a=oleschoenburg

## Description

* Sets the default encoding to UTF-8
* Switches the prod base image to `eclipse-temurin:17-jre-focal` which sets a system locale


## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #8579



Co-authored-by: Ole Schönburg <[email protected]>
@ghost ghost closed this as completed in 950c0f9 Jan 13, 2022
ghost pushed a commit that referenced this issue Jan 13, 2022
8585: [Backport stable/1.3] fix: set default encoding to UTF-8 r=oleschoenburg a=github-actions[bot]

# Description
Backport of #8580 to `stable/1.3`.

relates to #8579

Co-authored-by: Ole Schönburg <[email protected]>
@npepinpe npepinpe added the version:1.3.1 Marks an issue as being completely or in parts released in 1.3.1 label Jan 17, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes an issue or PR as a bug version:1.3.1 Marks an issue as being completely or in parts released in 1.3.1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants