Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

utf8::utf16to8 and utf8::unchecked::utf16to8 produce different output #111

Closed
GPUex opened this issue Oct 27, 2023 · 2 comments
Closed

utf8::utf16to8 and utf8::unchecked::utf16to8 produce different output #111

GPUex opened this issue Oct 27, 2023 · 2 comments

Comments

@GPUex
Copy link

GPUex commented Oct 27, 2023

As title mentions, checked and unchecked variations of utf16to8 produce different output.
Simple test case:

#include <iostream>
#include <format>
#include "utf8.h"

std::string u16_to_u8_unchecked(const std::u16string& u16)
{
	std::string str;
	utf8::unchecked::utf16to8(u16.begin(), u16.end(), std::back_inserter(str));
	return str;
}

int main()
{
	std::u16string u16 = u"Simple test!";

	std::string u8_checked = utf8::utf16to8(u16);
	std::string u8_unchecked = u16_to_u8_unchecked(u16);

	std::cout << std::format("{}\n{}\n", u8_checked, u8_unchecked);
	std::cout << std::format("{} {} {}\n", u8_checked == u8_unchecked, u8_checked.size(), u8_unchecked.size());

	return 0;
}

Produces this output:

Simple test!
Simple test
false 12 11

Library version 4.0
Platform:
Windows 11 (10.0.22621.2428)
Visual Studio 2022 (17.7.6) (compiler version 19.37.32825.0)

@nemtrif
Copy link
Owner

nemtrif commented Oct 28, 2023

Ugh, yes. That was introduced by my fix of issue #78 . Looking at it now...

nemtrif added a commit that referenced this issue Oct 28, 2023
Fixing regression caused by the fix for #78, which leads to utf8::unchecked::utf16to8() chopping off the last character in many cases.
@nemtrif
Copy link
Owner

nemtrif commented Oct 28, 2023

Fixed at 4.0.1. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants