Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use non-latin symbols with pq.CopyIn for jsonb type #1023

Open
wuzzapcom opened this issue Jan 12, 2021 · 3 comments
Open

Can't use non-latin symbols with pq.CopyIn for jsonb type #1023

wuzzapcom opened this issue Jan 12, 2021 · 3 comments

Comments

@wuzzapcom
Copy link

Description

I'm implementing bulk insert using pq.CopyIn feature. One of column is jsonb type and it contain non-latin symbols, for example:

{ "text": {
  "title": "название"
  }
}

This json can be inserted with regular INSERT.
When I've tried to wrap the code with pq.CopyIn, I got the following error:

invalid input syntax for type json

When I insert same json with latin-only strings, all works properly.

code example

func bulk() {
	tx, err := db.Begin()

	stmt, err := tx.Prepare(pq.CopyIn(
		"example", "payload_jsonb",
	))

	stmt.Exec(Payload{Title: "название"})

        stmt.Exec()
}

what I want to see

Inserted successfully

what I get

invalid input syntax for type json

Same code with Payload{Title: "title"} inserts successfully.

Research

I've discovered that this issue happens because input in pq.CopyIn mode is encoded as text using appendEncodedText.
So cyrillic symbols are encoded like this:

{
    "text": {
        "title":"\321\202\320\260\320\271\321\202\320\273"
     }
}

Apparently PostgreSQL does not accept such json and returns the error.
This is the reason why no issue appears on same json with latin value of the field title.

On the other hand I've checked how regular INSERT handle this case. It uses encode. So json is just not encoded with encodeBytea.

Are there any workarounds for this issue?

@lragnarsson
Copy link

Ran into the same problem, did you find any workarounds? @wuzzapcom

@wuzzapcom
Copy link
Author

Unfortunately, I didn’t

@drrossum
Copy link

I seem to run into the same problem. I bisected this to commit 4a7d9870ef6a5a247ef532192988b01bf13cc110

When I revert this commit from the latest release this works again.

Does this solve your problem, too?

drrossum referenced this issue Jan 22, 2022
fixed server version detection for builds like "11.2 (Debian 11.2-1.pgdg90+1)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants