Skip to content
This repository has been archived by the owner on Feb 9, 2019. It is now read-only.

String Redis cache value is limited to length < 2^16 #182

Open
ildar15 opened this issue Jan 11, 2019 · 2 comments
Open

String Redis cache value is limited to length < 2^16 #182

ildar15 opened this issue Jan 11, 2019 · 2 comments

Comments

@ildar15
Copy link

ildar15 commented Jan 11, 2019

String Redis cache is using DataInputStream / DataOutputStream readUTF / write UTF for string serialization. This is stepping on Java's inherent limitation of these functions, namely, they use 2 byte header and cannot process strings longer than 2^16 bytes
see for example
https://www.drillio.com/en/2009/java-encoded-string-too-long-64kb-limit/
I created a test case which exercises that limitation: simply using string of 70K bytes will fail string test.

@ildar15
Copy link
Author

ildar15 commented Jan 11, 2019

This issue can be fixed in two way:

  1. the easy way to allow large strings into the cache is simply use String as an Object. The drawback of this approach is that other tools besides Java (Scala) will not be able to access redis value directly.
  2. another approach will be just to avoid using writeUTF and convert String to byte array directly.
    I tested number

Number 1 certainly works and I can submit patch for that solution. I will be testing second approach, which is more favorable (flexible) in my opinion. One can certainly try to support both provided we have correct default.

@ildar15 ildar15 changed the title String cache value is limited to length < 2^16 String Redis cache value is limited to length < 2^16 Jan 11, 2019
@ildar15
Copy link
Author

ildar15 commented Jan 27, 2019

I forked the project here

https://github.com/ildar15/play-plugins

and implemented the solution to this problem. The writeUTF was eliminated, and the value is stored as text, without length header. The result is transparent to the client, but will invalidate existing string caches. You will need to skip the first two bytes of the returned result if you need to read old cache values.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant