'decimal' codec can't encode character u'\uff00'

[jasconius]

Bug Report

App crashes when dragging certain items into fit

Expected behavior:

Item goes into fit

Actual behavior:

Crash with error report

Detailed steps to reproduce:

Was using a Tormenter. Tried to add ancillary armor rep to empty low slot, crash

Fits involved in EFT format (Edit > To Clipboard > EFT):

Release or development git branch? Please note the release version or commit hash:

Release (latest as of May 10)

Operating system and version (eg: Windows 10, OS X 10.9, OS X 10.11, Ubuntu 16.10):

Mac OS 10.12.4

Other relevant information:

EXCEPTION: 'decimal' codec can't encode character u'\uff00' in position 4: invalid decimal Unicode string

File "/Applications/pyfa.app/Contents/Resources/lib/python2.7/site-packages.zip/gui/builtinViews/fittingView.py", line 114, in OnData
File "/Applications/pyfa.app/Contents/Resources/lib/python2.7/site-packages.zip/gui/builtinViews/fittingView.py", line 204, in handleListDrag

[blitzmann]

@jasconius does this happen with other modules? Did you drag it from the market list, or from something like cargo? Unfortunately I can't seem to reproduce :(

Can you send me your /.pyfa directory zipped to [email protected]? It contains logs, database, and user settings that might prove useful in determining what's happened.

[jasconius]

Yeah it happened with some pulse lasers too. It's only partially reproducible.

The modules were dragged from the market tab. Into both empty slots and slots with stuff already in it (to replace)

Only way to recover was to reboot, and after a reboot, sometimes you could repeat the exact same action and it would not error, but sometimes it would. Some sort of race condition...?

I'll see if I can find the directory, it's worth noting that this happened on fits which were created on the new build, not edits of pre-existing fits

[blitzmann]

It's just a very weird error. It doesn't make any sense to me why it would happen sometimes and not other. is it always the same error?

Are you using the mac-deprecated or mac build? I tried both, couldn't reproduce, but will try some more since you state it's not easy to reproduce. It could also possibly be related to other issues that may have been fixed, though I am skeptical of that since it's not a database issue. Very strange. I'll await your files - if you have trouble finding them, let me know. :)

[ghesla]

I received a very similar error this morning when dragging a script into the cargo hold of a fit.
Mac OS 10.12.5

edit - right-click on script and select "Add to Cargo" worked with no issue.

OS version: Darwin-16.6.0-x86_64-i386-64bit
Python: 2.7.10
wxPython: 3.0.2.0
SQLAlchemy: 1.0.9
Logbook: 1.0.0
pyfa version: 1.29.2 Stable - YC119.5 1.0
pyfa root: /Applications/pyfa.app/Contents/Resources
save path: /Users/{redacted}/.pyfa
fs encoding: utf-8

EXCEPTION: 'decimal' codec can't encode character u'\ucd00' in position 5: invalid decimal Unicode string

File "/Applications/pyfa.app/Contents/Resources/lib/python2.7/site-packages.zip/gui/cargoView.py", line 41, in OnData
File "/Applications/pyfa.app/Contents/Resources/lib/python2.7/site-packages.zip/gui/cargoView.py", line 82, in handleListDrag

[jasconius]

I just sent my pyfa lib zip file. Still happening, seemingly even more with the x.2 release.

I will note that the PC version is generally stable. But this mac version is not usable

[blitzmann]

Thanks for the files guys. With the extra logging I enabled in 1.29.2, I found some oddities:

This is the way pyfa does drag and drop: we send over a data object, which is a simple string. That string needs to identify where the drag is coming from (market) and the item that we're dragging (eg 12345). This is concatenated to a string that looks like "market:12345". On the drop, that is split so that we get the two identifiers. The error happens when we try to convert the number back over to an integer - there seems to be extra characters appended to the string for unknown reasons. As far as I can tell, this is only happening on OS X (and quite possibly only on OS X 10.12, as I can't reproduce on my 10.7.5 dev VM).

Question to you folks: does this error happen when you drag from cargo to fitting window as well, and vise versa?

I will poke around a bit and send those that emailed me updated OS X builds to try out, along with some additional logging. I'll also poke the wxpython user group and see if there's anyone that can give me an answer as to why this happens. This will probably take some time guys, so please bear with me! In the meantime, double click and then move the module to the desired position :)

[jasconius]

Do you know roughly what file/line we're dealing with?

I work with with Python every day on Mac I could take a look.

How much trouble am I going to run into getting the source spun up in a fresh virtual environment? Any dependencies outside of pip?

[blitzmann]

@jasconius

I know with with Python every day on Mac I could take a look.

OMFG I would love you long time if you're a python dev working on mac. I run off a shitty, slow VM and am not a mac guy, so any help in that arena is much appreciated! It'll be a lot easier to chat on Slack moving forward if you're interested in helping with this: https://pyfainvite.azurewebsites.net/

How much trouble am I going to run into getting the source spun up in a fresh virtual environment? Any dependencies outside of pip?

wxPython is the only one - http://downloads.sourceforge.net/wxpython/wxPython3.0-osx-3.0.2.0-cocoa-py2.7.dmg

I should also mention that there are known issues between running pyfa from source vs running from the builds that I do (which are also shitty) - there are times in which there is a bug with the packaged version, but not when running via source (usually to do with GUI stuff).

As for the files...

When the user drags something form market, this function is run:

Pyfa/gui/marketBrowser.py

Line 277 in cff3d0e

def startDrag(self, event):

The drop target for the fitting window is located here:

Pyfa/gui/builtinViews/fittingView.py

Line 111 in cff3d0e

def OnData(self, x, y, t):

As you can see, there's something funky going on between the time we set the PyTextDataObject and the time we get it in the drop target. Due to the funky unicode characters seemingly appended to the data, the int() fails.

I assume you already have pyCharm or another IDE to use in python development. :) Going to try to hop on my 10.12 Hackintosh later tonight hopefully and see if I can also reproduce the error myself.

[jasconius]

Yeah I use PyCharm on latest OS X, I develop python full time on Mac, but it's web, so I don't have very much py GUI experience, but I know the essentials. I just need this to work because when I get bored in the office I like playing with Pyfa on my Mac :)

This only started happening (to the best of my knowledge) on the latest release (1.29.x), so I may also look at the commit logs and see what changed.

But when you said that you were getting garbage unicode during integer conversions that kinda triggered me and is probably something I could sniff out with minimal codebase knowledge.

[blitzmann]

This only started happening (to the best of my knowledge) on the latest release (1.29.x), so I may also look at the commit logs and see what changed.

1.29.x introduced a new error dialog. It may have always silently failed, which is my current thought on why this was never brought up before. I don't think 1.29.x touched any of the drag and drop stuff

[jasconius]

I will get on slack at some point, but as a point of curiosity, is there anything about the flow of execution in your error reporter that might stop a UI action from being completed thoroughly?

I accept your point about it failing silently, but in 1.28.x, let's say that would happen silently, i would never experience a loss of functionality though. Whereas in 1.29, once I get this error, the app becomes unusable.

My thought was that there must be some state-management issue with how the error dialog works that is causing once polite errors to become show-stoppers.

But that's a different matter entirely. I'll look into this type conversion issue first.

[jasconius]

Initial inspection of the problem leads me to suspect that the source of this issue might be within the wx lib on Mac OS itself.

The python level integer and string conversions are always successful as one would expect. The problem clearly occurs somewhere between SetData() and GetData() on the wx objects.

I inspected the problematic strings in the debugger myself and they are typical garbage data that just screams "bad buffer" somewhere in the wx C source.

I did a quick glance into the wx 3.0 source to see where this data really goes, but didn't get very far before finding a deep maze of pointer hi-jinx.

I have a couple of easy-win suggestions on how to resolve this problem, but I'll give it one more honest debug session with a clear mind before going about creating a pull request.

[blitzmann]

@jasconius

I have a couple of easy-win suggestions on how to resolve this problem, but I'll give it one more honest debug session with a clear mind before going about creating a pull request.

Excellent. I was thinking that it was a wx issue as well, but having someone confirm that is a great step forward. I'm not opposed to hacks in this situation, as long as we can get it working properly. My first thought is to strip out any character that is not a decimal, and then int() it. But that would be problematic if wx is attaching a decimal to the thing. Could also do some sort of delimitation if we know that wx is always returning bad data at the end (instead of market:12345, do market:12345:, that way a split(':') will always get us the number in the 1 index, and index 2 will be junk, if any)

[blitzmann]

@jasconius provided a fix for this issue, should be out tomorrow with 1.29.3 :) Thanks @jasconius!