1.17.1win doesn't import XML fitting library

[VarialFlayer]

Just got a new computer, trying to copy my fitting library over (~590 fits). Importing window hangs with the animation repeating. Tried several ways of starting Pyfa (admin, cmd line) and have tried letting it run the importation for ~4h with no luck.

[blitzmann]

Please post the xml file on pastebin or somewhere so that I may test it. It might have to do with the recent importation issues that have been reported.

[VarialFlayer]

http://pastebin.com/6LjZpGEV here is the pasted XML

On Fri, Jan 1, 2016 at 12:06 AM, blitzmann [email protected] wrote:

Please post the xml file on pastebin or somewhere so that I may test it.
It might have to do with the recent importation issues that have been
reported.

—
Reply to this email directly or view it on GitHub
#489 (comment).

[thorr18]

There are 9200 lines of whitespace before the final tag ?

[thorr18]

For me, loading this XML file fails right away with the same traceback even for very old versions of pyfa.

Jan  2 16:22:56  pyfa[76318] <Notice>: 2016-01-02 16:22:56,411 service.fit              WARNING  Unicode BOM not found in file /Users/bem/Documents/untitled folder/500eveFits.xml.
Jan  2 16:22:56  pyfa[76318] <Notice>: 2016-01-02 16:22:56,412 service.fit              WARNING  Attempting to decode file /Users/bem/Documents/untitled folder/500eveFits.xml using UTF-8 page.
Jan  2 16:22:56  pyfa[76318] <Notice>: 2016-01-02 16:22:56,412 service.fit              WARNING  File /Users/bem/Documents/untitled folder/500eveFits.xml decoded using UTF-8 page.
Jan  2 16:22:56  pyfa[76318] <Notice>: Exception in thread Thread-6:
Jan  2 16:22:56  pyfa[76318] <Notice>: Traceback (most recent call last):
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "threading.pyc", line 810, in __bootstrap_inner
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "/Users/bem/factory/Py/Pyfa/build/dloadedBuilds/pyfa1171.app/Contents/Resources/lib/python2.7/site-packages.zip/service/fit.py", line 68, in run
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "/Users/bem/factory/Py/Pyfa/build/dloadedBuilds/pyfa1171.app/Contents/Resources/lib/python2.7/site-packages.zip/service/fit.py", line 892, in importFitFromFiles
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "/Users/bem/factory/Py/Pyfa/build/dloadedBuilds/pyfa1171.app/Contents/Resources/lib/python2.7/site-packages.zip/service/port.py", line 106, in importAuto
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "/Users/bem/factory/Py/Pyfa/build/dloadedBuilds/pyfa1171.app/Contents/Resources/lib/python2.7/site-packages.zip/service/port.py", line 477, in importXml
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "xml/dom/minidom.pyc", line 1928, in parseString
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "xml/dom/expatbuilder.pyc", line 940, in parseString
Jan  2 16:22:56  pyfa[76318] <Notice>:   File "xml/dom/expatbuilder.pyc", line 223, in parseString
Jan  2 16:22:56  pyfa[76318] <Notice>: ExpatError: not well-formed (invalid token): line 3, column 1
Jan  2 16:22:56  pyfa[76318] <Notice>: 

[blitzmann]

Traceback (most recent call last):
  File "C:\Users\Ryan\Copy\WinPython-32bit-2.7.10.1\python-2.7.10\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:/Users/Ryan/Copy/Git/blitzmann/Pyfa\service\fit.py", line 68, in run
    fits = sFit.importFitFromFiles(self.paths, self.callback)
  File "C:/Users/Ryan/Copy/Git/blitzmann/Pyfa\service\fit.py", line 892, in importFitFromFiles
    _, fitsImport = Port.importAuto(srcString, path, callback=callback, encoding=codec_found)
  File "C:/Users/Ryan/Copy/Git/blitzmann/Pyfa\service\port.py", line 126, in importAuto
    return "XML", cls.importXml(string, callback, encoding)
  File "C:/Users/Ryan/Copy/Git/blitzmann/Pyfa\service\port.py", line 497, in importXml
    doc = xml.dom.minidom.parseString(text.encode(encoding))
  File "C:\Users\Ryan\Copy\WinPython-32bit-2.7.10.1\python-2.7.10\lib\xml\dom\minidom.py", line 1928, in parseString
    return expatbuilder.parseString(string)
  File "C:\Users\Ryan\Copy\WinPython-32bit-2.7.10.1\python-2.7.10\lib\xml\dom\expatbuilder.py", line 940, in parseString
    return builder.parseString(string)
  File "C:\Users\Ryan\Copy\WinPython-32bit-2.7.10.1\python-2.7.10\lib\xml\dom\expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
ExpatError: not well-formed (invalid token): line 3, column 1

Can't look into it more now, but I can reproduce. =)

[blitzmann]

Looking at the XML, it seems it has a lot of - in front of tags, eg:

<?xml version="1.0"?>

-<fittings>


-<fitting name="Armageddon - Swiss Army Geddon PVE">

<description value=""/>

Not sure if this is a problem with how it was uploaded with pastebin or what. Try to upload the XML document directly to evefiles.com or something. What version of pyfa did you export this XML file with? Do you still have the fitting database that the export represents? I just exported 500+ of my own fits and am not running into this specific issue, but am running into another issue:

2016-01-02 21:53:52,997 service.fit              WARNING  Unicode BOM not found in file C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml.
2016-01-02 21:53:52,997 service.fit              WARNING  Attempting to decode file C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml using cp1252 page.
2016-01-02 21:53:52,999 service.fit              WARNING  Error unicode decoding C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml from page cp1252, trying next codec
2016-01-02 21:53:52,999 service.fit              WARNING  Attempting to decode file C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml using utf-16 page.
2016-01-02 21:53:53,003 service.fit              WARNING  Error unicode decoding C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml from page utf-16, trying next codec
2016-01-02 21:53:53,003 service.fit              WARNING  Attempting to decode file C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml using cp1252 page.
2016-01-02 21:53:53,005 service.fit              WARNING  Error unicode decoding C:\Users\Ryan\Desktop\pyfa-fits-20160103_025329.xml from page cp1252, trying next codec
Exception in thread Thread-9:
Traceback (most recent call last):
  File "C:\Users\Ryan\Copy\WinPython-32bit-2.7.10.1\python-2.7.10\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Users\Ryan\Copy\Git\blitzmann\Pyfa\service\fit.py", line 68, in run
    fits = sFit.importFitFromFiles(self.paths, self.callback)
  File "C:\Users\Ryan\Copy\Git\blitzmann\Pyfa\service\fit.py", line 892, in importFitFromFiles
    _, fitsImport = Port.importAuto(srcString, path, callback=callback, encoding=codec_found)
  File "C:\Users\Ryan\Copy\Git\blitzmann\Pyfa\service\port.py", line 126, in importAuto
    return "XML", cls.importXml(string, callback, encoding)
  File "C:\Users\Ryan\Copy\Git\blitzmann\Pyfa\service\port.py", line 497, in importXml
    doc = xml.dom.minidom.parseString(text.encode(encoding))
TypeError: encode() argument 1 must be string, not None

[blitzmann]

Also, just to get you set up on the new computer, the easiest way is to go to C:/Users/<name>/.pyfa/ and copy the saveddata.db to the same path on the new computer. This is your user database and contains any user-information.

[VarialFlayer]

The weird spacing is from the paste, when I look at the file directly it
doesn't have it. It's the newest version of Pyfa, and I still have the old
system it's installed on so I'll give the .db copy a go. evefiles isn't
letting me upload the XML there, is there any other way to send it to you?

On Sat, Jan 2, 2016 at 10:02 PM, blitzmann [email protected] wrote:

Also, just to get you set up on the new computer, the easiest way is to go
to C:/Users//.pyfa/ and copy the saveddata.db to the same path on
the new computer. This is your user database and contains any
user-information.

—
Reply to this email directly or view it on GitHub
#489 (comment).

[blitzmann]

Any kind of file sharing / cloud service. Dropbox, copy, google docs, etc.

Otherwise shoot me an email [email protected]

[VarialFlayer]

https://drive.google.com/open?id=0B6GlcFPYDA-1WDZ3aDIxYWJFTVk

On Sun, Jan 3, 2016 at 12:07 AM, blitzmann [email protected] wrote:

Any kind of file sharing / cloud service. Dropbox, copy, google docs, etc.

—
Reply to this email directly or view it on GitHub
#489 (comment).

[blitzmann]

Thanks! Running that gets me the same error that I got: encode() argument 1 must be string, not None. It seems that the charset is not correctly identified for whatever reason. I'll look into it more tomorrow and hopefully develop a fix. =)

[blitzmann]

The reason this happens seems to be because we forget to test for utf-8:

attempt_codecs = (defcodepage, "utf-16", "cp1252")

Although we attempt to find utf-8 with BOM earlier in the logic. This is fixed simply by adding utf-8 into the codecs to check for. I am not sure how this has been missed for so long, but there you have it.

I also might add an error message for things like this to prevent lock-up and give more info to the user:

This is tentative and may not make it to release as it's screwing with dialogs, which the OS X client always has issues with, so if it doesn't work on that platform it might be abandoned for now.

[thorr18]

I think this fix also took care of #456