Mac Ship searching and loading behaving slowly

[JoshuaNotowitz]

ship search and category select is behaving very slowly,
i made a video example of searching for a fit on a mac, http://youtu.be/hFYIsKgdGRs
if you need information about my computer for reference i can provide it as needed.

[blitzmann]

My god, I knew there were Mac performance issues, but I had no idea they were this bad. Did this just recently start? The problem with Mac is that neither Kadesh nor I have a mac to test with, so we are shooting blind. I have an old OS X on virtualbox, however it doesn't run nearly as slow as this.

May I ask how quick market search is?

Also, are you computer savvy? Would you be able to run pyfa through a python interpreter and test it out? Soing so would show us if it's a problem with the packaged file or something in the application.

Also, can you please send me your fit database file (it's in ~/.pyfa, called saveddata.db). It may be that you have so many fits, it's having difficulties with the queries.

[blitzmann]

@danmane any luck pinpointing mac performance issues?

[JoshuaNotowitz]

The market search works perfectly fine
[snipped saveddata.db file download for privacy]

[blitzmann]

Thanks. You have 557 fits in your database. I'll try to test it on my mac VM and see what happens. Odd that market search works.

Have you always had this issue? And would you be able to install and run a python interpreter (I think mac comes with python 2.7, you would just need to download and install the other dependencies to try to run from source)?

[JoshuaNotowitz]

I've had the problem for a while and i ran into problems while installing dependencies in Yosemite using brew
i had trouble installing virtualenv while trying to install sqlalchemy

[blitzmann]

I've been able to somewhat duplicate long ship menu loading times on my mac VM, although it's nowhere near as bad as your video (I'm using Mountain Lion IIRC). So at least that's a start.

I wonder if, in the process of querying fits, if they are all calculated. That would probably cause a ton of lag, but I dunno why it would affect OS X so badly.

If you can get an interpreter running, that would be great. otherwise I'll try to work off what we have now. I got it running through my interpreter and it's showing the same amount of lag from what I can tell, so it's hopefully not due to the skeleton package we use.

Thanks!

[JoshuaNotowitz]

I'm not savvy enough to get an interpreter running, I did notice faster load times when search phrases after 3 characters returns only a few fits.

[blitzmann]

Yeah, it seems to work faster with loading less fits. I'm not sure if this is a sql issue or an implementation issue - I'll do some testing in the next week to try to determine what the bottleneck is. On windows and linux it works flawlessly =/

[blitzmann]

Also, for those interested in helping, there's a class in utils/timer.py that will time a block of code. Just import it using

from utils.timer import Timer

and use it in this way:

with Timer(verbose=True) as t:
    # code to test

It will print out the time it took to run that code when the block is finished.

@JoshuaNotowitz, do you mind if I remove your character information and distribute your database file? it provides a nice test bench as it has a lots of fits (500+). Just don't want any supersecret fits to be disclosed due to this.

Quick test in windows testing "shi" search: 150ms to return bare query (in eos.db.saveddata.queries.searchFits). Subsequent searches take only a few ms presumably since results are cached. Will try OS X next.

here's the block:

def searchFits(nameLike, where=None, eager=None):
    from utils.timer import Timer
    if not isinstance(nameLike, basestring):
        raise TypeError("Need string as argument")
    # Prepare our string for request
    nameLike = u"%{0}%".format(sqlizeString(nameLike))

    #Add any extra components to the search to our where clause
    filter = processWhere(Fit.name.like(nameLike, escape="\\"), where)
    eager = processEager(eager)

    with Timer(verbose=True) as t:
        with sd_lock:
            fits = saveddata_session.query(Fit).options(*eager).filter(filter).all()

    return fits

EDIT:

OS X returns 226ms for "shi", and then 3ms for a repeat. So it's not the database query from what I can tell. Will continue branching out.

[blitzmann]

Was able to narrow it down to around line 919 of shipBrowser, adding the fit GUI widget:

for ID, name, shipID, shipName, booster, timestamp in fitList:
    self.lpane.AddWidget(FitItem(self.lpane, ID, (shipName, name, booster, timestamp), shipID))

That block takes 3.5 seconds on my os x, whereas on Windows it takes 29ms.

We're getting into GUI drawing territory now which is not something I'm too familiar with.

[blitzmann]

Got a gui profiler working:

http://gyazo.com/588299bb9615157b87802dcd41beee42 for windows

http://gyazo.com/c79def99df1d42442eb38ee4cf7149b9 for mac

Looks like FontFromPixelSize does not work very efficiently in the mac, and causes a lot of hold up not only in the ship browser but where ever it is used. This is a good place to start. Not sure how we might fix this other than maybe declaring font in a central font's file (something we already kinda do in the wx3 branch). That may work well as we only specify font in few select cases

[blitzmann]

Commited a patch that hopefully clears up the issue. Instead of using wx.FontFromPixelSize, I switched over to just wx.Font which doesn't display the slowness that the other one exhibits. I assume wx.FontFromPixelSize was originally chosen because it's easy to have the same font size across platforms with it, so moving to wx.Font we have to determine which point size to use based on platform (something that we would have to do eventually anyway with wx3).

I have a test OS X package if people can test it. it works great for me. http://dl.eve-files.com/media/1412/pyfa-20141212-88074c-mac.zip

Please pay attention to fonts and see if there's any difference to what you're used to, and let me know if it works for you.

[JoshuaNotowitz]

If it would still help with testing you can distribute the file

[blitzmann]

@JoshuaNotowitz have you tried the new build? If so, how is it running for you?

[JoshuaNotowitz]

the new build is great, the searching is very fast. the ship loading is still the same speed but it wasn't nearly as slow as the search so its not that bad.

[blitzmann]

@JoshuaNotowitz Yeah, I'm aware that there are still issues with loading a fit. I don't think is an inherit problem with the Mac tho - some of my fits can take ~1-2 seconds to load depending on number of modules, complexity of effects, booster, projections, etc. I think it's just simply a limitation of the current pyfa engine.

If you have a PC with windows / linux on it, try to compare the speed of loading the same exact fit between the two and note any difference in speed. My Mac VM does seem a bit slower than my desktop, but everything in a VM seems slower, so I cannot really tell.

Anyway, gonna close this issue as the ship menu is fixed. Feel free to continue discussion here if you find any other Mac lag. =)