ENH: improve performance of df.to_csv GH3054

[ghost]

Needs more testing before merging.

Following SO question mentioned in #3054:

In [7]: def df_to_csv(df,fname):
   ...:     fh=open(fname,'w')
   ...:     fh.write(','.join(df.columns) + '\n')
   ...:     for row in df.itertuples(index=False):
   ...:         slist = [str(x) for x in row]
   ...:         ss = ','.join(slist) + "\n"
   ...:         fh.write(ss)
   ...:     fh.close()
   ...: 
   ...: aa=pd.DataFrame({'A':range(100000)})
   ...: aa['B'] = aa.A + 1.0
   ...: aa['C'] = aa.A + 2.0
   ...: aa['D'] = aa.A + 3.0
   ...: 
   ...: %timeit -r10 aa.to_csv('/tmp/junk1',index=False)   
   ...: %timeit -r10 df_to_csv(aa,'/tmp/junk2') 
   ...: from hashlib import sha1
   ...: print sha1(open("/tmp/junk1").read()).hexdigest()
   ...: print sha1(open("/tmp/junk2").read()).hexdigest()

current pandas	with PR	example code
2.3 s	1.29 s	1.28 s

wins:

• convert numpy numerics to native types to eliminate expensive numpy specific
  stringify calls.
• if number of columns is < 10000, precompute the cols loop range rather
  then creating and walking a generator at each iteration of the inner loop.
• some cargo cult stuff that's probably in the noise.

[jreback]

Case 2

aa=pd.DataFrame({'A':range(100000),'B' : pd.Timestamp('20010101')})
aa.ix[100:2000,'A'] = np.nan

Current pandas

In [4]: %timeit aa.to_csv('test.csv')
1 loops, best of 3: 1.57 s per loop

case 2

In [7]: %timeit aa.to_csv('test.csv')
1 loops, best of 3: 2.66 s per loop

With change (starting with yours + my patch)

In [3]: %timeit aa.to_csv('test.csv')
1 loops, best of 3: 825 ms per loop

case 2

In [5]: %timeit aa.to_csv('test.csv')
1 loops, best of 3: 1.96 s per loop

[ghost]

Thanks jeff. the patch was garbled by GH. Can you pull this branch, add your patch
as a new commit and push to your fork? I'll pick up the commit and add it to the PR.

[jreback]

jreback@5c4d0bf

[ghost]

After jeff's commit:

current pandas	with PR	example code
2.3 s	0.58s	1.28 s

[ghost]

L1389 sure looks like one loop too many.

[jreback]

I played around more with this, but all of the pre-allocation and array indexing seems to be working, (even the map around asscalar makes a big diff), so unless we write this loop in cython, not sure how much better we can do

[ghost]

Another 25%:

current pandas	with PR	example code
2.3 s	0.44s	1.28 s

[jreback]

I get about 20% more with cython.....(on my test case that has lots of dates).....
jreback@5335a80
jreback@fa12e63

[jreback]

I think you win! (I get 0.47s) with the same test (an 2.3s with current pandas) :(
hmm....though If i were to pre-allocate the rows...., I am using a single row and overwriting (and copy to the writer)

[ghost]

wait, I'm following your lead and cythonizing the latest python version. let's see...

[ghost]

Remember we're benchmarking on different machines, timings differ. I get 0.407 with your cython branch.

[ghost]

310ms

[jreback]

whoosh
it's the preallocation of rows
pretty good 8x speedup

[ghost]

hang on, I'm working on a linux kernel patch...

[ghost]

the list() cast in the call into cython eliminates some test failures that
occur when passing in the index ndarray, have no idea why.

another 20% (relative):

current pandas	with PR	example code
2.3 s	0.35s	1.28 s

any more tricks? probably heavily IO bound at this point.
edit: not really.

[jreback]

could try doing a tolist() on the numeric arrays (in helper, when creating the series)
so that can eliminate the np.asscalar test and conversions

[ghost]

If it works, post the commit hash and i'll update.

[jreback]

try this out

jreback@d78f4f6

[ghost]

240ms. very nice.
But this doubles the memory footprint doesn't it? definitely add as option.

[jreback]

your wish is my command :) had the same issue in HDFStore, so chunked it

jreback@55adfb7

[ghost]

very good, and that also bounds the memory used by the list() copy in the call to cython.

I hereby declare these games over. 10x will have to do...

👍

[jreback]

AWESOME!

[jreback]

typos in RELEASE.rst the 3059 link is pointing to 3039, and df.to_csv

•Improved performance of dv.to_csv() by up to 10x in some cases. (GH3059)

[ghost]

fixed.

[jreback]

fyi vs 0.10.1

Results:
                                            t_head  t_baseline      ratio
name                                                                     
frame_get_dtype_counts                      0.0988    217.2718     0.0005
frame_wide_repr                             0.5526    216.5370     0.0026
groupby_first_float32                       3.0029    341.3520     0.0088
groupby_last_float32                        3.1525    339.0419     0.0093
frame_to_csv2                             190.5260   2244.4260     0.0849
indexing_dataframe_boolean                 13.6755    126.9212     0.1077
write_csv_standard                         38.1940    234.2570     0.1630
frame_reindex_axis0                         0.3215      1.1042     0.2911
frame_to_csv_mixed                        369.0670   1123.0412     0.3286
frame_to_csv                              112.2720    226.7549     0.4951
frame_mult                                 22.6785     42.7152     0.5309
frame_add                                  24.3593     41.8012     0.5827
frame_reindex_upcast                       11.8235     17.0124     0.6950
frame_fancy_lookup_all                     15.0496     19.4497     0.7738

[ghost]

We should jigger the code so the first column of the test names spells "AwesomeRelease".

If I had to pick one thing, I'd say iloc/loc is the best addition in 0.11, though.
ix and I have been playing "20 questions" for far too long.

[jreback]

hahah....fyi I put up a commit on the centered mov stuff when you have a chance....all fixed except for rolling_count, which I am not quite sure what the results should be...

[ghost]

btw, the final perf tweak was all about avoiding iteritems and it's reliance
on icol. there might be a fast path hiding there which would get you another
0.0 entry on test_perf.sh.

[jreback]

I saw your data pre-allocation, did you change iteritems somewhere?

fyi....as a side issue, that's the problem in the duplicate col names, essentially the col label maps to the column in the block, but there isn't a simple map, in theory should be a positional map, but very tricky

[ghost]

@jreback , can you follow up on the SO question, and make a pandas user happy?

[ghost]

Your refactor calculated series using iteritems, and that was the cause of the O(ncols)
behaviour I noted. Eliminating that in favor of yanking the data out directly from the blocks
turned that into ~ O(1), but breaks encpsulation. would be nice to use a proper interface
rather then rifle the bowels of the underlying block manager.

Come to think of it, iteritems is col oriented, I wonder if all that could have
been avoided... with iterrows. oh boy.

[jreback]

iterrows does a lot of stuff....and the block based approach is better anyhow...I will answer the so question

[ghost]

The tests I added for duplicate columns was buggy, and didn't catch the fact
that dupe columns are disabled for to_csv.

v10.1 to_csv() mangles dupe names into "dupe.1,dupe.2," etc. That's an ok workaround,
but what's the fundamental reason we can't just do it straight? is there one?

correction: it's read_csv that does the mangling.

[jreback]

the issue is that the blocks have a 2-d of the items in a particular block, the mapping between where it is in the block and the frame is depedent on a unique name of the columns (in the case of a mi this is fine of course).

There isn't a positional map (from columns in the frame to in the block). Keeping one is pretty tricky.

Wes has a failry complicated routing to find the correct column even with the duplicates, and it succeeds unless they are across blocks (which is the case that I am testing here).

I suppose you could temporarily do a rename, on the frame, (no idea if that works), then iterate on the blocks. which will solve the problem. As I said, 0.10.1 actually prints the data out twice. I think raising is ok here, very unlikely to happen, and if it does you can just put a mi in the first place.

[ghost]

I don't follow. If you can display a repr for a frame with dupe columns, you can write it out to csv.

[jreback]

can't display a repr either......I will create an issue for this..

[jreback]

#3092

[ghost]

We're probably thinking of different things. what do you mean you can't repr a frame
with dupe columns?

In [19]: pd.DataFrame([[1,1]],columns=['a','a'])
Out[19]: 
   a  a
0  1  1

on the other hand

In [4]: df.to_csv("/tmp/a")
    807         if len(set(self.cols)) != len(self.cols):
--> 808             raise Exception("duplicate columns are not permitted in to_csv")
    809 
    810         self.colname_map = dict((k,i) for i,k in  enumerate(obj.columns))

Exception: duplicate columns are not permitted in to_csv

That's completely messed up.

[ghost]

I think I can rejig CSVWriter to do this properly, or at least discover what it is that I'm
failing to understand.

[jreback]

a bit contrived, but this is the case that's the issue (yours we should allow)
the problem is that detecting this case is hard

In [32]: df1 = pd.DataFrame([[1]],columns=['a'],dtype='float64')

In [33]: df2 = pd.DataFrame([[1]],columns=['a'],dtype='int64')

In [34]: df3 = pd.concat([df1,df2],axis=1)

In [35]: df3.columns
Out[35]: Index([a, a], dtype=object)

In [36]: df3.index
Out[36]: Int64Index([0], dtype=int64)

In [37]: df3
Out[37]: ----------------------------------------------
Exception: ('Cannot have duplicate column names split across dtypes', u'occurred at index a')

[ghost]

So:

• the test that raises an exception should be finer grained, and fail only if dupes
  are across blocks.
• I should fix up the way the data object is constructed to handle dupe columns

Then we can think about the general case.

[ghost]

continued in #3095.

Thank you #3059, it's been fun.

[ghost]

dupe columns common case fixed in master via 1f138a4.

[ghost]

Changed 'legacy' keyword to engine=='python', to be consistent with c_parser.
in case it sticks around.

[dragoljub]

Guys this is truly amazing work. With the performance of reading/writing CSVs Pandas has become truly has enterprise leading I/O performance. Recently I was reading some zipped CSV files to DF at ~40MB/sec thinking to myself how much faster this is than many distributed 'Hadoop' solutions I've seen... :)