PERF: Improve performance of StataReader's processing of missing values

[bashtage]

Improve performance of StataReader when converting columns
with missing values

xref #25772

• closes #N/A
• tests added / passed N/A, perf only
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jreback]

can you add an asv that covers this case?

[jreback]

don't use inplace

[bashtage]

I used inplace intentionally to try and minimize memory consumption, which can be an issue when loading some Stata files. Is it going away?

[jreback]

its not idiomatic, nor does it actually do anything here (it rarely does); we don't use it internally if at all possible.

[pep8speaks]

Hello @bashtage! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-03-20 13:16:12 UTC

[jreback]

can you merge master

[codecov]

Codecov Report

Merging #25780 into master will decrease coverage by 49.51%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #25780       +/-   ##
===========================================
- Coverage   91.26%   41.74%   -49.52%     
===========================================
  Files         172      172               
  Lines       52965    52965               
===========================================
- Hits        48337    22109    -26228     
- Misses       4628    30856    +26228

Flag	Coverage Δ
#multiple	?
#single	41.74% <ø> (-0.01%)	⬇️

Impacted Files	Coverage Δ
pandas/io/formats/latex.py	0% <0%> (-100%)	⬇️
pandas/io/sas/sas_constants.py	0% <0%> (-100%)	⬇️
pandas/tseries/plotting.py	0% <0%> (-100%)	⬇️
pandas/tseries/converter.py	0% <0%> (-100%)	⬇️
pandas/io/formats/html.py	0% <0%> (-99.36%)	⬇️
pandas/core/groupby/categorical.py	0% <0%> (-95.46%)	⬇️
pandas/io/sas/sas7bdat.py	0% <0%> (-91.17%)	⬇️
pandas/io/sas/sas_xport.py	0% <0%> (-90.15%)	⬇️
pandas/core/tools/numeric.py	10.44% <0%> (-89.56%)	⬇️
pandas/io/json/normalize.py	8.16% <0%> (-88.78%)	⬇️
... and 129 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e54e55...2eda26b. Read the comment docs.

[codecov]

Codecov Report

Merging #25780 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #25780   +/-   ##
=======================================
  Coverage   91.27%   91.27%           
=======================================
  Files         173      173           
  Lines       53002    53002           
=======================================
  Hits        48375    48375           
  Misses       4627     4627

Flag	Coverage Δ
#multiple	89.83% <ø> (ø)	⬆️
#single	41.77% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37d04a3...e5f3c06. Read the comment docs.

[jreback]

lgtm. ping on green.

[bashtage]

@jreback I think the failures are unrelated to my changes.

[jreback]

master is passing, so maybe you have an older build. pls merge master and let's see.

[bashtage]

@jreback Green.

[jreback]

thanks @bashtage does this fully close the issue?

[bashtage]

No, there is still an issue with duplicate valuelables which can't be handeled by categories which require unique labels. There are 2 solutions to that: 1. mung the value label and preserve the data (an integer) or 2. assign all values with the same label to be the same. 1 is probably better since there isn't much loss of fidelity (can just find the munged names and fix, plus the underlying integer values are still correct for the labeled categories). But this needs more thought and work.