Extrapolate benefits

[hdoupe]

This initial PR is a prototype that uses SSI data to demonstrate how benefits data could be incorporated into the Records object as discussed in the first proposal here. Ultimately, if we decide to go this route, the benefits SNAP, Social Security, Medicare, and Medicaid would also be included. Currently, the data set benefit_extrapolation.csv.gz only includes tax and SSI benefit data, but could easily be expanded to include participation and benefits data on each filing unit for each program from 2014 to 2026.

This is only a demo of potential code modification to the Tax-Calculator. @Amy-Xu and I are very interested in hearing other opinions on this.

@martinholmer

[martinholmer]

@hdoupe said:

This initial PR is a prototype that uses SSI data to demonstrate how benefits data could be incorporated into the Records object as discussed in the first proposal here. Ultimately, if we decide to go this route, the benefits SNAP, Social Security, Medicare, and Medicaid would also be included. Currently, the data set benefit_extrapolation.csv.gz only includes tax and SSI benefit data, but could easily be expanded to include participation and benefits data on each filing unit for each program from 2014 to 2026.

This is only a demo of potential code modification to the Tax-Calculator. @Amy-Xu and I are very interested in hearing other opinions on this.

My first reaction has nothing to do with the logic of the approach in #1500, which I'll discuss in a subsequent comment. I think your choice of file and variable names needs to change.

Why not just benefits.csv.gz? I don't see what the file name benefits_extrapolation.csv.gz adds except letters.

And more importantly, can you change welfare to benefits?
You and the others working on the UBI project are probably too young to realize that "welfare" has been a highly-charged term in American politics for over fifty years. It has traditionally been used to describe means-tested benefit programs, such as SNAP (nee food stamps), TANF (nee AFDC), and Medicaid (except for when it pays nursing home bills for the elderly, which somehow is construed by the beneficiaries and their adult children as not being "welfare"). Those receiving non-means-tested social insurance benefits from OASDI (social security) and from Medicare would be shocked to hear their benefits being referred to as "welfare".

@Amy-Xu

[Amy-Xu]

@martinholmer said:

Those receiving non-means-tested social insurance benefits from OASDI (social security) and from Medicare would be shocked to hear their benefits being referred to as "welfare".

Thanks for pointing this out. Benefit definitely sounds more proper than welfare for our analysis.

[hdoupe]

@martinholmer Thanks for your feedback. I made the changes that you suggested.

[martinholmer]

Pull request #1500 includes the following lines:

        # set ssi participation and benefits data for current_year
        # in the future we would set corresponding attributes for
        # SNAP, SS, Medicare, and Medicaid
        # minimum year allowed is 2014
        if self.benefits is not None:
            year = max(2014, self.current_year)
            part_col_name = "Participation_{}".format(year)
            self.ssi_participation = self.benefits.loc[:, part_col_name]
            ben_col_name = "Benefit_{}".format(year)
            self.ssi_benefits = self.benefits.loc[:, ben_col_name]

I have a couple of basic questions about this code.

First, I don't understand why you are setting 2013 benefit variables to benefit values for 2014.
Why not set the values of self.ssi_benefits and self.ssi_participation to None in 2013?

Second, I understand that the Benefit_2014 column of the benefits.csv.gz is the dollar total of SSI benefits received by each filing unit, right? What is Participation_2014? What information does it add? We already know that filing units with zero for Benefit_2014 do not participate in that benefit program. What am I missing?

Third, how big is the benefits.csv.gz file when only SSI information is included?

@Amy-Xu @hdoupe

[hdoupe]

@martinholmer said

First, I don't understand why you are setting 2013 benefit variables to benefit values for 2014.
Why not set the values of self.ssi_benefits and self.ssi_participation to None in 2013?

That sounds sensible to me.

Second, I understand that the Benefit_2014 column of the benefits.csv.gz is the dollar total of SSI benefits received by each filing unit, right? What is Participation_2014? What information does it add? We already know that filing units with zero for Benefit_2014 do not participate in that benefit program. What am I missing?

Participation_Year tells you how many people in each tax unit receive benefits. For example,

In [7]: df.Participation_2014.value_counts()
Out[7]: 
0    442763
1     11855
2      1515
3       240
4        62
6        17
5        11
7         2
Name: Participation_2014, dtype: int64

@Amy-Xu is this interpretation correct?

Third, how big is the benefits.csv.gz file when only SSI information is included?

HDoupe-MacBook-Pro:taxcalc henrydoupe$ ls -lh benefits.csv.gz 
-rw-r--r--  1 henrydoupe  staff   1.3M Aug  4 11:01 benefits.csv.gz

[martinholmer]

@hdoupe said:

Participation_YEAR tells you how many people in each tax unit receive benefits. For example,

In [7]: df.Participation_2014.value_counts()
Out[7]: 
0    442763
1     11855
2      1515
3       240
4        62
6        17
5        11
7         2
Name: Participation_2014, dtype: int64

OK, but you need to name this variable something else to avoid confusing people who have experience studying transfer programs. I think you want something like this for the
variable names in the Records object:

ssi_benefits
ssi_recipients

and perhaps something like this in the benefits.csv.gz file:

ssi_ben_YEAR
ssi_rec_YEAR

The term recipients (or beneficiaries) is universally used to describe how many people receive a benefit. Participation is almost always a zero-one variable.

Now a question about substance. What are the two filing units like that contain seven SSI recipients? That sounds pretty unusual. When you look at those two filing units, what is it about their situation that leads to seven of the filing unit's members getting SSI benefits? Are we sure this is not some kind of data-preparation error?

And finally, just to confirm, you're saying the benefits.csv.gz file (with only the SSI variables for 2014-2026) is about 1.3 megabytes, right?

@Amy-Xu @andersonfrailey

[hdoupe]

@martinholmer said

OK, but you need to name this variable something else to avoid confusing people who have experience studying transfer programs. I think you want something like this for the
variable names in the Records object:

ssi_benefits ssi_recipients

and perhaps something like this in the benefits.csv.gz file:

ssi_ben_YEAR ssi_rec_YEAR

The term recipients (or beneficiaries) is universally used to describe how many people receive a benefit. Participation is almost always a zero-one variable.

That sounds reasonable to me. I'll make the suggested changes.

Now a question about substance. What are the two filing units like that contain seven SSI recipients? That sounds pretty unusual. When you look at those two filing units, what is it about their situation that leads to seven of the filing unit's members getting SSI benefits? Are we sure this is not some kind of data-preparation error?

That is worrisome. @Amy-Xu could you look into this?

And finally, just to confirm, you're saying the benefits.csv.gz file (with only the SSI variables for 2014-2026) is about 1.3 megabytes, right?

Yes, it is about 1.3 MB. Sorry for not making that clear.

[Amy-Xu]

@martinholmer asked:

When you look at those two filing units, what is it about their situation that leads to seven of the filing unit's members getting SSI benefits? Are we sure this is not some kind of data-preparation error?

I'll look into this issue on raw CPS first, but eventually might need @andersonfrailey help to see whether those two tax units have so many recipients.

[hdoupe]

@andersonfrailey @Amy-Xu have you figured out what's going on here:

@Amy-Xu said

@martinholmer asked:

When you look at those two filing units, what is it about their situation that leads to seven of the filing unit's members getting SSI benefits? Are we sure this is not some kind of data-preparation error?

I'll look into this issue on raw CPS first, but eventually might need @andersonfrailey help to see whether those two tax units have so many recipients.

[Amy-Xu]

@hdoupe @martinholmer So I looked into the imputed SSI data. There's one household headed by a never-married single mother, with no income (fwsval) and 7 children. All SSI participation are imputed. I wouldn't be surprised if in the tax unit dataset this household weight was split to two.

This is a very rare case though.

[hdoupe]

@Amy-Xu Yes, this seems like a pretty rare case.

What do you mean when you say:

I wouldn't be surprised if in the tax unit dataset this household weight was split to two.

[Amy-Xu]

@hdoupe What I had in mind was split-to-match-income, but then I realized that was for CPS-PUF match. This is a pure CPS dataset. Then I'm not sure why there're two records like this. @andersonfrailey Is there a way to check whether the two records are from household [h_seq==59152]? And if so, why are there two records instead of one?

[martinholmer]

@Amy-Xu said:

So I looked into the imputed SSI data. There's one household headed by a never-married single mother, with no income (fwsval) and 7 children. <details of those eight people are here>

I don't understand the SSI eligibility rules well enough to understand how her seven children are SSI recipients. Under which provisions of the SSI program do the kids qualify to receive SSI benefits even though their mother does not receive SSI benefits?

@hdoupe

[andersonfrailey]

@andersonfrailey Is there a way to check whether the two records are from household [h_seq==59152]? And if so, why are there two records instead of one?

@Amy-Xu because the CPS dataset is a combination of three CPS files h_seq wouldn't be the best way to find unique households. I'll take a look at the record and see if I can figure out what happened.

[Amy-Xu]

@martinholmer Thanks for catching this. I got confused as well since children have to be blind or disabled to be qualify for SSI. But none of the kids in this family has been marked as disabled or blind. I look into my imputation code, and it turns out that we use a combined disability indicator that includes natural disability and work disability. Natural disability is based on CPS disability variables, and work disability is based on CPS ASEC work disability definition. In this set of work disability rules, item 4 says if someone is younger than 65, and is covered by medicare, then this person is classified as work disabled.

In this particular family, everyone is less than 65 years old and (somehow) covered by medicare, which I frankly don't know this is a reporting error or what. Thus they all fulfill the disability requirement in my imputation routine, although in reality I don't think they belong to 'work disability' category or meet SSI eligibility rules.

It is indeed possible that only children get SSI but not parents, when children are disabled. However, we have all children of this family imputed mainly because the number of children getting SSI is way too low, so children are more likely get imputed than adults.

I'm aware this imputation routine is far from perfect, and will keep improving the algorithm. Martin, do you think it's ok for us just to remove this family for now since it is a tiny fraction of all imputed individuals?

[martinholmer]

@Amy-Xu said in #1500:

Thanks for catching this. I got confused as well since children have to be blind or disabled to be qualify for SSI. But none of the kids in this family has been marked as disabled or blind. I look into my imputation code, and it turns out that we use a combined disability indicator that includes natural disability and work disability. Natural disability is based on CPS disability variables, and work disability is based on CPS ASEC work disability definition. In this set of work disability rules, item 4 says if someone is younger than 65, and is covered by medicare, then this person is classified as work disabled.

It sounds to me as if "someone younger than 65 and covered by Medicare" is referring to adults not children.

In this particular family, everyone is less than 65 years old and (somehow) covered by medicare, which I frankly don't know this is a reporting error or what. Thus they all fulfill the disability requirement in my imputation routine [see caveat above], although in reality I don't think they belong to 'work disability' category or meet SSI eligibility rules.

It is indeed possible that only children get SSI but not parents, when children are disabled. However, we have all children of this family imputed mainly because the number of children getting SSI is way too low, so children are more likely get imputed than adults.

What do you mean by imputed? What is being imputed? Medicare receipt?

I'm aware this imputation routine is far from perfect, and will keep improving the algorithm. Martin, do you think it's ok for us just to remove this family for now since it is a tiny fraction of all imputed individuals?

No, I don't think this family should be removed. That would make your CPS sample be different from your benefits sample, which I don't think you want to do. Plus, its not just this one filing unit: there is another one with 7 SSI beneficiaries, and many with 6 or 5 SSI beneficiaries. So, it seems as if this one filing unit is the tip of an iceberg of poor imputation. I think you should look at the imputation algorithm again. Here is the tally done by @hdoupe:

In [7]: df.Participation_2014.value_counts()
Out[7]: 
0    442763
1     11855
2      1515
3       240
4        62
6        17
5        11
7         2
Name: Participation_2014, dtype: int64

[Amy-Xu]

It sounds to me as if "someone younger than 65 and covered by Medicare" is referring to adults not children.

I'm adding adult to the constraints to see what I can get.

What do you mean by imputed? What is being imputed? Medicare receipt?

SSI benefits and participation, because original CPS SSI data is under-reported, and we imputed participants and benefits to match administrative totals. Medicare coverage is a part of CPS ASEC -- no imputation from our end.

[martinholmer]

@Amy-Xu said:

It's sensible to keep this extrapolation routine on a separate branch, but I'm not sure how that would impact a near-term goal of incorporating CPS tax-unit and UBI reform on Tax-Brain.

You have to be kidding about that being a "near-term goal" given the state of of TaxBrain. There are TaxBrain bugs that were reported months ago that have not yet been fixed, so I don't see adding something major to TaxBrain as very likely in the "near-term".

[martinholmer]

@Amy-Xu said:

I very appreciate the fact that you spot the issue with SSI data, but to say CPS benefit data has not been through even a minimal check is not accurate. We have checked aggregates of each benefit program and posted those numbers in this UBI-examples issue. Benefit distribution wise, I have always been wanting to check at tax-unit level; however, there's no such official tax-unit distribution to compare with. In term of the raw CPS benefits, we have been through several rounds of review with Dan, Jamie Hall from Heritage, and Kevin Corinth who used to be a scholar at AEI. For the five programs added in CPS tax unit data, I don't see any major modification in short run, unless anyone finds errors.

OK, there's no public record of the review process you describe above. So, I stand corrected: the CPS benefit has undergone more than minimal review. But given that I could quickly spot problems with the SSI imputations after those reviews should make you wonder exactly how effective that review process was.

[martinholmer]

@Amy-Xu said:

In addition, to say this CPS benefit data will only produce meaningless results is also not accurate. Anderson has updated the code with this CPS benefit dataset for first UBI example. Most results resemble what has been published in the working paper. Certainly it needs a few more tweaks, but it is not meaningless.

But the key question is how sensible are the imputed benefit data. To say that the imputed benefit data produce about the same results as earlier benefit data (used in the working paper) says nothing about the quality of the imputed data. It just means that the imputed data haven't changed much since the working paper.

Anyway, my proposed approach (leaving open pull request #1500 until more progress is made) does nothing to slow down your data-imputation checking or improvement process and does nothing to slow down your code development work.

[MattHJensen]

It sounds like the primary objections to proceeding with this PR are:

1. The uncertain nature of the data.
2. The limited applicability of the data.
3. The size of the data.

With regard to the "uncertain nature of the data", I agree with @Amy-Xu that improving the data quality should be and is an ongoing process, rather than a one-time task or objective. As Amy notes, we have already incorporated several rounds of feedback from experts in the field, and we will continue to do so. Yes, the file should be considered 'beta,' and yes, we want to continue to find new ways to improve the file, but that does not mean that the file is not useful as is. Generally, I think the appropriate question about quality should be, "are these data better than the next best alternative, and is the documentation about the state of the data clear." The only alternative to this file is the raw CPS data, and I am quite certain that these data are better than the raw CPS. The C-TAM documentation is clear about the state of the project, as is the CPS.csv documentation.

With regard to the "limited applicability of the data", I strongly disagree with the premise. These data are applicable to all users who want to repeal and replace benefit programs with tax programs or a new program like a UBI. Given the strong interaction and conceptual similarity between benefit programs and tax programs, these seem basic and essential. The forthcoming MTR data will further improve the quality of our behavioral analyses for even tax-only reforms. We currently systematically misstate the behavioral responses to tax reforms because we don't include MTRs from state and local taxes and benefit programs.

With regard to the "size of the data", I agree that this is a problem.

Given that the size of the data is the fundamental problem here, at least in my thinking as described above, I doesn't seem that leaving the data in a separate branch is the right approach. It provides no path forward for dealing with that fundamental problem.

I wonder if it would make more sense to include these data in a conda package that would be an optional dependency for tax-calculator. Perhaps we made a mistake by not following this approach for the cps.csv file more generally. Including data of any sort in the tax-calculator repository creates a fundamental tension between adding new, useful, variables and minimizing the size of the repo.

[MattHJensen]

You have to be kidding about that being a "near-term goal" given the state of of TaxBrain. There are TaxBrain bugs that were reported months ago that have not yet been fixed, so I don't see adding something major to TaxBrain as very likely in the "near-term".

This needs to be accomplished by the end of January for external reasons, so ideally it would be accomplished before the December holidays. I think we are on schedule to do that, even while prioritizing the bug fixes ahead of time. In fact, I don't think we would get the CPS integration done any sooner even if we prioritized it ahead of the bug fixes, as the bug fixes are providing training for all of us --in particular Hank and Sean -- about webapp-public.

[MattHJensen]

I said:

This needs to be accomplished by the end of January for external reasons

@Amy-Xu reminded me via email that one of the good suggestions in Producing Open Source Software is to provide clarity about grant requirements as early as possible.

To that end: it is a grant requirement for TaxBrain to have the capability to repeal and replace benefit programs with a UBI by the end of March. To facilitate, that, I would like to have a version of those capabilities up by the end of January, which we will clearly mark as "alpha" or "beta".

My apologies for not being more clear about these requirements earlier in the process.

I recommend anyone who hasn't read the book to do so-- it contains numerous good suggestions.

[martinholmer]

@MattHJensen said in the discussion of pull request #1500:

I wonder if it would make more sense to include these [CPS-benefit] data in a conda package that would be an optional dependency for tax-calculator. Perhaps we made a mistake by not following this approach for the cps.csv file more generally.

I think moving the cps_benefits.csv.gz file out of PR#1500 and putting it in its own public conda package (like puf.csv is in its own private conda package) makes sense. TaxBrain can get it from that new conda package just like it gets puf.csv from its own conda package. I would feel much better about the future #1500 is pointing to if cps_benefits.csv.gz were removed from #1500.

My view is that we did not make a mistake when we included cps.csv.gz and its weights file in the Tax-Calculator repository. It has replaced, in many of the unit tests, the old 1991 PUF file, which was in the repository, and it is automatically available to users of the Tax-Calculator CLI tc. These two advantages alone outweigh, in my mind, whatever negatives there may be in terms of increased taxcalc package size.

@Amy-Xu @andersonfrailey @hdoupe

[Amy-Xu]

The size of cps_benefits.csv.gz is a part of my initial concerns. If moving this datafile to a separate conda package works for TB, it is certainly a very sensible solution.

[martinholmer]

@Amy-Xu said in #1500:

Do you think we can proceed with the current extrapolation routine, while open an issue in taxdata repo for the benefit portion of CPS tax unit data?

I have a couple of questions about the CPS-benefits data file.

• I don't see any code in the taxdata repo that generates the cps_benefits.csv file that was initially included in PR#1500. Did I miss something? Where is this file created?

• When I tabulate cps.csv and cps_benefits.csv, I see that they both have 456465 data rows and that cps_benefits.csv has 156 columns, but none of those columns is RECID. So, how can you tell which benefits go with which filing unit in cps.csv?

@MattHJensen @andersonfrailey @hdoupe

[Amy-Xu]

@martinholmer asked:

I don't see any code in the taxdata repo that generates the cps_benefits.csv file that was initially included in PR#1500. Did I miss something? Where is this file created?

That file is produced by Hank @hdoupe, using the code in taxdata PR #108, which has not been merged into taxdata repo yet. Even though the title only mentions SSI, I believe it has already been capable of handling all five programs we included in the cps_benefits.csv file.

When I tabulate cps.csv and cps_benefits.csv, I see that they both have 456465 data rows and that cps_benefits.csv has 156 columns, but none of those columns is RECID. So, how can you tell which benefits go with which filing unit in cps.csv?

This cps_benefits.csv works in the same way as weights files we have for puf and cps. As you can see, those weights files don't have any ID column either, because the sequence of records is guaranteed to match puf or cps in its production process. Same applies to the benefit file here.

[martinholmer]

@Amy-Xu said:

That file is produced by Hank @hdoupe, using the code in taxdata PR #108, which has not been merged into taxdata repo yet. Even though the title only mentions SSI, I believe it has already been capable of handling all five programs we included in the cps_benefits.csv file.

But that pull request does not contain the cps_benefits.csv file produced by the script. Where is that file?

[hdoupe]

@martinholmer said

But pull request does not contain the cps_benefits.csv file produced by the script. Where is that file?

This is the same file. I just copied it into the Tax-Calculator repo with what I thought was a better name. Should I change the name in extrapolate.py to cps_benefits.csv.gz or change the name in this PR to benefit_extrapolation.csv.gz?

[martinholmer]

@hdoupe said:

This is the same file. I just copied it into the Tax-Calculator repo with what I thought was a better name.

Fine. But where is the latest version of the CPS-benefit data file? The one with at most four (not seven) SSI recipients per filing unit? Your pull request #108 contains only one Python file and no csv output.

[hdoupe]

@martinholmer Both TC PR #1500 and taxdata PR#108 have been updated.

[martinholmer]

@hdoupe said:

Both TC PR #1500 and taxdata PR#108 have been updated.

Thanks very much. But given the recent conversation in #1500 about putting cps_benefits.csv.gz in its own repository for distribution (like we do with the puf.csv file), I'm puzzled why you updated #1500 with the newest version of the cps_benefits.csv.gz file. I would have thought you would have deleted it from #1500 now that it is available in taxdata where it is generated.

I had interpreted #1500 comments in the last day or so by @MattHJensen, @martinholmer and @Amy-Xu as all endorsing the idea of moving the cps_benefits.csv.gz file out of pull request #1500 (because it is not going to be part of the Tax-Calculator repository, like puf.csv is not part of the Tax-Calculator repository).

Am I confused and we do not have a consensus on the issue of how to distribute the cps_benefits.csv.gz file? Or do we have a consensus?

[Amy-Xu]

@martinholmer I think we do have a consensus here. Since the dataset is already available on the taxdata PR, Hank @hdoupe, could you remove the cps_benefits.csv.gz file from this PR?

[hdoupe]

@Amy-Xu @martinholmer All set. Sorry for causing the confusion.

[martinholmer]

@hdoupe said in #1500:

Sorry for causing the confusion.

No need to feel bad: this has been an extended conversation that has gone on while you've been focusing most of your creative energy on fixing TaxBrain. You're making substantial progress on both fronts, so all is good. And your TaxBrain work revealed one of my bugs, so its me who has been "causing the confusion" with respect to the new Tax-Calculator error/warning messages.

[MattHJensen]

@Amy-Xu, could you propose a repository name for the data files and suggest who should have write access initially? I will create it in Open Source Economics and set the appropriate permissions.

[Amy-Xu]

It seems to me the name should include a few key words including benefit, tax unit, and CPS. But I'm not sure whether CPS should be there since we're about to add institutional ACS data. So may just Tax-Unit Benefit data, and then add in the description section that this dataset is mainly based on CPS.

In terms of write access, Hank is the only person who directly deal with he dataset, so he should have write access. And you can add whoever else you think is appropriate.

[MattHJensen]

@Amy-Xu and @hdoupe, see https://github.com/open-source-economics/tax-unit_benefits. I gave you both write access. We can change the name anytime if you'd like.

[hdoupe]

Closing in favor of #1719