-
Notifications
You must be signed in to change notification settings - Fork 57
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Issues resolved: 1) qrsolve(XX, XY) suffers from numerical inaccuracies on some cases, so we fix to qrsolve(X, Y) if we don't have weights (Note: this might be a bit slower as the XX and XY are already precomputed. It could also be optimized but for now let's leave it as it is. 2) The methods used to find out omitted variables were different in different points. When demeaning, we looked if the regressors were close to zero, but used absolute values (1e-8) which doesn't work if there is a huge scaling difference between Y and X. Now we will assumme the var has been absorbed by the absvars if the ratio z'z/w'w < (1e-9) where z=old variable and w= demeaned variable. This 1e-9 will also increase with tolerance() as it's just tolerance*1e-1 We also use invsym() as the main driver for whether to drop or not, as that is what it's used on the built-in tools. Then the inputs of qrsolve() will exclude the omitted variables, to prevent any issue.
- Loading branch information
1 parent
9503eee
commit 79a8c01
Showing
7 changed files
with
204 additions
and
60 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| noi cscript "reghdfe: ensure we omit the correct variables" adofile reghdfe | ||
|
|
||
| * Setup | ||
| local included_e /// | ||
| scalar: N rmse tss rss r2 r2_a df_r df_m ll ll_0 /// mss F | ||
| matrix: trim_b trim_V /// | ||
| macros: wexp wtype | ||
|
|
||
| * [TEST] Simplest case | ||
| sysuse auto, clear | ||
| gen z = 0 | ||
| reghdfe price z, noab | ||
| matrix b = e(b) | ||
| assert b[1,1]==0 | ||
| matrix list b | ||
| _ms_omit_info b | ||
| assert r(k_omit)==1 | ||
|
|
||
|
|
||
| * [TEST] Simple case | ||
| * https://github.com/sergiocorreia/reghdfe/issues/93 | ||
|
|
||
| sysuse auto, clear | ||
| gen lowrep = rep78<4 | ||
| loc lhs price | ||
| loc rhs lowrep | ||
| loc absvars rep78 | ||
|
|
||
| * 1. Run benchmark | ||
| areg `lhs' `rhs', absorb(`absvars') | ||
| matrix list e(b) | ||
|
|
||
| trim_cons | ||
| local bench_df_a = e(df_a) | ||
| storedresults save benchmark e() | ||
|
|
||
| * 2. Run reghdfe | ||
| reghdfe `lhs' `rhs', absorb(`absvars') keepsingletons verbose(-1) | ||
| matrix list e(b) | ||
| notrim | ||
| storedresults compare benchmark e(), tol(1e-12) include(`included_e') | ||
| assert `bench_df_a'==e(df_a)-1 | ||
|
|
||
| * [TEST] https://github.com/sergiocorreia/reghdfe/issues/95 | ||
| clear | ||
| set obs 100 | ||
| set seed 123 | ||
| gen double y = runiform() * 1e8 | ||
| gen double z = runiform() | ||
| gen byte c = 1 | ||
|
|
||
| loc lhs z | ||
| loc rhs y | ||
| loc absvars c | ||
|
|
||
| * 1. Run benchmark | ||
| areg `lhs' `rhs', absorb(`absvars') | ||
| matrix list e(b) | ||
|
|
||
| trim_cons | ||
| local bench_df_a = e(df_a) | ||
| storedresults save benchmark e() | ||
|
|
||
| * 2. Run reghdfe | ||
| reghdfe `lhs' `rhs', absorb(`absvars') keepsingletons verbose(-1) | ||
| matrix list e(b) | ||
| notrim | ||
| storedresults compare benchmark e(), tol(1e-12) include(`included_e') | ||
| assert `bench_df_a'==e(df_a)-1 | ||
|
|
||
|
|
||
| * [TEST] https://github.com/sergiocorreia/reghdfe/issues/94 | ||
| clear | ||
| input w x y unity | ||
| w x y unity | ||
| 1 -1.10565 -1.223344 1 | ||
| 1 -1.109139 -1.127615 1 | ||
| 0 -.8497003 -1.207231 1 | ||
| 1 -.2734231 -1.294683 1 | ||
| 0 .0063879 -.8343916 1 | ||
| 1 .7669702 -.8245103 1 | ||
| end | ||
|
|
||
| gen z = 0 | ||
| reghdfe y z, noab | ||
|
|
||
| reghdfe y i.w##i.unity x, a(unity) tol(1e-12) | ||
| matrix list e(b) | ||
| matrix list e(V) | ||
|
|
||
| areg y i.w##i.unity x, a(unity) | ||
| matrix list e(b) | ||
| matrix list e(V) | ||
|
|
||
|
|
||
| loc lhs y | ||
| loc rhs i.w##i.unity x | ||
| loc absvars unity | ||
|
|
||
| * 1. Run benchmark | ||
| areg `lhs' `rhs', absorb(`absvars') | ||
| matrix list e(b) | ||
|
|
||
| trim_cons | ||
| local bench_df_a = e(df_a) | ||
| storedresults save benchmark e() | ||
|
|
||
| * 2. Run reghdfe | ||
| reghdfe `lhs' `rhs', absorb(`absvars') keepsingletons verbose(-1) | ||
| matrix list e(b) | ||
| notrim | ||
| matrix b = e(b) | ||
| matrix V = e(V) | ||
| assert abs(SR__benchmark_trim_b[1,2] - b[1,1]) < 1e-8 | ||
| assert abs(SR__benchmark_trim_b[1,6] - b[1,4]) < 1e-8 | ||
| assert abs(SR__benchmark_trim_V[2,2] - V[1,1]) < 1e-8 | ||
| assert abs(SR__benchmark_trim_V[6,6] - V[4,4]) < 1e-8 | ||
|
|
||
| * Done! | ||
|
|
||
| storedresults drop benchmark | ||
| clear matrix | ||
| exit |
Oops, something went wrong.