Issue #221: Add marginal model #426

athowes · 2024-11-07T12:43:46Z

Description

This PR implements the marginal model as in #221.

The file marginal_model-scratch.R implements a prototype outside of functions
This prototype is implemented into the standard structure
Uses primarycensored function primarycensored_lpmf which is passed into via a wrapper in marginal_model/functions.stan
- Uses only the analytic versions of this distribution
Adds tests for new marginal_model

Possible steps forward here:

But uses non vectorised version of LPMF: want to extend to be using vectorised version of LPMF
Could want to extend to being able to use non analytic solution in a natural way

Checklist

My PR is based on a package issue and I have explicitly linked it.
I have included the target issue or issues in the PR title in the for Issue(s) issue-numbers: PR title
I have read the contribution guidelines.
I have tested my changes locally.
I have added or updated unit tests where necessary.
I have updated the documentation if required.
My code follows the established coding standards.
I have added a news item linked to this PR.
I have reviewed CI checks for this PR and addressed them as far as I am able.

inst/cohort-scratch.R

inst/stan/marginal_model/functions.stan

…many primarycensored functions..

seabbs · 2024-12-03T10:05:42Z

@athowes there are a few bows to tie here and I am investigating the CI issues on mac and windows (it looks likes its also on main so presumably unrelated) but I think this is at the point its worth taking a look at. See https://github.com/epinowcast/epidist/actions/runs/12112515981/job/33766065135 for the same failure on main.

Updates in bullets:

Added a marginalised likelihood model based on primarycensored. This can be specified using as_epidist_marginal_model(). This is currently limited to Weibull, Log-Normal, and Gamma distributions with uniform primary censoring but this will be generalised in future releases. This is waiting on A lookup in R to find the stan number associated with a probability distribution primarycensored#175.
Added a transform_data s3 method to allow for data to be transformed for specific models. This is specifically useful for the marginal model at the moment as it allows reducing the data to its unique strata.
Switched over to using the marginal model everywhere
Added helper functions for new variables to avoid code duplication in vignettes
Added a heuristic for the marginal model that adapts the relative observation time to be Inf when it is much longer than the maximum delay. This is required as otherwise there are numerical instability errors from try to divide by something very near one (i.e the cdf at long observation times). In primarycensored this is a warning but here with more naive users I think it makes sense to make an informed choice and flag that that has happened.

seabbs

LGTM (self-review)

NEWS.md

R/marginal_model.R

inst/stan/latent_model/functions.stan

tests/testthat/setup.R

tests/testthat/test-marginal_model.R

vignettes/ebola.Rmd

vignettes/faq.Rmd

Co-authored-by: Adam Howes <[email protected]>

seabbs

Following comments from @athowes a follow review from me. LGTM so merging

* Add cohort model template * Fix to previous commit (name as cohort_model) * Generate simulated cohort data * Add unweighted and weighted direct models * Thinking about custom family for pcd function * functions component of stanvars * Add transformed parameters for cohort model * Progress on implementing PCD model * Set q as vector * Get rid of params input * This would work, apart from it's the CDF. For the PMF need to import many primarycensored functions.. * Almost working with "import all functions" strategy * Wrap up on attempt * Rename to marginal model * Move towards single wrap function with others imported * Just running into some C++ errors.. * This doesn't change anything * Move to marginal model name and lint * Create aggregate data inside model conversion function for now * First draft on moving marginal_model into functions * Run document * Tests working up to valid Stan code * Regex version of marginal model * Use prep_marginal_obs * Improve assert for marginal model * Add pkgdown and document * Clean up scratch implementation * remove scratch file * update data format, formula, and family * update stan code * basic working version * add transformm data methods * start exploring the ebola example * add a helper to find meaningful relative_obs_times * get the full ebola vignette working with new variable requirements * improve return messages * update approx vignette * get ebole vignette passing by checking pp and related inputs * add marginal model integration tests * expand post processing tests * add marginal model * use the right transform data keyword * add ... pass through to make constructors correct * fix .summarise_n_by_formula test so error message is as expected * drop not required .row_id * check using ... properly * make the progress messages prettier for reducing data complexit: * check post process tests again * add a test for the specific transform data method * add some tests for the generic transform data method * put transform data tests in the correct folder * add a news update * change vignette language to talk about marginal model * update the FAQ to use the marginal variables * call it transformed_data not trans_data * change the error message to make it clear its a epidist limitation * update stan docs * Update NEWS.md * Update NEWS.md Co-authored-by: Adam Howes <[email protected]> * Update NEWS.md Co-authored-by: Adam Howes <[email protected]> * Update inst/stan/latent_model/functions.stan Co-authored-by: Adam Howes <[email protected]> * Update setup.R --------- Co-authored-by: Sam <[email protected]> Former-commit-id: 0c41c481bcd9767252b562e8fa704e6b43995070 [formerly 267310c44261cc0e4cae488103b7f2028b4f0153] Former-commit-id: 277996547d66fd5d6260710b620f67f7ed3475d6

seabbs reviewed Nov 13, 2024

View reviewed changes

inst/cohort-scratch.R Outdated Show resolved Hide resolved

seabbs reviewed Nov 13, 2024

View reviewed changes

inst/cohort-scratch.R Outdated Show resolved Hide resolved

athowes force-pushed the cohort-model branch from 6e4c0e1 to 6ccab62 Compare November 13, 2024 16:29

athowes mentioned this pull request Nov 13, 2024

Import all function dependency of Stan function epinowcast/primarycensored#171

Open

athowes changed the title ~~Issue #221: Add cohort model~~ Issue #221: Add marginal model Nov 14, 2024

athowes force-pushed the cohort-model branch 2 times, most recently from a415d16 to aa24b86 Compare November 15, 2024 21:04

This comment was marked as off-topic.

Sign in to view

athowes force-pushed the cohort-model branch from bd6c7be to 1ab0450 Compare November 22, 2024 14:17

seabbs reviewed Nov 22, 2024

View reviewed changes

inst/stan/marginal_model/functions.stan Outdated Show resolved Hide resolved

seabbs reviewed Nov 22, 2024

View reviewed changes

inst/stan/marginal_model/functions.stan Outdated Show resolved Hide resolved

athowes commented Nov 22, 2024

View reviewed changes

inst/stan/marginal_model/functions.stan Outdated Show resolved Hide resolved

athowes force-pushed the cohort-model branch from 3b6630a to 0d71938 Compare November 25, 2024 12:36

This comment was marked as outdated.

Sign in to view

athowes requested a review from seabbs November 25, 2024 12:39

athowes force-pushed the cohort-model branch from 229527b to 4dff643 Compare November 25, 2024 14:40

athowes mentioned this pull request Nov 28, 2024

Issue #267: Refactor to allow custom event priors and marginalise the latent likelihood #474

Merged

9 tasks

athowes added 12 commits November 29, 2024 11:18

Add cohort model template

0d0be9f

Fix to previous commit (name as cohort_model)

810423a

Generate simulated cohort data

8f4c34a

Add unweighted and weighted direct models

8bcfbe7

Thinking about custom family for pcd function

e95316a

functions component of stanvars

2165646

Add transformed parameters for cohort model

602f015

Progress on implementing PCD model

8d4c737

Set q as vector

5590a0f

Get rid of params input

d005cba

This would work, apart from it's the CDF. For the PMF need to import …

31c3736

…many primarycensored functions..

Almost working with "import all functions" strategy

b4b71a0

epinowcast deleted a comment from codecov bot Dec 2, 2024

seabbs added 4 commits December 2, 2024 18:21

check post process tests again

6c529f1

add a test for the specific transform data method

9c0d9a2

add some tests for the generic transform data method

0712992

put transform data tests in the correct folder

2aabfd7

add a news update

ff89e3b

seabbs marked this pull request as ready for review December 3, 2024 10:23

seabbs added 5 commits December 3, 2024 10:44

change vignette language to talk about marginal model

3a3aed2

update the FAQ to use the marginal variables

2d1aef4

call it transformed_data not trans_data

2a61182

change the error message to make it clear its a epidist limitation

4f080c4

update stan docs

6293762

This comment was marked as outdated.

Sign in to view

seabbs previously approved these changes Dec 3, 2024

View reviewed changes

This was referenced Dec 3, 2024

rstan compilation failing on Windows and Mac CI on main #487

Open

Expand marginal model distribution support #488

Open

Add support for marginal exponential primary event priors #489

Open

athowes commented Dec 3, 2024

View reviewed changes

seabbs dismissed their stale review via 7062e52 December 3, 2024 13:15

seabbs and others added 5 commits December 3, 2024 13:15

Update NEWS.md

7062e52

Update NEWS.md

79f0a88

Co-authored-by: Adam Howes <[email protected]>

Update NEWS.md

292cfb3

Co-authored-by: Adam Howes <[email protected]>

Update inst/stan/latent_model/functions.stan

ef17a1a

Co-authored-by: Adam Howes <[email protected]>

Update setup.R

0ce7617

seabbs approved these changes Dec 3, 2024

View reviewed changes

seabbs merged commit ce41f42 into main Dec 3, 2024
8 of 10 checks passed

seabbs deleted the cohort-model branch December 3, 2024 13:22

seabbs mentioned this pull request Dec 5, 2024

Cohort based model #221

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #221: Add marginal model #426

Issue #221: Add marginal model #426

athowes commented Nov 7, 2024 •

edited by seabbs

Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as outdated.

seabbs commented Dec 3, 2024 •

edited

Loading

This comment was marked as outdated.

seabbs left a comment

seabbs left a comment •

edited

Loading

Issue #221: Add marginal model #426

Issue #221: Add marginal model #426

Conversation

athowes commented Nov 7, 2024 • edited by seabbs Loading

Description

Checklist

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as outdated.

seabbs commented Dec 3, 2024 • edited Loading

This comment was marked as outdated.

seabbs left a comment

Choose a reason for hiding this comment

seabbs left a comment • edited Loading

Choose a reason for hiding this comment

athowes commented Nov 7, 2024 •

edited by seabbs

Loading

seabbs commented Dec 3, 2024 •

edited

Loading

seabbs left a comment •

edited

Loading