Model Selections for the Memory Profiling Task

[i-am-sijia]

This is for discussing and finalizing which models to choose for the memory profiling task. Memory profiling is part of the ActivitySim Phase 8 Interim tasks.

Original Criteria Defined in the Scope

• 3 models
• 1-zone system
• Full-scale
• Capture a variety of memory usage-relevant factors, such as # of TAZs, # of skims, size of the synthetic population(s)
• With chunking disabled
• With Sharrow implemented
• Executed on the same computing resources

Environment

Windows 10, 64-bit OS, 64 cores, 512 GB RAM

Suggestions

I have tabled the ActivitySim consortium example models I have looked into with their characteristics in this spreadsheet.

Suggest to select the following for memory profiling task:

1. prototype_arc
   • It has the most number of zones.
2. prototype_mwcog
   • It has transit pass subsidy, transit pass ownership, and telecommuting models that others don't.
3. prototype_sandag
   • It has the least number of synthetic households and population.

Other considerations:

• prototype_semcog is similar to prototype_mwcog.
• prototype_mtc has the smallest number of zones so that it might also be a good candidate for comparison.
• prototype_mtc_extended has vehicle type and vehicle allocation models that others don't.

Issues

Without chunking, even with Sharrow, I still cannot run full-scale prototype_mtc model with 100% sample, it needs more memory than 512 GB. I can run 25% sample without chunking. I foresee this issue persist in other models as well. Can we just revise the scope to profile a big enough sample (e.g. 25%) for the selected three models?

[joecastiglione]

• There are a very limited number of 1-zone examples that we can consider for this work.

• The MTC example has been our original and canonical example, so I think it should definitely be one of the models for testing.

• In developing the scope, the text re: a 512GB constraint was removed because we need to understand what is going on with is original and canonical example.

• I don't support testing with subsamples. It is really important that we run these as full scale examples. One of the reasons that we find ourselves dealing with these memory issues is because full scale examples have not been used enough in development and testing.

• I think ARC should be one of the examples, due to the large number of zones.

• I'm agnostic as to whether to use SANDAG or MWCOG as the third example

[i-am-sijia]

A few clarifications first:

• When I suggested running with smaller samples, I meant smaller samples of households, not zones. **We will always use the full size zones and skims for profiling.**
• One of our hypotheses is that the memory requirement scales linearly with household samples, so we don't necessarily need a 100% sample run for profiling. For example, we currently see the memory pattern in a 25% sample MTC run, with mandatory tour scheduling being the peak, and we expect a similar pattern for other runs. We can do a 5%, 10%, 20% run to confirm this hypothesis.
• We can try with virtual machines with larger RAMs for 100% runs, see if any fits.

My follow up question:

• @joecastiglione What is your hypothesis on things that a 100% sample run would capture but a smaller sample run would not? Do you expect some non-linearity with 100% sample?

On the call today, the group had expressed preferences on choosing:

• prototype_arc
• prototype_mwcog
• prototype_mtc_extended

[joecastiglione]

• We need to run a 100% sample so that we have a set of true benchmarks against which any future improvements will be measured.
• Use of 100% sample will also be a good data point for validating your hypothesis that memory requirement scales linearly with household samples
• We really should know how much memory is truly required to deploy full scale models, without chunking
• If we can't run full scale models on servers virtual or otherwise, it is a real problem and validates the importance of investigation

[wusun2]

Is this task the same as described in here: https://github.com/ActivitySim/activitysim/wiki/Future-Development#phase-8? If so, I didn't see the memory profiling is limited to 1-zone examples. Would it be possible to do one 2-zone example profiling?

[i-am-sijia]

Is this task the same as described in here: https://github.com/ActivitySim/activitysim/wiki/Future-Development#phase-8? If so, I didn't see the memory profiling is limited to 1-zone examples. Would it be possible to do one 2-zone example profiling?

That is outdated. Here is the fully executed SOW: https://github.com/ActivitySim/activitysim/wiki/Phase-8-Interim-Scope-of-Work. I recall at the time the scope was developed, only Sharrow 1-zone was ready and reviewed, that's why 2-zone is not part of the scope.

[jpn--]

• The prototype_mtc is our "canonical example", and the prototype_mtc_extended version includes the vehicle type model that is slightly atypical and not yet included in any other model, so it's a good choice. It also has the smallest zone system, which will reduce memory stress for skims, so we'll have more space to see what's going on with everything else.
• I am highly skeptical that the ARC model can be run without chunking on any machine this consortium could reasonably afford... AWS might have virtual servers with 10s of TB of RAM but they'll cost probably thousands of dollars to do a single model run.
• There's little to learn from profiling a full population. A reasonably sample, run with full skims, should tell us everything we need to know.

[guyrousseau]

Quick question for the memory profiling tests, will shadow pricing be turned off?

With the ARC model, a possible alternative to a "Run Without Chunking" could perhaps be "Train Chunking", where one would run the model once in training mode. After the model runs once in training mode, then run in production mode. I seem to remember something similar was performed with our ARC ActivitySim implementation back on November 19, 2021, which resulted in the attached. I think the settings.yaml were something like:

num_processors: 38
chunk_size: 75% of available memory (example, 50_000_000_000 for 50Gb)
chunk_training_mode: training
chunk_method: hybrid_uss

num_processors: 38
chunk_size: 75% of available memory (example, 50_000_000_000 for 50Gb)
chunk_training_mode: production
chunk_method: hybrid_uss

activitysim.log
mp_households_16-activitysim.log

[joecastiglione]

• For the purposes of this investigation, we don't need to iterate shadow pricing, but I don't think we want shadow pricing turned off.
• I do think we need to do these tests with chunking turned off

[joecastiglione]

• After considerable group discussion and consultant input, the scope was written to explicitly include full scale examples, with chunking disabled. I don't feel comfortable changing from this scope before the work has really even begun.
• That we can't run (full-scale, un-chunked) two of our primary examples on "any machine this consortium could reasonably afford" is a failure of this project, in my opinion.