Demonstrating Python Memory Allocator Reference Counting (and Related)

Demonstrating Python memory allocator reference counting and debug. The goal of this repository is to help demonstrate Python memory allocation using source controlled code. Work for this project was originally inspired by cytomining/CytoTable#75 which explores related topics.

Development

The following are suggested steps to get started with development for this project.

1. (Suggested) Install Python from pyenv (or another way).
2. Install Poetry
3. Install Docker Desktop
4. Run Poe the Poet workflow(s): e.g. poetry run poe run_all_tests (Poe the Poet is installed as a Poetry env dependency for the runner group)

Project outline

---
title: Project Overview
---
flowchart LR

pyproject["pyproject.toml"]
poeworkflow["poethepoet\nworkflows"]
dockerfile["Dockerfile"]
subgraph test_container["test container"]
    subgraph tests
        test_env["Python environment\n(for tests)"]
        test_modules["Test module(s)"]
    end
end
subgraph runner
    runner_env["Python environment\n(for runner)"]
    runner_cli["Runner CLI\n(Python Fire)"]
    pipeline["Container pipeline\n(through Dagger)"]
end


pyproject --> |defines| test_env
pyproject --> |defines| runner_env

test_env --> |runs| test_modules
test_modules -.-> |display\nresults through| pipeline
runner_env --> |enables| runner_cli
runner_cli --> |runs| pipeline

dockerfile -.-> |defines\ncontainer for| pipeline

pipeline --> |executes| test_container

pyproject --> poeworkflow
poeworkflow -.-> |declarative\nworkflows for| runner

Loading

See above for a quick overview of project components and their relationship. Poetry is used to define Python environment dependencies within dependency groups in a pyproject.toml file. Declarative Poe the Poet tasks may also be found in the same pyproject.toml file to help define reproducible workflows. A "runner" command-line interface (CLI) is provided through Python Fire to help enable the use of the container-based pipelines. Container-based pipelines are provided through Dagger's Python SDK to help isolate potential OS-specific distinctions for memory allocation work in Python. Testing workflows are designed to run "locally" within a developer's environment (for example, leveraging pyenv, poetry, and Docker Desktop) or within GitHub Actions images (dagger-io installs the necessary dependencies).

Test Modules

This project focuses on leveraging Python memory observability tools to illustrate what happens as code is executed. See the src/pymaccounter/tests folder for a full list of test modules. Each test module includes a description of what it tests and expects in a docstring near the top of the file.

Running Test Modules

Test modules may be executed individually or in groups. Test modules are provided in a list to be run by containerized pipelines. Each test provided in this way is run in an isolated container instance.

In addition to test module specification, a test module base directory and debug mode may also be specified. The test module base directory is where the container pipeline will look for test modules listed by name in the list. Debug mode may be used to view container pipeline debug log messages.

See the following examples for more details on the suggested way to run tests through this project.

• Individual test: poetry run python src/pymaccounter/runner.py '["test_baseline.py"]'
• Multiple tests: poetry run python src/pymaccounter/runner.py '["test_baseline.py", "test_multiply_gc.collect.py"]'
• Individual test with debug mode: poetry run python src/pymaccounter/runner.py '["test_baseline.py"]' --debug True
• Individual test with non-default base test directory specification: poetry run python src/pymaccounter/runner.py '["test_baseline.py"]' --test_dir 'src/another_test_dir'