2024-12-05 - Green Software Playbooks agenda

[bryaki02]

Date

2024-12-05 - 15:00 UTC - See the time in your timezone https://everytimezone.com

Roll Call

Please add a comment to this issue during the meeting to denote attendance.
Any untracked attendees will be added by the GSF team below:

• Full Name, Affiliation, (optional) GitHub username

Previous Meeting

Notes from the previous meeting:

• Discussed the process for capturing the playbook instructions for the Data Engineer Directory

Agenda

• Convene & Roll Call
• Review submissions since last meeting
• Plan for new year to attract new contributors
• Review the agenda and suggest new agenda points
• [Agenda Item]
• AOB, Q&A & Adjourn

Any Other Business

[moin-oss]

I'm going to be a few minutes late to the meeting.

[f-mellinghoff]

Create a suggestion for : "Move archived data to appropriate storage (maybe cold storage is enough for some archived data)"

[moin-oss]

Will add a writeup on "minimizing the frequency of batch jobs" and determine how much this overlaps with "only load data where changes occurred, maybe think about event based triggers (only load delta, but also only start followup ETL processes, if some source data changed)"

[f-mellinghoff]

Green Software Playbooks – Data Engineering

Improvements to existing projects

Move data to appropriate storage (hot vs. cold storage)

Analyse your existing data and move all rarely accessed data which still needs to be stored (e.g. for compliance or legal reasons) to cold storage.
By default, most data is stored in hot storage which is typically meant for frequently used data where fast and reliable access is needed. But often projects also include data which must only be archived and will not be accessed regularly any longer. For this data cold storage with a slower access to the data is sufficient and it should be moved accordingly.
Establish a continuous process to judge if data can be moved to cold storage or needs to be kept in hot storage.

Green IT Advantages: By moving data to cold storage not only will the storage costs be reduced (though access costs increase), but it will also save energy as the servers on which the data is stored do not need to constantly be available.

Considerations during setup of a new project

Move data to appropriate storage (hot vs. cold storage)

Define a data strategy from the beginning of your project which defines which data storage option (hot vs. cold) should be used for which data.
Data stored in hot storage typically needs to be accessed frequently whereas cold storage data is not accessed regularly.
Establish a continuous process to judge if data can be moved to cold storage or needs to be kept in hot storage.

Green IT Advantages: By moving data to cold storage not only will the storage costs be reduced (though access costs increase), but it will also save energy as the servers on which the data is stored do not need to constantly be available.