Move time-sync to builtin feature

[zanete]

Why: Sub of #763
What: Refactor TimeSync so that it is an IF builtin feature and not a plugin. It accepts config from the top level manifest context and is automatically executed after regroup if the config exists.

Blocked by:

• Implement observe, regroup and compute as distinct execution phases #809

Scope of work:

• Refactor TimeSync so that it is an IF core feature rather than a plugin
• Config is set at the top level (i.e. the context portion of a manifest) instead of in the plugin initialize field
• IF recognizes that TimeSync config exists and automatically executes between regroup and compute

Acceptance criteria

• Time Sync is an IF builtin feature
  GIVEN TimeSync is moved
  WHEN I browse the IF repository
  THEN I should NOT find /src/builtins/time-sync.ts but I SHOULD find src/lib/time-sync.ts

• Time Sync does not require intiializing in a manifest, and its config comes from manifest context

  GIVEN TimeSync is moved to a core feature
  WHEN I run if-run -m demo.yml
  AND demo.yml contains the following:

name: time-sync
description: successful path
tags:
time-sync:
    start-time: '2023-12-12T00:00:00.000Z'
    end-time: '2023-12-12T00:01:00.000Z'
    interval: 5
    allow-padding: true
initialize:
  plugins:
    sum:
      method: Sum
      path: "builtin"
      global-config:
        input-parameters: ["energy-cpu", "energy-mem"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        observe:
        regroup:
        compute:
          - sum
      config:
      inputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 1
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:01.000Z'
          duration: 5
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          energy-cpu: 0.001
          energy-mem: 0.001

THEN it runs successfully and the output is:

name: time-sync
description: successful path
tags: null
time-sync:
    start-time: '2023-12-12T00:00:00.000Z'
    end-time: '2023-12-12T00:01:00.000Z'
    interval: 5
    allow-padding: true
initialize:
    sum:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - energy-cpu
          - energy-mem
        output-parameter: energy
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/uesr/Code/if/src/index.ts -m manifests/plugins/time-sync/success.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-107-generic
    node-version: 21.4.0
    date-time: 2024-06-17T08:23:25.105Z (UTC)
    dependencies:
      - '@babel/[email protected]'
      - '@babel/[email protected]'
      - '@commitlint/[email protected]'
      - '@commitlint/[email protected]'
      - '@grnsft/[email protected]'
      - '@jest/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - '@types/[email protected]'
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
      - [email protected]
  status: success
tree:
  children:
    child:
      pipeline:
        observe:
        regroup:
        compute:
          - sum
      config: null
      inputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 1
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:01.000Z'
          duration: 5
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          energy-cpu: 0.001
          energy-mem: 0.001
      outputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 5
          energy-cpu: 0.0018000000000000004
          energy-mem: 0.0018000000000000004
          energy: 0.0036000000000000008
        - timestamp: '2023-12-12T00:00:05.000Z'
          duration: 5
          energy-cpu: 0.0007714285714285716
          energy-mem: 0.0007714285714285716
          energy: 0.0015428571428571431
        - timestamp: '2023-12-12T00:00:10.000Z'
          duration: 5
          energy-cpu: 0.0004952380952380952
          energy-mem: 0.0004952380952380952
          energy: 0.0009904761904761905
        - timestamp: '2023-12-12T00:00:15.000Z'
          duration: 5
          energy-cpu: 0.0001666666666666667
          energy-mem: 0.0001666666666666667
          energy: 0.0003333333333333334
        - timestamp: '2023-12-12T00:00:20.000Z'
          duration: 5
          energy-cpu: 0.0001666666666666667
          energy-mem: 0.0001666666666666667
          energy: 0.0003333333333333334
        - timestamp: '2023-12-12T00:00:25.000Z'
          duration: 5
          energy-cpu: 0.0001666666666666667
          energy-mem: 0.0001666666666666667
          energy: 0.0003333333333333334
        - timestamp: '2023-12-12T00:00:30.000Z'
          duration: 5
          energy-cpu: 0.0001666666666666667
          energy-mem: 0.0001666666666666667
          energy: 0.0003333333333333334
        - timestamp: '2023-12-12T00:00:35.000Z'
          duration: 5
          energy-cpu: 0.0001666666666666667
          energy-mem: 0.0001666666666666667
          energy: 0.0003333333333333334
        - timestamp: '2023-12-12T00:00:40.000Z'
          duration: 5
          energy-cpu: 0.0001
          energy-mem: 0.0001
          energy: 0.0002
        - timestamp: '2023-12-12T00:00:45.000Z'
          duration: 5
          energy-cpu: 0
          energy-mem: 0
          energy: 0
        - timestamp: '2023-12-12T00:00:50.000Z'
          duration: 5
          energy-cpu: 0
          energy-mem: 0
          energy: 0
        - timestamp: '2023-12-12T00:00:55.000Z'
          duration: 5
          energy-cpu: 0
          energy-mem: 0
          energy: 0
        - timestamp: '2023-12-12T00:01:00.000Z'
          duration: 1
          energy-cpu: 0
          energy-mem: 0
          energy: 0

• TimeSync checks that there are no duplicate timestamps in each time series
  GIVEN the check is implemented
  WHEN I run the following manifest (first timestep is duplicated)

name: time-sync
description: successful path
tags:
time-sync:
      start-time: '2023-12-12T00:00:00.000Z'
      end-time: '2023-12-12T00:01:00.000Z'
      interval: 5
      allow-padding: true
initialize:
  plugins:
    sum:
      method: Sum
      path: "builtin"
      global-config:
        input-parameters: ["energy-cpu", "energy-mem"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        observe:
        regroup:
        compute:
          - sum
      config:
      inputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 1
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 1
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          energy-cpu: 0.001
          energy-mem: 0.001
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          energy-cpu: 0.001
          energy-mem: 0.001

THEN TimeSync throws an exception , such as TimeSeriesError: Duplicaste timestamps exist in <path-to-node>

• Time Sync executes immediately after regroup and immediately before compute
  GIVEN The changes are implemented
  WHEN I run a manifest that includes valid TimeSync config and regroup config in the context
  THEN IF should execute observe then regroup then timesync then compute

e.g. this manifest should trigger the intended behaviour

name: time-sync
description: successful path
tags:
time-sync:
      start-time: '2023-12-12T00:00:00.000Z'
      end-time: '2023-12-12T00:01:00.000Z'
      interval: 5
      allow-padding: true
initialize:
  plugins:
    sum:
      method: Sum
      path: "builtin"
      global-config:
        input-parameters: ["energy-cpu", "energy-mem"]
        output-parameter: "energy"
tree:
  children:
    child:
      pipeline:
        observe:
        regroup:
          - cloud/instance-type
        compute:
          - sum
      config:
      inputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 1
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: A1
        - timestamp: '2023-12-12T00:00:01.000Z'
          duration: 5
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: A1
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: A1
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: A1
        - timestamp: '2023-12-12T00:00:00.000Z'
          duration: 1
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: B1
        - timestamp: '2023-12-12T00:00:01.000Z'
          duration: 5
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: B1
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: B1
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          energy-cpu: 0.001
          energy-mem: 0.001
          cloud/instance-type: B1

[zanete]

@manushak @narekhovhannisyan please review ac to confirm you're ok with it

[zanete]

note that there is a dependency in #842 which will mean that during the implementation of this feature we need to review the aggregate feature and remove temp workaround code

[zanete]

expecting a PR today

[zanete]

need to fix the issues reported during the review

[zanete]

@manushak @MariamKhalatova please take a look at the linked PR 🙏

[zanete]

as discovered by running converted manifests for phased execution, it turns out that having this as a built in feature causes a lot of unwanted side effects for other plugins and to address all of them would require changing multiple plugins. the safest option for now is to revert back to it being a plugin.
blocked until we understand what is really needed, currently phased execution is being reworked to support this being a plugin.
cc @jmcook1186

[zanete]

As I understood from @jawache this one can be backlogged for the time being until we have more experience making real life manifests and better awareness of the related frictions