Gaps in data when scrolling

[mwolter805]

Hello,
Great integration, thanks for your work on this! I've used plotly in the past and was excited to see it available for home assistant.

I'm attempting to create a bar chart for energy usage of several circuits in my house. I think I've cobbled it together well enough that it displays the info, however, there are several gaps in the data scrolling.

I've searched through the discussions and haven't found anything related to this. It appears the first period for the chart is lost and when scrolling it is never obtained. I've seen reference in other discussions regarding HA not data for the first period. Is there a way to fill the gaps filled?

Thanks again for all your work on this.

In the animation below, I scroll to an earlier time period, then click the 1 day button and the chart is missing bars for 7am, 4am and 1am

However, if I were to click the button for 1d directly, this data for 4am and 1am appears. Still missing 7am though.

Plotly YAML used to generate the graph above

type: custom:plotly-graph
hours_to_show: 6h
entities:
  - entity: sensor.emporia_vue_2_1_circuit_1_energy
    visible: legendonly
  - entity: sensor.emporia_vue_2_1_circuit_2_energy
  - entity: sensor.emporia_vue_2_1_circuit_3_energy
  - entity: sensor.emporia_vue_2_1_circuit_4_energy
  - entity: sensor.emporia_vue_2_1_circuit_5_energy
  - entity: sensor.emporia_vue_2_1_circuit_6_energy
  - entity: sensor.emporia_vue_2_1_circuit_7_energy
  - entity: sensor.emporia_vue_2_1_circuit_8_energy
  - entity: sensor.emporia_vue_2_1_circuit_9_energy
  - entity: sensor.emporia_vue_2_1_circuit_10_energy
  - entity: sensor.emporia_vue_2_1_circuit_11_energy
    visible: legendonly
  - entity: sensor.emporia_vue_2_1_circuit_12_energy
  - entity: sensor.emporia_vue_2_1_circuit_13_energy
  - entity: sensor.emporia_vue_2_1_circuit_14_energy
    visible: legendonly
fn: |
  $fn({getFromConfig, vars})=> {
    const range = getFromConfig("visible_range");
    const width = range[1] - range[0];
    vars.scroll = (label, p) => ({
      args: [
        {
          layout: {
            "xaxis.range": [range[0] + width*p, range[1] + width*p],
          }
        }, {
          transition: {
            duration: 150,
          }
        }
      ],
      label,
      method: "animate",
    })
    vars.zoom = (label, h) => ({
      args: [
        {
          layout: {
            "xaxis.range": [Date.now()-1000*60*60*h, Date.now()],
          }
        }
      ],
      label,
      method: "animate",
    })
  }
layout:
  height: 400
  barmode: group
  xaxis:
    autorange: true
  updatemenus:
    - buttons:
        - $fn({vars}) => vars.scroll( '<', -.5)
        - $fn({vars}) => vars.scroll( '>', .5)
      direction: right
      active: -1
      pad:
        r: 10
        t: 10
      type: buttons
      x: 1
      xanchor: right
      'y': 1
      yanchor: top
    - buttons:
        - $fn({vars}) => vars.zoom( '1y', 24*365)
        - $fn({vars}) => vars.zoom( '1m', 24*30)
        - $fn({vars}) => vars.zoom( '1w', 24*7)
        - $fn({vars}) => vars.zoom( '1d', 24)
        - $fn({vars}) => vars.zoom( '1h', 1)
      direction: right
      active: -1
      pad:
        r: 10
        t: 10
      type: buttons
      x: 0
      xanchor: left
      'y': 1
      yanchor: top
defaults:
  entity:
    statistic: state
    filters:
      - fn: |-
          (params) => {
            const ys = [];
            ys.push(0); 
            for (let i = 1; i < params.statistics.length; i++){
              if (params.statistics[i-1].state > params.statistics[i].state) {
                ys.push(params.statistics[i].state);
              } else {
                ys.push(params.statistics[i].state-params.statistics[i-1].state);
              }
            };
            return { ys };
          }
      - filter: i>0
    type: bar
    width: $fn () => 60*60*60*1
    period:
      6h: hour
      7d: day
      6M: month
  yaxes:
    fixedrange: true
autorange_after_scroll: true
config:
  scrollZoom: false
  doubleClickDelay: 600

[dbuezas]

That's odd. There used to ve a bug in HA that would explain this, but it was fixed many months ago. Can you confirm your Home Assistant is up to date?

Happy to hear you are enjoying this integration! You're making good used of advanced features, solid!

On a side note, you may like $ex (from "expression") which is the same as $fn but you can skip the parameters and arrow (internally those ar added ant then converted into a $fn). Particularly for small functions like width: $fn () => 60*60*60*1 which becomes width: $ex 60*60*60*1.

[mwolter805]

Hello @dbuezas,
Yes, using the most recent HA version 2023.9.3.

[mwolter805]

@dbuezas would it be best if this was converted to an issue? Is there any more data/logs that should be included?

[dbuezas]

Yes, this looks like a bug. Potentially in HA, but I can't be sure.

I won't be able to work on this for a while though

[deltamelter]

I'm noticing a lot of "random" gaps, even without scrolling.
This series is hourly data points so should be equally spread and no gaps but I see bunches of columns followed by gaps like this.
when the data is from history the gaps stay. When looking at future data, the gaps seem to "move" around.

[dbuezas]

Oh, that's weird. I assume this is a bar chart and they are so thin that plotly doesn't render them all.
Try removing the first one and resampling so the bars are wider

- entity
  filters:
     - filter: "i>1"
     - resample: 10m

[deltamelter]

set resample to 60m because that's the normal base period, seems to have ignored the stacking groups now though.
At least most of the gaps and clumps are gone now and the bar width is closer to what I wanted 😁

[dbuezas]

That looks much better!
Are you sure the stacking groups stopped working? it may well be that they now work better because the dates match exactly.

[deltamelter]

maybe I misunderstood the function of stacking groups but I set up as stackgroup: one and stackgroup: two so I expected them not to stack as they are series from different groups.

[dbuezas]

Maybe that's only for filled line chart

[mwolter805]

@dbuezas the resample filter appears to have helped my issue as well. With the resample set to 60m and zoom set at 1d the graph accurately shows an entire day (nearly) in hour increments.

However, my goal is to use the buttons to change the zoom and scroll and resample doesn't appear to be honoring the period specified when the zoom changes.

Is there a way to have the resample change for each zoom button level? Would like the resample filter to honor the sum to the periods specified

Thanks

Correctly Displaying 1d in hour increments (periods)

Incorrectly Displaying 1w (appears to be displaying 1h periods)

Plotly YAML used to generate the graph above

type: custom:plotly-graph
hours_to_show: 24h
entities:
  - entity: sensor.emporia_vue_2_1_circuit_1_energy
    visible: legendonly
  - entity: sensor.emporia_vue_2_1_circuit_2_energy
  - entity: sensor.emporia_vue_2_1_circuit_3_energy
  - entity: sensor.emporia_vue_2_1_circuit_4_energy
  - entity: sensor.emporia_vue_2_1_circuit_5_energy
  - entity: sensor.emporia_vue_2_1_circuit_6_energy
  - entity: sensor.emporia_vue_2_1_circuit_7_energy
  - entity: sensor.emporia_vue_2_1_circuit_8_energy
  - entity: sensor.emporia_vue_2_1_circuit_9_energy
  - entity: sensor.emporia_vue_2_1_circuit_10_energy
  - entity: sensor.emporia_vue_2_1_circuit_11_energy
    visible: legendonly
  - entity: sensor.emporia_vue_2_1_circuit_12_energy
  - entity: sensor.emporia_vue_2_1_circuit_13_energy
  - entity: sensor.emporia_vue_2_1_circuit_14_energy
    visible: legendonly
fn: |
  $fn({getFromConfig, vars})=> {
    const range = getFromConfig("visible_range");
    const width = range[1] - range[0];
    vars.xrange = (h) => {
      if (h === null) h = width/1000/60/60;
      let start = new Date((range[0] + range[1])/2.0);
      start.setHours(0, 0, 0, 0);
      if (h >= 24*7) start.setFullYear(start.getFullYear(), start.getMonth(), start.getDate()-start.getDay());
      if (h >= 24*30) start.setDate(1);
      if (h >= 24*365) start.setMonth(0);
      return [start.getTime(), start.getTime() + 1000*60*60*h];
    }
    vars.scroll = (label, p) => ({
      args: [
        {
          layout: {
            "xaxis.range": [range[0] + width*p, range[1] + width*p],
          }
        }, {
          transition: {
            duration: 150,
          }
        }
      ],
      label,
      method: "animate",
    })
    vars.zoom = (label, h, w) => ({
      args: [
        {
          layout: {
            "xaxis.range": vars.xrange(h),
          }
        }, {
          defaults: {
            entity: {
              "width": [w],
              "statistic": 'state'
            }
          }
        }
      ],
      label,
      method: "animate",
    })
  }
layout:
  height: 400
  barmode: group
  xaxis:
    autorange: true
  updatemenus:
    - buttons:
        - $fn({vars}) => vars.scroll( '<', -.5)
        - $fn({vars}) => vars.scroll( '>', .5)
      direction: right
      active: -1
      pad:
        r: 10
        t: 10
      type: buttons
      x: 1
      xanchor: right
      'y': 1
      yanchor: top
    - buttons:
        - $fn({vars}) => vars.zoom( '1y', 24*365, 60*60*60*1)
        - $fn({vars}) => vars.zoom( '1m', 24*30, 60*60*60*1)
        - $fn({vars}) => vars.zoom( '1w', 24*7, 60*60*60*10)
        - $fn({vars}) => vars.zoom( '1d', 24, 60*60)
        - $fn({vars}) => vars.zoom( '1h', 1, 60*60*60*100)
      direction: right
      active: -1
      pad:
        r: 10
        t: 10
      type: buttons
      x: 0
      xanchor: left
      'y': 1
      yanchor: top
defaults:
  entity:
    statistic: state
    filters:
      - fn: |-
          (params) => {
            const ys = [];
            ys.push(0); 
            for (let i = 1; i < params.statistics.length; i++){
              if (params.statistics[i-1].state > params.statistics[i].state) {
                ys.push(params.statistics[i].state);
              } else {
                ys.push(params.statistics[i].state-params.statistics[i-1].state);
              }
            };
            return { ys };
          }
      - filter: i>0
      - resample: 60m
    type: bar
    period:
      1h: 5minute
      6h: hour
      1d: hour
      7d: day
      1M: week
      1y: month
  yaxes:
    fixedrange: true
autorange_after_scroll: true
config:
  scrollZoom: false
  doubleClickDelay: 600

[dbuezas]

perfect! you could use

const range = get('visible_range');
const time_span = range[1]-range[0];

so it updates when you zoom in and out or scroll

[dbuezas]

Also here's a nice more clear pattern to avoid those numbers like 2592000000

const range = get('visible_range');
const time_span = range[1] - range[0];
const days = 1000 * 60 * 60 * 24;

if (time_span >= 30 * days) return "1M";
if (time_span >= 1 * days) return "1d"; 
return "1h";

[dbuezas]

Wait, why are you using both a custom adaptive period AND a resample?

[CendaL]

I thought it would solve my issue with missing data but I was wrong.

I'm saving data from couple of power meters into variables and then subtracting them from total consumption to see "other" consumption. Sometimes this "other meter" is wrong and I have a graph where I can see these mistakes - basically compare saved "other meter" value with again calculated one. I'm checking it for hourly, daily and monthly statistics. The problem starts when I add a new power meter or when some data is missing - then dates on x axis are not the same and resample does not help. I will probably have to do manual adjustment to have the same data points for all meters.

[dbuezas]

Oh, yes, I need some sort of an align filter