Non null flat line joining when no data

[Thlerry]

Describe the bug
Graph joins non null values, instead of showing zero / null.
I had no power for several hours today between 7am and 12:30pm roughly. So I would have expected the graph to show a dip at zero during that time, but it seems plotly joins the last known value, with the first new one, even several hours apart. Is there a way to prevent this, and show the fact that there was no value during that time, and keep the graph at zero ?
See screenshot below

Screenshots

yaml

type: custom:plotly-graph
entities:
  - entity: sensor.tb_house_consumption_kw
    name: Grid
    fill: tozeroy
    line:
      color: '#77aaff'
      shape: spline
    show_value: false
    extend_to_present: false
    on_legend_click: {}
    on_click: {}
    on_dblclick: {}
    filters:
      - resample: 10m
  - entity: sensor.tb_solaredge_ac_power
    name: Solar
    fill: tozeroy
    line:
      shape: spline
      color: '#ff8800'
    show_value: false
    extend_to_present: false
    on_legend_click: {}
    on_click: {}
    on_dblclick: {}
    filters:
      - resample: 10m
  - entity: sensor.tb_house_solar_only_consumption_kw
    name: Self
    fill: tozeroy
    line:
      color: '#00eeee'
      shape: spline
    show_value: false
    extend_to_present: false
    on_legend_click: {}
    on_click: {}
    on_dblclick: {}
    filters:
      - resample: 10m
hours_to_show: current_day
refresh_interval: 1
config:
  scrollZoom: false
  displayModeBar: false
  staticPlot: true

Additional context
When this happened, I had no electricity during the gap, so HA was not running at all, so nothing should have been recorded in the DB ?

[Thlerry]

In the console I see a lot of the following errors:

[Thlerry]

I clear the console, and did a refresh and this is what I see:

[Thlerry]

Not sure how to get the information you need to debug ?
Would you be able to help ?

[dbuezas]

Check out this section: https://github.com/dbuezas/lovelace-plotly-graph-card#debugging

With it you can dump the data to the console for analysis

[dbuezas]

The issue may be the resample filter

[dbuezas]

I looked at my implementation of resampling, and it should actually add a bunch of nulls, but try without it to narrow this down :)

[Thlerry]

With resampling:

Without Resampling:

[Thlerry]

here is the graph with nofill / no resampling. This looks correct.

[Thlerry]

And this is nofill but with resampling.
So it seems one issue is with the resampling, and one issue with the fill option ?

[dbuezas]

I see. Maybe you can map all the null values to the number zero before resampling?

[dbuezas]

This would give a reasonable result after resampling, even if you use fill

[Thlerry]

Is there an easy way to do that, maybe with a filter ?

Would that work ?

filters:
  - map_y_numbers: 'y == null ? 0 : y'
  - resample: 2m

[dbuezas]

Yes,
But don't use map_y_numbers, that one will already eliminate the nulls

Use map_y instead

[Thlerry]

OK, I gave it a go, but still not working.

Here is a simplified graph with various options. Any option used in filters: seems to mess up the line / fill.

[dbuezas]

Try

map_y: isNaN(+y) ? 0 : y

Maybe y is "unavailable" or something like that.
You may want to check out the "debugging" section in the readme to see the data in the console. Way easier to debug these kinds of issues.

BTW, map_y_numbers is expected to remove nulls, since they are not numbers

edit: fixed typo isNan => isNaN

[Thlerry]

I am getting this error with isNan function ?

[Thlerry]

I think checking against null or undefined works because the graph changes.
It starts from 0 after the gap, but still links the last valid value to 0. (See below)

How do you use the debugger to check further ? I looked at the debugging section but not sure what to do in the debugger (not used to it)

[Thlerry]

Were you able to reproduce it ?
You can take any sensor, stop HA for a few minutes to create a gap in values, and restart HA. Pretty easy.

[dbuezas]

just a typo, I updated my message above

[dbuezas]

Oh I see, you are right, it is working. So it is not that the sensor has "unavailable" datapoints, they are just missing. In this case, joining the lines is actually reasonable behaviour. What you can do is add an extra fake datapoint

filters:
  - fn: |- 
      ({xs, ys}) => ({
          xs: xs.flatMap((x, i) => ys[i+1] === null ? [x,x]:[x]),
          ys: ys.flatMap((y, i) => ys[i+1] === null ? [y,null]:[y])
      })

This will insert an extra undefined to the ys array, but set its date to that of the last known value. This should make the gap as wide as the whole missing chunk

[dbuezas]

Edit for future people: add the mapping from undefined to zero after this filter

[Thlerry]

I tried to add the fn above, but the syntax doesn't seem correct, and HA keeps spinning.
I tried to put everything on the same line, and still doesn't work:

And the multiline version not working:

[dbuezas]

Add an extra indentation (2 spaces) to all lines inside -fn

[dbuezas]

(ok, that's equivalent to the single line)
Now add the extra filter that maps undefineds to zero. You now only have the "filll" issue, which isn't a bug bit just how plotly behaves

[Thlerry]

I tried but this is still not working as expected:

[dbuezas]

Sorry the value is null, not undefined. I updated the comment marked as answer above (use === null instead). It will also fix the wrong drop on the last datapoint

[Thlerry]

That looks much better ! Thank you !
Just a last question, in the comparison to null what's the difference between '==' and '===' operator? both seem to work ?

[dbuezas]

That's a JavaScript thing.

• === means same type and value
• == means _convert operands to same type and then compare`.

The problem in this case is that 0==false==undefined==null=="", so the comparison will match:

• The end of the array (since ys[i+1] is undefined when i is already the index of the last element), and
• The value of the sensor actually being zero

And we don't want that.

If you want a headache, here are the details: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Equality_comparisons_and_sameness

[Thlerry]

Thanks again for your help and explanations !

[citoki]

Interesting topic @Thlerry , I've also experimented with Plotly Graphs for SolarEdge system here: #320

[thoath2]

Hello. I have a similar problem. My heat pump failed on December 15th, 2024 from 6:00 a.m. to 2:00 p.m. and did not deliver any values, but a line was displayed. I have now configured the filter you recommended (entity: "Sensor.wp_aussentemperature"), but it still does not work. I do something wrong.
Here is the standard-diagramm of HA:

And this is plotly-Graph:

And here is the yaml-code:

type: custom:plotly-graph
title: Temperaturen (WP-Allgemein)
defaults:
  yaxes:
    fixedrange: true
hours_to_show: current_day
refresh_interval: 10
min_y_axis: -50
max_y_axis: 10000
entities:
  - entity: sensor.wp_aussentemperatur
    name: |
      $ex {
            return (
                    "Außen "
                    + "<span style='font-size: 16px; color: rgb(154 205 50)'>" 
                    + ys[ys.length - 1].toFixed(1) 
                    +  " °C   </span>"
                   )
          }
    showlegend: true
    statistic: mean
    filters:
      - fn: |-
          ({xs, ys}) => ({
              xs: xs.flatMap((x, i) => ys[i+1] === null ? [x,x]:[x]),
              ys: ys.flatMap((y, i) => ys[i+1] === null ? [y,null]:[y])
          })
    extend_to_present: false
    period: 5minute
    fill: null
    fillcolor: rgba(154, 205, 50, 0.5)
    line:
      color: rgba(154, 205, 50, 1)
      width: 2
  - entity: sensor.wmz_solar_forward_water_temperature
    name: |
      $ex {
            return (
                    "Solar-Vorlauf "
                    + "<span style='font-size: 16px; color: rgb(255 165 0)'>" 
                    + ys[ys.length - 1].toFixed(1) 
                    +  " °C   </span>"
                   )
          }
    showlegend: true
    stackgroup: null
    statistic: mean
    filters:
      - map_y: "y < 0 ? 0 : (1 * y)"
    extend_to_present: false
    period: 5minute
    fill: null
    fillcolor: rgba(255, 165, 0, 0.8)
    line:
      color: rgba(255, 165, 0, 1)
      width: 2
  - entity: sensor.wp_hp1_flow_line_temperature
    name: |
      $ex {
            return (
                    "WP-Vorlauf "
                    + "<span style='font-size: 16px; color: rgb(255 69 0)'>" 
                    + ys[ys.length - 1].toFixed(1) 
                    +  " °C   </span>"
                   )
          }
    showlegend: true
    statistic: mean
    filters:
      - map_y: "y < 0 ? 0 : (1 * y)"
    extend_to_present: false
    period: 5minute
    fill: null
    fillcolor: rgba(255, 69, 0, 0.5)
    line:
      color: rgba(255, 69, 0, 1)
      width: 2
  - entity: sensor.wp_hp1_return_line_temperature
    name: |
      $ex {
            return (
                    "WP-Rücklauf "
                    + "<span style='font-size: 16px; color: rgb(70 130 180)'>" 
                    + ys[ys.length - 1].toFixed(1) 
                    +  " °C   </span>"
                   )
          }
    showlegend: true
    statistic: mean
    filters:
      - map_y: "y < 0 ? 0 : (1 * y)"
    extend_to_present: false
    period: 5minute
    fill: null
    fillcolor: rgba(70, 130, 180, 0.5)
    line:
      color: rgba(70, 130, 180, 1)
      width: 2
layout:
  height: 400
  shapes:
    - type: line
      name: |
        $ex {
              var aktDate = new Date(); //Date.now();
              var aktHour = aktDate.getHours();
              var strAktHour = "0" + aktHour 
              strAktHour = strAktHour.substr(strAktHour.length - 2)
              var aktMinute = aktDate.getMinutes();
              var strAktMinute = "0" + aktMinute 
              strAktMinute = strAktMinute.substr(strAktMinute.length - 2)
              var strHM = strAktHour + ":" + strAktMinute
              return (
                      "Akt. Zeitpunkt "                    
                      + [strHM] + " "                   
                     )
            }
      showlegend: true
      x0: $ex Date.now()
      x1: $ex Date.now()
      y0: 0
      y1: 1
      xref: x
      yref: paper
      line:
        color: grey
        dash: dot
        width: 2

Have you an idea, what i make wrong?