Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add_histogram and add_figure causes UnicodeDecodeError, TypeError and IndexError when trying to visualize in Tensorboard #6937

Open
Marius-Juston opened this issue Nov 15, 2024 · 1 comment
Open

add_histogram and add_figure causes UnicodeDecodeError, TypeError and IndexError when trying to visualize in Tensorboard #6937

Marius-Juston opened this issue Nov 15, 2024 · 1 comment

Comments

@Marius-Juston
Copy link

Marius-Juston commented Nov 15, 2024 •
edited
Loading

Environment information (required)

Using Windows 11 Pro Version 10.0.22631 Build 22631 running on WSL2 Ubuntu 22.04.05 with Python 3.11

/home/marius/apecs_library/diagnose_tensorboard.py:32: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
  import pipes

Diagnostics

Diagnostics output 
--- check: autoidentify
INFO: diagnose_tensorboard.py version b9865891333c4b6c896b078b5edaa7bbeaddb80f

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=11, micro=9, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='DESKTOP-IOUM5DH', release='5.15.153.1-microsoft-standard-WSL2', version='#1 SMP Fri Mar 29 23:14:13 UTC 2024', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
WARNING: no installation among: ['tb-nightly', 'tensorboard', 'tensorflow-tensorboard']
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.18.0'

--- check: tensorflow_python_version
Traceback (most recent call last):
  File "/home/marius/apecs_library/diagnose_tensorboard.py", line 501, in main
    suggestions.extend(check())
                       ^^^^^^^
  File "/home/marius/apecs_library/diagnose_tensorboard.py", line 81, in wrapper
    result = fn()
             ^^^^
  File "/home/marius/apecs_library/diagnose_tensorboard.py", line 265, in tensorflow_python_version
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

--- check: tensorboard_data_server_version
INFO: data server binary: '/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.7.0'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/marius/miniconda3/envs/APECS/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'DESKTOP-IOUM5DH.'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=77675, st_dev=2080, st_nlink=2, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1731652522, st_mtime=1731663189, st_ctime=1731663189)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py @ file:///croot/absl-py_1714140470852/work
action-msgs==1.2.1
action-tutorials-interfaces==0.20.5
action-tutorials-py==0.20.5
actionlib-msgs==4.2.4
actuator-msgs==0.0.1
alabaster==0.7.16
ament-cmake-test==1.3.10
ament-copyright==0.12.11
ament-cppcheck==0.12.11
ament-cpplint==0.12.11
ament-flake8==0.12.11
ament-index-python==1.4.0
ament-lint==0.12.11
ament-lint-cmake==0.12.11
ament-package==0.14.0
ament-pep257==0.12.11
ament-uncrustify==0.12.11
ament-xmllint==0.12.11
angles==1.15.0
apecs @ git+https://github.com/AUVSL/apecs_library.git@9ab6934997dfed2a01b31a65f3da5cad00197d54
apecs-cpp==0.0.0
Babel==2.15.0
backports.tarfile==1.2.0
black==24.10.0
Brotli @ file:///croot/brotli-split_1714483155106/work
build==1.2.1
builtin-interfaces==1.2.1
camera-calibration==3.0.6
certifi @ file:///croot/certifi_1725551672989/work/certifi
cffi==1.16.0
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
clarabel @ file:///home/conda/feedstock_root/build_artifacts/clarabel_1709397162848/work
click==8.1.7
cloudpickle @ file:///croot/cloudpickle_1683040006038/work
composition-interfaces==1.2.1
contourpy @ file:///croot/contourpy_1700583582875/work
cryptography==42.0.8
cv-bridge==3.2.1
cvxpy @ file:///home/conda/feedstock_root/build_artifacts/cvxpy-split_1724065180972/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
demo-nodes-py==0.20.5
diagnostic-msgs==4.2.4
docker-pycreds==0.4.0
docutils==0.20.1
domain-coordinator==0.10.0
ecos @ file:///home/conda/feedstock_root/build_artifacts/ecos_1725950139015/work
example-interfaces==0.9.3
examples-rclpy-executors==0.15.2
examples-rclpy-minimal-action-client==0.15.2
examples-rclpy-minimal-action-server==0.15.2
examples-rclpy-minimal-client==0.15.2
examples-rclpy-minimal-publisher==0.15.2
examples-rclpy-minimal-service==0.15.2
examples-rclpy-minimal-subscriber==0.15.2
Farama-Notifications @ file:///croot/farama-notifications_1683058702908/work
filelock @ file:///croot/filelock_1700591183607/work
fonttools @ file:///croot/fonttools_1713551344105/work
fsspec==2024.6.1
geometry-msgs==4.2.4
gitdb==4.0.11
GitPython==3.1.43
gmpy2 @ file:///work/ci_py311/gmpy2_1676839849213/work
gps-msgs==2.0.4
graphviz==0.20.3
grpcio @ file:///croot/grpc-suite_1701982571719/work
gymnasium @ file:///home/conda/feedstock_root/build_artifacts/gymnasium-split_1729640470784/work
idna @ file:///croot/idna_1714398848350/work
image-geometry==3.2.1
imagesize==1.4.1
importlib_metadata==8.0.0
interactive-markers==2.3.2
jaraco.classes==3.4.0
jaraco.context==5.3.0
jaraco.functools==4.0.1
jax-jumpy @ file:///croot/jax-jumpy_1683230527192/work
jeepney==0.8.0
Jinja2 @ file:///croot/jinja2_1706733616596/work
joblib @ file:///croot/joblib_1713976767370/work
keyring==25.2.1
kiwisolver @ file:///work/ci_py311/kiwisolver_1676827230232/work
laser-geometry==2.4.0
launch==1.0.6
launch-ros==0.19.7
launch-testing==1.0.6
launch-testing-ros==0.19.7
launch-xml==1.0.6
launch-yaml==1.0.6
lifecycle-msgs==1.2.1
logging-demo==0.20.5
map-msgs==2.1.0
Markdown @ file:///work/ci_py311/markdown_1676837083112/work
markdown-it-py==3.0.0
MarkupSafe @ file:///croot/markupsafe_1704205993651/work
matplotlib @ file:///croot/matplotlib-suite_1713336378214/work
mdit-py-plugins==0.4.1
mdurl==0.1.2
message-filters==4.3.5
mkl-fft @ file:///croot/mkl_fft_1695058164594/work
mkl-random @ file:///croot/mkl_random_1695059800811/work
mkl-service==2.4.0
more-itertools==10.3.0
mpmath @ file:///croot/mpmath_1690848262763/work
mypy-extensions==1.0.0
natsort @ file:///home/conda/feedstock_root/build_artifacts/natsort_1687263093793/work
nav-msgs==4.2.4
networkx @ file:///croot/networkx_1690561992265/work
nh3==0.2.18
numpy @ file:///croot/numpy_and_numpy_base_1708638617955/work/dist/numpy-1.26.4-cp311-cp311-linux_x86_64.whl#sha256=5f96f274d410a1682519282ae769c877d32fdbf171aa8badec7bf5e1d3a1748a
osqp @ file:///home/conda/feedstock_root/build_artifacts/osqp_1719722524380/work
osrf-pycommon==2.0.2
packaging @ file:///croot/packaging_1710807400464/work
pandas==2.2.2
pathspec==0.12.1
pcl-msgs==1.0.0
pendulum-msgs==0.20.5
pillow @ file:///croot/pillow_1714398848491/work
pip==23.3.1
pkginfo==1.10.0
platformdirs==4.2.2
ply==3.11
protobuf==3.20.3
psutil==6.0.0
pycparser==2.22
Pygments==2.18.0
pyparsing @ file:///work/ci_py311/pyparsing_1677811559502/work
pyproject_hooks==1.1.0
PyQt5==5.15.10
PyQt5-sip @ file:///croot/pyqt-split_1698769088074/work/pyqt_sip
PySocks @ file:///work/ci_py311/pysocks_1676822712504/work
python-dateutil @ file:///croot/python-dateutil_1715108784345/work
python-qt-binding==1.1.2
pytz==2024.1
PyYAML @ file:///croot/pyyaml_1698096049011/work
qdldl @ file:///home/conda/feedstock_root/build_artifacts/qdldl-python_1725950149104/work
qt-dotgraph==2.2.3
qt-gui==2.2.3
qt-gui-cpp==2.2.3
qt-gui-py-common==2.2.3
quality-of-service-demo-py==0.20.5
rcl-interfaces==1.2.1
rclpy==3.3.14
rcutils==5.1.6
readme_renderer==44.0
requests @ file:///croot/requests_1707355572290/work
requests-toolbelt==1.0.0
resource-retriever==3.1.2
rfc3986==2.0.0
rich==13.7.1
rmw-dds-common==1.6.0
ros-gz-interfaces==0.244.15
ros-ign-interfaces==0.244.15
ros2action==0.18.11
ros2bag==0.15.12
ros2cli==0.18.11
ros2component==0.18.11
ros2doctor==0.18.11
ros2interface==0.18.11
ros2launch==0.19.7
ros2lifecycle==0.18.11
ros2multicast==0.18.11
ros2node==0.18.11
ros2param==0.18.11
ros2pkg==0.18.11
ros2run==0.18.11
ros2service==0.18.11
ros2topic==0.18.11
rosbag2-interfaces==0.15.12
rosbag2-py==0.15.12
rosgraph-msgs==1.2.1
rosidl-adapter==3.1.5
rosidl-cli==3.1.5
rosidl-cmake==3.1.5
rosidl-generator-c==3.1.5
rosidl-generator-cpp==3.1.5
rosidl-generator-py==0.14.4
rosidl-parser==3.1.5
rosidl-runtime-py==0.9.3
rosidl-typesupport-c==2.0.1
rosidl-typesupport-cpp==2.0.1
rosidl-typesupport-fastrtps-c==2.2.2
rosidl-typesupport-fastrtps-cpp==2.2.2
rosidl-typesupport-introspection-c==3.1.5
rosidl-typesupport-introspection-cpp==3.1.5
rpyutils==0.2.1
rqt-action==2.0.1
rqt-bag==1.1.5
rqt-bag-plugins==1.1.5
rqt-console==2.0.3
rqt-graph==1.3.1
rqt-gui==1.1.7
rqt-gui-py==1.1.7
rqt-msg==1.2.0
rqt-plot==1.1.2
rqt-publisher==1.5.0
rqt-py-common==1.1.7
rqt-py-console==1.0.2
rqt-reconfigure==1.1.2
rqt-service-caller==1.0.5
rqt-shell==1.0.2
rqt-srv==1.0.3
rqt-topic==1.5.0
scikit-learn @ file:///croot/scikit-learn_1714164736727/work
scipy @ file:///croot/scipy_1714069776893/work/dist/scipy-1.13.0-cp311-cp311-linux_x86_64.whl#sha256=5ea9bb3db7de954ca26bfbeb18592fabbe71154f6bc567a9ad09df45dd38f54d
scs @ file:///home/conda/feedstock_root/build_artifacts/scs_1725944826688/work
SecretStorage==3.3.3
sensor-msgs==4.2.4
sensor-msgs-py==4.2.4
sentry-sdk==2.13.0
setproctitle==1.3.3
setuptools==68.2.2
shape-msgs==4.2.4
sip @ file:///croot/sip_1698675935381/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
smmap==5.0.1
snowballstemmer==2.2.0
sphinxcontrib-applehelp==1.0.8
sphinxcontrib-devhelp==1.0.6
sphinxcontrib-htmlhelp==2.0.6
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.8
sphinxcontrib-serializinghtml==1.1.10
sros2==0.10.5
statistics-msgs==1.2.1
std-msgs==4.2.4
std-srvs==4.2.4
stereo-msgs==4.2.4
sympy @ file:///croot/sympy_1724938189289/work
teleop-twist-keyboard==2.4.0
tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1727435705122/work/tensorboard-2.18.0-py3-none-any.whl#sha256=107ca4821745f73e2aefa02c50ff70a9b694f39f790b11e6f682f7d326745eab
tensorboard-data-server @ file:///croot/tensorboard-data-server_1681498183723/work/tensorboard_data_server-0.7.0-py3-none-manylinux2014_x86_64.whl
tf2-geometry-msgs==0.25.8
tf2-kdl==0.25.8
tf2-msgs==0.25.8
tf2-py==0.25.8
tf2-ros-py==0.25.8
tf2-tools==0.25.8
theora-image-transport==2.5.2
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
topic-monitor==0.20.5
torch==2.5.1
torch-tb-profiler==0.4.3
torchaudio==2.5.1
torchvision==0.20.1
torchviz==0.0.2
tornado @ file:///croot/tornado_1696936946304/work
trajectory-msgs==4.2.4
triton==3.1.0
turtlesim==1.4.2
twine==5.1.1
typing_extensions @ file:///croot/typing_extensions_1705599297034/work
tzdata==2024.1
unicodedata2 @ file:///croot/unicodedata2_1713212950228/work
unique-identifier-msgs==2.2.1
urllib3 @ file:///croot/urllib3_1715635851070/work
vision-msgs==4.1.1
visualization-msgs==4.2.4
wandb==0.17.8
Werkzeug @ file:///croot/werkzeug_1706210078083/work
wheel==0.43.0
xacro==2.0.8
zipp==3.19.2

Next steps

No action items identified. Please copy ALL of the above output,
including the lines containing only backticks, into your GitHub issue
or comment. Be sure to redact any sensitive information.

Issue description

I am trying to plot the weights of my network and illustrate it's distribution, histogram and show a heatmap of the weight over time to be able to validate some aspects of my network; however, I am getting issues displaying the data in Tensorboard tensorboard --logdir=test --load_fast=false (without the load_fast=false I would not anything at all) shows the following errors when you navigate to Histogram and Distribution tabs, the Image tabs does not show anything.
image

from typing import Optional

import matplotlib.pyplot as plt
import numpy as np
import torch
from matplotlib.axes import Axes
from torch import Tensor
from torch.utils.tensorboard import SummaryWriter

torch.set_float32_matmul_precision("high")

def generate_histogram(
    summary_writer: SummaryWriter, name: str, input_tensor: Tensor, epoch: int, n=100
):
    values = input_tensor.flatten().cpu().numpy()

    min_ = values.min().item()
    max_ = values.max().item()

    bins = np.linspace(min_, max_, n)

    counts, limits = np.histogram(values, bins=bins)
    sum_sq = values.dot(values)

    summary_writer.add_histogram_raw(
        tag=name,
        min=min_,
        max=max_,
        num=len(values),
        sum=values.sum().item(),
        sum_squares=sum_sq,
        bucket_limits=limits[1:].tolist(),
        bucket_counts=counts.tolist(),
        global_step=epoch,
    )


@torch.no_grad()
def plot_matrix(
    summary_writer: SummaryWriter,
    name: str,
    input_tensor: Tensor,
    epoch: int,
    index: int,
    sub_index: Optional[int] = None,
    image: bool = True,
):
    name = f"{index}/{sub_index}/{name}" if sub_index is not None else f"{index}/{name}"

    input_tensor = torch.zeros((64, 64)).uniform_(0, 1)

    print(name)

    # generate_histogram(summary_writer, name, input_tensor, epoch)
    summary_writer.add_histogram(name, input_tensor, global_step=epoch)

    if image:
        fig, ax = plt.subplots()
        ax: Axes
        ax.imshow(torch.atleast_2d(input_tensor).cpu())

        summary_writer.add_figure(name, fig, global_step=epoch)

    summary_writer.add_scalar(name, torch.linalg.norm(input_tensor), global_step=epoch)

    summary_writer.flush()

def main():
    summary_writer = SummaryWriter(log_dir="temp")

    w = torch.empty((64, 64)).uniform_(0, 1)

    plot_matrix(summary_writer, "W", w, 0, 0, 0, True)


if __name__ == "__main__":
    main()

Whenever I try to open the Distribution tab I get the following error:

E1115 03:31:15.958985 140704193963584 _internal.py:187] Error on request:
Traceback (most recent call last):
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/serving.py", line 364, in run_wsgi
    execute(self.server.app)
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/serving.py", line 325, in execute
    application_iter = app(environ, start_response)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 528, in __call__
    return self._app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 569, in wrapper
    return wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/security_validator.py", line 91, in __call__
    return self._application(environ, start_response_proxy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/path_prefix.py", line 68, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/experiment_id.py", line 73, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/empty_path_redirect.py", line 43, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/client_feature_flags.py", line 55, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/auth_context_middleware.py", line 38, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 551, in _route_request
    return self.exact_routes[clean_path](environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 188, in application
    resp = f(*args[:-2] + (request,))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/distribution/distributions_plugin.py", line 114, in distributions_route
    (body, mime_type) = self.distributions_impl(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/distribution/distributions_plugin.py", line 87, in distributions_impl
    [self._compress(histogram) for histogram in histograms],
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/distribution/distributions_plugin.py", line 87, in <listcomp>
    [self._compress(histogram) for histogram in histograms],
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/distribution/distributions_plugin.py", line 93, in _compress
    converted_buckets = compressor.compress_histogram(buckets)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/distribution/compressor.py", line 122, in compress_histogram
    counts = buckets[:, 2]
             ^^^^^^^^^^^^^
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

image

Whenever I try to open the Histogram tab I get the following error:

E1115 03:32:28.110896 139718423467584 _internal.py:187] Error on request:
Traceback (most recent call last):
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/serving.py", line 364, in run_wsgi
    execute(self.server.app)
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/serving.py", line 325, in execute
    application_iter = app(environ, start_response)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 528, in __call__
    return self._app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 569, in wrapper
    return wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/security_validator.py", line 91, in __call__
    return self._application(environ, start_response_proxy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/path_prefix.py", line 68, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/experiment_id.py", line 73, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/empty_path_redirect.py", line 43, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/client_feature_flags.py", line 55, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/auth_context_middleware.py", line 38, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 551, in _route_request
    return self.exact_routes[clean_path](environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 188, in application
    resp = f(*args[:-2] + (request,))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/histogram/histograms_plugin.py", line 146, in histograms_route
    return http_util.Respond(request, body, mime_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/http_util.py", line 150, in Respond
    json_util.Cleanse(content, encoding), ensure_ascii=not charset_match
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 64, in Cleanse
    return [Cleanse(i, encoding) for i in obj]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 64, in <listcomp>
    return [Cleanse(i, encoding) for i in obj]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 64, in Cleanse
    return [Cleanse(i, encoding) for i in obj]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 64, in <listcomp>
    return [Cleanse(i, encoding) for i in obj]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 64, in Cleanse
    return [Cleanse(i, encoding) for i in obj]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 64, in <listcomp>
    return [Cleanse(i, encoding) for i in obj]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/json_util.py", line 62, in Cleanse
    return obj.decode(encoding)
           ^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

and after much longer I get:

E1115 03:39:08.982144 139925135554112 _internal.py:187] Error on request:
Traceback (most recent call last):
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/serving.py", line 364, in run_wsgi
    execute(self.server.app)
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/serving.py", line 325, in execute
    application_iter = app(environ, start_response)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 528, in __call__
    return self._app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 569, in wrapper
    return wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/security_validator.py", line 91, in __call__
    return self._application(environ, start_response_proxy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/path_prefix.py", line 68, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/experiment_id.py", line 73, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/empty_path_redirect.py", line 43, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/client_feature_flags.py", line 55, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/auth_context_middleware.py", line 38, in __call__
    return self._application(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/backend/application.py", line 551, in _route_request
    return self.exact_routes[clean_path](environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 188, in application
    resp = f(*args[:-2] + (request,))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 393, in _serve_time_series
    response = self._time_series_impl(ctx, experiment, series_requests)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 407, in _time_series_impl
    responses = [
                ^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 408, in <listcomp>
    self._get_time_series(ctx, experiment, request)
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 479, in _get_time_series
    run_to_series = self._get_run_to_histogram_series(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 564, in _get_run_to_histogram_series
    values = [
             ^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 568, in <listcomp>
    "bins": self._format_histogram_datum_bins(datum),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marius/miniconda3/envs/APECS/lib/python3.11/site-packages/tensorboard/plugins/metrics/metrics_plugin.py", line 537, in _format_histogram_datum_bins
    bins = [{"min": x[0], "max": x[1], "count": x[2]} for x in numpy_list]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'float' object is not iterable

When I try to open the Images tab nothing shows up at all.
@Marius-Juston
Copy link
Author

It seems that add_image does work when I do:

normalized = (input_tensor - input_tensor.min()) / (
            input_tensor.max() - input_tensor.min()
        )

summary_writer.add_image(
            name, torch.atleast_2d(normalized)[None, :], global_step=epoch
        )

instead of which still gives incoherent outputs:

fig, ax = plt.subplots()
ax.imshow(torch.atleast_2d(input_tensor).cpu())

summary_writer.add_figure(name, fig, global_step=epoch)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Marius-Juston