Quantum Leap

Bug fix

• Fixed issue causing the Windows version of the Scalyr Agent to not start when starting in forked mode.

Prime

Critical bug fix

• Updated bundled certificates used to verify Scalyr TLS certificate. Existing bundled certificates had expired causing some customers to not be able to verify the TLS connection.

Features

• Added ability to specify an extra configuration snippet directory (in addition to agent.d). You may specify the extra directory by setting the SCALYR_EXTRA_CONFIG_DIR environment variable to the path of your desired directory. This has been added mainly to help mounting additional configuration in Kubernetes.

Bug fix

• Fix timing issue in kubernetes_monitor that could result in not collecting logs from very short lived containers.

Orion

Features

• The Kubernetes and Kubernetes event monitors now allow you to restrict the namespaces for which the Scalyr Agent will collect logs and events. This may be controlled via the SCALYR_K8S_INCLUDE_NAMESPACES environment variable or k8s_include_namespaces configuration option. See release notes for more information.

Bugs

• Fixed Kubernetes monitor to verify by default TLS connections made to the local Kubelet. This was causing an excessive amount of warnings to be emitted to the stdout. See release notes for details on how to disable verification if this causes issues.
• Fixed issue preventing the Apache monitor from running under Python 2.
• Fixed some issues causing some logging to fail due to incorrect arguments.

Nostromo

Features

• The Agent no longer enforces all events must have monotonically increasing timestamps when pushed into Scalyr. The is now possible due to changes in the server. This allow better use of the default timestamps, such as in Docker and Kubernetes.
• Log lines collected via the Kubernetes or Docker monitors will now have their default timestamps set to the timestamp generated by K8s/Docker. As long as a parser does not override the timestamp, the K8s and Docker timestamps will be used in Scalyr.
• Agent logs with severity higher than INFO will now be output to stdout as well as the agent.log file when running with the --no-fork flag.
• The Journald monitor now supports a detect_escaped_strings option which automatically detects if the details field is already an escaped string, and if so, just emits it directly to the log instead of re-escaping it. This option is specified using the journald_logs array. See the journald monitor documentation for more details.

Bugs

• Fixed monitor configuration bug that prevented config options that did not have a default nor were required from being properly set via environment variables.

Optimizations

• Optimize RFC3339 date strings parsing. This should result in better throughput under highly loaded scenarios (many lines per second) when using Docker / Kubernetes monitor.
• Speed up event serialization under highly loaded scenarios by optimizing json encoding and encoding of event attributes.
• We now default to orjson JSON library under Python 3 (if the library is available). orjson is substantially faster than ujson for encoding.

Millenium Falcon

Features

• Major update of code base to support running under Python 2 and 3. See release notes for more information.
• Agent now supports Python 2.6, 2.7 and >= 3.5
• When installing using RPM or Debian, Python 2 will be used by default unless unavailable.
• The Python version used to run the Agent may be controlled using the scalyr-switch-python command.
• The K8s manifest files have been changed to run the Agent in the scalyr namespace instead of the default namespace. When updating an existing Scalyr Agent K8s instance, you must follow the upgrade instructions described in the upgrade notes.
• RPM and Debian packages no longer declare dependency on Python to promote cross-distribution compatibility. The dependency is now verified at package install time.
• Added option to scalyr-agent-2 status -v to emit JSON (--format=[text|json]).
• Add new metric_name_blacklist supported attribute to each monitor configuration section. With this attribute, user can define a list of metric names which should be excluded and not shipped to Scalyr.
• Add support for orjson JSON library when running under Python 3. This library offers significantly better performance and can be enabled by setting json_library config option to orjson and installing orjson Python package using pip.

Bugs

• Fix authentication issue in kubernetes_monitor when accessing kublet API.
• Better error messages issued when missing required certificate files.
• Updated Kubernetes manifest file to use apps/v1 for DaemonSet API version instead of beta version.
• Update scalyr client code to log raw uncompressed body under debug log level to aid with troubleshooting.
• Metric type for app.disk.requests.{read,write} metrics has been fixed.
• Fix iostat monitor so it also works with newer versions of Linux kernel.
• Fix invalid extra field for two tomcat.runtime.threads metrics in the Tomcat monitor (all of the metrics had type set to max whereas one should have type set to active and the other to busy).

Minor updates

• The /etc/init.d/scalyr-agent-2 symlink is no longer marked as a conf file in Debian packages.
• Update of embedded ecsda library to 0.13.3
• Docker support now requires the docker 4.1 client library
• Changed which signal is used to execute scalyr-agent-2 status -v under Linux to improve handling of SIGINT. Previously SIGINT was used, now SIGUSR1 is used.
• When running in foreground mode (--no-fork flag), SIGINT signal (aka CTRL+C) now starts the graceful shutdown procedure.
• Two new metrics (app.io.wait, app.mem.majflt) are now emitted by the Linux process monitor. If you want those metrics to be excluded for your monitors, you can utilize new metric_name_blacklist monitor config option.

Testing updates

• Numerous changes to improve testing and coverage reporting

Lady MacBeth

Features

• Improved support for handling long log lines captured by kubernetes_monitor and docker_monitor. When parsing docker logs, the serialized JSON objects can now be up to internal_parse_max_line_size bytes (defaults to 64KB). This replaces the requirement they be less than max_line_size. The log field extracted from the JSON object is still subjected to the max_line_size limit.
• Improved handling when kubernetes_monitor fails due to an uninitialized K8s cache. If a kubernetes_monitor becomes stuck for 2 minutes and cannot initialize, the agent will terminate its container to allow for K8s to restart it.

Bugs

• Add additional diagnostic information to help customers troubleshoot issues in the kubernetes_monitor due to failures in K8s cache initialization.
• Fix bug due to protected access in the journald_monitor.

Karrajor

Bugs

• Fixed error in Windows release caused by missing glob2 module
• Fixed removal of stream attribute from logs uploaded using Docker and K8s monitors

Jupiter 2

Bugs:

• Fixed error in syslog-monitor causing monitor to stop working without docker-py installed on host system

Icarus

Features

• Added support to journald plugin to create separate log files (with distinct configuration such as parsers) based on unit field.
• Add supported for ** in file path glob patterns recursively drill down into sub-directories.
• Changed to new API format that allows for more efficient encoding of log file attributes

Hermes

Bugs

• Fixed bug causing SSL connection failures on certain Windows configurations.