hid-subdrivers.txt

How to make a new subdriver to support another USB/HID UPS
----------------------------------------------------------

Overall concept
~~~~~~~~~~~~~~~

USB (Universal Serial Port) devices can be divided into several
different classes (audio, imaging, mass storage etc). Almost all UPS
devices belong to the "HID" class, which means "Human Interface
Device", and also includes things like keyboards and mice. What HID
devices have in common is a particular (and very flexible) interface
for reading and writing information (such as X/Y coordinates and
button states, in the case of a mouse, or voltages and status information,
in the case of a UPS).

The NUT "usbhid-ups" driver is a meta-driver that handles all HID UPS
devices. It consists of a core driver that handles most of the work of
talking to the USB hardware, and several sub-drivers to handle
specific UPS manufacturers (MGE, APC, and Belkin are currently
supported).  Adding support for a new HID UPS device is easy, because
it requires only the creation of a new sub-driver.

There are a few USB UPS devices that are not true HID devices. These
devices typically implement some version of the manufacturer's serial
protocol over USB (which is a really dumb idea, by the way). An
example is the original Tripplite USB interface (USB idProduct = 0001). Its HID
descriptor is only 52 bytes long (compared to several hundred bytes for a true
PDC HID UPS).  Such devices are *not* supported by the usbhid-ups driver, and
are not covered in this document. If you need to add support for such a device,
read new-drivers.txt and see the "tripplite_usb" driver for inspiration.

HID Usage Tree
~~~~~~~~~~~~~~

From the point of view of writing a HID subdriver, a HID device
consists of a bunch of variables.  Some variables (such as the current
input voltage) are read-only, whereas other variables (such as the
beeper enabled/disabled/muted status) can be read and written. These
variables are usually grouped together and arranged in a hierarchical
tree shape, similar to directories in a file system. This tree is
called the "usage" tree. For example, here is part of the usage tree
for a typical APC device. Variable components are separated by ".".
Typical values for each variable are also shown for illustrative
purposes.

[width="35%"]
|================================================
|UPS.Battery.Voltage            |  11.4 V
|UPS.Battery.ConfigVoltage      |  12 V
|UPS.Input.Voltage              |  117 V
|UPS.Input.ConfigVoltage        |  120 V
|UPS.AudibleAlarmControl        |  2 (=enabled)
|UPS.PresentStatus.Charging     |  1 (=yes)
|UPS.PresentStatus.Discharging  |  0 (=no)
|UPS.PresentStatus.ACPresent    |  1 (=yes)
|================================================

As you can see, variables that describe the battery status might be
grouped together under "Battery", variables that describe the input
power might be grouped together under "Input", and variables that
describe the current UPS status might be grouped together under
"PresentStatus". All of these variables are grouped together under
"UPS".

This hierarchical organization of data has the advantage of being very
flexible; for example, if some device has more than one battery, then
similar information about each battery could be grouped under
"Battery1", "Battery2" and so forth. If your UPS can also be used as a
toaster, then information about the toaster function might be grouped
under "Toaster", rather than "UPS".

However, the disadvantage is that each manufacturer will have their
own idea about how the usage tree should be organized, and usbhid-ups
needs to know about all of them. This is why manufacturer specific
subdrivers are needed.

To make matters more complicated, usage tree components (such as
"UPS", "Battery", or "Voltage") are internally represented not as
strings, but as numbers (called "usages" in HID terminology). These
numbers are defined in the "HID Usage Tables", available from
http://www.usb.org/developers/hidpage/.  The standard usages for UPS
devices are defined in a document called "Usage Tables for HID Power
Devices" (the Power Device Class [PDC] specification).

For example:

--------------------------------------------------------------------------------
 0x00840010 = UPS
 0x00840012 = Battery
 0x00840030 = Voltage
 0x00840040 = ConfigVoltage
 0x0084001a = Input
 0x0084005a = AudibleAlarmControl
 0x00840002 = PresentStatus
 0x00850044 = Charging
 0x00850045 = Discharging
 0x008500d0 = ACPresent
--------------------------------------------------------------------------------

Thus, the above usage tree is internally represented as:

--------------------------------------------------------------------------------
 00840010.00840012.00840030
 00840010.00840012.00840040
 00840010.0084001a.00840030
 00840010.0084001a.00840040
 00840010.0084005a
 00840010.00840002.00850044
 00840010.00840002.00850045
 00840010.00840002.008500d0
--------------------------------------------------------------------------------

To make matters worse, most manufacturers define their own additional
usages, even in cases where standard usages could have been used. for
example Belkin defines `00860040` = ConfigVoltage (which is incidentally
a violation of the USB PDC specification, as `00860040` is reserved for
future use).

Thus, subdrivers generally need to provide:

- manufacturer-specific usage definitions,
- a mapping of HID variables to NUT variables.

Moreover, subdrivers might have to provide additional functionality,
such as custom implementations of specific instant commands (load.off,
shutdown.restart), and conversions of manufacturer specific data
formats.


Usage macros in drivers/hidtypes.h
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The `drivers/hidtypes.h` header provides a number of macro names
for entries in the standard usage tables for Power Device
`USAGE_POW_<SOMETHING>` and Battery System `USAGE_BAT_<SOMETHING>`
data pages.

If NUT codebase would ever need to refresh those macros, here is
some background information (based on NUT issue #1189 and PR #1290):

These data were parsed from (a very slightly updated version of)
https://github.com/abend0c1/hidrdd/blob/master/rd.conf file, which
incorporates the complete USB-IF usage definitions for Power Device
and Battery System pages (among many others), so we didn't have to
extract the names and values from the USB-IF standards documents
(did check it all by eye though).

The file was processed with the following chain of commands:

------
:; grep -e '^0084' -e '^0085' rd.conf \
   | sed 's/,.*$//;s/ *$//' \
   | sed 's/ /_/g;s/_/ /' \
   | tr '[:lower:]' '[:upper:]' \
   | sed 's/\(0085.... \)/\1USAGE_BAT_/;s/\(0084.... \)/\1USAGE_POW_/;s/\([A-Z_]*\)_PAGE/PAGE_\1/' \
   | awk '{print "#define "$2" 0x"$1}'
------


Writing a subdriver
~~~~~~~~~~~~~~~~~~~

In preparation for writing a subdriver for a device that is currently
unsupported, run usbhid-ups with the following command line:

	drivers/usbhid-ups -DD -u root -x explore -x vendorid=XXXX \
		-x port=auto -s ups

(substitute your device's 4-digit VendorID instead of "XXXX").
This will produce a bunch of debugging information, including a number
of lines starting with "Path:" that describe the device's usage tree.
This information forms the initial basis for a new subdriver.

You should save this information to a file, e.g.:

	drivers/usbhid-ups -DD -u root -x explore -x vendorid=XXXX \
		-x port=auto -s ups -d1 2>&1 | tee /tmp/info

You can now create an initial "stub" subdriver for your device by using
helper script `scripts/subdriver/gen-usbhid-subdriver.sh`.

NOTE: this only creates a driver code "stub" which needs to be further
customized to be actually useful (see "Customization" below).

Use the script as follows:

	scripts/subdriver/gen-usbhid-subdriver.sh < /tmp/info

where /tmp/info is the file where you previously saved the debugging
information.

This script prompts you for a name for the subdriver; use only letters
and digits, and use natural capitalization such as "Belkin" (not
"belkin" or "BELKIN"). The script may prompt you for additional
information.

You should put the generated files into the drivers/ subdirectory, and
update `usbhid-ups.c` by adding the appropriate `#include` line and by
updating the definition of `subdriver_list` in `usbhid-ups.c`. You must
also add the subdriver to USBHID_UPS_SUBDRIVERS in `drivers/Makefile.am`
and call `autoreconf` and/or `./configure` from the top-level NUT directory.
You can then recompile `usbhid-ups`, and start experimenting with the new
subdriver.


Updating a subdriver
~~~~~~~~~~~~~~~~~~~~

You may have a device from vendor (and maybe model) whose support `usbhid-ups`
already claims. However, you may feel that the driver does not represent all
data points that your device serves. This may be possible, as vendors tend to
use the same identifiers for unrelated products, as well as produce revisions
of devices with same marketed name but different internals (due to chip and
other components availability, cost optimization, etc.) Even without sinister
implications, UPS firmwares evolve and so bugs and features can get added,
fixed and removed over time with truly the same hardware being involved.

In this case you should follow the same instructions as above for "Writing
a subdriver", but specify the same subdriver name as the one which supports
your device family already.

Then compare the generated source file with the one already committed to NUT
codebase, paying close attention to `..._hid2nut[]` table which maps "usage"
names to NUT data points. There may be several "usage" values served by
different device models or firmware versions, that provide same information
for a NUT data point, such as `input.voltage`. For the `hid2nut` mapping
tables, first hit wins (so you may e.g. prefer to check values with better
precision first).

Using a GUI tool with partial-line difference matching and highlighting,
such as Meld or WinMerge, is recommended for this endeavour.

For new data points in `hid2nut` tables be sure to not invent new names,
but use standard ones from `docs/nut-names.txt` file. Temporarily, the
`experimental.*` namespace may be used.
If you need to standardize a name for some concept not addressed yet,
please do so via nut-upsdev mailing list discussion.


Customization
~~~~~~~~~~~~~

The initially generated subdriver code is only a stub,
and will not implement any useful functionality (in particular, it
will be unable to shut down the UPS). In the beginning, it simply
attempts to monitor some UPS variables. To make this driver useful,
you must examine the NUT variables of the form "unmapped.*" in the
hid_info_t data structure, and map them to actual NUT variables and
instant commands. There are currently no step-by-step instructions for
how to do this. Please look at the files to see how the currently implemented
subdrivers are written:

- apc-hid.c/h
- belkin-hid.c/h
- cps-hid.c/h
- explore-hid.c/h
- libhid.c/h
- liebert-hid.c/h
- mge-hid.c/h
- powercom-hid.c/h
- tripplite-hid.c/h

NOTE: To test existing data points (including those not yet translated
to standard NUT mappings conforming to <<nut-names,NUT command and
variable naming scheme>>), you can use custom drivers built after you
`./configure --with-unmapped-data-points`.
Production driver builds must not include any non-standard names.

Fixing report descriptors
~~~~~~~~~~~~~~~~~~~~~~~~~

It is a fact of life that fellow developers make mistakes, and firmware
authors do too. In some cases there are inconsistencies about bytes seen
on the wire vs. their logical values, such value range and signedness if
interpreting them according to standard.

NUT drivers now include a way to detect and fix up known issues in such
flawed USB report descriptors, side-stepping the standard similarly where
deemed needed. A pointer to such hook method is part of the `subdriver_t`
structure detailing each `usbhid-ups` subdriver nuances, defaulting to
a `fix_report_desc()` trivial implementation.

For some practical examples, see e.g. `apc_fix_report_desc()` method in the
`drivers/apc-hid.c` file, and `cps_fix_report_desc()` in `drivers/cps-hid.c`
file.

Finally note that such fix-ups may be not applicable to all devices or
firmware versions for what they assume their target audience is. If you
suspect that the fix-up method is actually causing problems, you can quickly
disable it with `disable_fix_report_desc` driver option for `usbhid-ups`.
If the problem does dissipate, please find a way to identify your "fixed"
hardware/firmware vs. those models where existing fix-up method should be
applied, and post a pull request so the NUT driver would handle both cases.


Investigating report descriptors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Beside looking for problems with report descriptor processing in NUT code,
it is important to make sure what data the device actually serves on the
wire, and if it is logically consistent with the protocol requirements.

While here, keep in mind that USB protocol on the wire has a specified
order of bytes involved, while processing on your computer may lay them
out differently due to bitness and endianness of the current binary build.
General NUT codebase (`libhid.c`, `hidparser.c`) aims to abstract this,
so application code like drivers can deal with their native numeric data
types, but when troubleshooting, do not rule out possibility of flaws
there as well. And certainly do not code any assumptions about ordered
multiple-byte ranges in a protocol buffer.

For a deep dive into the byte stream, you will need additional tools:

* get/build/install link:https://regina-rexx.sourceforge.io[regina-rexx]
* get/install link:https://github.com/abend0c1/hidrdd[HIDRDD] (uses REXX
  as the interpreter)

Typical troubleshooting of suspected firmware/protocol issues goes like this:

* Turn the NUT `usbhid-ups` driver debug verbosity level up to 5 (or more)
  and restart the driver, so it would record the HEX dump of report descriptor
* Look for reports from the driver of any problems it has already detected
  and possibly amended (LogMin/LogMax, report descriptor fix-ups)
* Extract the HEX dump of the report descriptor from USB driver output from
  the first step above, and run it through HIDRDD (and/or REXX directly,
  per example below).
* Look at the HIDRDD output, with reference to any documents related to your
  device and the USB/HID power devices class available in NUT documentation,
  e.g. at https://www.networkupstools.org/ups-protocols.html
* Especially look for inconsistencies in the USB HID report descriptors (RD):
  * between the min/max (logical and physical) values,
  * the sizes of the report fields they apply to,
  * the expected physical values (e.g., supply and output voltages,
    over-voltage/under-voltage transfer points, ...)
* If you're seeing unexpected values for particular variables, look at the
  raw data that is being sent, decide whether it makes sense in the context
  of the logical and physical min/max values from the report descriptor.
* Read the NUT code, tracing through how each value gets processed looking
  for where the result deviates from expectations...
* Think, code, test, rinse, repeat, post a PR :)

.Example direct use of REXX
===========================
Example adapted from https://github.com/networkupstools/nut/issues/2039

Run a NUT `usb-hid` driver with at least debug verbosity level 3 (`-DDD`)
to get a report descriptor dump starting with a line like this:
----
   3.670755     [D3] Report Descriptor: (909 bytes) => 05 84 09 04 a1 01 ...
----

...and copy-paste those reported lines as input into `rexx` tool, which
would generate a C source file including human-worded description and
a relevant data structure:

----
:; rexx rd.rex -d --hex 05 84 09 04 a1 01 85 01 09 18 ... 55 b1 02 c0 c0 c0

//--------------------------------------------------------------------------------
// Decoded Application Collection
//--------------------------------------------------------------------------------

/*
05 84        (GLOBAL) USAGE_PAGE         0x0084 Power Device Page
09 04        (LOCAL)  USAGE              0x00840004 UPS (Application Collection)
A1 01        (MAIN)   COLLECTION         0x01 Application (Usage=0x00840004: Page=Power Device Page, Usage=UPS, Type=Application Collection)
85 01          (GLOBAL) REPORT_ID          0x01 (1)
09 18          (LOCAL)  USAGE              0x00840018 Outlet System (Physical Collection)
...
*/

// All structure fields should be byte-aligned...
#pragma pack(push,1)

//--------------------------------------------------------------------------------
// Power Device Page featureReport 01 (Device <-> Host)
//--------------------------------------------------------------------------------

typedef struct
{
  uint8_t  reportId;                                 // Report ID = 0x01 (1)
                                                     // Collection: CA:UPS CP:OutletSystem CP:Outlet
  int8_t   POW_UPSOutletSystemOutletSwitchable;      // Usage 0x0084006C: Switchable, Value =  to 
  int8_t   POW_UPSOutletSystemOutletDelayBeforeStartup; // Usage 0x00840056: Delay Before Startup, Value = -1 to 60
  int8_t   POW_UPSOutletSystemOutletDelayBeforeShutdown; // Usage 0x00840057: Delay Before Shutdown, Value = -1 to 60
  int8_t   POW_UPSOutletSystemOutletDelayBeforeReboot; // Usage 0x00840055: Delay Before Reboot, Value = -1 to 60
  int8_t   POW_UPSOutletSystemOutletSwitchable_1;    // Usage 0x0084006C: Switchable, Value = -1 to 60
  int8_t   POW_UPSOutletSystemOutletDelayBeforeStartup_1; // Usage 0x00840056: Delay Before Startup, Value = -1 to 60
  int8_t   POW_UPSOutletSystemOutletDelayBeforeShutdown_1; // Usage 0x00840057: Delay Before Shutdown, Value = -1 to 60
  int8_t   POW_UPSOutletSystemOutletDelayBeforeReboot_1; // Usage 0x00840055: Delay Before Reboot, Value = -1 to 60
} featureReport01_t;

#pragma pack(pop)
----
===========================


Shutting down the UPS
~~~~~~~~~~~~~~~~~~~~~

It is desirable to support shutting down the UPS. Usually (for
devices that follow the HID Power Device Class specification), this
requires sending the UPS two commands. One for shutting down the UPS
(with an 'offdelay') and one for restarting it (with an 'ondelay'),
where offdelay < ondelay. The two NUT commands for which this is
relevant, are 'shutdown.return' and 'shutdown.stayoff'.

Since the one-to-one mapping above doesn't allow sending two HID
commands to the UPS in response to sending one NUT command to the
driver, this is handled by the driver. In order to make this work,
you need to define the following four NUT values:

	ups.delay.start    (variable, R/W)
	ups.delay.shutdown (variable, R/W)
	load.off.delay     (command)
	load.on.delay      (command)

If the UPS supports it, the following variables can be used to show
the countdown to start/shutdown:

	ups.timer.start    (variable, R/O)
	ups.timer.shutdown (variable, R/O)

The `load.on` and `load.off` commands will be defined implicitly by
the driver (using a delay value of '0'). Define these commands
yourself, if your UPS requires a different value to switch on/off
the load without delay.

Note that the driver expects the `load.off.delay` and `load.on.delay`
to follow the HID Power Device Class specification, which means that
the `load.on.delay` command should NOT switch on the load in the
absence of mains power. If your UPS switches on the load regardless of
the mains status, DO NOT define this command. You probably want to
define the `shutdown.return` and/or `shutdown.stayoff` commands in
that case. Commands defined in the subdriver will take precedence over
the ones that are composed in the driver.

When running the driver with the '-k' flag, it will first attempt to
send a `shutdown.return` command and if that fails, will fallback to
`shutdown.reboot`.