User
@@ -2923,12 +2909,7 @@
Getting Started
-
-
Attention
-
Supported Architectures
-
EdgeX Foundry is a hardware and operating system agnostic IoT / edge platform. It is was built to run on Intel and ARM hardware. It runs on various distributions and / or versions of Linux, Unix, MacOS, Windows, etc.
-
However, the EdgeX Foundry community only supports the platform on Intel (x86, x86_64) and ARM64 hardware. Adopters may build EdgeX for ARM32, but the community does not support or provide pre-built artifacts such as Docker containers, snaps, etc. for ARM32. Building EdgeX for ARM32 will typically require some modifications to the build process. Some services may not compile on ARM32 and/or require removal of components such as ZMQ that are not available or compile on ARM32.
-
EdgeX Foundry is operating system and architecture agnostic. The community releases artifacts for common architectures. However, it is possible to build the components for other platforms. See the platform requirements reference page for details.
To get started you need to get EdgeX Foundry either as a User or as a Developer/Contributor.
User
If you want to get the EdgeX platform and run it (but do not
diff --git a/2.2/search/search_index.json b/2.2/search/search_index.json
index 4d5a64e8c2..76d65a961b 100644
--- a/2.2/search/search_index.json
+++ b/2.2/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Introduction EdgeX 2.x Want to know what's new in EdgeX 2.x releases (Ireland/Jakarta/etc)? If you are already familiar with EdgeX, look for the EdgeX 2.x emoji ( Edgey - the EdgeX mascot) throughout the documentation - like the one on this page outlining what's new in the latest 2.x releases. These sections will give you a summary of what's new in each area of the documentation. EdgeX Foundry is an open source, vendor neutral, flexible, interoperable, software platform at the edge of the network, that interacts with the physical world of devices , sensors, actuators, and other IoT objects. In simple terms, EdgeX is edge middleware - serving between physical sensing and actuating \"things\" and our information technology (IT) systems. The EdgeX platform enables and encourages the rapidly growing community of IoT solution providers to work together in an ecosystem of interoperable components to reduce uncertainty, accelerate time to market, and facilitate scale. By bringing this much-needed interoperability, EdgeX makes it easier to monitor physical world items, send instructions to them, collect data from them, move the data across the fog up to the cloud where it may be stored, aggregated, analyzed, and turned into information, actuated, and acted upon. So EdgeX enables data to travel northwards towards the cloud or enterprise and back to devices, sensors, and actuators. The initiative is aligned around a common goal: the simplification and standardization of the foundation for tiered edge computing architectures in the IoT market while still enabling the ecosystem to provide significant value-added differentiation. If you don't need further description and want to immediately use EdgeX Foundry use this link: Getting Started Guide EdgeX Foundry Use Cases Originally built to support industrial IoT needs, EdgeX today is used in a variety of use cases to include: Building automation \u2013 helping to manage shared workspace facilities Oil/gas \u2013 closed loop control of a gas supply valve Retail \u2013 multi-sensor reconciliation for loss prevention at the point of sale Water treatment \u2013 monitor and control chemical dosing Consumer IoT \u2013 the open source HomeEdge project is using elements of EdgeX as part of its smart home platform EdgeX Foundry Architectural Tenets EdgeX Foundry was conceived with the following tenets guiding the overall architecture: EdgeX Foundry must be platform agnostic with regard to Hardware (x86, ARM) Operating system (Linux, Windows, MacOS, ...) Distribution (allowing for the distribution of functionality through micro services at the edge, on a gateway, in the fog, on cloud, etc.) Deployment/orchestration (Docker, Snaps, K8s, roll-your-own, ... ) Protocols ( north or south side protocols) EdgeX Foundry must be extremely flexible Any part of the platform may be upgraded, replaced or augmented by other micro services or software components Allow services to scale up and down based on device capability and use case EdgeX Foundry should provide \" reference implementation \" services but encourages best of breed solutions EdgeX Foundry must provide for store and forward capability To support disconnected/remote edge systems To deal with intermittent connectivity EdgeX Foundry must support and facilitate \"intelligence\" moving closer to the edge in order to address Actuation latency concerns Bandwidth and storage concerns Operating remotely concerns EdgeX Foundry must support brown and green device/sensor field deployments EdgeX Foundry must be secure and easily managed Deployments EdgeX was originally built by Dell to run on its IoT gateways . While EdgeX can and does run on gateways, its platform agnostic nature and micro service architecture enables tiered distributed deployments. In other words, a single instance of EdgeX\u2019s micro services can be distributed across several host platforms. The host platform for one or many EdgeX micro services is called a node. This allows EdgeX to leverage compute, storage, and network resources wherever they live on the edge. Its loosely-coupled architecture enables distribution across nodes to enable tiered edge computing. For example, thing communicating services could run on a programmable logic controller (PLC), a gateway, or be embedded in smarter sensors while other EdgeX services are deployed on networked servers or even in the cloud. The scope of a deployment could therefore include embedded sensors, controllers, edge gateways, servers and cloud systems. EdgeX micro services can be deployed across an array of compute nodes to maximize resources while at the same time position more processing intelligence closer to the physical edge. The number and the function of particular micro services deployed on a given node depends on the use case and capability of the hardware and infrastructure. Apache 2 License EdgeX is distributed under Apache 2 License backed by the Apache Foundation. Apache 2 licensing is very friendly (\u201cpermissive\u201d) to open and commercial interests. It allows users to use the software for any purpose. It allows users to distribute, modify or even fork the code base without seeking permission from the founding project. It allows users to change or extend the code base without having to contribute back to the founding project. It even allows users to build commercial products without concerns for profit sharing or royalties to go back to the Linux Foundation or open source project organization. EdgeX Foundry Service Layers EdgeX Foundry is a collection of open source micro services. These micro services are organized into 4 service layers, and 2 underlying augmenting system services. The Service Layers traverse from the edge of the physical realm (from the Device Services Layer), to the edge of the information realm (that of the Application Services Layer), with the Core and Supporting Services Layers at the center. The 4 Service Layers of EdgeX Foundry are as follows: Core Services Layer Supporting Services Layer Application Services Layer Device Services Layer The 2 underlying System Services of EdgeX Foundry are as follows: Security System Management Core Services Layer Core services provide the intermediary between the north and south sides of EdgeX. As the name of these services implies, they are \u201ccore\u201d to EdgeX functionality. Core services is where most of the innate knowledge of what \u201cthings\u201d are connected, what data is flowing through, and how EdgeX is configured resides in an EdgeX instance. Core consists of the following micro services: Core data: a persistence repository and associated management service for data collected from south side objects. Command: a service that facilitates and controls actuation requests from the north side to the south side. Metadata: a repository and associated management service of metadata about the objects that are connected to EdgeX Foundry. Metadata provides the capability to provision new devices and pair them with their owning device services. Registry and Configuration: provides other EdgeX Foundry micro services with information about associated services within EdgeX Foundry and micro services configuration properties (i.e. - a repository of initialization values). Core services provide intermediary communications between the things and the IT systems. Supporting Services Layer The supporting services encompass a wide range of micro services to include edge analytics (also known as local analytics). Normal software application duties such as scheduler, and data clean up (also known as scrubbing in EdgeX) are performed by micro services in the supporting services layer. These services often require some amount of core services in order to function. In all cases, supporting service can be considered optional \u2013 that is they can be left out of an EdgeX deployment depending on use case needs and system resources. Supporting services include: Rules Engine: the reference implementation edge analytics service that performs if-then conditional actuation at the edge based on sensor data collected by the EdgeX instance. This service may be replaced or augmented by use case specific analytics capability. Scheduler: an internal EdgeX \u201cclock\u201d that can kick off operations in any EdgeX service. At a configuration specified time, the service will call on any EdgeX service API URL via REST to trigger an operation. For example, the scheduler service periodically calls on core data APIs to clean up old sensed events that have been successfully exported out of EdgeX. Alerts and Notifications: provides EdgeX services with a central facility to send out an alert or notification. These are notices sent to another system or to a person monitoring the EdgeX instance (internal service communications are often handled more directly). Application Services Layer Application services are the means to extract, process/transform and send sensed data from EdgeX to an endpoint or process of your choice. EdgeX today offers application service examples to send data to many of the major cloud providers (Amazon IoT Hub, Google IoT Core, Azure IoT Hub, IBM Watson IoT\u2026), to MQTT(s) topics, and HTTP(s) REST endpoints. Application services are based on the idea of a \"functions pipeline\". A functions pipeline is a collection of functions that process messages (in this case EdgeX event messages) in the order specified. The first function in a pipeline is a trigger. A trigger begins the functions pipeline execution. A trigger, for example, is something like a message landing in a message queue. Each function then acts on the message. Common functions include filtering, transformation (i.e. to XML or JSON), compression, and encryption functions. The function pipeline ends when the message has gone through all the functions and is set to a sink. Putting the resulting message into an MQTT topic to be sent to Azure or AWS is an example of a sink completing an application service. Device Services Layer Device services connect \u201cthings\u201d \u2013 that is sensors and devices \u2013 into the rest of EdgeX. Device services are the edge connectors interacting with the \"things\" that include, but are not limited to: alarm systems, heating and air conditioning systems in homes and office buildings, lights, machines in any industry, irrigation systems, drones, currently automated transit such as some rail systems, currently automated factories, and appliances in your home. In the future, this may include driverless cars and trucks, traffic signals, fully automated fast food facilities, fully automated self-serve grocery stores, devices taking medical readings from patients, etc. Device services may service one or a number of things or devices (sensor, actuator, etc.) at one time. A device that a device service manages, could be something other than a simple, single, physical device. The device could be another gateway (and all of that gateway's devices), a device manager, a device aggregator that acts as a device, or collection of devices, to EdgeX Foundry. The device service communicates with the devices, sensors, actuators, and other IoT objects through protocols native to each device object. The device service converts the data produced and communicated by the IoT object into a common EdgeX Foundry data structure, and sends that converted data into the core services layer, and to other micro services in other layers of EdgeX Foundry. EdgeX comes with a number of device services speaking many common IoT protocols such as Modbus, BACnet, MQTT, etc. System Services Layer Security Infrastructure Security elements of EdgeX Foundry protect the data and control of devices, sensors, and other IoT objects managed by EdgeX Foundry. Based on the fact that EdgeX is a \"vendor-neutral open source software platform at the edge of the network\", the EdgeX security features are also built on a foundation of open interfaces and pluggable, replaceable modules. There are two major EdgeX security components. A security store, which is used to provide a safe place to keep the EdgeX secrets. Examples of EdgeX secrets are the database access passwords used by the other services and tokens to connect to cloud systems. An API gateway serves as the reverse proxy to restrict access to EdgeX REST resources and perform access control related works. System Management System Management facilities provide the central point of contact for external management systems to start/stop/restart EdgeX services, get the status/health of a service, or get metrics on the EdgeX services (such as memory usage) so that the EdgeX services can be monitored. Software Development Kits (SDKs) Two types of SDKs are provided by EdgeX to assist in creating north and south side services \u2013 specifically to create application services and device services. SDKs for both the north and south side services make connecting new things or new cloud/enterprise systems easier by providing developers all the scaffolding code that takes care of the basic operations of the service. Thereby allowing developers to focus on specifics of their connectivity to the south or north side object without worrying about all the raw plumbing of a micro service. SDKs are language specific; meaning an SDK is written to create services in a particular programming language. Today, EdgeX offers the following SDKs: Golang Device Service SDK C Device Service SDK Golang Application Functions SDK How EdgeX Works Sensor Data Collection EdgeX\u2019s primary job is to collect data from sensors and devices and make that data available to north side applications and systems. Data is collected from a sensor by a device service that speaks the protocol of that device. Example: a Modbus device service would communicate in Modbus to get a pressure reading from a Modbus pump. The device service translates the sensor data into an EdgeX event object. The device service can then either: put the event object on a message bus (which may be implemented via Redis Streams or MQTT). Subscribers to the event message on the message bus can be application services or core data or both (see step 1.1 below). send the event object to the core data service via REST communications (see step 1.2). When core data receives the event (either via message bus or REST), it persists the sensor data in the local edge database. EdgeX uses Redis as our persistence store. There is an abstraction in place to allow you to use another database (which has allowed other databases to be used in the past). Persistence is not required and can be turned off. Data is persisted in EdgeX at the edge for two basics reasons: Edge nodes are not always connected. During periods of disconnected operations, the sensor data must be saved so that it can be transmitted northbound when connectivity is restored. This is referred to as store and forward capability. In some cases, analytics of sensor data needs to look back in history in order to understand the trend and to make the right decision based on that history. If a sensor reports that it is 72\u00b0 F right now, you might want to know what the temperature was ten minutes ago before you make a decision to adjust a heating or cooling system. If the temperature was 85\u00b0 F, you may decide that adjustments to lower the room temperature you made ten minutes ago were sufficient to cool the room. It is the context of historical data that are important to local analytic decisions. When core data receives event objects from the device service via REST, it will put sensor data events on a message topic destined for application services. Redis Pub/Sub is used as the messaging infrastructure by default (step 2). MQTT or ZMQ can also be used as the messaging infrastructure between core data and the application services. The application service transforms the data as needed and pushes the data to an endpoint. It can also filter, enrich, compress, encrypt or perform other functions on the event before sending it to the endpoint (step 3). The endpoint could be an HTTP/S endpoint, an MQTT topic, a cloud system (cloud topic), etc. Edge Analytics and Actuation In edge computing, simply collecting sensor data is only part of the job of an edge platform like EdgeX. Another important job of an edge platform is to be able to: Analyze the incoming sensor data locally Act quickly on that analysis Edge or local analytics is the processing that performs an assessment of the sensor data collected at the edge (\u201clocally\u201d) and triggers actuations or actions based on what it sees. Why edge analytics ? Local analytics are important for two reasons: Some decisions cannot afford to wait for sensor collected data to be fed back to an enterprise or cloud system and have a response returned. Additionally, some edge systems are not always connected to the enterprise or cloud \u2013 they have intermittent periods of connectivity. Local analytics allows systems to operate independently, at least for some stretches of time. For example: a shipping container\u2019s cooling system must be able to make decisions locally without the benefit of Internet connectivity for long periods of time when the ship is at sea. Local analytics also allow a system to act quickly in a low latent fashion when critical to system operations. As an extreme case, imagine that your car\u2019s airbag fired on the basis of data being sent to the cloud and analyzed for collisions. Your car has local analytics to prevent such a potentially slow and error prone delivery of the safety actuation in your automobile. EdgeX is built to act locally on data it collects from the edge. In other words, events are processed by local analytics and can be used to trigger action back down on a sensor/device. Just as application services prepare data for consumption by north side cloud systems or applications, application services can process and get EdgeX events (and the sensor data they contain) to any analytics package (see step 4). By default, EdgeX ships with a simple rules engine (the default EdgeX rules engine is eKuiper \u2013 an open source rules engine and now a sister project in LF Edge). Your own analytics package (or ML agent) could replace or augment the local rules engine. The analytic package can explore the sensor event data and make a decision to trigger actuation of a device. For example, it could check that the pressure reading of an engine is greater than 60 PSI. When such a rule is determined to be true, the analytic package calls on the core command service to trigger some action, like \u201copen a valve\u201d on some controllable device (see step 5). The core command service gets the actuation request and determines which device it needs to act on with the request; then calling on the owning device service to do the actuation (see step 6). Core command allows developers to put additional security measures or checks in place before actuating. The device service receives the request for actuation, translates that into a protocol specific request and forwards the request to the desired device (see step 7). Project Release Cadence Typically, EdgeX releases twice a year; once in the spring and once in the fall. Bug fix releases may occur more often. Each EdgeX release has a code name. The code name follows an alphabetic pattern similar to Android (code names sequentially follow the alphabet). The code name of each release is named after some geographical location in the world. The honor of naming an EdgeX release is given to a community member deemed to have contributed significantly to the project. A release also has a version number. The release version follows sematic versioning to indicate the release is major or minor in scope. Major releases typically contain significant new features and functionality and are not always backward compatible with prior releases. Minor releases are backward compatible and usually contain bug fixes and fewer new features. See the project Wiki for more information on releases, versions and patches . Release Schedule Version Barcelona Oct 2017 0.5.0 California Jun 2017 0.6.0 Delhi Oct 2018 0.7.0 Edinburgh Jul 2019 1.0.0 Fuji Nov 2019 1.1.0 Geneva May 2020 1.2.0 Hanoi November 2020 1.3.0 Ireland Spring 2021 2.0.0 Jakarta Fall 2021 2.1.0 Kamukura Spring 2022 TBD Levski Fall 2022 TBD Note : minor releases of the Device Services and Application Services (along with their associated SDKs) can be release independently. Graphical User Interface, the command line interface (CLI) and other tools can be released independently. EdgeX community members convene in a meeting right at the time of a release to plan the next release and roadmap future releases. See the Project Wiki for more detailed information on releases and roadmap . EdgeX 2.0 The Ireland Release The Ireland release, available June 2021, is the second major version of EdgeX. Highlights of the 2.0 release include: A new and improved set of service APIs, which eliminate a lot of technical debt and setting EdgeX up for new features in the future (such as allowing for more message based communications) Direct device service to application service communications via message bus (bypassing core data if desired or allowing it to be a secondary subscriber) Simplified device profiles Improved security New, improved and more comprehensive graphical user interface (for development and demonstration purposes) New device services for CoAP, GPIO, and LLRP (RFID protocol) An LLRP inventory application service Improved application service capability and functions (to include new filter functions) Cleaner/simpler Docker image naming and facilities to create custom Docker Compose files EdgeX 2.0 provides adopters with a platform that Has an improved API that addresses edge application needs of today and tomorrow Is more efficient and lighter (depending on use case) Is more reliable and offers better quality of service (less REST, more messaging and incorporating a number of bug fixes) Has eliminated a lot of technical debt accumulated over 4 years EdgeX History and Naming EdgeX Foundry began as a project chartered by Dell IoT Marketing and developed by the Dell Client Office of the CTO as an incubation project called Project Fuse in July 2015. It was initially created to run as the IoT software application on Dell\u2019s introductory line of IoT gateways. Dell entered the project into open source through the Linux Foundation on April 24, 2017. EdgeX was formally announced and demonstrated at Hanover Messe 2017. Hanover Messe is one of the world's largest industrial trade fairs. At the fair, the Linux Foundation also announced the association of 50 founding member organizations \u2013 the EdgeX ecosystem \u2013 to help further the project and the goals of creating a universal edge platform. The name \u2018foundry\u2019 was used to draw parallels to Cloud Foundry . EdgeX Foundry is meant to be a foundry for solutions at the edge just like Cloud Foundry is a foundry for solutions in the cloud. Cloud Foundry was originated by VMWare (Dell Technologies is a major shareholder of VMWare - recall that Dell Technologies was the original creator of EdgeX). The \u2018X\u2019 in EdgeX represents the transformational aspects of the platform and allows the project name to be trademarked and to be used in efforts such as certification and certification marks. The EdgeX Foundry Logo represents the nature of its role as transformation engine between the physical OT world and the digital IT world. The EdgeX community selected the octopus as the mascot or \u201cspirit animal\u201d of the project at its inception. Its eight arms and the suckers on the arms represent the sensors. The sensors bring the data into the octopus. Actually, the octopus has nine brains in a way. It has millions of neurons running down each arm; functioning as mini-brains in each of those arms. The arms of the octopus serve as \u201clocal analytics\u201d like that offered by EdgeX. The mascot is affectionately called \u201cEdgey\u201d by the community.","title":"Introduction"},{"location":"#introduction","text":"EdgeX 2.x Want to know what's new in EdgeX 2.x releases (Ireland/Jakarta/etc)? If you are already familiar with EdgeX, look for the EdgeX 2.x emoji ( Edgey - the EdgeX mascot) throughout the documentation - like the one on this page outlining what's new in the latest 2.x releases. These sections will give you a summary of what's new in each area of the documentation. EdgeX Foundry is an open source, vendor neutral, flexible, interoperable, software platform at the edge of the network, that interacts with the physical world of devices , sensors, actuators, and other IoT objects. In simple terms, EdgeX is edge middleware - serving between physical sensing and actuating \"things\" and our information technology (IT) systems. The EdgeX platform enables and encourages the rapidly growing community of IoT solution providers to work together in an ecosystem of interoperable components to reduce uncertainty, accelerate time to market, and facilitate scale. By bringing this much-needed interoperability, EdgeX makes it easier to monitor physical world items, send instructions to them, collect data from them, move the data across the fog up to the cloud where it may be stored, aggregated, analyzed, and turned into information, actuated, and acted upon. So EdgeX enables data to travel northwards towards the cloud or enterprise and back to devices, sensors, and actuators. The initiative is aligned around a common goal: the simplification and standardization of the foundation for tiered edge computing architectures in the IoT market while still enabling the ecosystem to provide significant value-added differentiation. If you don't need further description and want to immediately use EdgeX Foundry use this link: Getting Started Guide","title":"Introduction"},{"location":"#edgex-foundry-use-cases","text":"Originally built to support industrial IoT needs, EdgeX today is used in a variety of use cases to include: Building automation \u2013 helping to manage shared workspace facilities Oil/gas \u2013 closed loop control of a gas supply valve Retail \u2013 multi-sensor reconciliation for loss prevention at the point of sale Water treatment \u2013 monitor and control chemical dosing Consumer IoT \u2013 the open source HomeEdge project is using elements of EdgeX as part of its smart home platform","title":"EdgeX Foundry Use Cases"},{"location":"#edgex-foundry-architectural-tenets","text":"EdgeX Foundry was conceived with the following tenets guiding the overall architecture: EdgeX Foundry must be platform agnostic with regard to Hardware (x86, ARM) Operating system (Linux, Windows, MacOS, ...) Distribution (allowing for the distribution of functionality through micro services at the edge, on a gateway, in the fog, on cloud, etc.) Deployment/orchestration (Docker, Snaps, K8s, roll-your-own, ... ) Protocols ( north or south side protocols) EdgeX Foundry must be extremely flexible Any part of the platform may be upgraded, replaced or augmented by other micro services or software components Allow services to scale up and down based on device capability and use case EdgeX Foundry should provide \" reference implementation \" services but encourages best of breed solutions EdgeX Foundry must provide for store and forward capability To support disconnected/remote edge systems To deal with intermittent connectivity EdgeX Foundry must support and facilitate \"intelligence\" moving closer to the edge in order to address Actuation latency concerns Bandwidth and storage concerns Operating remotely concerns EdgeX Foundry must support brown and green device/sensor field deployments EdgeX Foundry must be secure and easily managed","title":"EdgeX Foundry Architectural Tenets"},{"location":"#deployments","text":"EdgeX was originally built by Dell to run on its IoT gateways . While EdgeX can and does run on gateways, its platform agnostic nature and micro service architecture enables tiered distributed deployments. In other words, a single instance of EdgeX\u2019s micro services can be distributed across several host platforms. The host platform for one or many EdgeX micro services is called a node. This allows EdgeX to leverage compute, storage, and network resources wherever they live on the edge. Its loosely-coupled architecture enables distribution across nodes to enable tiered edge computing. For example, thing communicating services could run on a programmable logic controller (PLC), a gateway, or be embedded in smarter sensors while other EdgeX services are deployed on networked servers or even in the cloud. The scope of a deployment could therefore include embedded sensors, controllers, edge gateways, servers and cloud systems. EdgeX micro services can be deployed across an array of compute nodes to maximize resources while at the same time position more processing intelligence closer to the physical edge. The number and the function of particular micro services deployed on a given node depends on the use case and capability of the hardware and infrastructure.","title":"Deployments"},{"location":"#apache-2-license","text":"EdgeX is distributed under Apache 2 License backed by the Apache Foundation. Apache 2 licensing is very friendly (\u201cpermissive\u201d) to open and commercial interests. It allows users to use the software for any purpose. It allows users to distribute, modify or even fork the code base without seeking permission from the founding project. It allows users to change or extend the code base without having to contribute back to the founding project. It even allows users to build commercial products without concerns for profit sharing or royalties to go back to the Linux Foundation or open source project organization.","title":"Apache 2 License"},{"location":"#edgex-foundry-service-layers","text":"EdgeX Foundry is a collection of open source micro services. These micro services are organized into 4 service layers, and 2 underlying augmenting system services. The Service Layers traverse from the edge of the physical realm (from the Device Services Layer), to the edge of the information realm (that of the Application Services Layer), with the Core and Supporting Services Layers at the center. The 4 Service Layers of EdgeX Foundry are as follows: Core Services Layer Supporting Services Layer Application Services Layer Device Services Layer The 2 underlying System Services of EdgeX Foundry are as follows: Security System Management","title":"EdgeX Foundry Service Layers"},{"location":"#core-services-layer","text":"Core services provide the intermediary between the north and south sides of EdgeX. As the name of these services implies, they are \u201ccore\u201d to EdgeX functionality. Core services is where most of the innate knowledge of what \u201cthings\u201d are connected, what data is flowing through, and how EdgeX is configured resides in an EdgeX instance. Core consists of the following micro services: Core data: a persistence repository and associated management service for data collected from south side objects. Command: a service that facilitates and controls actuation requests from the north side to the south side. Metadata: a repository and associated management service of metadata about the objects that are connected to EdgeX Foundry. Metadata provides the capability to provision new devices and pair them with their owning device services. Registry and Configuration: provides other EdgeX Foundry micro services with information about associated services within EdgeX Foundry and micro services configuration properties (i.e. - a repository of initialization values). Core services provide intermediary communications between the things and the IT systems.","title":"Core Services Layer"},{"location":"#supporting-services-layer","text":"The supporting services encompass a wide range of micro services to include edge analytics (also known as local analytics). Normal software application duties such as scheduler, and data clean up (also known as scrubbing in EdgeX) are performed by micro services in the supporting services layer. These services often require some amount of core services in order to function. In all cases, supporting service can be considered optional \u2013 that is they can be left out of an EdgeX deployment depending on use case needs and system resources. Supporting services include: Rules Engine: the reference implementation edge analytics service that performs if-then conditional actuation at the edge based on sensor data collected by the EdgeX instance. This service may be replaced or augmented by use case specific analytics capability. Scheduler: an internal EdgeX \u201cclock\u201d that can kick off operations in any EdgeX service. At a configuration specified time, the service will call on any EdgeX service API URL via REST to trigger an operation. For example, the scheduler service periodically calls on core data APIs to clean up old sensed events that have been successfully exported out of EdgeX. Alerts and Notifications: provides EdgeX services with a central facility to send out an alert or notification. These are notices sent to another system or to a person monitoring the EdgeX instance (internal service communications are often handled more directly).","title":"Supporting Services Layer"},{"location":"#application-services-layer","text":"Application services are the means to extract, process/transform and send sensed data from EdgeX to an endpoint or process of your choice. EdgeX today offers application service examples to send data to many of the major cloud providers (Amazon IoT Hub, Google IoT Core, Azure IoT Hub, IBM Watson IoT\u2026), to MQTT(s) topics, and HTTP(s) REST endpoints. Application services are based on the idea of a \"functions pipeline\". A functions pipeline is a collection of functions that process messages (in this case EdgeX event messages) in the order specified. The first function in a pipeline is a trigger. A trigger begins the functions pipeline execution. A trigger, for example, is something like a message landing in a message queue. Each function then acts on the message. Common functions include filtering, transformation (i.e. to XML or JSON), compression, and encryption functions. The function pipeline ends when the message has gone through all the functions and is set to a sink. Putting the resulting message into an MQTT topic to be sent to Azure or AWS is an example of a sink completing an application service.","title":"Application Services Layer"},{"location":"#device-services-layer","text":"Device services connect \u201cthings\u201d \u2013 that is sensors and devices \u2013 into the rest of EdgeX. Device services are the edge connectors interacting with the \"things\" that include, but are not limited to: alarm systems, heating and air conditioning systems in homes and office buildings, lights, machines in any industry, irrigation systems, drones, currently automated transit such as some rail systems, currently automated factories, and appliances in your home. In the future, this may include driverless cars and trucks, traffic signals, fully automated fast food facilities, fully automated self-serve grocery stores, devices taking medical readings from patients, etc. Device services may service one or a number of things or devices (sensor, actuator, etc.) at one time. A device that a device service manages, could be something other than a simple, single, physical device. The device could be another gateway (and all of that gateway's devices), a device manager, a device aggregator that acts as a device, or collection of devices, to EdgeX Foundry. The device service communicates with the devices, sensors, actuators, and other IoT objects through protocols native to each device object. The device service converts the data produced and communicated by the IoT object into a common EdgeX Foundry data structure, and sends that converted data into the core services layer, and to other micro services in other layers of EdgeX Foundry. EdgeX comes with a number of device services speaking many common IoT protocols such as Modbus, BACnet, MQTT, etc.","title":"Device Services Layer"},{"location":"#system-services-layer","text":"Security Infrastructure Security elements of EdgeX Foundry protect the data and control of devices, sensors, and other IoT objects managed by EdgeX Foundry. Based on the fact that EdgeX is a \"vendor-neutral open source software platform at the edge of the network\", the EdgeX security features are also built on a foundation of open interfaces and pluggable, replaceable modules. There are two major EdgeX security components. A security store, which is used to provide a safe place to keep the EdgeX secrets. Examples of EdgeX secrets are the database access passwords used by the other services and tokens to connect to cloud systems. An API gateway serves as the reverse proxy to restrict access to EdgeX REST resources and perform access control related works. System Management System Management facilities provide the central point of contact for external management systems to start/stop/restart EdgeX services, get the status/health of a service, or get metrics on the EdgeX services (such as memory usage) so that the EdgeX services can be monitored.","title":"System Services Layer"},{"location":"#software-development-kits-sdks","text":"Two types of SDKs are provided by EdgeX to assist in creating north and south side services \u2013 specifically to create application services and device services. SDKs for both the north and south side services make connecting new things or new cloud/enterprise systems easier by providing developers all the scaffolding code that takes care of the basic operations of the service. Thereby allowing developers to focus on specifics of their connectivity to the south or north side object without worrying about all the raw plumbing of a micro service. SDKs are language specific; meaning an SDK is written to create services in a particular programming language. Today, EdgeX offers the following SDKs: Golang Device Service SDK C Device Service SDK Golang Application Functions SDK","title":"Software Development Kits (SDKs)"},{"location":"#how-edgex-works","text":"","title":"How EdgeX Works"},{"location":"#sensor-data-collection","text":"EdgeX\u2019s primary job is to collect data from sensors and devices and make that data available to north side applications and systems. Data is collected from a sensor by a device service that speaks the protocol of that device. Example: a Modbus device service would communicate in Modbus to get a pressure reading from a Modbus pump. The device service translates the sensor data into an EdgeX event object. The device service can then either: put the event object on a message bus (which may be implemented via Redis Streams or MQTT). Subscribers to the event message on the message bus can be application services or core data or both (see step 1.1 below). send the event object to the core data service via REST communications (see step 1.2). When core data receives the event (either via message bus or REST), it persists the sensor data in the local edge database. EdgeX uses Redis as our persistence store. There is an abstraction in place to allow you to use another database (which has allowed other databases to be used in the past). Persistence is not required and can be turned off. Data is persisted in EdgeX at the edge for two basics reasons: Edge nodes are not always connected. During periods of disconnected operations, the sensor data must be saved so that it can be transmitted northbound when connectivity is restored. This is referred to as store and forward capability. In some cases, analytics of sensor data needs to look back in history in order to understand the trend and to make the right decision based on that history. If a sensor reports that it is 72\u00b0 F right now, you might want to know what the temperature was ten minutes ago before you make a decision to adjust a heating or cooling system. If the temperature was 85\u00b0 F, you may decide that adjustments to lower the room temperature you made ten minutes ago were sufficient to cool the room. It is the context of historical data that are important to local analytic decisions. When core data receives event objects from the device service via REST, it will put sensor data events on a message topic destined for application services. Redis Pub/Sub is used as the messaging infrastructure by default (step 2). MQTT or ZMQ can also be used as the messaging infrastructure between core data and the application services. The application service transforms the data as needed and pushes the data to an endpoint. It can also filter, enrich, compress, encrypt or perform other functions on the event before sending it to the endpoint (step 3). The endpoint could be an HTTP/S endpoint, an MQTT topic, a cloud system (cloud topic), etc.","title":"Sensor Data Collection"},{"location":"#edge-analytics-and-actuation","text":"In edge computing, simply collecting sensor data is only part of the job of an edge platform like EdgeX. Another important job of an edge platform is to be able to: Analyze the incoming sensor data locally Act quickly on that analysis Edge or local analytics is the processing that performs an assessment of the sensor data collected at the edge (\u201clocally\u201d) and triggers actuations or actions based on what it sees. Why edge analytics ? Local analytics are important for two reasons: Some decisions cannot afford to wait for sensor collected data to be fed back to an enterprise or cloud system and have a response returned. Additionally, some edge systems are not always connected to the enterprise or cloud \u2013 they have intermittent periods of connectivity. Local analytics allows systems to operate independently, at least for some stretches of time. For example: a shipping container\u2019s cooling system must be able to make decisions locally without the benefit of Internet connectivity for long periods of time when the ship is at sea. Local analytics also allow a system to act quickly in a low latent fashion when critical to system operations. As an extreme case, imagine that your car\u2019s airbag fired on the basis of data being sent to the cloud and analyzed for collisions. Your car has local analytics to prevent such a potentially slow and error prone delivery of the safety actuation in your automobile. EdgeX is built to act locally on data it collects from the edge. In other words, events are processed by local analytics and can be used to trigger action back down on a sensor/device. Just as application services prepare data for consumption by north side cloud systems or applications, application services can process and get EdgeX events (and the sensor data they contain) to any analytics package (see step 4). By default, EdgeX ships with a simple rules engine (the default EdgeX rules engine is eKuiper \u2013 an open source rules engine and now a sister project in LF Edge). Your own analytics package (or ML agent) could replace or augment the local rules engine. The analytic package can explore the sensor event data and make a decision to trigger actuation of a device. For example, it could check that the pressure reading of an engine is greater than 60 PSI. When such a rule is determined to be true, the analytic package calls on the core command service to trigger some action, like \u201copen a valve\u201d on some controllable device (see step 5). The core command service gets the actuation request and determines which device it needs to act on with the request; then calling on the owning device service to do the actuation (see step 6). Core command allows developers to put additional security measures or checks in place before actuating. The device service receives the request for actuation, translates that into a protocol specific request and forwards the request to the desired device (see step 7).","title":"Edge Analytics and Actuation"},{"location":"#project-release-cadence","text":"Typically, EdgeX releases twice a year; once in the spring and once in the fall. Bug fix releases may occur more often. Each EdgeX release has a code name. The code name follows an alphabetic pattern similar to Android (code names sequentially follow the alphabet). The code name of each release is named after some geographical location in the world. The honor of naming an EdgeX release is given to a community member deemed to have contributed significantly to the project. A release also has a version number. The release version follows sematic versioning to indicate the release is major or minor in scope. Major releases typically contain significant new features and functionality and are not always backward compatible with prior releases. Minor releases are backward compatible and usually contain bug fixes and fewer new features. See the project Wiki for more information on releases, versions and patches . Release Schedule Version Barcelona Oct 2017 0.5.0 California Jun 2017 0.6.0 Delhi Oct 2018 0.7.0 Edinburgh Jul 2019 1.0.0 Fuji Nov 2019 1.1.0 Geneva May 2020 1.2.0 Hanoi November 2020 1.3.0 Ireland Spring 2021 2.0.0 Jakarta Fall 2021 2.1.0 Kamukura Spring 2022 TBD Levski Fall 2022 TBD Note : minor releases of the Device Services and Application Services (along with their associated SDKs) can be release independently. Graphical User Interface, the command line interface (CLI) and other tools can be released independently. EdgeX community members convene in a meeting right at the time of a release to plan the next release and roadmap future releases. See the Project Wiki for more detailed information on releases and roadmap . EdgeX 2.0","title":"Project Release Cadence"},{"location":"#the-ireland-release","text":"The Ireland release, available June 2021, is the second major version of EdgeX. Highlights of the 2.0 release include: A new and improved set of service APIs, which eliminate a lot of technical debt and setting EdgeX up for new features in the future (such as allowing for more message based communications) Direct device service to application service communications via message bus (bypassing core data if desired or allowing it to be a secondary subscriber) Simplified device profiles Improved security New, improved and more comprehensive graphical user interface (for development and demonstration purposes) New device services for CoAP, GPIO, and LLRP (RFID protocol) An LLRP inventory application service Improved application service capability and functions (to include new filter functions) Cleaner/simpler Docker image naming and facilities to create custom Docker Compose files EdgeX 2.0 provides adopters with a platform that Has an improved API that addresses edge application needs of today and tomorrow Is more efficient and lighter (depending on use case) Is more reliable and offers better quality of service (less REST, more messaging and incorporating a number of bug fixes) Has eliminated a lot of technical debt accumulated over 4 years","title":"The Ireland Release"},{"location":"#edgex-history-and-naming","text":"EdgeX Foundry began as a project chartered by Dell IoT Marketing and developed by the Dell Client Office of the CTO as an incubation project called Project Fuse in July 2015. It was initially created to run as the IoT software application on Dell\u2019s introductory line of IoT gateways. Dell entered the project into open source through the Linux Foundation on April 24, 2017. EdgeX was formally announced and demonstrated at Hanover Messe 2017. Hanover Messe is one of the world's largest industrial trade fairs. At the fair, the Linux Foundation also announced the association of 50 founding member organizations \u2013 the EdgeX ecosystem \u2013 to help further the project and the goals of creating a universal edge platform. The name \u2018foundry\u2019 was used to draw parallels to Cloud Foundry . EdgeX Foundry is meant to be a foundry for solutions at the edge just like Cloud Foundry is a foundry for solutions in the cloud. Cloud Foundry was originated by VMWare (Dell Technologies is a major shareholder of VMWare - recall that Dell Technologies was the original creator of EdgeX). The \u2018X\u2019 in EdgeX represents the transformational aspects of the platform and allows the project name to be trademarked and to be used in efforts such as certification and certification marks. The EdgeX Foundry Logo represents the nature of its role as transformation engine between the physical OT world and the digital IT world. The EdgeX community selected the octopus as the mascot or \u201cspirit animal\u201d of the project at its inception. Its eight arms and the suckers on the arms represent the sensors. The sensors bring the data into the octopus. Actually, the octopus has nine brains in a way. It has millions of neurons running down each arm; functioning as mini-brains in each of those arms. The arms of the octopus serve as \u201clocal analytics\u201d like that offered by EdgeX. The mascot is affectionately called \u201cEdgey\u201d by the community.","title":"EdgeX History and Naming"},{"location":"V2TopLevelMigration/","text":"V2 Migration Guide EdgeX 2.0 Many backward breaking changes occurred in the EdgeX 2.0 (Ireland) release which may require some migration depending on your use case. This section describes how to migrate from V1 to V2 at a high level and refers the reader to the appropriate detail documents. The areas to consider for migrating are: Custom Compose File Database Custom Configuration Custom Device Service Custom Device Profile Custom Pre-Defined Device Custom Applications Service Security eKuiper Rules Custom Compose File The compose files for V2 have many changes from their V1 counter parts. If you have customized a V1 compose file to add additional services and/or add or modify configuration overrides, it is highly recommended that you start with the appropriate V2 compose file and re-add your customizations. It is very likely that the sections for your additional services will need to be migrated to have the proper environment overrides. Best approach is to use one of the V2 service sections that closest matches your service as a template. The latest V2 compose files can be found here: https://github.com/edgexfoundry/edgex-compose/tree/ireland Compose Builder If the add on service(s) in your custom compose file are EdgeX released device or app services, it is highly recommended that you use the Compose Builder to generate your custom compose file. The latest V2 Compose Builder can be found here: https://github.com/edgexfoundry/edgex-compose/tree/ireland/compose-builder#readme Database There currently is no migration path for the data stored in the database. The V2 data collections are stored separately from the V1 data collections in the Redis database. Redis is now the only supported database, i.e. support for Mongo has been removed. Note Since the V1 data and V2 data are stored separately, one could create a migration tool and upstream it to the EdgeX community. Warning If the database is not cleared before starting the V2 services, the old V1 data will still reside in the database taking up useful memory. It is recommended that you first wipe the database clean before starting V2 Services. That is unless you create a DB migration tool, in which case you will not want to clear the V1 data until it has been migrated. See Clearing Redis Database section below for details on how to clear the Redis database. The following sections describe what you need to be aware for the different services that create data in the database. Core Data The Event/Reading data stored by Core Data is considered transient and of little value once it has become old. The V2 versions of these data collections will be empty until new Events/Readings are received from V2 Device Services. The V1 ValueDescriptors have been removed in V2. Core Metadata Most of the data stored by Core Metadata will be recreated when the V2 versions of the Device Services start-up. The statically declared devices will automatically be created and device discovery will find and add existing devices. Any device profiles, devices, provision watchers created manually via the V1 REST APIs will have to be recreated using the V2 REST API. Any manually-applied AdministrativeState settings will also need to be re-applied. Support Notifications Any Subscriptions created via the V1 REST API will have to be recreated using the V2 REST API. The Notification and Transmission collections will be empty until new notifications are sent using EdgeX 2.0 Support Scheduler The statically declared Interval and IntervalAction will be created automatically. Any Interval and/or IntervalAction created via the V1 REST API will have to be recreated using the V2 REST API. If you have created a custom configuration with additional statically declared Interval s and IntervalActions see the TOML File section under Custom Configuration below. Application Services Application services use the database only when the Store and Forward capability is enabled. If you do not use this capability you can skip this section. This data collection only has data when that data could not be exported. It is recommended not to upgrade to V2 while the Store and Forward data collection is not empty or you are certain the data is no longer needed. You can determine if the Store and Forward data collection is empty by setting the Application Service's log level to DEBUG and look for the following message which is logged every RetryInterval : msg=\" 0 stored data items found for retrying\" Clearing Redis Database Docker When running EdgeX in Docker the simplest way to clear the database is to remove the db-data volume after stopping the V1 EdgeX services. docker-compose -f down docker volume rm $(docker volume ls -q | grep db-data) Now when the V2 EdgeX services are started the database will be cleared of the old v1 data. Snaps Because there are no tools to migrate EdgeX configuration and database, it's not possible to update the edgexfoundry snap from a V1 version to a V2 version. You must remove the V1 snap first, and then install a V2 version of the snap (available from the 2.0 track in the Snap Store). This will result in starting fresh with EdgeX V2 and all V1 data removed. Local If you are running EdgeX locally, i.e. not in Docker or snaps and in non-secure mode you can use the Redis CLI to clear the database. The CLI would have been installed when you installed Redis locally. Run the following command to clear the database: redis-cli FLUSHDB This will not work if running EdgeX V1 in running in secure mode since you will not have the random generated Redis password unless you created an Admin password when you installed Redis. Custom Configuration Consul If you have customized any EdgeX service's configuration (core, support, device, etc.) via Consul, those customization will need to be re-applied to those services' configuration in Consul once the V2 versions have started and pushed their configuration into Consul. The V2 services now use 2.0 in the Consul path rather than 1.0 . See the TOML File section below for details on migrating configuration for each of the EdgeX services. Example Consul path for V2 .../kv/edgex/core/2.0/core-data/ The same applies for custom device and application service once they have been migrated following the guides referenced in the Custom Device Service and Custom Applications Service sections below. Warning If the Consul data is not cleared prior to running the V2 services, the V1 configuration will remain and be taking up useful memory. The configuration data in Consul can be cleared by deleting the .../kv/edgex/ node with the curl command below prior to starting EdgeX 2.0. Consul is secured in EdgeX 2.0 secure-mode which will make running the command below require an access token if not done prior. curl --request DELETE http://localhost:8500/v1/kv/edgex?recurse=true` TOML File If you have custom configuration TOML files for any EdgeX service (core, support, device, etc.) that configuration will need to be migrated. See V2 Migration of Common Configuration for the details on migrating configuration common to all EdgeX services. The following are where you can find the configuration migration specifics for individual core/support the services Core Data Core Metadata Core Command Support Notifications Support Scheduler System Management Agent (DEPRECATED) Application Services Device Services (common) Device MQTT Device Camera Custom Environment Overrides If you have custom environment overrides for configuration impacted by the V2 changes you will also need to migrate your overrides to use the new name or value depending on what has changed. Refer to the links above and/or below for details for migration of common and/or the service specific configuration to determine if your overrides require migrating. Custom Device Service If you have custom Device Services they will need to be migrated to the V2 version of the Device SDK. See Device Service V2 Migration Guide for complete details. Custom Device Profile If you have custom V1 Device Profile(s) for one of the EdgeX Device Services they will need to be migrated to the V2 version of Device Profiles. See Device Service V2 Migration Guide for complete details. Custom Pre-Defined Device If you have custom V1 Pre-Defined Device(s) for one of the EdgeX Device Services they will need to be migrated to the V2 version of Pre-Defined Devices. See Device Service V2 Migration Guide for complete details. Custom Applications Service If you have custom Application Services they will need to be migrated to the V2 version of the App Functions SDK. See Application Services V2 Migration Guide for complete details. Security Settings If you have an add-on service running in secure mode you will need to set addition security service environment variables in EdgeX V2. See Configuring Add-on Service for more details. API Gateway configuration The API gateway has different tools to set TLS and acquire access tokens. See Configuring API Gateway section for complete details. Secure Consul Consul is now secured when running EdgeX 2.0 in secured mode. See Secure Consul section for complete details. Secured API Gateway Admin Port The API Gateway Admin port is now secured when running EdgeX 2.0 in secured mode. See API Gateway Admin Port (TBD) section for complete details. eKuiper Rules If you have rules defined in the eKuiper rules engine that utilize the meta() directive, you will need to migrate your rule(s) to use the new V2 meta names. The following are the meta names that have changed, added or removed. device => deviceName name => resourceName profileName ( new ) pushed ( removed ) created ( removed - use origin) modified ( removed - use origin) floatEncoding ( removed ) Example V1 to V2 rule migration V1 Rule: { \"id\": \"ruleInt64\", \"sql\": \"SELECT Int64 FROM demo WHERE meta(device) = \\\"Random-Integer-Device\\\" \", \"actions\": [ { \"mqtt\": { \"server\": \"tcp://edgex-mqtt-broker:1883\", \"topic\": \"result\", \"clientId\": \"demo_001\" } } ] } V2 Rule: { \"id\": \"ruleInt64\", \"sql\": \"SELECT Int64 FROM demo WHERE meta(deviceName) = \\\"Random-Integer-Device\\\" \", \"actions\": [ { \"mqtt\": { \"server\": \"tcp://edgex-mqtt-broker:1883\", \"topic\": \"result\", \"clientId\": \"demo_001\" } } ] }","title":"V2 Migration Guide"},{"location":"V2TopLevelMigration/#v2-migration-guide","text":"EdgeX 2.0 Many backward breaking changes occurred in the EdgeX 2.0 (Ireland) release which may require some migration depending on your use case. This section describes how to migrate from V1 to V2 at a high level and refers the reader to the appropriate detail documents. The areas to consider for migrating are: Custom Compose File Database Custom Configuration Custom Device Service Custom Device Profile Custom Pre-Defined Device Custom Applications Service Security eKuiper Rules","title":"V2 Migration Guide"},{"location":"V2TopLevelMigration/#custom-compose-file","text":"The compose files for V2 have many changes from their V1 counter parts. If you have customized a V1 compose file to add additional services and/or add or modify configuration overrides, it is highly recommended that you start with the appropriate V2 compose file and re-add your customizations. It is very likely that the sections for your additional services will need to be migrated to have the proper environment overrides. Best approach is to use one of the V2 service sections that closest matches your service as a template. The latest V2 compose files can be found here: https://github.com/edgexfoundry/edgex-compose/tree/ireland","title":"Custom Compose File"},{"location":"V2TopLevelMigration/#compose-builder","text":"If the add on service(s) in your custom compose file are EdgeX released device or app services, it is highly recommended that you use the Compose Builder to generate your custom compose file. The latest V2 Compose Builder can be found here: https://github.com/edgexfoundry/edgex-compose/tree/ireland/compose-builder#readme","title":"Compose Builder"},{"location":"V2TopLevelMigration/#database","text":"There currently is no migration path for the data stored in the database. The V2 data collections are stored separately from the V1 data collections in the Redis database. Redis is now the only supported database, i.e. support for Mongo has been removed. Note Since the V1 data and V2 data are stored separately, one could create a migration tool and upstream it to the EdgeX community. Warning If the database is not cleared before starting the V2 services, the old V1 data will still reside in the database taking up useful memory. It is recommended that you first wipe the database clean before starting V2 Services. That is unless you create a DB migration tool, in which case you will not want to clear the V1 data until it has been migrated. See Clearing Redis Database section below for details on how to clear the Redis database. The following sections describe what you need to be aware for the different services that create data in the database.","title":"Database"},{"location":"V2TopLevelMigration/#core-data","text":"The Event/Reading data stored by Core Data is considered transient and of little value once it has become old. The V2 versions of these data collections will be empty until new Events/Readings are received from V2 Device Services. The V1 ValueDescriptors have been removed in V2.","title":"Core Data"},{"location":"V2TopLevelMigration/#core-metadata","text":"Most of the data stored by Core Metadata will be recreated when the V2 versions of the Device Services start-up. The statically declared devices will automatically be created and device discovery will find and add existing devices. Any device profiles, devices, provision watchers created manually via the V1 REST APIs will have to be recreated using the V2 REST API. Any manually-applied AdministrativeState settings will also need to be re-applied.","title":"Core Metadata"},{"location":"V2TopLevelMigration/#support-notifications","text":"Any Subscriptions created via the V1 REST API will have to be recreated using the V2 REST API. The Notification and Transmission collections will be empty until new notifications are sent using EdgeX 2.0","title":"Support Notifications"},{"location":"V2TopLevelMigration/#support-scheduler","text":"The statically declared Interval and IntervalAction will be created automatically. Any Interval and/or IntervalAction created via the V1 REST API will have to be recreated using the V2 REST API. If you have created a custom configuration with additional statically declared Interval s and IntervalActions see the TOML File section under Custom Configuration below.","title":"Support Scheduler"},{"location":"V2TopLevelMigration/#application-services","text":"Application services use the database only when the Store and Forward capability is enabled. If you do not use this capability you can skip this section. This data collection only has data when that data could not be exported. It is recommended not to upgrade to V2 while the Store and Forward data collection is not empty or you are certain the data is no longer needed. You can determine if the Store and Forward data collection is empty by setting the Application Service's log level to DEBUG and look for the following message which is logged every RetryInterval : msg=\" 0 stored data items found for retrying\"","title":"Application Services"},{"location":"V2TopLevelMigration/#clearing-redis-database","text":"","title":"Clearing Redis Database"},{"location":"V2TopLevelMigration/#docker","text":"When running EdgeX in Docker the simplest way to clear the database is to remove the db-data volume after stopping the V1 EdgeX services. docker-compose -f down docker volume rm $(docker volume ls -q | grep db-data) Now when the V2 EdgeX services are started the database will be cleared of the old v1 data.","title":"Docker"},{"location":"V2TopLevelMigration/#snaps","text":"Because there are no tools to migrate EdgeX configuration and database, it's not possible to update the edgexfoundry snap from a V1 version to a V2 version. You must remove the V1 snap first, and then install a V2 version of the snap (available from the 2.0 track in the Snap Store). This will result in starting fresh with EdgeX V2 and all V1 data removed.","title":"Snaps"},{"location":"V2TopLevelMigration/#local","text":"If you are running EdgeX locally, i.e. not in Docker or snaps and in non-secure mode you can use the Redis CLI to clear the database. The CLI would have been installed when you installed Redis locally. Run the following command to clear the database: redis-cli FLUSHDB This will not work if running EdgeX V1 in running in secure mode since you will not have the random generated Redis password unless you created an Admin password when you installed Redis.","title":"Local"},{"location":"V2TopLevelMigration/#custom-configuration","text":"","title":"Custom Configuration"},{"location":"V2TopLevelMigration/#consul","text":"If you have customized any EdgeX service's configuration (core, support, device, etc.) via Consul, those customization will need to be re-applied to those services' configuration in Consul once the V2 versions have started and pushed their configuration into Consul. The V2 services now use 2.0 in the Consul path rather than 1.0 . See the TOML File section below for details on migrating configuration for each of the EdgeX services. Example Consul path for V2 .../kv/edgex/core/2.0/core-data/ The same applies for custom device and application service once they have been migrated following the guides referenced in the Custom Device Service and Custom Applications Service sections below. Warning If the Consul data is not cleared prior to running the V2 services, the V1 configuration will remain and be taking up useful memory. The configuration data in Consul can be cleared by deleting the .../kv/edgex/ node with the curl command below prior to starting EdgeX 2.0. Consul is secured in EdgeX 2.0 secure-mode which will make running the command below require an access token if not done prior. curl --request DELETE http://localhost:8500/v1/kv/edgex?recurse=true`","title":"Consul"},{"location":"V2TopLevelMigration/#toml-file","text":"If you have custom configuration TOML files for any EdgeX service (core, support, device, etc.) that configuration will need to be migrated. See V2 Migration of Common Configuration for the details on migrating configuration common to all EdgeX services. The following are where you can find the configuration migration specifics for individual core/support the services Core Data Core Metadata Core Command Support Notifications Support Scheduler System Management Agent (DEPRECATED) Application Services Device Services (common) Device MQTT Device Camera","title":"TOML File"},{"location":"V2TopLevelMigration/#custom-environment-overrides","text":"If you have custom environment overrides for configuration impacted by the V2 changes you will also need to migrate your overrides to use the new name or value depending on what has changed. Refer to the links above and/or below for details for migration of common and/or the service specific configuration to determine if your overrides require migrating.","title":"Custom Environment Overrides"},{"location":"V2TopLevelMigration/#custom-device-service","text":"If you have custom Device Services they will need to be migrated to the V2 version of the Device SDK. See Device Service V2 Migration Guide for complete details.","title":"Custom Device Service"},{"location":"V2TopLevelMigration/#custom-device-profile","text":"If you have custom V1 Device Profile(s) for one of the EdgeX Device Services they will need to be migrated to the V2 version of Device Profiles. See Device Service V2 Migration Guide for complete details.","title":"Custom Device Profile"},{"location":"V2TopLevelMigration/#custom-pre-defined-device","text":"If you have custom V1 Pre-Defined Device(s) for one of the EdgeX Device Services they will need to be migrated to the V2 version of Pre-Defined Devices. See Device Service V2 Migration Guide for complete details.","title":"Custom Pre-Defined Device"},{"location":"V2TopLevelMigration/#custom-applications-service","text":"If you have custom Application Services they will need to be migrated to the V2 version of the App Functions SDK. See Application Services V2 Migration Guide for complete details.","title":"Custom Applications Service"},{"location":"V2TopLevelMigration/#security","text":"","title":"Security"},{"location":"V2TopLevelMigration/#settings","text":"If you have an add-on service running in secure mode you will need to set addition security service environment variables in EdgeX V2. See Configuring Add-on Service for more details.","title":"Settings"},{"location":"V2TopLevelMigration/#api-gateway-configuration","text":"The API gateway has different tools to set TLS and acquire access tokens. See Configuring API Gateway section for complete details.","title":"API Gateway configuration"},{"location":"V2TopLevelMigration/#secure-consul","text":"Consul is now secured when running EdgeX 2.0 in secured mode. See Secure Consul section for complete details.","title":"Secure Consul"},{"location":"V2TopLevelMigration/#secured-api-gateway-admin-port","text":"The API Gateway Admin port is now secured when running EdgeX 2.0 in secured mode. See API Gateway Admin Port (TBD) section for complete details.","title":"Secured API Gateway Admin Port"},{"location":"V2TopLevelMigration/#ekuiper-rules","text":"If you have rules defined in the eKuiper rules engine that utilize the meta() directive, you will need to migrate your rule(s) to use the new V2 meta names. The following are the meta names that have changed, added or removed. device => deviceName name => resourceName profileName ( new ) pushed ( removed ) created ( removed - use origin) modified ( removed - use origin) floatEncoding ( removed ) Example V1 to V2 rule migration V1 Rule: { \"id\": \"ruleInt64\", \"sql\": \"SELECT Int64 FROM demo WHERE meta(device) = \\\"Random-Integer-Device\\\" \", \"actions\": [ { \"mqtt\": { \"server\": \"tcp://edgex-mqtt-broker:1883\", \"topic\": \"result\", \"clientId\": \"demo_001\" } } ] } V2 Rule: { \"id\": \"ruleInt64\", \"sql\": \"SELECT Int64 FROM demo WHERE meta(deviceName) = \\\"Random-Integer-Device\\\" \", \"actions\": [ { \"mqtt\": { \"server\": \"tcp://edgex-mqtt-broker:1883\", \"topic\": \"result\", \"clientId\": \"demo_001\" } } ] }","title":"eKuiper Rules"},{"location":"api/Ch-APIIntroduction/","text":"Introduction Each of the EdgeX services (core, supporting, management, device and application) implement a RESTful API. This section provides details about each service's API. You will see there is a common set of API's that all services implement, which are: Version Metrics Config Ping Each Edgex Service's RESTful API is documented via Swagger. A link is provided to the swagger document in the service specific documentation. Also included in this API Reference are a couple 3rd party services (Configuration/Registry and Rules Engine). These services do not implement the above common APIs and don't not have swagger documentation. Links are provided to their appropriate documentation. See the left side navigation for complete list of services to access their API Reference. EdgeX 2.0 For EdgeX 2.0 all the EdgeX services use new DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all or /label/{label}, provide offset and limit query parameters.","title":"Introduction"},{"location":"api/Ch-APIIntroduction/#introduction","text":"Each of the EdgeX services (core, supporting, management, device and application) implement a RESTful API. This section provides details about each service's API. You will see there is a common set of API's that all services implement, which are: Version Metrics Config Ping Each Edgex Service's RESTful API is documented via Swagger. A link is provided to the swagger document in the service specific documentation. Also included in this API Reference are a couple 3rd party services (Configuration/Registry and Rules Engine). These services do not implement the above common APIs and don't not have swagger documentation. Links are provided to their appropriate documentation. See the left side navigation for complete list of services to access their API Reference. EdgeX 2.0 For EdgeX 2.0 all the EdgeX services use new DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all or /label/{label}, provide offset and limit query parameters.","title":"Introduction"},{"location":"api/applications/Ch-APIAppFunctionsSDK/","text":"Application Services The App Functions SDK is provided to help build Application Services by assembling triggers, pre-existing functions and custom functions of your making into a functions pipeline. This functions pipeline processes messages received by the configured trigger. See Application Functions SDK for more details on this SDK. The App Functions SDK provides a RESTful API that all Application Services inherit from the SDK. Application Service SDK V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the App Functions SDK has changed to use DTOs (Data Transfer Objects) for all responses and for POST requests. One exception is the /api/v2/trigger endpoint that is enabled when the Trigger is configured to be http . This endpoint accepts any data POSTed to it.","title":"Application Services"},{"location":"api/applications/Ch-APIAppFunctionsSDK/#application-services","text":"The App Functions SDK is provided to help build Application Services by assembling triggers, pre-existing functions and custom functions of your making into a functions pipeline. This functions pipeline processes messages received by the configured trigger. See Application Functions SDK for more details on this SDK. The App Functions SDK provides a RESTful API that all Application Services inherit from the SDK. Application Service SDK V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the App Functions SDK has changed to use DTOs (Data Transfer Objects) for all responses and for POST requests. One exception is the /api/v2/trigger endpoint that is enabled when the Trigger is configured to be http . This endpoint accepts any data POSTed to it.","title":"Application Services"},{"location":"api/applications/Ch-APIRulesEngine/","text":"Rules Engine EdgeX Foundry Rules Engine Microservice receives data from the instance of App Service Configurable running the rules-engine profile (aka app-rules-engine) via the EdgeX MessageBus. EdgeX uses eKuiper for the rules engine, which is a separate LF Edge project. See the eKuiper Website for more details on this rules engine. eKuiper's documentation","title":"Rules Engine"},{"location":"api/applications/Ch-APIRulesEngine/#rules-engine","text":"EdgeX Foundry Rules Engine Microservice receives data from the instance of App Service Configurable running the rules-engine profile (aka app-rules-engine) via the EdgeX MessageBus. EdgeX uses eKuiper for the rules engine, which is a separate LF Edge project. See the eKuiper Website for more details on this rules engine. eKuiper's documentation","title":"Rules Engine"},{"location":"api/core/Ch-APICoreCommand/","text":"Core Command EdgeX Foundry's Command microservice is a conduit for other services to trigger action on devices and sensors through their managing Device Services. See Core Command for more details about this service. The service provides an API to get the list of commands that can be issued for all devices or a single device. Commands are divided into two groups for each device: GET commands are issued to a device or sensor to get a current value for a particular attribute on the device, such as the current temperature provided by a thermostat sensor, or the on/off status of a light. SET commands are issued to a device or sensor to change the current state or status of a device or one of its attributes, such as setting the speed in RPMs of a motor, or setting the brightness of a dimmer light. Core Command V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Core Command has changed to use DTOs (Data Transfer Objects) for all responses and for all PUT requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Core Command"},{"location":"api/core/Ch-APICoreCommand/#core-command","text":"EdgeX Foundry's Command microservice is a conduit for other services to trigger action on devices and sensors through their managing Device Services. See Core Command for more details about this service. The service provides an API to get the list of commands that can be issued for all devices or a single device. Commands are divided into two groups for each device: GET commands are issued to a device or sensor to get a current value for a particular attribute on the device, such as the current temperature provided by a thermostat sensor, or the on/off status of a light. SET commands are issued to a device or sensor to change the current state or status of a device or one of its attributes, such as setting the speed in RPMs of a motor, or setting the brightness of a dimmer light. Core Command V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Core Command has changed to use DTOs (Data Transfer Objects) for all responses and for all PUT requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Core Command"},{"location":"api/core/Ch-APICoreConfigurationAndRegistry/","text":"Configuration and Registry EdgeX uses the 3rd party Consul microservice as the implementations for Configuration and Registry. The RESTful APIs are provided by Consul directly, and several communities supply Consul client libraries for different programming languages, including Go (official), Python, Java, PHP, Scala, Erlang/OTP, Ruby, Node.js, and C#. EdgeX 2.0 New for Edgex 2.0 is Secure Consul when running EdgeX in secure mode. See the Secure Consul section for more details. For the client libraries of different languages, please refer to the list on this page: https://www.consul.io/downloads_tools.html Configuration Management For the current API documentation, please refer to the official Consul web site: https://www.consul.io/intro/getting-started/kv.html https://www.consul.io/docs/agent/http/kv.html Service Registry For the current API documentation, please refer to the official Consul web site: https://www.consul.io/intro/getting-started/services.html https://www.consul.io/docs/agent/http/catalog.html https://www.consul.io/docs/agent/http/agent.html https://www.consul.io/docs/agent/checks.html https://www.consul.io/docs/agent/http/health.html Service Registration While each microservice is starting up, it will connect to Consul to register its endpoint information, including microservice ID, address, port number, and health checking method. After that, other microservices can locate its URL from Consul, and Consul has the ability to monitor its health status. The RESTful API of registration is described on the following Consul page: https://www.consul.io/docs/agent/http/agent.html#agent_service_register Service Deregistration Before microservices shut down, they have to deregister themselves from Consul. The RESTful API of deregistration is described on the following Consul page: https://www.consul.io/docs/agent/http/agent.html#agent_service_deregister Service Discovery Service Discovery feature allows client micro services to query the endpoint information of a particular microservice by its microservice IDor list all available services registered in Consul. The RESTful API of querying service by microservice IDis described on the following Consul page: https://www.consul.io/docs/agent/http/catalog.html#catalog_service The RESTful API of listing all available services is described on the following Consul page: https://www.consul.io/docs/agent/http/agent.html#agent_services Health Checking Health checking is a critical feature that prevents using services that are unhealthy. Consul provides a variety of methods to check the health of services, including Script + Interval, HTTP + Interval, TCP + Interval, Time to Live (TTL), and Docker + Interval. The detailed introduction and examples of each checking methods are described on the following Consul page: https://www.consul.io/docs/agent/checks.html The health checks should be established during service registration. Please see the paragraph on this page of Service Registration section. Consul UI Consul has UI which allows you to view the health of registered services and view/edit services' individual configuration. Learn more about the UI on the following Consul page: https://learn.hashicorp.com/tutorials/consul/get-started-explore-the-ui EdgeX 2.0 Please note that as of EdgeX 2.0, Consul can be secured. When EdgeX is running in secure mode with secure Consul , you must provide Consul's access token to get to the UI referenced above. See How to get Consul ACL token for details.","title":"Configuration and Registry"},{"location":"api/core/Ch-APICoreConfigurationAndRegistry/#configuration-and-registry","text":"EdgeX uses the 3rd party Consul microservice as the implementations for Configuration and Registry. The RESTful APIs are provided by Consul directly, and several communities supply Consul client libraries for different programming languages, including Go (official), Python, Java, PHP, Scala, Erlang/OTP, Ruby, Node.js, and C#. EdgeX 2.0 New for Edgex 2.0 is Secure Consul when running EdgeX in secure mode. See the Secure Consul section for more details. For the client libraries of different languages, please refer to the list on this page: https://www.consul.io/downloads_tools.html","title":"Configuration and Registry"},{"location":"api/core/Ch-APICoreConfigurationAndRegistry/#configuration-management","text":"For the current API documentation, please refer to the official Consul web site: https://www.consul.io/intro/getting-started/kv.html https://www.consul.io/docs/agent/http/kv.html","title":"Configuration Management"},{"location":"api/core/Ch-APICoreConfigurationAndRegistry/#service-registry","text":"For the current API documentation, please refer to the official Consul web site: https://www.consul.io/intro/getting-started/services.html https://www.consul.io/docs/agent/http/catalog.html https://www.consul.io/docs/agent/http/agent.html https://www.consul.io/docs/agent/checks.html https://www.consul.io/docs/agent/http/health.html Service Registration While each microservice is starting up, it will connect to Consul to register its endpoint information, including microservice ID, address, port number, and health checking method. After that, other microservices can locate its URL from Consul, and Consul has the ability to monitor its health status. The RESTful API of registration is described on the following Consul page: https://www.consul.io/docs/agent/http/agent.html#agent_service_register Service Deregistration Before microservices shut down, they have to deregister themselves from Consul. The RESTful API of deregistration is described on the following Consul page: https://www.consul.io/docs/agent/http/agent.html#agent_service_deregister Service Discovery Service Discovery feature allows client micro services to query the endpoint information of a particular microservice by its microservice IDor list all available services registered in Consul. The RESTful API of querying service by microservice IDis described on the following Consul page: https://www.consul.io/docs/agent/http/catalog.html#catalog_service The RESTful API of listing all available services is described on the following Consul page: https://www.consul.io/docs/agent/http/agent.html#agent_services Health Checking Health checking is a critical feature that prevents using services that are unhealthy. Consul provides a variety of methods to check the health of services, including Script + Interval, HTTP + Interval, TCP + Interval, Time to Live (TTL), and Docker + Interval. The detailed introduction and examples of each checking methods are described on the following Consul page: https://www.consul.io/docs/agent/checks.html The health checks should be established during service registration. Please see the paragraph on this page of Service Registration section.","title":"Service Registry"},{"location":"api/core/Ch-APICoreConfigurationAndRegistry/#consul-ui","text":"Consul has UI which allows you to view the health of registered services and view/edit services' individual configuration. Learn more about the UI on the following Consul page: https://learn.hashicorp.com/tutorials/consul/get-started-explore-the-ui EdgeX 2.0 Please note that as of EdgeX 2.0, Consul can be secured. When EdgeX is running in secure mode with secure Consul , you must provide Consul's access token to get to the UI referenced above. See How to get Consul ACL token for details.","title":"Consul UI"},{"location":"api/core/Ch-APICoreData/","text":"Core Data EdgeX Foundry Core Data microservice includes the Events/Readings database collected from devices /sensors and APIs to expose this database to other services. Its APIs to provide access to Add, Query and Delete Events/Readings. See Core Data for more details about this service. Core Data V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Core Data has changed to use DTOs (Data Transfer Objects) for all responses and for all POST requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Core Data"},{"location":"api/core/Ch-APICoreData/#core-data","text":"EdgeX Foundry Core Data microservice includes the Events/Readings database collected from devices /sensors and APIs to expose this database to other services. Its APIs to provide access to Add, Query and Delete Events/Readings. See Core Data for more details about this service. Core Data V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Core Data has changed to use DTOs (Data Transfer Objects) for all responses and for all POST requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Core Data"},{"location":"api/core/Ch-APICoreMetadata/","text":"Core Metadata The Core Metadata microservice includes the device/sensor metadata database and APIs to expose this database to other services. In particular, the device provisioning service deposits and manages device metadata through this service's API. See Core Metadata for more details about this service. Core Metadata V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Core Metadata has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Core Metadata"},{"location":"api/core/Ch-APICoreMetadata/#core-metadata","text":"The Core Metadata microservice includes the device/sensor metadata database and APIs to expose this database to other services. In particular, the device provisioning service deposits and manages device metadata through this service's API. See Core Metadata for more details about this service. Core Metadata V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Core Metadata has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Core Metadata"},{"location":"api/devices/Ch-APIDeviceSDK/","text":"Device Services The EdgeX Foundry Device Service Software Development Kit (SDK) takes the Developer through the step-by-step process to create an EdgeX Foundry Device Service microservice. See Device Service SDK for more details on this SDK. The Device Service SDK provides a RESTful API that all Device Services inherit from the SDK. Device SDK V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Device Service SDK has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT requests.","title":"Device Services"},{"location":"api/devices/Ch-APIDeviceSDK/#device-services","text":"The EdgeX Foundry Device Service Software Development Kit (SDK) takes the Developer through the step-by-step process to create an EdgeX Foundry Device Service microservice. See Device Service SDK for more details on this SDK. The Device Service SDK provides a RESTful API that all Device Services inherit from the SDK. Device SDK V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Device Service SDK has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT requests.","title":"Device Services"},{"location":"api/management/Ch-APISystemManagement/","text":"System Management Agent EdgeX 2.0 System Management Agent has been deprecated for EdgeX 2.0. While it is still available, it may be removed in a future release and no further develop is planned for it. The EdgeX System Management Agent (SMA) microservice exposes the EdgeX management service API to 3rd party systems. In other words, the Agent serves as a proxy for system management service API calls into each micro service. See System Management Agent for more details about this service. System Management V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the System Management Agent has changed to use DTOs (Data Transfer Objects) for all responses and for all POST requests.","title":"System Management Agent"},{"location":"api/management/Ch-APISystemManagement/#system-management-agent","text":"EdgeX 2.0 System Management Agent has been deprecated for EdgeX 2.0. While it is still available, it may be removed in a future release and no further develop is planned for it. The EdgeX System Management Agent (SMA) microservice exposes the EdgeX management service API to 3rd party systems. In other words, the Agent serves as a proxy for system management service API calls into each micro service. See System Management Agent for more details about this service. System Management V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the System Management Agent has changed to use DTOs (Data Transfer Objects) for all responses and for all POST requests.","title":"System Management Agent"},{"location":"api/support/Ch-APISupportNotifications/","text":"Support Notifications When a person or a system needs to be informed of something discovered on the node by another microservice on the node, EdgeX Foundry's Support Notifications microservice delivers that information. Examples of Alerts and Notifications that other services might need to broadcast include sensor data detected outside of certain parameters, usually detected by a Rules Engine service, or a system or service malfunction usually detected by system management services. See Support Notifications for more details about this service. Support Notifications V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Support Notifications has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Support Notifications"},{"location":"api/support/Ch-APISupportNotifications/#support-notifications","text":"When a person or a system needs to be informed of something discovered on the node by another microservice on the node, EdgeX Foundry's Support Notifications microservice delivers that information. Examples of Alerts and Notifications that other services might need to broadcast include sensor data detected outside of certain parameters, usually detected by a Rules Engine service, or a system or service malfunction usually detected by system management services. See Support Notifications for more details about this service. Support Notifications V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Support Notifications has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Support Notifications"},{"location":"api/support/Ch-APISupportScheduler/","text":"Support Scheduler EdgeX Foundry's Support Scheduler microservice to schedule actions to occur on specific intervals. See Support Scheduler for more details about this service. Support Scheduler V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Support Scheduler has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Support Scheduler"},{"location":"api/support/Ch-APISupportScheduler/#support-scheduler","text":"EdgeX Foundry's Support Scheduler microservice to schedule actions to occur on specific intervals. See Support Scheduler for more details about this service. Support Scheduler V2 API Swagger Documentation EdgeX 2.0 For EdgeX 2.0 the REST API provided by the Support Scheduler has changed to use DTOs (Data Transfer Objects) for all responses and for all POST/PUT/PATCH requests. All query APIs (GET) which return multiple objects, such as /all, provide offset and limit query parameters.","title":"Support Scheduler"},{"location":"design/","text":"Architecture Decision Records Folder This folder contains EdgeX Foundry decision records (ADR) and legacy design / requirement documents. /design /adr (architecture decision Records) /legacy-design (legacy design documents) /legacy-requirements (legacy requirement documents) At the root of the ADR folder (/design/adr) are decisions that are relevant to multiple parts of the project (aka \ufffd cross cutting concerns ). Sub folders under the ADR folder contain decisions relevant to the specific area of the project and essentially set up along working group lines (security, core, application, etc.). Naming and Formatting ADR documents are requested to follow RFC (request for comments) naming standard. Specifically, authors should name their documents with a sequentially increasing integer (or serial number) and then the architectural design topic: (sequence number - topic). Example: 0001-SeparateConfigurationInterface. The sequence is a global sequence for all EdgeX ADR. Per RFC and Michael Nygard suggestions the makeup of the ADR document should generally include: Title Status (proposed, accepted, rejected, deprecated, superseded, etc.) Context and Proposed Design Decision Consequences/considerations References Document history is maintained via Github history. Ownership EdgeX WG chairman own the sub folder and included documents associated to their work group. The EdgeX TSC chair/vice chair are responsible for the root level, cross cutting concern documents. Review and Approval ADR\u2019s shall be submitted as PRs to the appropriate edgex-docs folder based on the Architecture Decision Records Folder section above. The status of the PR (inside the document) shall be listed as proposed during this period. The PRs shall be left open (not merged) so that comments against the PR can be collected during the proposal period. The PRs can be approved and merged only after a formal vote of approval is conducted by the TSC. On approval of the ADR by the TSC, the status of the ADR should be changed to accepted . If the ADR is not approved by the TSC, the status in the document should be changed to rejected and the PR closed. Legacy A separate folder (/design/legacy-design) is used for legacy design/architecture decisions. A separate folder (/design/legacy-requirements) is used for legacy requirements documents. WG chairman take the responsibility for posting legacy material in to the applicable folders. Table of Contents A README with a table of contents for current documents is located here . Legacy Design and Requirements have their own Table of Contents as well and are located in their respective directories at /legacy-design and /legacy-requirements . Document authors are asked to keep the TOC updated with each new document entry.","title":"Architecture Decision Records Folder"},{"location":"design/#architecture-decision-records-folder","text":"This folder contains EdgeX Foundry decision records (ADR) and legacy design / requirement documents. /design /adr (architecture decision Records) /legacy-design (legacy design documents) /legacy-requirements (legacy requirement documents) At the root of the ADR folder (/design/adr) are decisions that are relevant to multiple parts of the project (aka \ufffd cross cutting concerns ). Sub folders under the ADR folder contain decisions relevant to the specific area of the project and essentially set up along working group lines (security, core, application, etc.).","title":"Architecture Decision Records Folder"},{"location":"design/#naming-and-formatting","text":"ADR documents are requested to follow RFC (request for comments) naming standard. Specifically, authors should name their documents with a sequentially increasing integer (or serial number) and then the architectural design topic: (sequence number - topic). Example: 0001-SeparateConfigurationInterface. The sequence is a global sequence for all EdgeX ADR. Per RFC and Michael Nygard suggestions the makeup of the ADR document should generally include: Title Status (proposed, accepted, rejected, deprecated, superseded, etc.) Context and Proposed Design Decision Consequences/considerations References Document history is maintained via Github history.","title":"Naming and Formatting"},{"location":"design/#ownership","text":"EdgeX WG chairman own the sub folder and included documents associated to their work group. The EdgeX TSC chair/vice chair are responsible for the root level, cross cutting concern documents.","title":"Ownership"},{"location":"design/#review-and-approval","text":"ADR\u2019s shall be submitted as PRs to the appropriate edgex-docs folder based on the Architecture Decision Records Folder section above. The status of the PR (inside the document) shall be listed as proposed during this period. The PRs shall be left open (not merged) so that comments against the PR can be collected during the proposal period. The PRs can be approved and merged only after a formal vote of approval is conducted by the TSC. On approval of the ADR by the TSC, the status of the ADR should be changed to accepted . If the ADR is not approved by the TSC, the status in the document should be changed to rejected and the PR closed.","title":"Review and Approval"},{"location":"design/#legacy","text":"A separate folder (/design/legacy-design) is used for legacy design/architecture decisions. A separate folder (/design/legacy-requirements) is used for legacy requirements documents. WG chairman take the responsibility for posting legacy material in to the applicable folders.","title":"Legacy"},{"location":"design/#table-of-contents","text":"A README with a table of contents for current documents is located here . Legacy Design and Requirements have their own Table of Contents as well and are located in their respective directories at /legacy-design and /legacy-requirements . Document authors are asked to keep the TOC updated with each new document entry.","title":"Table of Contents"},{"location":"design/TOC/","text":"ADR Table of Contents Name/Link Short Description 0001 Registry Refactor Separate out Registry and Configuration APIs 0002 Array Datatypes Allow Arrays to be held in Readings 0003 V2 API Principles Principles and Goals of V2 API Design 0004 Feature Flags Feature Flag Implementation 0005 Service Self Config Init Service Self Config Init & Config Seed Removal 0006 Metrics Collection Collection of service telemetry data 0007 Release Automation Overview of Release Automation Flow for EdgeX 0008 Secret Distribution Creation and Distribution of Secrets 0009 Secure Bootstrapping Secure Bootstrapping of EdgeX 0011 Device Service REST API The REST API for Device Services in EdgeX v2.x 0012 Device Service Filters Device Service event/reading filters 0013 Device Service Events via Message Bus Device Services send Events via Message Bus 0014 Secret Provider for All Secret Provider for All EdgeX Services 0015 Encryption between microservices Details conditions under which TLS is or is not used 0016 Container Image Guidelines Documents best practices for security of docker images 0017 Securing access to Consul Access control and authorization strategy for Consul 0018 Service Registry Service registry usage for EdgeX services 0019 EdgeX-CLI V2 EdgeX-CLI V2 Implementation 0020 Delay start services (SPIFFE/SPIRE) Secret store tokens for delayed start services 0021 Device Profile Changes Rules on device profile modifications 0022 Unit of Measure Unit of Measure 0023 North South Messaging Provide for messaging from north side systems through command down to device services","title":"ADR Table of Contents"},{"location":"design/TOC/#adr-table-of-contents","text":"Name/Link Short Description 0001 Registry Refactor Separate out Registry and Configuration APIs 0002 Array Datatypes Allow Arrays to be held in Readings 0003 V2 API Principles Principles and Goals of V2 API Design 0004 Feature Flags Feature Flag Implementation 0005 Service Self Config Init Service Self Config Init & Config Seed Removal 0006 Metrics Collection Collection of service telemetry data 0007 Release Automation Overview of Release Automation Flow for EdgeX 0008 Secret Distribution Creation and Distribution of Secrets 0009 Secure Bootstrapping Secure Bootstrapping of EdgeX 0011 Device Service REST API The REST API for Device Services in EdgeX v2.x 0012 Device Service Filters Device Service event/reading filters 0013 Device Service Events via Message Bus Device Services send Events via Message Bus 0014 Secret Provider for All Secret Provider for All EdgeX Services 0015 Encryption between microservices Details conditions under which TLS is or is not used 0016 Container Image Guidelines Documents best practices for security of docker images 0017 Securing access to Consul Access control and authorization strategy for Consul 0018 Service Registry Service registry usage for EdgeX services 0019 EdgeX-CLI V2 EdgeX-CLI V2 Implementation 0020 Delay start services (SPIFFE/SPIRE) Secret store tokens for delayed start services 0021 Device Profile Changes Rules on device profile modifications 0022 Unit of Measure Unit of Measure 0023 North South Messaging Provide for messaging from north side systems through command down to device services","title":"ADR Table of Contents"},{"location":"design/adr/0001-Registy-Refactor/","text":"Registry Refactoring Design Status Context Proposed Design Decision Consequences References Status Approved Context Currently the Registry Client in go-mod-registry module provides Service Configuration and Service Registration functionality. The goal of this design is to refactor the go-mod-registry module for separation of concerns. The Service Registry functionality will stay in the go-mod-registry module and the Service Configuration functionality will be separated out into a new go-mod-configuration module. This allows for implementations for deferent providers for each, another aspect of separation of concerns. Proposed Design Provider Connection information An aspect of using the current Registry Client is \" Where do the services get the Registry Provider connection information? \" Currently all services either pull this connection information from the local configuration file or from the edgex_registry environment variable. Device Services also have the option to specify this connection information on the command line. With the refactoring for separation of concerns, this issue changes to \" Where do the services get the Configuration Provider connection information? \" There have been concerns voiced by some in the EdgeX community that storing this Configuration Provider connection information in the configuration which ultimately is provided by that provider is not the right design. This design proposes that all services will use the command line option approach with the ability to override with an environment variable. The Configuration Provider information will not be stored in each service's local configuration file. The edgex_registry environment variable will be deprecated. The Registry Provider connection information will continue to be stored in each service's configuration either locally or from the Configuration Provider same as all other EdgeX Client and Database connection information. Command line option changes The new -cp/-configProvider command line option will be added to each service which will have a value specified using the format {type}.{protocol}://{host}:{port} e.g consul.http://localhost:8500 . This new command line option will be overridden by the edgex_configuration_provider environment variable when it is set. This environment variable's value has the same format as the command line option value. If no value is provided to the -cp/-configProvider option, i.e. just -cp , and no environment variable override is specified, the default value of consul.http://localhost:8500 will be used. if -cp/-configProvider not used and no environment variable override is specified the local configuration file is used, as is it now. All services will log the Configuration Provider connection information that is used. The existing -r/-registry command line option will be retained as a Boolean flag to indicate to use the Registry. Bootstrap Changes All services in the edgex-go mono repo use the new common bootstrap functionality. The plan is to move this code to a go module for the Device Service and App Functions SDKs to also use. The current bootstrap modules pkg/bootstrap/configuration/registry.go and pkg/bootstrap/container/registry.go will be refactored to use the new Configuration Client and be renamed appropriately. New bootstrap modules will be created for using the revised version of Registry Client . The current use of useRegistry and registryClient for service configuration will be change to appropriate names for using the new Configuration Client . The current use of useRegistry and registryClient for service registration will be retained for service registration. Call to the new Unregister() API will be added to shutdown code for all services. Config-Seed Changes The conf-seed service will have similar changes for specifying the Configuration Provider connection information since it doesn't use the common bootstrap package. Beyond that it will have minor changes for switching to using the Configuration Client interface, which will just be imports and appropriate name refactoring. Config Endpoint Changes Since the Configuration Provider connection information will no longer be in the service's configuration struct, the config endpoint processing will be modified to add the Configuration Provider connection information to the resulting JSON create from service's configuration. Client Interfaces changes Current Registry Client This following is the current Registry Client Interface type Client interface { Register () error HasConfiguration () ( bool , error ) PutConfigurationToml ( configuration * toml . Tree , overwrite bool ) error PutConfiguration ( configStruct interface {}, overwrite bool ) error GetConfiguration ( configStruct interface {}) ( interface {}, error ) WatchForChanges ( updateChannel chan <- interface {}, errorChannel chan <- error , configuration interface {}, waitKey string ) IsAlive () bool ConfigurationValueExists ( name string ) ( bool , error ) GetConfigurationValue ( name string ) ([] byte , error ) PutConfigurationValue ( name string , value [] byte ) error GetServiceEndpoint ( serviceId string ) ( types . ServiceEndpoint , error ) IsServiceAvailable ( serviceId string ) error } New Configuration Client This following is the new Configuration Client Interface which contains the Service Configuration specific portion from the above current Registry Client . type Client interface { HasConfiguration () ( bool , error ) PutConfigurationFromToml ( configuration * toml . Tree , overwrite bool ) error PutConfiguration ( configStruct interface {}, overwrite bool ) error GetConfiguration ( configStruct interface {}) ( interface {}, error ) WatchForChanges ( updateChannel chan <- interface {}, errorChannel chan <- error , configuration interface {}, waitKey string ) IsAlive () bool ConfigurationValueExists ( name string ) ( bool , error ) GetConfigurationValue ( name string ) ([] byte , error ) PutConfigurationValue ( name string , value [] byte ) error } Revised Registry Client This following is the revised Registry Client Interface, which contains the Service Registry specific portion from the above current Registry Client . The UnRegister() API has been added per issue #20 type Client interface { Register () error UnRegister () error IsAlive () bool GetServiceEndpoint ( serviceId string ) ( types . ServiceEndpoint , error ) IsServiceAvailable ( serviceId string ) error } Client Configuration Structs Current Registry Client Config The following is the current struct used to configure the current Registry Client type Config struct { Protocol string Host string Port int Type string Stem string ServiceKey string ServiceHost string ServicePort int ServiceProtocol string CheckRoute string CheckInterval string } New Configuration Client Config The following is the new struct the will be used to configure the new Configuration Client from the command line option or environment variable values. The Service Registry portion has been removed from the above existing Registry Client Config type Config struct { Protocol string Host string Port int Type string BasePath string ServiceKey string } New Registry Client Config The following is the revised struct the will be used to configure the new Registry Client from the information in the service's configuration. This is mostly unchanged from the existing Registry Client Config , except that the Stem for configuration has been removed type Config struct { Protocol string Host string Port int Type string ServiceKey string ServiceHost string ServicePort int ServiceProtocol string CheckRoute string CheckInterval string } Provider Implementations The current Consul implementation of the Registry Client will be split up into implementations for the new Configuration Client in the new go-mod-configuration module and the revised Registry Client in the existing go-mod-registry module. Decision It was decided to move forward with the above design After initial ADR was approved, it was decided to retain the -r/--registry command-line flag and not add the Enabled field in the Registry provider configuration. Consequences Once the refactoring of go-mod-registry and go-mod-configuration are complete, they will need to be integrated into the new go-mod-bootstrap. Part of this integration will be the Command line option changes above. At this point the edgex-go services will be integrated with the new Registry and Configuration providers. The App Services SDK and Device Services SDK will then need to integrate go-mod-bootstrap to take advantage of these new providers. References Registry Abstraction - Decouple EdgeX services from Consul (Previous design)","title":"Registry Refactoring Design"},{"location":"design/adr/0001-Registy-Refactor/#registry-refactoring-design","text":"Status Context Proposed Design Decision Consequences References","title":"Registry Refactoring Design"},{"location":"design/adr/0001-Registy-Refactor/#status","text":"Approved","title":"Status"},{"location":"design/adr/0001-Registy-Refactor/#context","text":"Currently the Registry Client in go-mod-registry module provides Service Configuration and Service Registration functionality. The goal of this design is to refactor the go-mod-registry module for separation of concerns. The Service Registry functionality will stay in the go-mod-registry module and the Service Configuration functionality will be separated out into a new go-mod-configuration module. This allows for implementations for deferent providers for each, another aspect of separation of concerns.","title":"Context"},{"location":"design/adr/0001-Registy-Refactor/#proposed-design","text":"","title":"Proposed Design"},{"location":"design/adr/0001-Registy-Refactor/#provider-connection-information","text":"An aspect of using the current Registry Client is \" Where do the services get the Registry Provider connection information? \" Currently all services either pull this connection information from the local configuration file or from the edgex_registry environment variable. Device Services also have the option to specify this connection information on the command line. With the refactoring for separation of concerns, this issue changes to \" Where do the services get the Configuration Provider connection information? \" There have been concerns voiced by some in the EdgeX community that storing this Configuration Provider connection information in the configuration which ultimately is provided by that provider is not the right design. This design proposes that all services will use the command line option approach with the ability to override with an environment variable. The Configuration Provider information will not be stored in each service's local configuration file. The edgex_registry environment variable will be deprecated. The Registry Provider connection information will continue to be stored in each service's configuration either locally or from the Configuration Provider same as all other EdgeX Client and Database connection information.","title":"Provider Connection information"},{"location":"design/adr/0001-Registy-Refactor/#command-line-option-changes","text":"The new -cp/-configProvider command line option will be added to each service which will have a value specified using the format {type}.{protocol}://{host}:{port} e.g consul.http://localhost:8500 . This new command line option will be overridden by the edgex_configuration_provider environment variable when it is set. This environment variable's value has the same format as the command line option value. If no value is provided to the -cp/-configProvider option, i.e. just -cp , and no environment variable override is specified, the default value of consul.http://localhost:8500 will be used. if -cp/-configProvider not used and no environment variable override is specified the local configuration file is used, as is it now. All services will log the Configuration Provider connection information that is used. The existing -r/-registry command line option will be retained as a Boolean flag to indicate to use the Registry.","title":"Command line option changes"},{"location":"design/adr/0001-Registy-Refactor/#bootstrap-changes","text":"All services in the edgex-go mono repo use the new common bootstrap functionality. The plan is to move this code to a go module for the Device Service and App Functions SDKs to also use. The current bootstrap modules pkg/bootstrap/configuration/registry.go and pkg/bootstrap/container/registry.go will be refactored to use the new Configuration Client and be renamed appropriately. New bootstrap modules will be created for using the revised version of Registry Client . The current use of useRegistry and registryClient for service configuration will be change to appropriate names for using the new Configuration Client . The current use of useRegistry and registryClient for service registration will be retained for service registration. Call to the new Unregister() API will be added to shutdown code for all services.","title":"Bootstrap Changes"},{"location":"design/adr/0001-Registy-Refactor/#config-seed-changes","text":"The conf-seed service will have similar changes for specifying the Configuration Provider connection information since it doesn't use the common bootstrap package. Beyond that it will have minor changes for switching to using the Configuration Client interface, which will just be imports and appropriate name refactoring.","title":"Config-Seed Changes"},{"location":"design/adr/0001-Registy-Refactor/#config-endpoint-changes","text":"Since the Configuration Provider connection information will no longer be in the service's configuration struct, the config endpoint processing will be modified to add the Configuration Provider connection information to the resulting JSON create from service's configuration.","title":"Config Endpoint Changes"},{"location":"design/adr/0001-Registy-Refactor/#client-interfaces-changes","text":"","title":"Client Interfaces changes"},{"location":"design/adr/0001-Registy-Refactor/#current-registry-client","text":"This following is the current Registry Client Interface type Client interface { Register () error HasConfiguration () ( bool , error ) PutConfigurationToml ( configuration * toml . Tree , overwrite bool ) error PutConfiguration ( configStruct interface {}, overwrite bool ) error GetConfiguration ( configStruct interface {}) ( interface {}, error ) WatchForChanges ( updateChannel chan <- interface {}, errorChannel chan <- error , configuration interface {}, waitKey string ) IsAlive () bool ConfigurationValueExists ( name string ) ( bool , error ) GetConfigurationValue ( name string ) ([] byte , error ) PutConfigurationValue ( name string , value [] byte ) error GetServiceEndpoint ( serviceId string ) ( types . ServiceEndpoint , error ) IsServiceAvailable ( serviceId string ) error }","title":"Current Registry Client"},{"location":"design/adr/0001-Registy-Refactor/#new-configuration-client","text":"This following is the new Configuration Client Interface which contains the Service Configuration specific portion from the above current Registry Client . type Client interface { HasConfiguration () ( bool , error ) PutConfigurationFromToml ( configuration * toml . Tree , overwrite bool ) error PutConfiguration ( configStruct interface {}, overwrite bool ) error GetConfiguration ( configStruct interface {}) ( interface {}, error ) WatchForChanges ( updateChannel chan <- interface {}, errorChannel chan <- error , configuration interface {}, waitKey string ) IsAlive () bool ConfigurationValueExists ( name string ) ( bool , error ) GetConfigurationValue ( name string ) ([] byte , error ) PutConfigurationValue ( name string , value [] byte ) error }","title":"New Configuration Client"},{"location":"design/adr/0001-Registy-Refactor/#revised-registry-client","text":"This following is the revised Registry Client Interface, which contains the Service Registry specific portion from the above current Registry Client . The UnRegister() API has been added per issue #20 type Client interface { Register () error UnRegister () error IsAlive () bool GetServiceEndpoint ( serviceId string ) ( types . ServiceEndpoint , error ) IsServiceAvailable ( serviceId string ) error }","title":"Revised Registry Client"},{"location":"design/adr/0001-Registy-Refactor/#client-configuration-structs","text":"","title":"Client Configuration Structs"},{"location":"design/adr/0001-Registy-Refactor/#current-registry-client-config","text":"The following is the current struct used to configure the current Registry Client type Config struct { Protocol string Host string Port int Type string Stem string ServiceKey string ServiceHost string ServicePort int ServiceProtocol string CheckRoute string CheckInterval string }","title":"Current Registry Client Config"},{"location":"design/adr/0001-Registy-Refactor/#new-configuration-client-config","text":"The following is the new struct the will be used to configure the new Configuration Client from the command line option or environment variable values. The Service Registry portion has been removed from the above existing Registry Client Config type Config struct { Protocol string Host string Port int Type string BasePath string ServiceKey string }","title":"New Configuration Client Config"},{"location":"design/adr/0001-Registy-Refactor/#new-registry-client-config","text":"The following is the revised struct the will be used to configure the new Registry Client from the information in the service's configuration. This is mostly unchanged from the existing Registry Client Config , except that the Stem for configuration has been removed type Config struct { Protocol string Host string Port int Type string ServiceKey string ServiceHost string ServicePort int ServiceProtocol string CheckRoute string CheckInterval string }","title":"New Registry Client Config"},{"location":"design/adr/0001-Registy-Refactor/#provider-implementations","text":"The current Consul implementation of the Registry Client will be split up into implementations for the new Configuration Client in the new go-mod-configuration module and the revised Registry Client in the existing go-mod-registry module.","title":"Provider Implementations"},{"location":"design/adr/0001-Registy-Refactor/#decision","text":"It was decided to move forward with the above design After initial ADR was approved, it was decided to retain the -r/--registry command-line flag and not add the Enabled field in the Registry provider configuration.","title":"Decision"},{"location":"design/adr/0001-Registy-Refactor/#consequences","text":"Once the refactoring of go-mod-registry and go-mod-configuration are complete, they will need to be integrated into the new go-mod-bootstrap. Part of this integration will be the Command line option changes above. At this point the edgex-go services will be integrated with the new Registry and Configuration providers. The App Services SDK and Device Services SDK will then need to integrate go-mod-bootstrap to take advantage of these new providers.","title":"Consequences"},{"location":"design/adr/0001-Registy-Refactor/#references","text":"Registry Abstraction - Decouple EdgeX services from Consul (Previous design)","title":"References"},{"location":"design/adr/0004-Feature-Flags/","text":"Feature Flag Proposal Status Accepted Context Out of the proposal for releasing on time, the community suggested that we take a closer look at feature-flags. Feature-flags are typically intended for users of an application to turn on or off new or unused features. This gives user more control to adopt a feature-set at their own pace \u2013 i.e disabling store and forward in App Functions SDK without breaking backward compatibility. It can also be used to indicate to developers the features that are more often used than others and can provided valuable feedback to enhance and continue a given feature. To gain that insight of the use of any given feature, we would require not only instrumentation of the code but a central location in the cloud (i.e a TIG stack) for the telemetry to be ingested and in turn reported in order to provide the feedback to the developers. This becomes infeasible primarily because the cloud infrastructure costs, privacy concerns, and other unforeseen legal reasons for sending \u201cUsage Metrics\u201d of an EdgeX installation back to a central entity such as the Linux Foundation, among many others. Without the valuable feedback loop, feature-flags don\u2019t provide much value on their own and they certainly don\u2019t assist in increasing velocity to help us deliver on time. Putting aside one of the major value propositions listed above, feasibility of a feature flag \u201cmodule\u201d was still evaluated. The simplest approach would be to leverage configuration following a certain format such as FF_[NewFeatureName]=true/false. This is similar to what is done today. Turning on/off security is an example, turning on/off the registry is another. Expanding this further with a module could offer standardization of controlling a given feature such as featurepkg.Register(\u201cMyNewFeature\u201d) or featurepkg.IsOn(\u201cMyNewFeature\u201d) . However, this really is just adding complexity on top of the underlying configuration that is already implemented. If we were to consider doing something like this, it lends it self to a central management of features within the EdgeX framework\u2014either its own service or possibly added as part of the SMA. This could help address concerns around feature dependencies and compatibility. Feature A on Service X requires Feature B and Feature C on Service Y. Continuing down this path starts to beget a fairly large impact to EdgeX for value that cannot be fully realized. Decision The community should NOT pursue a full-fledged feature flag implementation either homegrown or off-the-shelf. However, it should be encouraged to develop features with a wholistic perspective and consider leveraging configuration options to turn them on/off. In other words, once a feature compiles, can work under common scenarios, but perhaps isn\u2019t fully tested with edge cases, but doesn\u2019t impact any other functionality, should be encouraged. Consequences Allows more focus on the many more competing priorities for this release. Minimal impact to development cycles and release schedule","title":"Feature Flag Proposal"},{"location":"design/adr/0004-Feature-Flags/#feature-flag-proposal","text":"","title":"Feature Flag Proposal"},{"location":"design/adr/0004-Feature-Flags/#status","text":"Accepted","title":"Status"},{"location":"design/adr/0004-Feature-Flags/#context","text":"Out of the proposal for releasing on time, the community suggested that we take a closer look at feature-flags. Feature-flags are typically intended for users of an application to turn on or off new or unused features. This gives user more control to adopt a feature-set at their own pace \u2013 i.e disabling store and forward in App Functions SDK without breaking backward compatibility. It can also be used to indicate to developers the features that are more often used than others and can provided valuable feedback to enhance and continue a given feature. To gain that insight of the use of any given feature, we would require not only instrumentation of the code but a central location in the cloud (i.e a TIG stack) for the telemetry to be ingested and in turn reported in order to provide the feedback to the developers. This becomes infeasible primarily because the cloud infrastructure costs, privacy concerns, and other unforeseen legal reasons for sending \u201cUsage Metrics\u201d of an EdgeX installation back to a central entity such as the Linux Foundation, among many others. Without the valuable feedback loop, feature-flags don\u2019t provide much value on their own and they certainly don\u2019t assist in increasing velocity to help us deliver on time. Putting aside one of the major value propositions listed above, feasibility of a feature flag \u201cmodule\u201d was still evaluated. The simplest approach would be to leverage configuration following a certain format such as FF_[NewFeatureName]=true/false. This is similar to what is done today. Turning on/off security is an example, turning on/off the registry is another. Expanding this further with a module could offer standardization of controlling a given feature such as featurepkg.Register(\u201cMyNewFeature\u201d) or featurepkg.IsOn(\u201cMyNewFeature\u201d) . However, this really is just adding complexity on top of the underlying configuration that is already implemented. If we were to consider doing something like this, it lends it self to a central management of features within the EdgeX framework\u2014either its own service or possibly added as part of the SMA. This could help address concerns around feature dependencies and compatibility. Feature A on Service X requires Feature B and Feature C on Service Y. Continuing down this path starts to beget a fairly large impact to EdgeX for value that cannot be fully realized.","title":"Context"},{"location":"design/adr/0004-Feature-Flags/#decision","text":"The community should NOT pursue a full-fledged feature flag implementation either homegrown or off-the-shelf. However, it should be encouraged to develop features with a wholistic perspective and consider leveraging configuration options to turn them on/off. In other words, once a feature compiles, can work under common scenarios, but perhaps isn\u2019t fully tested with edge cases, but doesn\u2019t impact any other functionality, should be encouraged.","title":"Decision"},{"location":"design/adr/0004-Feature-Flags/#consequences","text":"Allows more focus on the many more competing priorities for this release. Minimal impact to development cycles and release schedule","title":"Consequences"},{"location":"design/adr/0005-Service-Self-Config/","text":"Service Self Config Init & Config Seed Removal Status approved - TSC vote on 3/25/20 for Geneva release NOTE: this ADR does not address high availability considerations and concerns. EdgeX, in general, has a number of unanswered questions with regard to HA architecture and this design adds to those considerations. Context Since its debut, EdgeX has had a configuration seed service (config-seed) that, on start of EdgeX, deposits configuration for all the services into Consul (our configuration/registry service). For development purposes, or on resource constrained platforms, EdgeX can be run without Consul with services simply reading configuration from the filesystem. While this process has nominally worked for several releases of EdgeX, there has always been some issues with this extra initialization process (config-seed), not least of which are: - race conditions on the part of the services, as they bootstrap, coming up before the config-seed completes its deposit of configuration into Consul - how to deal with \"overrides\" such as environmental variable provided configuration overrides. As the override is often specific to a service but has to be in place for config-seed in order to take effect. - need for an additional service that is only there for init and then dies (confusing to users) NOTE - for historical purposes, it should be noted that config-seed only writes configuration into the configuration/registry service (Consul) once on the first start of EdgeX. On subsequent starts of EdgeX, config-seed checks to see if it has already populated the configuration/registry service and will not rewrite configuration again (unless the --overwrite flag is used). The design/architectural proposal, therefore, is: - removal of the config-seed service (removing cmd/config-seed from the edgex-go repository) - have each EdgeX micro service \"self seed\" - that is seed Consul with their own required configuration on bootstrap of the service. Details of that bootstrapping process are below. Command Line Options All EdgeX services support a common set of command-line options, some combination of which are required on startup for a service to interact with the rest of EdgeX. Command line options are not set by any configuration. Command line options include: --configProvider or -cp (the configuration provider location URL - prefixed with consul. - for example: -cp=consul.http://localhost:8500 ) --overwrite or -o (overwrite the configuration in the configuration provider) --file or -f (the configuration filename - configuration.toml is used by default if the configuration filename is not provided) --profile or -p (the name of a sub directory in the configuration directory in which a profile-specific configuration file is found. This has no default. If not specified, the configuration file is read from the configuration directory) --confdir or -c (the directory where the configuration file is found - ./res is used by default if the confdir is not specified, where \".\" is the convention on Linux/Unix/MacOS which means current directory) --registry or -r (string indicating use of the registry) The distinction of command line options versus configuration will be important later in this ADR. Two command line options (-o for overwrite and -r for registry) are not overridable by environmental variables. NOTES: Use of the --overwrite command line option should be used sparingly and with expert knowledge of EdgeX; in particular knowledge of how it operates and where/how it gets its configuration on restarts, etc. Ordinarily, --overwrite is provided as a means to support development needs. Use of --overwrite permanently in production enviroments is highly discouraged. Configuration Initialization Each service has (or shall have if not providing it already) a local configuration file. The service may use the local configuration file on initialization of the service (aka bootstrap of the service) depending on command line options and environmental variables (see below) provided at startup. Using a configuration provider When the configuration provider is specified, the service will call on the configuration provider (Consul) and check if the top-level (root) namespace for the service exists. If configuratation at the top-level (root) namespace exists, it indicates that the service has already populated its configuration into the configuration provider in a prior startup. If the service finds the top-level (root) namespace is already populated with configuration information it will then read that configuration information from the configuration provider under namespace for that service (and ignore what is in the local configuration file). If the service finds the top-level (root) namespace is not populated with configuration information, it will read its local configuration file and populate the configuration provider (under the namespace for the service) with configuration read from the local configuration file. A configuration provider can be specified with a command line argument (the -cp / --configProvider) or environment variable (the EDGEX_CONFIGURATION_PROVIDER environmental variable which overrides the command line argument). NOTE: the environmental variables are typically uppercase but there have been inconsistencies in environmental variable casing (example: edgex_registry). This should be considered and made consistent in a future major release. Using the local configuration file When a configuration provider isn't specified, the service just uses the configuration in its local configuration file. That is the service uses the configuration in the file associated with the profile, config filename and config file directory command line options or environmental variables. In this case, the service does not contact the configuration service (Consul) for any configuration information. NOTE: As the services now self seed and deployment specific changes can be made via environment overrides, it will no longer be necessary to have a Docker profile configuration file in each of the service directories (example: https://github.com/edgexfoundry/edgex-go/blob/master/cmd/core-data/res/docker/configuration.toml). See Consequences below. It will still be possible for users to use the profile mechanism to specify a Docker configuration, but it will no longer be required and not the recommended approach to providing Docker container specific configuration. Overrides Environment variables used to override configuration always take precedence whether configuration is being sourced locally or read from the config provider/Consul. Note - this means that a configuration value that is being overridden by an environment variable will always be the source of truth, even if the same configuration is changed directly in Consul. The name of the environmental variable must match the path names in Consul. NOTES: - Environmental variables overrides remove the need to change the \"docker\" profile in the res/docker/configuration.toml files - Allowing removal of 50% of the existing configuration.toml files. - The override rules in EdgeX between environmental variables and command line options may be counter intuitive compared to other systems. There appears to be no standard practice. Indeed, web searching \"Reddit & Starting Fights Env Variables vs Command Line Args\" will layout the prevailing differences. - Environment variables used for configuration overrides are named by prepending the the configuration element with the configuration section inclusive of sub-path, where sub-path's \".\"s are replaced with underscores. These configuration environment variable overrides must be specified using camel case. Here are two examples: Registry_Host for [Registry] Host = 'localhost' Clients_CoreData_Host for [Clients] [Clients.CoreData] Host = 'localhost' - Going forward, environmental variables that override command line options should be all uppercase. All values overriden get logged (indicating which configuration value or op param and the new value). Decision These features have been implemented (with some minor changes to be done) for consideration here: https://github.com/edgexfoundry/go-mod-bootstrap/compare/master...lenny-intel:SelfSeed2. This code branch will be removed once this ADR is approved and implemented on master. The implementation for self-seeding services and environmental overrides is already implemented (for Fuji) per this document in the application services and device services (and instituted in the SDKs of each). Backward compatibility Several aspects of this ADR contain backward compatibility issues for the device service and application service SDKs. Therefore, for the upcoming minor release, the following guidelines and expections are added to provide for backward compatibility. --registry= for Device SDKs As earlier versions of the device service SDKs accepted a URI for --registry, if specified on the command line, use the given URI as the address of the configuration provider. If both --configProvider and --registry specify URIs, then the service should log an error and exit. --registry (no \u2018=\u2019) and w/o --configProvider for both SDKs If a configProvider URI isn't specified, but --registry (w/out a URI) is specified, then the service will use the Registry provider information from its local configuration file for both configuration and registry providers. Env Var: edgex_registry= for all services (currently has been removed) Add it back and use value as if it was EDGEX_CONFIGURATION_PROVIDER and enable use of registry with same settings in URL. Default to http as it is in Fuji. Consequences Docker compose files will need to be changed to remove config seed. The main Snap will need to be changed to remove config seed. Config seed code (currently in edgex-go repo) is to be removed. Any service specific environmental overrides currently on config seed need to be moved to the specific service(s). The Docker configuration files and directory (example: https://github.com/edgexfoundry/edgex-go/blob/master/cmd/core-data/res/docker/configuration.toml) that are used to populate the config seed for Docker containers can be eliminated from all the services. In cmd/security-secretstore-setup, there is only a docker configuration.toml. This file will be moved rather than deleted. Documentation would need to reflect removal of config seed and \"self seeding\" process. Removes any potential issue with past race conditions (as experienced with the Edinburgh release) as each service is now responsible for its own configuration. There are still high availability concerns that need to be considered and not covered in this ADR at this time. Removes some confusion on the part of users as to why a service (config-seed) starts and immediately exits. Minimal impact to development cycles and release schedule Configuration endpoints in all services need to ensure the environmental variables are reflected in the configuration data returned (this is a system management impact). Docker files will need to be modified to remove setting profile=docker Docker compose files will need to be changed to add environmental overrides for removal of docker profiles. These should go in the global environment section of the compose files for those overrides that apply to all services. Example: # all common shared environment variables defined here: x-common-env-variables: &common-variables EDGEX_SECURITY_SECRET_STORE: \"false\" EDGEX_CONFIGURATION_PROVIDER: consul.http://edgex-core-consul:8500 Clients_CoreData_Host: edgex-core-data Clients_Logging_Host: edgex-support-logging Logging_EnableRemote: \"true\"","title":"Service Self Config Init & Config Seed Removal"},{"location":"design/adr/0005-Service-Self-Config/#service-self-config-init-config-seed-removal","text":"","title":"Service Self Config Init & Config Seed Removal"},{"location":"design/adr/0005-Service-Self-Config/#status","text":"approved - TSC vote on 3/25/20 for Geneva release NOTE: this ADR does not address high availability considerations and concerns. EdgeX, in general, has a number of unanswered questions with regard to HA architecture and this design adds to those considerations.","title":"Status"},{"location":"design/adr/0005-Service-Self-Config/#context","text":"Since its debut, EdgeX has had a configuration seed service (config-seed) that, on start of EdgeX, deposits configuration for all the services into Consul (our configuration/registry service). For development purposes, or on resource constrained platforms, EdgeX can be run without Consul with services simply reading configuration from the filesystem. While this process has nominally worked for several releases of EdgeX, there has always been some issues with this extra initialization process (config-seed), not least of which are: - race conditions on the part of the services, as they bootstrap, coming up before the config-seed completes its deposit of configuration into Consul - how to deal with \"overrides\" such as environmental variable provided configuration overrides. As the override is often specific to a service but has to be in place for config-seed in order to take effect. - need for an additional service that is only there for init and then dies (confusing to users) NOTE - for historical purposes, it should be noted that config-seed only writes configuration into the configuration/registry service (Consul) once on the first start of EdgeX. On subsequent starts of EdgeX, config-seed checks to see if it has already populated the configuration/registry service and will not rewrite configuration again (unless the --overwrite flag is used). The design/architectural proposal, therefore, is: - removal of the config-seed service (removing cmd/config-seed from the edgex-go repository) - have each EdgeX micro service \"self seed\" - that is seed Consul with their own required configuration on bootstrap of the service. Details of that bootstrapping process are below.","title":"Context"},{"location":"design/adr/0005-Service-Self-Config/#command-line-options","text":"All EdgeX services support a common set of command-line options, some combination of which are required on startup for a service to interact with the rest of EdgeX. Command line options are not set by any configuration. Command line options include: --configProvider or -cp (the configuration provider location URL - prefixed with consul. - for example: -cp=consul.http://localhost:8500 ) --overwrite or -o (overwrite the configuration in the configuration provider) --file or -f (the configuration filename - configuration.toml is used by default if the configuration filename is not provided) --profile or -p (the name of a sub directory in the configuration directory in which a profile-specific configuration file is found. This has no default. If not specified, the configuration file is read from the configuration directory) --confdir or -c (the directory where the configuration file is found - ./res is used by default if the confdir is not specified, where \".\" is the convention on Linux/Unix/MacOS which means current directory) --registry or -r (string indicating use of the registry) The distinction of command line options versus configuration will be important later in this ADR. Two command line options (-o for overwrite and -r for registry) are not overridable by environmental variables. NOTES: Use of the --overwrite command line option should be used sparingly and with expert knowledge of EdgeX; in particular knowledge of how it operates and where/how it gets its configuration on restarts, etc. Ordinarily, --overwrite is provided as a means to support development needs. Use of --overwrite permanently in production enviroments is highly discouraged.","title":"Command Line Options"},{"location":"design/adr/0005-Service-Self-Config/#configuration-initialization","text":"Each service has (or shall have if not providing it already) a local configuration file. The service may use the local configuration file on initialization of the service (aka bootstrap of the service) depending on command line options and environmental variables (see below) provided at startup. Using a configuration provider When the configuration provider is specified, the service will call on the configuration provider (Consul) and check if the top-level (root) namespace for the service exists. If configuratation at the top-level (root) namespace exists, it indicates that the service has already populated its configuration into the configuration provider in a prior startup. If the service finds the top-level (root) namespace is already populated with configuration information it will then read that configuration information from the configuration provider under namespace for that service (and ignore what is in the local configuration file). If the service finds the top-level (root) namespace is not populated with configuration information, it will read its local configuration file and populate the configuration provider (under the namespace for the service) with configuration read from the local configuration file. A configuration provider can be specified with a command line argument (the -cp / --configProvider) or environment variable (the EDGEX_CONFIGURATION_PROVIDER environmental variable which overrides the command line argument). NOTE: the environmental variables are typically uppercase but there have been inconsistencies in environmental variable casing (example: edgex_registry). This should be considered and made consistent in a future major release. Using the local configuration file When a configuration provider isn't specified, the service just uses the configuration in its local configuration file. That is the service uses the configuration in the file associated with the profile, config filename and config file directory command line options or environmental variables. In this case, the service does not contact the configuration service (Consul) for any configuration information. NOTE: As the services now self seed and deployment specific changes can be made via environment overrides, it will no longer be necessary to have a Docker profile configuration file in each of the service directories (example: https://github.com/edgexfoundry/edgex-go/blob/master/cmd/core-data/res/docker/configuration.toml). See Consequences below. It will still be possible for users to use the profile mechanism to specify a Docker configuration, but it will no longer be required and not the recommended approach to providing Docker container specific configuration.","title":"Configuration Initialization"},{"location":"design/adr/0005-Service-Self-Config/#overrides","text":"Environment variables used to override configuration always take precedence whether configuration is being sourced locally or read from the config provider/Consul. Note - this means that a configuration value that is being overridden by an environment variable will always be the source of truth, even if the same configuration is changed directly in Consul. The name of the environmental variable must match the path names in Consul. NOTES: - Environmental variables overrides remove the need to change the \"docker\" profile in the res/docker/configuration.toml files - Allowing removal of 50% of the existing configuration.toml files. - The override rules in EdgeX between environmental variables and command line options may be counter intuitive compared to other systems. There appears to be no standard practice. Indeed, web searching \"Reddit & Starting Fights Env Variables vs Command Line Args\" will layout the prevailing differences. - Environment variables used for configuration overrides are named by prepending the the configuration element with the configuration section inclusive of sub-path, where sub-path's \".\"s are replaced with underscores. These configuration environment variable overrides must be specified using camel case. Here are two examples: Registry_Host for [Registry] Host = 'localhost' Clients_CoreData_Host for [Clients] [Clients.CoreData] Host = 'localhost' - Going forward, environmental variables that override command line options should be all uppercase. All values overriden get logged (indicating which configuration value or op param and the new value).","title":"Overrides"},{"location":"design/adr/0005-Service-Self-Config/#decision","text":"These features have been implemented (with some minor changes to be done) for consideration here: https://github.com/edgexfoundry/go-mod-bootstrap/compare/master...lenny-intel:SelfSeed2. This code branch will be removed once this ADR is approved and implemented on master. The implementation for self-seeding services and environmental overrides is already implemented (for Fuji) per this document in the application services and device services (and instituted in the SDKs of each).","title":"Decision"},{"location":"design/adr/0005-Service-Self-Config/#backward-compatibility","text":"Several aspects of this ADR contain backward compatibility issues for the device service and application service SDKs. Therefore, for the upcoming minor release, the following guidelines and expections are added to provide for backward compatibility. --registry= for Device SDKs As earlier versions of the device service SDKs accepted a URI for --registry, if specified on the command line, use the given URI as the address of the configuration provider. If both --configProvider and --registry specify URIs, then the service should log an error and exit. --registry (no \u2018=\u2019) and w/o --configProvider for both SDKs If a configProvider URI isn't specified, but --registry (w/out a URI) is specified, then the service will use the Registry provider information from its local configuration file for both configuration and registry providers. Env Var: edgex_registry= for all services (currently has been removed) Add it back and use value as if it was EDGEX_CONFIGURATION_PROVIDER and enable use of registry with same settings in URL. Default to http as it is in Fuji.","title":"Backward compatibility"},{"location":"design/adr/0005-Service-Self-Config/#consequences","text":"Docker compose files will need to be changed to remove config seed. The main Snap will need to be changed to remove config seed. Config seed code (currently in edgex-go repo) is to be removed. Any service specific environmental overrides currently on config seed need to be moved to the specific service(s). The Docker configuration files and directory (example: https://github.com/edgexfoundry/edgex-go/blob/master/cmd/core-data/res/docker/configuration.toml) that are used to populate the config seed for Docker containers can be eliminated from all the services. In cmd/security-secretstore-setup, there is only a docker configuration.toml. This file will be moved rather than deleted. Documentation would need to reflect removal of config seed and \"self seeding\" process. Removes any potential issue with past race conditions (as experienced with the Edinburgh release) as each service is now responsible for its own configuration. There are still high availability concerns that need to be considered and not covered in this ADR at this time. Removes some confusion on the part of users as to why a service (config-seed) starts and immediately exits. Minimal impact to development cycles and release schedule Configuration endpoints in all services need to ensure the environmental variables are reflected in the configuration data returned (this is a system management impact). Docker files will need to be modified to remove setting profile=docker Docker compose files will need to be changed to add environmental overrides for removal of docker profiles. These should go in the global environment section of the compose files for those overrides that apply to all services. Example: # all common shared environment variables defined here: x-common-env-variables: &common-variables EDGEX_SECURITY_SECRET_STORE: \"false\" EDGEX_CONFIGURATION_PROVIDER: consul.http://edgex-core-consul:8500 Clients_CoreData_Host: edgex-core-data Clients_Logging_Host: edgex-support-logging Logging_EnableRemote: \"true\"","title":"Consequences"},{"location":"design/adr/0006-Metrics-Collection/","text":"EdgeX Metrics Collection Status Approved Original proposal 10/24/2020 Approved by the TSC on 3/2/22 Metric (or telemetry) data is defined as the count or rate of some action, resource, or circumstance in the EdgeX instance or specific service. Examples of metrics include: the number of EdgeX Events sent from core data to an application service the number of requests on a service API the average time it takes to process a message through an application service The number of errors logged by a service Control plane events (CPE) are defined as events that occur within an EdgeX instance. Examples of CPE include: a device was provisioned (added to core metadata) a service was stopped service configuration has changed CPE should not be confused with core data Events. Core data Events represent a collection (one or more) of sensor/device readings. Core data Events represent sensing of some measured state of the physical world (temperature, vibration, etc.). CPE represents the detection of some happening inside of the EdgeX software. This ADR outlines metrics (or telemetry) collection and handling. Note This ADR initially incorporated metrics collection and control plane event processing. The EdgeX architects felt the scope of the design was too large to cover under one ADR. Control plane event processing will be covered under a separate ADR in the future. Context System Management services (SMA and executors) currently provide a limited set of \u201cmetrics\u201d to requesting clients (3rd party applications and systems external to EdgeX). Namely, it provides requesting clients with service CPU and memory usage; both metrics about the resource utilization of the service (the executable) itself versus metrics that are about what is happening inside of the service. Arguably, the current system management metrics can be provided by the container engine and orchestration tools (example: by Docker engine) or by the underlying OS tooling. Info The SMA has been deprecated (since Ireland release) and will be removed in a future, yet named, release. Going forward, users of EdgeX will want to have more insights \u2013 that is more metrics telemetry \u2013 on what is happening directly in the services and the tasks that they are preforming. In other words, users of EdgeX will want more telemetry on service activities to include: sensor data collection (how much, how fast, etc.) command requests handled (how many, to which devices, etc.) sensor data transformation as it is done in application services (how fast, what is filtered, etc) sensor data export (how much is sent, how many exports have failed, etc. ) API requests (how often, how quickly, how many success versus failed attempts, etc.) bootstrapping time (time to come up and be available to other services) activity processing time (amount of time it takes to perform a particular service function - such as respond to a command request) Definitions Metric (or telemetry) data is defined as the count or rate of some action, resource, or circumstance in the EdgeX instance or specific service. Examples of metrics include: the number of EdgeX Events sent from core data to an application service via message bus (or via device service to application service in Ireland and beyond) the number of requests on a service API the average time it takes to process a message through an application service The number of errors logged by a service The collection and dissemination of metric data will require internal service level instrumentation (relevant to that service) to capture and send data about relevant EdgeX operations. EdgeX does not currently offer any service instrumentation. Metric Use As a first step in implementation of metrics data, EdgeX will make metric data available to other subscribing 3rd party applications and systems, but will not necessarily consume or use this information itself. In the future, EdgeX may consume its own metric data. For example, EdgeX may, in the future, use a metric on the number of EdgeX events being sent to core data (or app services) as the means to throttle back device data collection. In the future, EdgeX application services may optionally subscribe to a service's metrics messages bus (by attaching to the appropriate message pipe for that service). Thus allowing additional filtering, transformation, endpoint control of metric data from that service. At the point where this feature is supported, consideration would need to be made as to whether all events (sensor reading messages and metric messages) go through the same application services. At this time, EdgeX will not persist the metric data (except as it may be retained as part of a message bus subsystem such as in an MQTT broker). Consumers of metric data are responsible for persisting the data if needed, but this is external to EdgeX. Persistence of metric information may be considered in the future based on requirements and adopter demand for such a feature. In general, EdgeX metrics are meant to provide internal services and external applications and systems better information about what is happening \"inside\" EdgeX services and the associated devices with which it communicates. Requirements Services will push specified metrics collected for that service to a specified (by configuration) message endpoint (as supported by the EdgeX message bus implementation; currently either Redis Pub/Sub or MQTT implementations are supported) Each service will have configuration that specifies a message endpoint for the service metrics. The metrics message topic communications may be secured or unsecured (just as application services provide the means to export to secured or unsecured message pipes today). The configuration will be placed in the Writable area. When a user wishes to change the configuration dynamically (such as turning on/off a metric), then Consul's UI can be used to change it. Services will have configuration which indicates what metrics are available from the service. Services will have configuration which allows EdgeX system managers to select which metrics are on or off - in other words providing configuration that determines what metrics are collected and reported by default. When a metric is turned off (the default setting) the service does not report the metric. When a metric is turned on the service collects and sends the metric to the designated message topic. Metrics collection must be pushed to the designated message topic on some appointed schedule. The schedule would be designated by configuration and done in a way similar to auto events in device services. For the initial implementation, there will be just one scheduled time when all metrics will be collected and pushed to the designated message topic. In the future, there may be a desire to set up a separate schedule for each metric, but this was deemed too complex for the initial implementation. Info Initially, it was proposed that metrics be associated with a \"level\" and allow metrics to be turned on or off by level (like levels associated to log messages in logging). The level of metrics data seems arbitrary at this time and considered too complex for initial implementation. This may be reconsidered in a future release and based on new requirements/use cases. It was also proposed to categorize or label metrics - essentially allowing grouping of various metrics. This would allow groups of metrics to be turned on or off, and allow metrics to be organized per the group when reporting. At this time, this feature is also considered beyond the scope of the initial implementation and to be reconsidered in a future release based on requirements/use case needs. It was also proposed that each service offer a REST API to provide metrics collection information (such as which metrics were being collected) and the ability to turn the collection on or off dynamically. This is deemed out of scope for the first implementation and may be brought back if there are use case requirements / demand for it. Requested Metrics The following is a list of example metrics requested by the EdgeX community and adopters for various service areas. Again, metrics would generally be collected and pushed to the message topic in some configured interval (example: 1/5/15 minutes or other defined interval). This is just a sample of metrics thought relevant by each work group. It may not reflect the metrics supported by the implementation. The exact metrics collected by each service will be determined by the service implementers (or SDK implementers in the case of the app functions and device service SDKs). General The following metrics apply to all (or most) services. Service uptime (time since last service boot) Cumulative number of API requests succeeded / failed / invalid (2xx vs 5xx vs 4xx) Avg response time (in milliseconds or appropriate unit of measure) on APIs Avg and Max request size Core/Supporting Latency (measure of time) an event takes to get through core data Latency (measure of time) a command request takes to get to a device service Indication of health \u2013 that events are being processed during a configurable period Number of events in persistence Number of readings in persistence Number of validation failures (validation of device identification) Number of notification transactions Number of notifications handled Number of failed notification transmissions Number of notifications in retry status Application Services Processing time for a pipeline; latency (measure of time) an event takes to get through an application service pipeline DB access times How often are we failing export to be sent to db to be retried at a later time What is the current store and forward queue size How much data (size in KBs or MBs) of packaged sensor data is being sent to an endpoint (or volume) Number of invalid messages that triggered pipeline Number of events processed Device Services Number of devices managed by this DS Device Requests (which may be more informative than reading counts and rates) Note It is envisioned that there may be additional specific metrics for each device service. For example, the ONVIF camera device service may report number of times camera tampering was detected. Security Security metrics may be more difficult to ascertain as they are cross service metrics. Given the nature of this design (on a per service basis), global security metrics may be out of scope or security metrics collection has to be copied into each service (leading to lots of duplicate code for now). Also, true threat detection based on metrics may be a feature best provided by 3rd party based on particular threats and security profile needs. Number of API requests denied due to wrong access token (Kong) per service and within a given time Number of secrets accessed per service name Count of any accesses and failures to the data persistence layer Count of service start and restart attempts Design Proposal Collect and Push Architecture Metric data will be collected and cached by each service. At designated times (kicked off by configurable schedule), the service will collect telemetry data from the cache and push it to a designated message bus topic. Metrics Messaging Cached metric data, at the designated time, will be marshaled into a message and pushed to the pre-configured message bus topic. Each metric message consists of several key/value pairs: - a required name (the name of the metric) such as service-uptime - a required value which is the telemetry value collected such as 120 as the number of hours the service has been up. - a required timestamp is the time (in Epoch timestamp/milliseconds format) at which the data was collected (similar in nature to the origin of sensed data). - an optional collection (array) of tags. The tags are sets of key/value pairs of strings that provide amplifying information about the telemetry. Tags may include: - originating service name - unit of measure associated with the telemetry value - value type of the value - additional values when the metric is more than just one value (example: when using a histogram, it would include min, max, mean and sum values) The metric name must be unique for that service. Because some metrics are reported from multiple services (such as service uptime), the name is not required to be unique across all services. All information (keys, values, tags, etc.) is in string format and placed in a JSON array within the message body. Here are some example representations: Example metric message body with a single value { \"name\" : \"service-up\" , \"value\" : \"120\" , \"timestamp\" : \"1602168089665570000\" , \"tags\" :{ \"service\" : \"coredata\" , \"uom\" : \"days\" , \"type\" : \"int64\" }} Example metric message body with multiple values { \"name\" : \"api-requests\" , \"value\" : \"24\" , \"timestamp\" : \"1602168089665570001\" , \"tags\" :{ \"service\" : \"coredata\" , \"uom\" : \"count\" , \"type\" : \"int64\" , \"mean\" : \"0.0665\" , \"rate1\" : \"0.111\" , \"rate5\" : \"0.150\" , \"rate15\" : \"0.111\" }} Info The key or metric name must be unique when using go-metrics as it requires the metric name to be unique per the registry. Metrics are considered immutable. Configuration Configuration, not unlike that provided in core data or any device service, will specify the message bus type and locations where the metrics messages should be sent. In fact, the message bus configuration will use (or reuse if the service is already using the message bus) the common message bus configuration as defined below. Common configuration for each service for message queue configuration (inclusive of metrics): [ MessageQueue ] Protocol = 'redis' ## or 'tcp' Host = 'localhost' Port = 5573 Type = 'redis' ## or 'mqtt' PublishTopicPrefix = \"edgex/events/core\" # standard and existing core or device topic for publishing [ MessageQueue.Optional ] # Default MQTT Specific options that need to be here to enable environment variable overrides of them # Client Identifiers ClientId = \"device-virtual\" # Connection information Qos = \"0\" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified Additional configuration must be provided in each service to provide metrics / telemetry specific configuration. This area of the configuration will likely be different for each type of service. Additional metrics collection configuration to be provided include: Trigger the collection of telemetry from the metrics cache and sending it into the appointed message bus. Define which metrics are available and which are turned off and on . All are false by default. The list of metrics can and likely will be different per service. The keys in this list are the metric name. True and false are used for on and off values. Specify the metrics topic prefix where metrics data will be published to (ex: providing the prefix /edgex/telemetry/topic name where the service and metric name [service-name]/[metric-name] will be appended per metric (allowing subscribers to filter by service or metric name) These metrics configuration options will be defined in the Writable area of configuration.toml so as to allow for dynamic changes to the configuration (when using Consul). Specifically, the [Writable].[Writable.Telemetry] area will dictate metrics collection configuration like this: [[ Writable ]] [[ Writable.Telemetry ]] Interval = \"30s\" PublishTopicPrefix = \"edgex/telemetry\" # // will be added to this Publish Topic prefix #available metrics listed here. All metrics should be listed off (or false) by default service-up = false api-requests = false Info It was discussed that in future EdgeX releases, services may want separate message bus connections. For example one for sensor data and one for metrics telemetry data. This would allow the QoS and other settings of the message bus connection to be different. This would allow sensor data collection, for example, to be messaged with a higher QoS than that of metrics. As an alternate approach, we could modify go-mod-messaging to allow setting QoS per topic (and thereby avoid multiple connections). For the initial release of this feature, the service will use the same connection (and therefore configuration) for metrics telemetry as well as sensor data. Library Support Each service will now need go-mod-messaging support (for GoLang services and the equivalent for C services). Each service would determine when and what metrics to collect and push to the message bus, but will use a common library chosen for each EdgeX language supported (Go or C currently) Use of go-metrics (a GoLang library to publish application metrics) would allow EdgeX to utilize (versus construct) a library utilized by over 7 thousand projects. It provides the means to capture various types of metrics in a registry (a sophisticated map). The metrics can then be published ( reported ) to a number of well known systems such as InfluxDB, Graphite, DataDog, and Syslog. go-metrics is a Go library made from original Java package https://github.com/dropwizard/metrics. A similar package would need to be selected (or created) for C. Per the Core WG meeting of 2/24/22 - it is important to provide an implementation that is the same in Go or C. The adopter of EdgeX should not see a difference in whether the metrics/telemetry is collected by a C or Go service. Configuration of metrics in a C or Go service should have the same structure. The C based metrics collection mechanism in C services (specifically as provided for in our C device service SDK) may operate differently \"under the covers\" but its configuration and resulting metrics messages on the EdgeX message bus must be formatted/organized the same. Considerations in the use of go-metrics This is a Golang only library. Using this library would not provide with any package to use for the C services. If there are expectations for parity between the services, this may be more difficult to achieve given the features of go-metrics. go-metrics will still require the EdgeX team to develop a bootstrapping apparatus to take the metrics configuration and register each of the metrics defined in the configuration in go-metrics. go-metrics would also require the EdgeX team to develop the means to periodically extract the metrics data from the registry and ship it via message bus (something the current go-metrics library does not do). While go-metrics offers the ability for data to be reported to other systems, it would required EdgeX to expose these capabilities (possibly through APIs) if a user wanted to export to these subsystems in addition to the message bus. Per the Kamakura Planning Meeting, it was noted that go-metrics is already a dependency in our Go code due to its use other 3rd party packages (see https://github.com/edgexfoundry/edgex-go/blob/4264632f3ddafb0cbc2089cffbea8c0719035c96/go.sum#L18). Community questions about go-metrics Per the Monthly Architect's meeting of 9/20/21): How it manages the telemetry data (persistence, in memory, database, etc.)? In memory - in a \"registry\"; essentially a key/value store where the key is the metric name Does it offer a query API (in order to easily support the ADR suggested REST API)? Yes - metrics are stored in a \"Registry\" (MetricRegistry - essentially a map). Get (or GetAll) methods provided to query for metrics What does the go-metrics package do so that its features can become requirements for C side? About a dozen types of metrics collection (simple gauge or counter to more sophisticated structures like Histograms) - all stored in a registry (map). How is the data made available? Report out (export or publish) to various integrated packages (InfluxDB, Graphite, DataDog, Syslog, etc.). Nothing to MQTT or other base message service. This would have to be implemented from scratch. Can the metric/telemetry count be reset if needed? Does this happen whenever it posts to the message bus? How would this work for REST? Yes, you can unregister and re-register the metric. A REST API would have to be constructed to call this capability. As an alternative to go-metrics, there is another library called OpenCensus . This is a multi-language metrics library, including Go and C++. This library is more feature rich. OpenCensus is also roughly 5x the size of the go-metrics library. Additional Open Questions Should consideration be given to allow metrics to be placed in different topics per name? If so, we will have to add to the topic name like we do for device name in device services? A future consideration Should consideration be given to incorporate alternate protocols/standards for metric collection such as https://opentelemetry.io/ or https://github.com/statsd/? Go metrics is already a library pulled into all Go services. These packages may be used in C side implementations. Decision Per the Monthly Architect's meeting of 12/13/21 - it was decided to use go-metrics for Go services over creating our own library or using open census. C services will either find/pick a package that provides similar functionality to go-metrics or implement internally something providing MVP capability. Use of go-metrics helps avoid too much service bloat since it is already in most Go services. Per the same Monthly Architect's meeting, it as decided to implement metrics in Go services first. Per the Monthly Architect's meeting of 1/24/22 - it was decided not to support a REST API on all services that would provide information on what metrics the service provides and the ability to turn them on / off. Instead, the decision was to use Writable configuration and allow Consul to be the means to change the configuration (dynamically). If an adopter chooses not to use Consul, then the configuration with regard to metrics collection, as with all configuration in this circumstance, would be static. If an external API need is requested in the future (such as from an external UI or tool), a REST API may be added. See older versions of this PR for ideas on implementation in this case. Per Core Working Group meeting of 2/24/22 (and in many other previous meetings on this ADR) - it was decided that the EdgeX approach should be one of push (via message bus/MQTT) vs. pull (REST API). Both approaches require each service to collect metric telemetry specific to that service. After collecting it, the service must either push it onto a message topic (as a message) or cache it (into memory or some storage mechanism depending on whether the storage needs to be durable or not) and allow for a REST API call that would cause the data to be pulled from that cache and provided in a response to the REST call. Given both mechanisms require the same collection process, the belief is that push is probably preferred today by adopters. In the future, if highly desired, a pull REST API could be added (along with a decision on how to cache the metrics telemetry for that pull). Per Core Working Group meeting of 2/24/22 - importantly , EdgeX is just making the metrics telemetry available on the internal EdgeX message bus. An adopter would need to create something to pull the data off this bus to use it in some way. As voiced by several on the call, it is important for the adopter to realize that today, \"we (EdgeX) are not providing the last mile in metrics data\". The adopter must provide that last mile which is to pick the data from the topic, make it available to their systems and do something with it. Per Core Working Group meeting of 2/24/22 (and in many other previous meetings on this ADR) - it was decided not to use Prometheus (or Prometheus library) as the means to provide for metrics. The reasons for this are many: Push vs pull is favored in the first implementation (see point above). Also see similar debate online for the pluses/minuses of each approach. EdgeX wants to make telemetry data available without dictating the specific mechanism for making the data more widely available. Specific debate centered on use of Prometheus as a popular collection library (to use inside of services to collect the data) as well as a monitoring system to watch/display the data. While Prometheus is popular open source approach, it was felt that many organizations choose to use InfluxDB/Grafana, DataDog, AppDynamics, a cloud provided mechanism, or their own home-grown solution to collect, analyse, visualize and otherwise use the telemetry. Therefore, rather than dictating the selection of the monitoring system, EdgeX would simply make the data available whereby and organization could choose their own monitoring system/tooling. It should be noted that the EdgeX approach merely makes the telemetry data available by message bus. A Prometheus approach would provide collection as well as backend system to otherwise collect, analyse, display, etc. the data. Therefore, there is typically work to be done by the adopter to get the telemetry data from the proposed EdgeX message bus solution and do something with it. There are some reporters that come with go-metrics that allow for data to be taken directly from go-metrics and pushed to an intermediary for Prometheus and other monitoring/telemetry platforms as referenced above. These capabilities may not be very well supported and is beyond the scope of this EdgeX ADR. However, even without reporters , it was felt a relatively straightforward exercise (on the part of the adopter) to create an application that listens to the EdgeX metrics message bus and makes that data available via pull REST API for Prometheus if desired. The Prometheus client libraries would have to be added to each service which would bloat the services (although they are available for both Go an C). The benefit of using go-metrics is that it is used already by Hashicorp Consul (so already in the Go services). Implementation Details for Go The go-metrics package offers the following types of metrics collection: Gauges: holds a single integer (int64) value. Example use: Number of notifications in retry status Operations to update the gauge and get the gauge's value Example code: g := metrics . NewGauge () g . Update ( 42 ) // set the value to 42 g . Update ( 10 ) // now set the value to 10 fmt . Println ( g . Value ()) // print out the current value in the gauge = 10 Counter: holds a integer (in64) count. A counter could be implemented with a Gauge. Example use: the current store and forward queue size Operations to increment, decrement, clear and get the counter's count (or value) c := metrics . NewCounter () c . Inc ( 1 ) // add one to the current counter c . Inc ( 10 ) // add 10 to the current counter, making it 11 c . Dec ( 5 ) // decrement the counter by 5, making it 6 fmt . Println ( c . Count ()) // print out the current count of the counter = 6 Meter: measures the rate (int64) of events over time (at one, five and fifteen minute intervals). Example use: the number or rate of requests on a service API Operations: provide the total count of events as well as the mean and rate at 1, 5, and 15 minute rates m := metrics . NewMeter () m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by fmt . Println ( m . Count ()) // prints 4 fmt . Println ( m . Rate1 ()) // prints 0.11075889086811593 fmt . Println ( m . Rate5 ()) // prints 0.1755318374350548 fmt . Println ( m . Rate15 ()) // prints 0.19136522498856992 fmt . Println ( m . RateMean ()) //prints 0.06665062941438574 Histograms: measure the statistical distribution of values (int64 values) in a collection of values. Example use: response times on APIs Operations: update and get the min, max, count, percentile, sample, sum and variance from the collection h := metrics . NewHistogram ( metrics . NewUniformSample ( 4 )) h . Update ( 10 ) h . Update ( 20 ) h . Update ( 30 ) h . Update ( 40 ) fmt . Println (( h . Max ())) // prints 40 fmt . Println ( h . Min ()) // prints 10 fmt . Println ( h . Mean ()) // prints 25 fmt . Println ( h . Count ()) // prints 4 fmt . Println ( h . Percentile ( 0.25 )) //prints 12.5 fmt . Println ( h . Variance ()) //prints 125 fmt . Println ( h . Sample ()) //prints &{4 {0 0} 4 [10 20 30 40]} Timer: measures both the rate a particular piece of code is called and the distribution of its duration Example use: how often an app service function gets called and how long it takes get through the function Operations: update and get min, max, count, rate1, rate5, rate15, mean, percentile, sum and variance from the collection t := metrics . NewTimer () t . Update ( 10 ) time . Sleep ( 15 * time . Second ) t . Update ( 20 ) time . Sleep ( 15 * time . Second ) t . Update ( 30 ) time . Sleep ( 15 * time . Second ) t . Update ( 40 ) time . Sleep ( 15 * time . Second ) fmt . Println (( t . Max ())) // prints 40 fmt . Println ( t . Min ()) // prints 10 fmt . Println ( t . Mean ()) // prints 25 fmt . Println ( t . Count ()) // prints 4 fmt . Println ( t . Sum ()) // prints 100 fmt . Println ( t . Percentile ( 0.25 )) //prints 12.5 fmt . Println ( t . Variance ()) //prints 125 fmt . Println ( t . Rate1 ()) // prints 0.1116017821771607 fmt . Println ( t . Rate5 ()) // prints 0.1755821073441404 fmt . Println ( t . Rate15 ()) // prints 0.1913711954736821 fmt . Println ( t . RateMean ()) //prints 0.06665773963998162 Note The go-metrics package does offer some variants of these like the GaugeFloat64 to hold 64 bit floats. Consequences Should there be a global configuration option to turn all metrics off/on? EdgeX doesn't yet have global config so this will have to be by service. Given the potential that each service publishes metrics to the same message topic, 0MQ is not implementation option unless each service uses a different 0MQ pipe (0MQ topics do not allow multiple publishers). Like the DS to App Services implementation, do we allow 0MQ to be used, but only if each service sends to a different 0MQ topic? Probably not. We need to avoid service bloat. EdgeX is not an enterprise system. How can we implement in a concise and economical way? Use of Go metrics helps on the Go side since this is already a module used by EdgeX modules (and brought in by default). Care and concern must be given to not cause too much bloat on the C side. SMA reports on service CPU, memory, configuration and provides the means to start/stop/restart the services. This is currently outside the scope of the new metric collection/monitoring. In the future, 3rd party mechanisms which offer the same capability as SMA may warrant all of SMA irrelevant. The existing notifications service serves to send a notification via alternate protocol outside of EdgeX. This communication service is provided as a generic communication instrument from any micro service and is independent of any type of data or concern. In the future, the notification service could be configured to be a subscriber of the metric messages and trigger appropriate external notification (via email, SMTP, etc.). Reference Possible standards for implementation Open Telemetry statsd go-metrics OpenCensus","title":"EdgeX Metrics Collection"},{"location":"design/adr/0006-Metrics-Collection/#edgex-metrics-collection","text":"","title":"EdgeX Metrics Collection"},{"location":"design/adr/0006-Metrics-Collection/#status","text":"Approved Original proposal 10/24/2020 Approved by the TSC on 3/2/22 Metric (or telemetry) data is defined as the count or rate of some action, resource, or circumstance in the EdgeX instance or specific service. Examples of metrics include: the number of EdgeX Events sent from core data to an application service the number of requests on a service API the average time it takes to process a message through an application service The number of errors logged by a service Control plane events (CPE) are defined as events that occur within an EdgeX instance. Examples of CPE include: a device was provisioned (added to core metadata) a service was stopped service configuration has changed CPE should not be confused with core data Events. Core data Events represent a collection (one or more) of sensor/device readings. Core data Events represent sensing of some measured state of the physical world (temperature, vibration, etc.). CPE represents the detection of some happening inside of the EdgeX software. This ADR outlines metrics (or telemetry) collection and handling. Note This ADR initially incorporated metrics collection and control plane event processing. The EdgeX architects felt the scope of the design was too large to cover under one ADR. Control plane event processing will be covered under a separate ADR in the future.","title":"Status"},{"location":"design/adr/0006-Metrics-Collection/#context","text":"System Management services (SMA and executors) currently provide a limited set of \u201cmetrics\u201d to requesting clients (3rd party applications and systems external to EdgeX). Namely, it provides requesting clients with service CPU and memory usage; both metrics about the resource utilization of the service (the executable) itself versus metrics that are about what is happening inside of the service. Arguably, the current system management metrics can be provided by the container engine and orchestration tools (example: by Docker engine) or by the underlying OS tooling. Info The SMA has been deprecated (since Ireland release) and will be removed in a future, yet named, release. Going forward, users of EdgeX will want to have more insights \u2013 that is more metrics telemetry \u2013 on what is happening directly in the services and the tasks that they are preforming. In other words, users of EdgeX will want more telemetry on service activities to include: sensor data collection (how much, how fast, etc.) command requests handled (how many, to which devices, etc.) sensor data transformation as it is done in application services (how fast, what is filtered, etc) sensor data export (how much is sent, how many exports have failed, etc. ) API requests (how often, how quickly, how many success versus failed attempts, etc.) bootstrapping time (time to come up and be available to other services) activity processing time (amount of time it takes to perform a particular service function - such as respond to a command request)","title":"Context"},{"location":"design/adr/0006-Metrics-Collection/#definitions","text":"Metric (or telemetry) data is defined as the count or rate of some action, resource, or circumstance in the EdgeX instance or specific service. Examples of metrics include: the number of EdgeX Events sent from core data to an application service via message bus (or via device service to application service in Ireland and beyond) the number of requests on a service API the average time it takes to process a message through an application service The number of errors logged by a service The collection and dissemination of metric data will require internal service level instrumentation (relevant to that service) to capture and send data about relevant EdgeX operations. EdgeX does not currently offer any service instrumentation.","title":"Definitions"},{"location":"design/adr/0006-Metrics-Collection/#metric-use","text":"As a first step in implementation of metrics data, EdgeX will make metric data available to other subscribing 3rd party applications and systems, but will not necessarily consume or use this information itself. In the future, EdgeX may consume its own metric data. For example, EdgeX may, in the future, use a metric on the number of EdgeX events being sent to core data (or app services) as the means to throttle back device data collection. In the future, EdgeX application services may optionally subscribe to a service's metrics messages bus (by attaching to the appropriate message pipe for that service). Thus allowing additional filtering, transformation, endpoint control of metric data from that service. At the point where this feature is supported, consideration would need to be made as to whether all events (sensor reading messages and metric messages) go through the same application services. At this time, EdgeX will not persist the metric data (except as it may be retained as part of a message bus subsystem such as in an MQTT broker). Consumers of metric data are responsible for persisting the data if needed, but this is external to EdgeX. Persistence of metric information may be considered in the future based on requirements and adopter demand for such a feature. In general, EdgeX metrics are meant to provide internal services and external applications and systems better information about what is happening \"inside\" EdgeX services and the associated devices with which it communicates.","title":"Metric Use"},{"location":"design/adr/0006-Metrics-Collection/#requirements","text":"Services will push specified metrics collected for that service to a specified (by configuration) message endpoint (as supported by the EdgeX message bus implementation; currently either Redis Pub/Sub or MQTT implementations are supported) Each service will have configuration that specifies a message endpoint for the service metrics. The metrics message topic communications may be secured or unsecured (just as application services provide the means to export to secured or unsecured message pipes today). The configuration will be placed in the Writable area. When a user wishes to change the configuration dynamically (such as turning on/off a metric), then Consul's UI can be used to change it. Services will have configuration which indicates what metrics are available from the service. Services will have configuration which allows EdgeX system managers to select which metrics are on or off - in other words providing configuration that determines what metrics are collected and reported by default. When a metric is turned off (the default setting) the service does not report the metric. When a metric is turned on the service collects and sends the metric to the designated message topic. Metrics collection must be pushed to the designated message topic on some appointed schedule. The schedule would be designated by configuration and done in a way similar to auto events in device services. For the initial implementation, there will be just one scheduled time when all metrics will be collected and pushed to the designated message topic. In the future, there may be a desire to set up a separate schedule for each metric, but this was deemed too complex for the initial implementation. Info Initially, it was proposed that metrics be associated with a \"level\" and allow metrics to be turned on or off by level (like levels associated to log messages in logging). The level of metrics data seems arbitrary at this time and considered too complex for initial implementation. This may be reconsidered in a future release and based on new requirements/use cases. It was also proposed to categorize or label metrics - essentially allowing grouping of various metrics. This would allow groups of metrics to be turned on or off, and allow metrics to be organized per the group when reporting. At this time, this feature is also considered beyond the scope of the initial implementation and to be reconsidered in a future release based on requirements/use case needs. It was also proposed that each service offer a REST API to provide metrics collection information (such as which metrics were being collected) and the ability to turn the collection on or off dynamically. This is deemed out of scope for the first implementation and may be brought back if there are use case requirements / demand for it.","title":"Requirements"},{"location":"design/adr/0006-Metrics-Collection/#requested-metrics","text":"The following is a list of example metrics requested by the EdgeX community and adopters for various service areas. Again, metrics would generally be collected and pushed to the message topic in some configured interval (example: 1/5/15 minutes or other defined interval). This is just a sample of metrics thought relevant by each work group. It may not reflect the metrics supported by the implementation. The exact metrics collected by each service will be determined by the service implementers (or SDK implementers in the case of the app functions and device service SDKs).","title":"Requested Metrics"},{"location":"design/adr/0006-Metrics-Collection/#general","text":"The following metrics apply to all (or most) services. Service uptime (time since last service boot) Cumulative number of API requests succeeded / failed / invalid (2xx vs 5xx vs 4xx) Avg response time (in milliseconds or appropriate unit of measure) on APIs Avg and Max request size","title":"General"},{"location":"design/adr/0006-Metrics-Collection/#coresupporting","text":"Latency (measure of time) an event takes to get through core data Latency (measure of time) a command request takes to get to a device service Indication of health \u2013 that events are being processed during a configurable period Number of events in persistence Number of readings in persistence Number of validation failures (validation of device identification) Number of notification transactions Number of notifications handled Number of failed notification transmissions Number of notifications in retry status","title":"Core/Supporting"},{"location":"design/adr/0006-Metrics-Collection/#application-services","text":"Processing time for a pipeline; latency (measure of time) an event takes to get through an application service pipeline DB access times How often are we failing export to be sent to db to be retried at a later time What is the current store and forward queue size How much data (size in KBs or MBs) of packaged sensor data is being sent to an endpoint (or volume) Number of invalid messages that triggered pipeline Number of events processed","title":"Application Services"},{"location":"design/adr/0006-Metrics-Collection/#device-services","text":"Number of devices managed by this DS Device Requests (which may be more informative than reading counts and rates) Note It is envisioned that there may be additional specific metrics for each device service. For example, the ONVIF camera device service may report number of times camera tampering was detected.","title":"Device Services"},{"location":"design/adr/0006-Metrics-Collection/#security","text":"Security metrics may be more difficult to ascertain as they are cross service metrics. Given the nature of this design (on a per service basis), global security metrics may be out of scope or security metrics collection has to be copied into each service (leading to lots of duplicate code for now). Also, true threat detection based on metrics may be a feature best provided by 3rd party based on particular threats and security profile needs. Number of API requests denied due to wrong access token (Kong) per service and within a given time Number of secrets accessed per service name Count of any accesses and failures to the data persistence layer Count of service start and restart attempts","title":"Security"},{"location":"design/adr/0006-Metrics-Collection/#design-proposal","text":"","title":"Design Proposal"},{"location":"design/adr/0006-Metrics-Collection/#collect-and-push-architecture","text":"Metric data will be collected and cached by each service. At designated times (kicked off by configurable schedule), the service will collect telemetry data from the cache and push it to a designated message bus topic.","title":"Collect and Push Architecture"},{"location":"design/adr/0006-Metrics-Collection/#metrics-messaging","text":"Cached metric data, at the designated time, will be marshaled into a message and pushed to the pre-configured message bus topic. Each metric message consists of several key/value pairs: - a required name (the name of the metric) such as service-uptime - a required value which is the telemetry value collected such as 120 as the number of hours the service has been up. - a required timestamp is the time (in Epoch timestamp/milliseconds format) at which the data was collected (similar in nature to the origin of sensed data). - an optional collection (array) of tags. The tags are sets of key/value pairs of strings that provide amplifying information about the telemetry. Tags may include: - originating service name - unit of measure associated with the telemetry value - value type of the value - additional values when the metric is more than just one value (example: when using a histogram, it would include min, max, mean and sum values) The metric name must be unique for that service. Because some metrics are reported from multiple services (such as service uptime), the name is not required to be unique across all services. All information (keys, values, tags, etc.) is in string format and placed in a JSON array within the message body. Here are some example representations: Example metric message body with a single value { \"name\" : \"service-up\" , \"value\" : \"120\" , \"timestamp\" : \"1602168089665570000\" , \"tags\" :{ \"service\" : \"coredata\" , \"uom\" : \"days\" , \"type\" : \"int64\" }} Example metric message body with multiple values { \"name\" : \"api-requests\" , \"value\" : \"24\" , \"timestamp\" : \"1602168089665570001\" , \"tags\" :{ \"service\" : \"coredata\" , \"uom\" : \"count\" , \"type\" : \"int64\" , \"mean\" : \"0.0665\" , \"rate1\" : \"0.111\" , \"rate5\" : \"0.150\" , \"rate15\" : \"0.111\" }} Info The key or metric name must be unique when using go-metrics as it requires the metric name to be unique per the registry. Metrics are considered immutable.","title":"Metrics Messaging"},{"location":"design/adr/0006-Metrics-Collection/#configuration","text":"Configuration, not unlike that provided in core data or any device service, will specify the message bus type and locations where the metrics messages should be sent. In fact, the message bus configuration will use (or reuse if the service is already using the message bus) the common message bus configuration as defined below. Common configuration for each service for message queue configuration (inclusive of metrics): [ MessageQueue ] Protocol = 'redis' ## or 'tcp' Host = 'localhost' Port = 5573 Type = 'redis' ## or 'mqtt' PublishTopicPrefix = \"edgex/events/core\" # standard and existing core or device topic for publishing [ MessageQueue.Optional ] # Default MQTT Specific options that need to be here to enable environment variable overrides of them # Client Identifiers ClientId = \"device-virtual\" # Connection information Qos = \"0\" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified Additional configuration must be provided in each service to provide metrics / telemetry specific configuration. This area of the configuration will likely be different for each type of service. Additional metrics collection configuration to be provided include: Trigger the collection of telemetry from the metrics cache and sending it into the appointed message bus. Define which metrics are available and which are turned off and on . All are false by default. The list of metrics can and likely will be different per service. The keys in this list are the metric name. True and false are used for on and off values. Specify the metrics topic prefix where metrics data will be published to (ex: providing the prefix /edgex/telemetry/topic name where the service and metric name [service-name]/[metric-name] will be appended per metric (allowing subscribers to filter by service or metric name) These metrics configuration options will be defined in the Writable area of configuration.toml so as to allow for dynamic changes to the configuration (when using Consul). Specifically, the [Writable].[Writable.Telemetry] area will dictate metrics collection configuration like this: [[ Writable ]] [[ Writable.Telemetry ]] Interval = \"30s\" PublishTopicPrefix = \"edgex/telemetry\" # // will be added to this Publish Topic prefix #available metrics listed here. All metrics should be listed off (or false) by default service-up = false api-requests = false Info It was discussed that in future EdgeX releases, services may want separate message bus connections. For example one for sensor data and one for metrics telemetry data. This would allow the QoS and other settings of the message bus connection to be different. This would allow sensor data collection, for example, to be messaged with a higher QoS than that of metrics. As an alternate approach, we could modify go-mod-messaging to allow setting QoS per topic (and thereby avoid multiple connections). For the initial release of this feature, the service will use the same connection (and therefore configuration) for metrics telemetry as well as sensor data.","title":"Configuration"},{"location":"design/adr/0006-Metrics-Collection/#library-support","text":"Each service will now need go-mod-messaging support (for GoLang services and the equivalent for C services). Each service would determine when and what metrics to collect and push to the message bus, but will use a common library chosen for each EdgeX language supported (Go or C currently) Use of go-metrics (a GoLang library to publish application metrics) would allow EdgeX to utilize (versus construct) a library utilized by over 7 thousand projects. It provides the means to capture various types of metrics in a registry (a sophisticated map). The metrics can then be published ( reported ) to a number of well known systems such as InfluxDB, Graphite, DataDog, and Syslog. go-metrics is a Go library made from original Java package https://github.com/dropwizard/metrics. A similar package would need to be selected (or created) for C. Per the Core WG meeting of 2/24/22 - it is important to provide an implementation that is the same in Go or C. The adopter of EdgeX should not see a difference in whether the metrics/telemetry is collected by a C or Go service. Configuration of metrics in a C or Go service should have the same structure. The C based metrics collection mechanism in C services (specifically as provided for in our C device service SDK) may operate differently \"under the covers\" but its configuration and resulting metrics messages on the EdgeX message bus must be formatted/organized the same. Considerations in the use of go-metrics This is a Golang only library. Using this library would not provide with any package to use for the C services. If there are expectations for parity between the services, this may be more difficult to achieve given the features of go-metrics. go-metrics will still require the EdgeX team to develop a bootstrapping apparatus to take the metrics configuration and register each of the metrics defined in the configuration in go-metrics. go-metrics would also require the EdgeX team to develop the means to periodically extract the metrics data from the registry and ship it via message bus (something the current go-metrics library does not do). While go-metrics offers the ability for data to be reported to other systems, it would required EdgeX to expose these capabilities (possibly through APIs) if a user wanted to export to these subsystems in addition to the message bus. Per the Kamakura Planning Meeting, it was noted that go-metrics is already a dependency in our Go code due to its use other 3rd party packages (see https://github.com/edgexfoundry/edgex-go/blob/4264632f3ddafb0cbc2089cffbea8c0719035c96/go.sum#L18). Community questions about go-metrics Per the Monthly Architect's meeting of 9/20/21): How it manages the telemetry data (persistence, in memory, database, etc.)? In memory - in a \"registry\"; essentially a key/value store where the key is the metric name Does it offer a query API (in order to easily support the ADR suggested REST API)? Yes - metrics are stored in a \"Registry\" (MetricRegistry - essentially a map). Get (or GetAll) methods provided to query for metrics What does the go-metrics package do so that its features can become requirements for C side? About a dozen types of metrics collection (simple gauge or counter to more sophisticated structures like Histograms) - all stored in a registry (map). How is the data made available? Report out (export or publish) to various integrated packages (InfluxDB, Graphite, DataDog, Syslog, etc.). Nothing to MQTT or other base message service. This would have to be implemented from scratch. Can the metric/telemetry count be reset if needed? Does this happen whenever it posts to the message bus? How would this work for REST? Yes, you can unregister and re-register the metric. A REST API would have to be constructed to call this capability. As an alternative to go-metrics, there is another library called OpenCensus . This is a multi-language metrics library, including Go and C++. This library is more feature rich. OpenCensus is also roughly 5x the size of the go-metrics library.","title":"Library Support"},{"location":"design/adr/0006-Metrics-Collection/#additional-open-questions","text":"Should consideration be given to allow metrics to be placed in different topics per name? If so, we will have to add to the topic name like we do for device name in device services? A future consideration Should consideration be given to incorporate alternate protocols/standards for metric collection such as https://opentelemetry.io/ or https://github.com/statsd/? Go metrics is already a library pulled into all Go services. These packages may be used in C side implementations.","title":"Additional Open Questions"},{"location":"design/adr/0006-Metrics-Collection/#decision","text":"Per the Monthly Architect's meeting of 12/13/21 - it was decided to use go-metrics for Go services over creating our own library or using open census. C services will either find/pick a package that provides similar functionality to go-metrics or implement internally something providing MVP capability. Use of go-metrics helps avoid too much service bloat since it is already in most Go services. Per the same Monthly Architect's meeting, it as decided to implement metrics in Go services first. Per the Monthly Architect's meeting of 1/24/22 - it was decided not to support a REST API on all services that would provide information on what metrics the service provides and the ability to turn them on / off. Instead, the decision was to use Writable configuration and allow Consul to be the means to change the configuration (dynamically). If an adopter chooses not to use Consul, then the configuration with regard to metrics collection, as with all configuration in this circumstance, would be static. If an external API need is requested in the future (such as from an external UI or tool), a REST API may be added. See older versions of this PR for ideas on implementation in this case. Per Core Working Group meeting of 2/24/22 (and in many other previous meetings on this ADR) - it was decided that the EdgeX approach should be one of push (via message bus/MQTT) vs. pull (REST API). Both approaches require each service to collect metric telemetry specific to that service. After collecting it, the service must either push it onto a message topic (as a message) or cache it (into memory or some storage mechanism depending on whether the storage needs to be durable or not) and allow for a REST API call that would cause the data to be pulled from that cache and provided in a response to the REST call. Given both mechanisms require the same collection process, the belief is that push is probably preferred today by adopters. In the future, if highly desired, a pull REST API could be added (along with a decision on how to cache the metrics telemetry for that pull). Per Core Working Group meeting of 2/24/22 - importantly , EdgeX is just making the metrics telemetry available on the internal EdgeX message bus. An adopter would need to create something to pull the data off this bus to use it in some way. As voiced by several on the call, it is important for the adopter to realize that today, \"we (EdgeX) are not providing the last mile in metrics data\". The adopter must provide that last mile which is to pick the data from the topic, make it available to their systems and do something with it. Per Core Working Group meeting of 2/24/22 (and in many other previous meetings on this ADR) - it was decided not to use Prometheus (or Prometheus library) as the means to provide for metrics. The reasons for this are many: Push vs pull is favored in the first implementation (see point above). Also see similar debate online for the pluses/minuses of each approach. EdgeX wants to make telemetry data available without dictating the specific mechanism for making the data more widely available. Specific debate centered on use of Prometheus as a popular collection library (to use inside of services to collect the data) as well as a monitoring system to watch/display the data. While Prometheus is popular open source approach, it was felt that many organizations choose to use InfluxDB/Grafana, DataDog, AppDynamics, a cloud provided mechanism, or their own home-grown solution to collect, analyse, visualize and otherwise use the telemetry. Therefore, rather than dictating the selection of the monitoring system, EdgeX would simply make the data available whereby and organization could choose their own monitoring system/tooling. It should be noted that the EdgeX approach merely makes the telemetry data available by message bus. A Prometheus approach would provide collection as well as backend system to otherwise collect, analyse, display, etc. the data. Therefore, there is typically work to be done by the adopter to get the telemetry data from the proposed EdgeX message bus solution and do something with it. There are some reporters that come with go-metrics that allow for data to be taken directly from go-metrics and pushed to an intermediary for Prometheus and other monitoring/telemetry platforms as referenced above. These capabilities may not be very well supported and is beyond the scope of this EdgeX ADR. However, even without reporters , it was felt a relatively straightforward exercise (on the part of the adopter) to create an application that listens to the EdgeX metrics message bus and makes that data available via pull REST API for Prometheus if desired. The Prometheus client libraries would have to be added to each service which would bloat the services (although they are available for both Go an C). The benefit of using go-metrics is that it is used already by Hashicorp Consul (so already in the Go services).","title":"Decision"},{"location":"design/adr/0006-Metrics-Collection/#implementation-details-for-go","text":"The go-metrics package offers the following types of metrics collection: Gauges: holds a single integer (int64) value. Example use: Number of notifications in retry status Operations to update the gauge and get the gauge's value Example code: g := metrics . NewGauge () g . Update ( 42 ) // set the value to 42 g . Update ( 10 ) // now set the value to 10 fmt . Println ( g . Value ()) // print out the current value in the gauge = 10 Counter: holds a integer (in64) count. A counter could be implemented with a Gauge. Example use: the current store and forward queue size Operations to increment, decrement, clear and get the counter's count (or value) c := metrics . NewCounter () c . Inc ( 1 ) // add one to the current counter c . Inc ( 10 ) // add 10 to the current counter, making it 11 c . Dec ( 5 ) // decrement the counter by 5, making it 6 fmt . Println ( c . Count ()) // print out the current count of the counter = 6 Meter: measures the rate (int64) of events over time (at one, five and fifteen minute intervals). Example use: the number or rate of requests on a service API Operations: provide the total count of events as well as the mean and rate at 1, 5, and 15 minute rates m := metrics . NewMeter () m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by m . Mark ( 1 ) // add one to the current meter value time . Sleep ( 15 * time . Second ) // allow some time to go by fmt . Println ( m . Count ()) // prints 4 fmt . Println ( m . Rate1 ()) // prints 0.11075889086811593 fmt . Println ( m . Rate5 ()) // prints 0.1755318374350548 fmt . Println ( m . Rate15 ()) // prints 0.19136522498856992 fmt . Println ( m . RateMean ()) //prints 0.06665062941438574 Histograms: measure the statistical distribution of values (int64 values) in a collection of values. Example use: response times on APIs Operations: update and get the min, max, count, percentile, sample, sum and variance from the collection h := metrics . NewHistogram ( metrics . NewUniformSample ( 4 )) h . Update ( 10 ) h . Update ( 20 ) h . Update ( 30 ) h . Update ( 40 ) fmt . Println (( h . Max ())) // prints 40 fmt . Println ( h . Min ()) // prints 10 fmt . Println ( h . Mean ()) // prints 25 fmt . Println ( h . Count ()) // prints 4 fmt . Println ( h . Percentile ( 0.25 )) //prints 12.5 fmt . Println ( h . Variance ()) //prints 125 fmt . Println ( h . Sample ()) //prints &{4 {0 0} 4 [10 20 30 40]} Timer: measures both the rate a particular piece of code is called and the distribution of its duration Example use: how often an app service function gets called and how long it takes get through the function Operations: update and get min, max, count, rate1, rate5, rate15, mean, percentile, sum and variance from the collection t := metrics . NewTimer () t . Update ( 10 ) time . Sleep ( 15 * time . Second ) t . Update ( 20 ) time . Sleep ( 15 * time . Second ) t . Update ( 30 ) time . Sleep ( 15 * time . Second ) t . Update ( 40 ) time . Sleep ( 15 * time . Second ) fmt . Println (( t . Max ())) // prints 40 fmt . Println ( t . Min ()) // prints 10 fmt . Println ( t . Mean ()) // prints 25 fmt . Println ( t . Count ()) // prints 4 fmt . Println ( t . Sum ()) // prints 100 fmt . Println ( t . Percentile ( 0.25 )) //prints 12.5 fmt . Println ( t . Variance ()) //prints 125 fmt . Println ( t . Rate1 ()) // prints 0.1116017821771607 fmt . Println ( t . Rate5 ()) // prints 0.1755821073441404 fmt . Println ( t . Rate15 ()) // prints 0.1913711954736821 fmt . Println ( t . RateMean ()) //prints 0.06665773963998162 Note The go-metrics package does offer some variants of these like the GaugeFloat64 to hold 64 bit floats.","title":"Implementation Details for Go"},{"location":"design/adr/0006-Metrics-Collection/#consequences","text":"Should there be a global configuration option to turn all metrics off/on? EdgeX doesn't yet have global config so this will have to be by service. Given the potential that each service publishes metrics to the same message topic, 0MQ is not implementation option unless each service uses a different 0MQ pipe (0MQ topics do not allow multiple publishers). Like the DS to App Services implementation, do we allow 0MQ to be used, but only if each service sends to a different 0MQ topic? Probably not. We need to avoid service bloat. EdgeX is not an enterprise system. How can we implement in a concise and economical way? Use of Go metrics helps on the Go side since this is already a module used by EdgeX modules (and brought in by default). Care and concern must be given to not cause too much bloat on the C side. SMA reports on service CPU, memory, configuration and provides the means to start/stop/restart the services. This is currently outside the scope of the new metric collection/monitoring. In the future, 3rd party mechanisms which offer the same capability as SMA may warrant all of SMA irrelevant. The existing notifications service serves to send a notification via alternate protocol outside of EdgeX. This communication service is provided as a generic communication instrument from any micro service and is independent of any type of data or concern. In the future, the notification service could be configured to be a subscriber of the metric messages and trigger appropriate external notification (via email, SMTP, etc.).","title":"Consequences"},{"location":"design/adr/0006-Metrics-Collection/#reference","text":"Possible standards for implementation Open Telemetry statsd go-metrics OpenCensus","title":"Reference"},{"location":"design/adr/0018-Service-Registry/","text":"Service Registry Status Context Existing Behavior Device Services Registry Client Interface Usage Core and Support Services Security Proxy Setup History Problem Statement Decision References Status Approved (by TSC vote on 3/25/21) Context An EdgeX system may be run with an optional service registry, the use of which (see the related ADR 0001-Registry-Refactor [1]) can be controlled on a per-service basis via the -r/-registry commmand line options. For the purposes of this ADR, a base assumption is that the registry has been enabled for all services. The default service registry used by EdgeX is Consul [2] from Hashicorp. Consul is also the default configuration provider for EdgeX. This ADR is meant to address the current usage of the registry by EdgeX services, and in particular whether the EdgeX services are using the registry to determine the location of peer services vs. using static per-service configuration. The reason this is being investigated is that there has been a proposal that EdgeX do away with the registry functionality, as the current implementation is not considered secure , due to the current configuration of Consul as used by the latest version of EdgeX (Hanoi/1.3.0). According to the original Service Name Design document (v6) [3] written during the California (0.6) release of EdgeX, all EdgeX Foundry microservices should be able to accomplish the following tasks: Register with the configuration/registration (referred to simply as \u201cthe registry\u201d for the rest of this document) provider (today Consul) Respond to availability requests Respond to shutdown requests by: Cleaning up resources in an orderly fashion Unregistering itself from the registry Get the address (host & port) of another EdgeX microservice by service name through the registry (when enabled) The purpose of this design is to ensure that services themselves advertise their location to the rest of the system by first self- registering. Most service registries (including Consul) implement some sort of health check mechanism. If a service is failing one or more health checks, the registry will stop reporting its availability when queried. Note - the design specifically excludes device services from this service lookup, as Core Metadata maintains a persistent store of DeviceService objects which provide service location for device services. Existing Behavior This section documents the existing behavior in the Hanoi (1.3.x) version of EdgeX. Device Services Device Virtual's behavior was first tested using the edgexfoundry snap (which is configured to always use the registry) by doing the following: $ sudo snap install edgexfoundry $ cp /var/snap/edgexfoundry/current/config/device-virtual/res/configuration.toml . I edited the file, removing the [Client.Data] section completely and copied the file back into place. Next I enabled device-virtual while monitoring the journal output. $ sudo cp configuration.toml /var/snap/edgexfoundry/current/config/device-virtual/res/ $ sudo snap set edgexfoundry device-virtual=on The following error was seen in the journal: level=INFO app=device-virtual source=httpserver.go:94 msg=\"Web server starting (0.0.0.0:49990)\" error: fatal error; Host setting for Core Data client not configured Next I followed the same steps, but instead of completely removing the client, I instead set the client ports to invalid values. In this case the service logged the following errors and exited: level=ERROR app=device-virtual source=service.go:149 msg=\"DeviceServicForName failed: Get \\\"http://localhost:3112/api/v1/deviceservice/name/device-virtual\\\": dial tcp 127.0.0.1:3112: connect: connection refused\" level=ERROR app=device-virtual source=init.go:45 msg=\"Couldn't register to metadata service: Get \\\"http://localhost:3112/api/v1/deviceservice/name/device-virtual\\\": dial tcp 127.0.0.1:3112: connect: connection refused\\n\" Note - in order to run this second test, the easiest way to do so is to remove and reinstall the snap vs. manually wiping out device-virtual's configuration in Consul. I could have also stopped the service, modified the configuration directly in Consul, and restarted the service. Registry Client Interface Usage Next the service's usage of the go-mod-registry Client interface was examined: type Client interface { // Registers the current service with Registry for discover and health check Register() error // Un-registers the current service with Registry for discover and health check Unregister() error // Simply checks if Registry is up and running at the configured URL IsAlive() bool // Gets the service endpoint information for the target ID from the Registry GetServiceEndpoint(serviceId string) (types.ServiceEndpoint, error) // Checks with the Registry if the target service is available, i.e. registered and healthy IsServiceAvailable(serviceId string) (bool, error) } Summary If a device service is started with the registry flag set: Both Device SDKs register with the registry on startup, and unregister from the registry on normal shutdown. The Go SDK (device-sdk-go) queries the registry to check dependent service availability and health (via IsServiceAvailable ) on startup. Regardless of the registry setting, the Go SDK always sources the addresses of its dependent services from the Client* configuration stanzas. The C SDK queries the registry for the addresses of its dependent services. It pings the services directly to determine their availbility and health. Core and Support Services The same approach was used for Core and Support services (i.e. reviewing the usage of go-mod-bootstrap's Client interface), and ironically, the SMA seems to be the only service in edgex-go that actually queries the registry for service location: ./internal/system/agent/getconfig/executor.go: ep, err := e.registryClient.GetServiceEndpoint(serviceName) ./internal/system/agent/direct/metrics.go: e, err := m.registryClient.GetServiceEndpoint(serviceName) In summary, other than the SMA's configuration and metrics logic, the Core and Support services behave in the same manner as device-sdk-go. Note - the SMA also has a longstanding issue #2486 where it continuousy logs errors if one (or more) of the Support Services are not running. As described in the issue, this could be avoided if the SMA used the registry to determine if the services were actually available. See related issue #1662 ('Look at Driving \"Default Services List\" via Configuration'). Security Proxy Setup The security-proxy-setup service also relies on static service address configuration to configure the server routes for each of the services accessible through the API Gateway (aka Kong). Although it uses the same TOML-based client config keys as the other services, these configuration values are only ever read from the security-proxy-setup's local configuration.toml file, as the security services have never supported using our configuration provider (aka Consul). Note - Another point worth mentioning with respect to security services is that in the Geneva and Hanoi releases the service health checks registered by the services (and the associated IsServiceAvailable method) are used to orchestrate the ordered startup of the security services via a set of Consul scripts. This additional orchestration is only performed when EdgeX is deployed via docker, and is slated to to be removed as part of the Ireland release. History After a bit of research reaching as far back as the California (0.6.1) release of EdgeX, I've managed to piece together why the current implementation works the way it does. This history focues solely on the core and support services. The California release of EdgeX was released in June of 2018 and was the first to include services written using Go. This version of EdgeX as well as versions through the Fuji release all relied on a bootstrapping service called core-config-seed which was responsible for seeding the configuration of all of the core and support services into Consul prior to any of the services being started. This release actually preceded usage of TOML for configuration files, and instead just used a flat key/value format, with keys converted from legacy Java property names (e.g. meta.db.device.url ) to Camel[Pascal]/Case (e.g. MetaDeviceServiceURL). I chose the config key mentioned above on purpose: MetaDeviceURL = \"http://edgex-core-metadata:48081/api/v1/device\" Not only did this config key provide the address of core metadata, it also provided the path of a specific REST endpoint. In later releases of EdgeX, the address of the service and the specific endpoint paths were de-coupled. Instead of following the Service Name design (which was finalized two months earlier), the initial implementation followed the legacy Java implementation and initialized its service clients for each required REST endpoint (belonging to another EdgeX service) directly from the associated *URL config key read from Consul (if enabled) or directly from the configuration file. The shared client initialization code also created an Endpoint monitor goroutine and passed it a go channel channel used by the service to receive updates to the REST API endpoint URL. This monitor goroutine effectively polled Consul every 15s (this became configurable in later versions) for the client's service address and if a change was detected, would write the updated endpoint URL to the given channel, effectively ensuring that the service started using the new URL. It wasn't till late in the Geneva development cycle that I noticed log messages which made me aware of the fact that every one of our services was making a REST call to check the address of a service endpoint every 15s, for every REST endpoint it used! An issue was filed (https://github.com/edgexfoundry/edgex-go/issues/2594), and the client monitoring was removed as part of the Geneva 1.2.1 release. Problem Statement The fundamental problem with the existing implementations (as decribed above), is that there is too much duplication of configuration across services. For instance, Core Data's service port can easily be changed by passing the environment variable SERVICE_PORT to the service on startup. This overrides the configuration read from the configuration provider, and will cause Core Data to listen on the new port, however it has no impact on any services which use Core Data, as the client config for each is read from the configuration provider (excluding security-proxy-setup). This means in order to change a service port, environment variable overrides (e.g. CLIENTS_COREDARA_PORT) need to set for every client service as well as security-proxy-setup (if required). Decision Update the core, support, and security-proxy-setup services to use go-mod-registry's Client.GetServiceEndpoint method (if started with the --registry option) to determine (a) if a service dependency is available and (b) use the returned address information to initialize client endpoints (or setup the correct route in the case of proxy-setup). The same changes also need to be applied to the App Functions SDK and Go Device SDK, with only minor changes required in the C Device SDK (see previous commments re: the current implementation). Note - this design only works if service registration occurs before the service initializes its clients. For instance, Core Data and Core Metadata both depend on the other, and thus if both defer service registration till after client initialization, neither will be able to successfully lookup the address of the other service. Consquences One impact of this decision is that since the security-proxy-setup service currently runs before any of the core and support services are started, it would not be possible to implement this proposal without also modifying the service to use a lazy initialization of the API Gateway's routes. As such, the implementation of this ADR will require more design work with respect to security-proxy-setup. Some of the issues include: Splitting the configuration of the API Gateway from the service route intialization logic, either by making the service long-running or splitting route initialization into it's own service. Handling registry and non-registry scenarios (i.e. add --registry command-line support to security-proxy-setup). Handling changes to service address information (i.e. dynamically update API Gateway routes if/when service addresses change). Finally the proxy-setup's configuration needs to be updated so that its Route entries use service-keys instead of arbitrary names (e.g. ( Route.core-data vs. Route.CoreData ). References [1] ADR 0001-Registry-Refactor [2] Consul [3] Service Name Design v6","title":"Service Registry"},{"location":"design/adr/0018-Service-Registry/#service-registry","text":"Status Context Existing Behavior Device Services Registry Client Interface Usage Core and Support Services Security Proxy Setup History Problem Statement Decision References","title":"Service Registry"},{"location":"design/adr/0018-Service-Registry/#status","text":"Approved (by TSC vote on 3/25/21)","title":"Status"},{"location":"design/adr/0018-Service-Registry/#context","text":"An EdgeX system may be run with an optional service registry, the use of which (see the related ADR 0001-Registry-Refactor [1]) can be controlled on a per-service basis via the -r/-registry commmand line options. For the purposes of this ADR, a base assumption is that the registry has been enabled for all services. The default service registry used by EdgeX is Consul [2] from Hashicorp. Consul is also the default configuration provider for EdgeX. This ADR is meant to address the current usage of the registry by EdgeX services, and in particular whether the EdgeX services are using the registry to determine the location of peer services vs. using static per-service configuration. The reason this is being investigated is that there has been a proposal that EdgeX do away with the registry functionality, as the current implementation is not considered secure , due to the current configuration of Consul as used by the latest version of EdgeX (Hanoi/1.3.0). According to the original Service Name Design document (v6) [3] written during the California (0.6) release of EdgeX, all EdgeX Foundry microservices should be able to accomplish the following tasks: Register with the configuration/registration (referred to simply as \u201cthe registry\u201d for the rest of this document) provider (today Consul) Respond to availability requests Respond to shutdown requests by: Cleaning up resources in an orderly fashion Unregistering itself from the registry Get the address (host & port) of another EdgeX microservice by service name through the registry (when enabled) The purpose of this design is to ensure that services themselves advertise their location to the rest of the system by first self- registering. Most service registries (including Consul) implement some sort of health check mechanism. If a service is failing one or more health checks, the registry will stop reporting its availability when queried. Note - the design specifically excludes device services from this service lookup, as Core Metadata maintains a persistent store of DeviceService objects which provide service location for device services.","title":"Context"},{"location":"design/adr/0018-Service-Registry/#existing-behavior","text":"This section documents the existing behavior in the Hanoi (1.3.x) version of EdgeX.","title":"Existing Behavior"},{"location":"design/adr/0018-Service-Registry/#device-services","text":"Device Virtual's behavior was first tested using the edgexfoundry snap (which is configured to always use the registry) by doing the following: $ sudo snap install edgexfoundry $ cp /var/snap/edgexfoundry/current/config/device-virtual/res/configuration.toml . I edited the file, removing the [Client.Data] section completely and copied the file back into place. Next I enabled device-virtual while monitoring the journal output. $ sudo cp configuration.toml /var/snap/edgexfoundry/current/config/device-virtual/res/ $ sudo snap set edgexfoundry device-virtual=on The following error was seen in the journal: level=INFO app=device-virtual source=httpserver.go:94 msg=\"Web server starting (0.0.0.0:49990)\" error: fatal error; Host setting for Core Data client not configured Next I followed the same steps, but instead of completely removing the client, I instead set the client ports to invalid values. In this case the service logged the following errors and exited: level=ERROR app=device-virtual source=service.go:149 msg=\"DeviceServicForName failed: Get \\\"http://localhost:3112/api/v1/deviceservice/name/device-virtual\\\": dial tcp 127.0.0.1:3112: connect: connection refused\" level=ERROR app=device-virtual source=init.go:45 msg=\"Couldn't register to metadata service: Get \\\"http://localhost:3112/api/v1/deviceservice/name/device-virtual\\\": dial tcp 127.0.0.1:3112: connect: connection refused\\n\" Note - in order to run this second test, the easiest way to do so is to remove and reinstall the snap vs. manually wiping out device-virtual's configuration in Consul. I could have also stopped the service, modified the configuration directly in Consul, and restarted the service.","title":"Device Services"},{"location":"design/adr/0018-Service-Registry/#registry-client-interface-usage","text":"Next the service's usage of the go-mod-registry Client interface was examined: type Client interface { // Registers the current service with Registry for discover and health check Register() error // Un-registers the current service with Registry for discover and health check Unregister() error // Simply checks if Registry is up and running at the configured URL IsAlive() bool // Gets the service endpoint information for the target ID from the Registry GetServiceEndpoint(serviceId string) (types.ServiceEndpoint, error) // Checks with the Registry if the target service is available, i.e. registered and healthy IsServiceAvailable(serviceId string) (bool, error) }","title":"Registry Client Interface Usage"},{"location":"design/adr/0018-Service-Registry/#summary","text":"If a device service is started with the registry flag set: Both Device SDKs register with the registry on startup, and unregister from the registry on normal shutdown. The Go SDK (device-sdk-go) queries the registry to check dependent service availability and health (via IsServiceAvailable ) on startup. Regardless of the registry setting, the Go SDK always sources the addresses of its dependent services from the Client* configuration stanzas. The C SDK queries the registry for the addresses of its dependent services. It pings the services directly to determine their availbility and health.","title":"Summary"},{"location":"design/adr/0018-Service-Registry/#core-and-support-services","text":"The same approach was used for Core and Support services (i.e. reviewing the usage of go-mod-bootstrap's Client interface), and ironically, the SMA seems to be the only service in edgex-go that actually queries the registry for service location: ./internal/system/agent/getconfig/executor.go: ep, err := e.registryClient.GetServiceEndpoint(serviceName) ./internal/system/agent/direct/metrics.go: e, err := m.registryClient.GetServiceEndpoint(serviceName) In summary, other than the SMA's configuration and metrics logic, the Core and Support services behave in the same manner as device-sdk-go. Note - the SMA also has a longstanding issue #2486 where it continuousy logs errors if one (or more) of the Support Services are not running. As described in the issue, this could be avoided if the SMA used the registry to determine if the services were actually available. See related issue #1662 ('Look at Driving \"Default Services List\" via Configuration').","title":"Core and Support Services"},{"location":"design/adr/0018-Service-Registry/#security-proxy-setup","text":"The security-proxy-setup service also relies on static service address configuration to configure the server routes for each of the services accessible through the API Gateway (aka Kong). Although it uses the same TOML-based client config keys as the other services, these configuration values are only ever read from the security-proxy-setup's local configuration.toml file, as the security services have never supported using our configuration provider (aka Consul). Note - Another point worth mentioning with respect to security services is that in the Geneva and Hanoi releases the service health checks registered by the services (and the associated IsServiceAvailable method) are used to orchestrate the ordered startup of the security services via a set of Consul scripts. This additional orchestration is only performed when EdgeX is deployed via docker, and is slated to to be removed as part of the Ireland release.","title":"Security Proxy Setup"},{"location":"design/adr/0018-Service-Registry/#history","text":"After a bit of research reaching as far back as the California (0.6.1) release of EdgeX, I've managed to piece together why the current implementation works the way it does. This history focues solely on the core and support services. The California release of EdgeX was released in June of 2018 and was the first to include services written using Go. This version of EdgeX as well as versions through the Fuji release all relied on a bootstrapping service called core-config-seed which was responsible for seeding the configuration of all of the core and support services into Consul prior to any of the services being started. This release actually preceded usage of TOML for configuration files, and instead just used a flat key/value format, with keys converted from legacy Java property names (e.g. meta.db.device.url ) to Camel[Pascal]/Case (e.g. MetaDeviceServiceURL). I chose the config key mentioned above on purpose: MetaDeviceURL = \"http://edgex-core-metadata:48081/api/v1/device\" Not only did this config key provide the address of core metadata, it also provided the path of a specific REST endpoint. In later releases of EdgeX, the address of the service and the specific endpoint paths were de-coupled. Instead of following the Service Name design (which was finalized two months earlier), the initial implementation followed the legacy Java implementation and initialized its service clients for each required REST endpoint (belonging to another EdgeX service) directly from the associated *URL config key read from Consul (if enabled) or directly from the configuration file. The shared client initialization code also created an Endpoint monitor goroutine and passed it a go channel channel used by the service to receive updates to the REST API endpoint URL. This monitor goroutine effectively polled Consul every 15s (this became configurable in later versions) for the client's service address and if a change was detected, would write the updated endpoint URL to the given channel, effectively ensuring that the service started using the new URL. It wasn't till late in the Geneva development cycle that I noticed log messages which made me aware of the fact that every one of our services was making a REST call to check the address of a service endpoint every 15s, for every REST endpoint it used! An issue was filed (https://github.com/edgexfoundry/edgex-go/issues/2594), and the client monitoring was removed as part of the Geneva 1.2.1 release.","title":"History"},{"location":"design/adr/0018-Service-Registry/#problem-statement","text":"The fundamental problem with the existing implementations (as decribed above), is that there is too much duplication of configuration across services. For instance, Core Data's service port can easily be changed by passing the environment variable SERVICE_PORT to the service on startup. This overrides the configuration read from the configuration provider, and will cause Core Data to listen on the new port, however it has no impact on any services which use Core Data, as the client config for each is read from the configuration provider (excluding security-proxy-setup). This means in order to change a service port, environment variable overrides (e.g. CLIENTS_COREDARA_PORT) need to set for every client service as well as security-proxy-setup (if required).","title":"Problem Statement"},{"location":"design/adr/0018-Service-Registry/#decision","text":"Update the core, support, and security-proxy-setup services to use go-mod-registry's Client.GetServiceEndpoint method (if started with the --registry option) to determine (a) if a service dependency is available and (b) use the returned address information to initialize client endpoints (or setup the correct route in the case of proxy-setup). The same changes also need to be applied to the App Functions SDK and Go Device SDK, with only minor changes required in the C Device SDK (see previous commments re: the current implementation). Note - this design only works if service registration occurs before the service initializes its clients. For instance, Core Data and Core Metadata both depend on the other, and thus if both defer service registration till after client initialization, neither will be able to successfully lookup the address of the other service.","title":"Decision"},{"location":"design/adr/0018-Service-Registry/#consquences","text":"One impact of this decision is that since the security-proxy-setup service currently runs before any of the core and support services are started, it would not be possible to implement this proposal without also modifying the service to use a lazy initialization of the API Gateway's routes. As such, the implementation of this ADR will require more design work with respect to security-proxy-setup. Some of the issues include: Splitting the configuration of the API Gateway from the service route intialization logic, either by making the service long-running or splitting route initialization into it's own service. Handling registry and non-registry scenarios (i.e. add --registry command-line support to security-proxy-setup). Handling changes to service address information (i.e. dynamically update API Gateway routes if/when service addresses change). Finally the proxy-setup's configuration needs to be updated so that its Route entries use service-keys instead of arbitrary names (e.g. ( Route.core-data vs. Route.CoreData ).","title":"Consquences"},{"location":"design/adr/0018-Service-Registry/#references","text":"[1] ADR 0001-Registry-Refactor [2] Consul [3] Service Name Design v6","title":"References"},{"location":"design/adr/0023-North-South-Messaging/","text":"North-South Messaging Status Approved by TSC Vote on 4/28/22 Context and Proposed Design Today, data flowing from sensors/devices (the \u201csouthside\u201d) through EdgeX to enterprise applications, databases and cloud-based systems (the \u201cnorthside\u201d) can be accomplished via REST or Message bus. That is, sensor or device data collected by a device service can be sent via REST or message bus to core data. Core data then relays the data to application services via message bus, but the sensor data can also be sent directly from device services to application services via message bus (bypassing core data). The message bus is implemented via Redis Pub/Sub (default) or via MQTT. From the application services, data can be sent to northside endpoints in any number of ways \u2013 including via MQTT. So, in summary, data can be collected from a sensor or device and be sent from the southside to the northside entirely using message bus technology when desired. Today, communications from a 3rd party system (enterprise application, cloud application, etc.) to EdgeX in order to acuate a device or get the latest information from a sensor is accomplished via REST. The 3rd party system makes a REST call of the command service which then relays a request to a device service also using REST. There is no built in means to make a message-based request of EdgeX or the devices/sensors it manages. Note, these REST calls are optionally made via the API Gateway in order to provide access control. In a future release of EdgeX, there is a desire to allow 3rd party systems to make requests of the southside via message bus. Specifically, a 3rd party system will send a command request to the command service via external message broker. The command service would then relay the request via message bus to the managing device service via one of the allowed internal message bus implementations (which could be MQTT or Redis Pub/Sub today). The device service would use the message to trigger action on the device/sensor as it does when it receives a REST request, and respond via message bus back to the command service. In turn, the command service would relay the response to the 3rd party system via external message bus. In summary, this ADR proposes that the core command service adds support for an external MQTT connection (in the same manner that app services provide an external MQTT connection), which will allow it to act as a bridge between the internal message bus (implemented via either MQTT or Redis Pub/Sub) and external MQTT message bus. Note For the purposes of this initial north-to-south message bus communications, external 3rd party communications to the command service will be limited to use of MQTT . Core Command as Message Bus Bridge The core command service will serve as the EdgeX entry point for external, north-to-south message bus requests to the south side. 3rd party systems should not be granted access to the EdgeX internal message bus. Therefore, in order to implement north to south communications via message bus (specifically MQTT), the command service needs to take messages from the 3rd party or external MQTT topics and pass them internally onto the EdgeX internal message bus where they can eventually be routed to the device services and then on to the devices/sensors (southside). In reverse, response messages from the southside will also be sent through the internal EdgeX message bus to the command service where they can then be bridged to the external MQTT topics and respond to the 3rd party system requester. Note Note that eKuiper is allowed access directly to the internal EdgeX message bus. This is a special circumstance of 3rd party external system communication as eKuiper is a sister project that is deemed the EdgeX reference implementation rules engine. In future releases of EdgeX, even eKuiper may be routed through an external to internal message bus bridge for better decoupling and security. Message Bus Subscriptions and Publishing The command service will require the means to publish messages to device services via the EdgeX message bus ( internal message bus ). It would use the messaging client (go-mod-messaging) to create a new MessageClient, connect to the message bus, and publish to designated request message topics (see topic configuration below). The command service will also need to connect to the EdgeX message bus ( internal message bus ) in order to receive responses from the device services after a request by message bus has been made. Again, core command will use the go-mod-messaging MessageClient to subscribe and receive response messages from the device services. In a similar fashion, device services will need to both subscribe and publish to the EdgeX message bus ( internal message bus ) to get command requests and push back any responses to the command service. Go lang device services will, like the command service, use the go-mod-messaging module and MessagingClient to get command requests and send command responses to and from the EdgeX message bus. C based device services will use a C alternative to subscribe and publish to the EdgeX message bus ( internal message bus ). Note, device services already use go-mod-messaging when publishing events/readings to the message bus ( internal message bus ). The command service will also need to subscribe to 3rd party MQTT topics ( external message bus ) in order to get command requests from the 3rd party system. The command service will then relay command requests on to the appropriate device service via the internal message bus (forming the message bus to message bus bridge). Likewise, the command service will accept responses from the device services on the EdgeX message bus ( internal message bus ) and then publish responses to the 3rd party system via the 3rd party MQTT topics ( external message bus ). Command Queries via Command Service Today, 3rd party systems can make a REST call of core command to get the possible commands that can be executed. There are two query REST API endpoints: /device/all (to get the commands for all devices) and device/name/{name} (to get the commands for a specific device by name). It stands to reason that if a 3rd party system wants to send commands via messaging that they would also want to get an understanding of what commands are available via messaging. For this reason, the core command service will also allow message requests to get all command or get all commands for a particular device name. In other words, the core command service must support command \"queries\" via messaging just as it supports command requests via messaging. In the case of command queries, the REST responses include the actual REST command endpoints. For example, the REST query would return core command paths, urls and parameters used to construct REST command requests (as shown in the example below). \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"get\" : true , \"path\" : \"/api/v2/device/name/testDevice1/command/coolingpoint1\" , \"url\" : \"http://localhost:59882\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] } ] When using messaging to make the \"queries\" the response message must return information about how to pass a message to the appropriate topic to make the command request. Therefore, the query response when using messaging would include something like the following: \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint/get\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] }, { \"name\" : \"coolingpoint1\" , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint1/set\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] } ] Note Per Core WG meeting of 4/7/22 - the JSON above serves as a general example. The implementation will have to address get/set (or read/write) differentiation, but this is considered an implementation detail to be resolved by the developers. Note The query response does not contain a URL since it is assumed that the broker address must already be known in order to make the query. Message Structure In REST based command requests (and responses), the HTTP request line contains important information such as the path or target of the request, and the HTTP method type (indicating a GET or PUT request). The HTTP status line provides the information such as the response code (ex: 200 for OK). The body or payload of the HTTP message contains the request details (such as parameters to a device PUT call) or response information (such as events and associated readings from a GET call). Since most message bus protocols lack a generic message header mechanism (as in HTTP), providing request/response metadata is accomplished by defining a message envelope object associated with each request/response. Therefore, messages described in this ADR must provide JSON envelope and payload objects for each request/response. The message topic names act like the HTTP paths and methods in REST requests. That is, the topic names specify the device receiver of any command request as paths do in the HTTP requests. Message Envelope The messages defined in this ADR are JSON formatted requests and responses that share a common base structure. The outer most JSON object represents the message envelope , which is used to convey metadata about request/response (e.g. a correlation identifier which will be added to any relayed request message as well as the response message envelope so that the 3rd party system will know to associate the responses to the original request). Note A Correlation ID (see this article for a more detailed description) is a unique value that is added to every request and response involved in a transaction which could include multiple requests/responses between one or more microservices. It's not meant to correlate requests to responses, its meant to label every message involved in a potentially multi-request transaction. A Request ID should be an identifier returned on the response to a request (providing traceability between single request/response). The envelope will also contain the API version (something provided in the HTTP path when using REST). Command requests in HTTP may also contain ds-pushevent and ds-returnevent query parameters (for GET commands). These will be optionally provided key/value pairs represented in the message envelope 's query parameters (and optionally allows for other parameters in the future). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"API\" : \"V2\" , \"queryParams\" : { \"ds-pushevent\" : \"yes\" , \"ds-returnevent\" : \"yes\" , } ... } Note As with REST requests, if the ds-returnvent was no , then a message with envelope would be returned but with no payload as there would be no events to return. Command Message Payload The request message payload to the command service and those relayed to the device service would mimic their HTTP/REST request body alternatives. The payload provides details needed in executing the command at the south side. In the example GET and PUT messages below, note the envelope wraps or encases the message payload . The payload may be empty (as is typical of GET requests). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"queryParams\" : { \"ds-pushevent\" : \"yes\" , \"ds-returnevent\" : \"yes\" , } } { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"payload\" : { \"AHU-TargetTemperature\" : \"28.5\" , \"AHU-TargetBand\" : \"4.0\" , \"AHU-TargetHumidity\" : { \"Accuracy\" : \"0.2-0.3% RH\" , \"Value\" : 59 } } } Note Payload could be empty and therefore optional in the message structure - and exemplified in the top example here. The response message payload would contain the response from the south side, which is typically EdgeX event/reading objects (in the case of GET requests) but would also include any error message details. Example response messages for a GET and PUT request are shown below. Again, note that the message envelope wraps the response payload . { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"errorCode\" : 0 , \"payload\" : { \"event\" : { \"apiVersion\" : \"v2\" , \"id\" : \"3fa85f64-5717-4562-b3fc-2c963f66afa6\" , \"deviceName\" : \"string\" , \"profileName\" : \"string\" , \"created\" : 0 , \"origin\" : 0 , \"readings\" : [ \"string\" ], \"tags\" : { \"Gateway-id\" : \"HoustonStore-000123\" , \"Latitude\" : \"29.630771\" , \"Longitude\" : \"-95.377603\" } } } } { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"errorCode\" : 1 , \"payload\" : { \"message\" : \"string\" } } Note Get command responses may include CBOR data. The message envelope (which has a content type indicator) will indicate that the payload is either CBOR or JSON. The same message envelope content type indicator that is used in REST communications will be used in this message bus communications. Alert Open discussions per working group meetings and reviews... Should we be validating the messages for version (V2 in this case)? Per @lenny-intel, do we validate incoming REST requests for the particular version of the APIs? Ans (per monthly architect's meeting of 3/28/22): yes we should be validating, but this is not done in REST either. We should implement validation for this message communications. We should also add validation in REST communications in the future (Levski release??). Add this to future planning meeting topics. Should API version be in the response at all? Per @iain-anderson, would the API version be implied in that a V2 request would mean a V2 response? Ans (per monthly architect's meeting of 3/28/22): the DTO already has API version so it will be in the payload automatically. Additionally, it was decided to add it to the message envelope. The reason for this is that when using REST, the path of the request includes the version. But in message communications which lacks the URL/path, it will be beneficial to also include the version in the envelope (in addition to the payload as part of the DTO) so that it is easily determined from the envelope without having to dig around in the payload. Per core WG meeting discussion of 2/24/22 - do we really need the request id (is it redundant based on already having correlation id)? Ans (per monthly architect's meeting of 3/28/22): request id and correlation id are two different things. Request id should be returned on the response to a request (providing traceability between single request/response). Correlation id is used to track across an entire transaction of many service request/responses (providing traceability across a \"transaction\" which is often across many services). So both are needed and should be kept. If we have request id, should it be in the payload? Ans (per monthly architect's meeting of 3/28/22): This should be kept in the message envelope (not the message payaload). Per core WG meeting discussion of 2/24/22 - do we have status code? If so should it mimic the REST/HTTP status code responses? Do we really want to mimic HTTP in our message bus approach? As suggested by @farshidtz, maybe we should just have an error boolean and then have the message indicate the error condition. Ans (per monthly architect's meeting of 3/28/22): We will use errorCode to provide the indication of error. The errorCode will be 0 (no error) or 1 (indicating error) as the two enums for error conditions today. In the future, if we determine the need for additional errors, we will add more \"enums\" or error code numbers. When there is an error (with errorCode set to 1), then the payload contains a message string indicating more information about the error. When there is no error (errorCode 0) then there is no message string in the payload. If we have a status code or error code, where does it belong? In the payload or in the envelope (as it would be in the header in REST)? As a reference, the IoTAAP MQTT to REST bridge provides a status code to message string translation as an example means to handle this problem. Should we use something similar? Ans (per monthly architect's meeting of 3/28/22): errorCode should be in the message envelope not the payload. When there is an error, the payload contains a single message string. Query Message Payload The request message payload to query the command service would mimic their HTTP/REST request body alternatives. The payload provides details needed in executing the command at the south side. In the example query to get all commands below, note the envelope wraps or encases the message payload . The payload will be empty. The query parameters will include the offset and limit (as per the REST counter parts). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"queryParams\" : { \"offset\" : 0 , \"limit\" : 20 , } } I n t he example query t o ge t comma n ds f or a speci f ic device by na me , t he device na me would be i n t he t opic , so t he query message would be wi t hou t i nf orma t io n (a n d removed fr om t he message as queryParams will be op t io nal ). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , } The response message payload for queries would contain the information necessary to make a message-based command request. An example response message is shown below. Again, note that the message envelope wraps the response payload . { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"errorCode\" : 0 , \"payload\" : { \"apiVersion\" : \"v2\" , \"deviceCoreCommands\" : [ { \"deviceName\" : \"testDevice1\" , \"profileName\" : \"testProfile\" , \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"get\" : true , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint1/get\" , \"url\" : \"broker.address:1883\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] } ] }, { \"deviceName\" : \"testDevice1\" , \"profileName\" : \"testProfile\" , \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"set\" : true , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint1/set\" , \"url\" : \"broker.address:1883\" , \"parameters\" : [ { \"resourceName\" : \"resource5\" , \"valueType\" : \"String\" }, { \"resourceName\" : \"resource6\" , \"valueType\" : \"Bool\" } ] } ] } ] } } Topic Naming 3rd party system topics The 3rd party system or application must publish command requests messages to an EdgeX specified MQTT topic ( external message bus ) and subscribe to responses from the same. Messages topics should follow the following pattern: Publishing command request topic: /edgex/command/request/// Subscribing command response topic: /edgex/command/response/# For queries, the following topics are used - Publishing query command request topic: /edgex/commandquery/request - Subscribing query command response topic: /edgex/commandquery/response command service topics The command service must subscribe to the request topics of the 3rd party MQTT topic ( external message bus ) to get command requests, publish those to a topic to send them to a device service via the EdgeX message bus ( internal message bus ), subscribe to response messages on topics from device services ( internal ), and then publish response messages to a topic on the 3rd party MQTT broker ( external ). Message topics for the command service would follow the following standard: Subscribing to 3rd party command request topics: edgex/command/request/# Publishing to device service request topic: edgex/command/request//// Subscribing to device service command response topics: edgex/command/response/# Publishing to 3rd party command response topic: edgex/command/response/// For queries, the following topics are used: Subscribing to 3rd party command query request topics: edgex/commandquery/request Publishing to 3rd party command query response topic: edgex/commandquery/response device service topics The device services must subscribe to the EdgeX command request topic ( internal message bus ) and publish response messages to an EdgeX command response topic. The following naming standard will be applied to these topic names: Subscribing to command request topic: edgex/command/request/# Publishing to command response topic: edgex/command/response//// Configuration Both the EdgeX command service and the device services must contain configuration needed to connect to and publish/subscribe to messages from topics on the EdgeX message bus ( internal ). This includes configuration to access the message bus when secure or insecure. The command service must also be provided configuration to connect to the 3rd party MQTT broker's topics ( external ). Because the communications may be done in a secure or insecure fashion, the core command service will need to be provided access to the 3rd party MQTT broker ( external ) Similar to EdgeX application services, the command service will have access to an external MQTT broker to get command requests and send 3rd parties a response. This will require the command service to have two message queue configuration settings (internal and external). command service configuration Example command service configuration is provided below. [MessageQueue] [InternalMessageQueue] Protocol = \"redis\" Host = \"localhost\" Port = 6379 Type = \"redis\" RequestTopicPrefix = \"edgex/command/request/\" # for publishing requests to the device service; /// will be added to this publish topic prefix ResponseTopic = \"edgex/command/response/#\u201d # for subscribing to device service responses AuthMode = \" usernamepassword \" # required for redis messagebus (secure or insecure). SecretName = \" redisdb \" [ExternalMQTT] Protocol = \" tcp \" Host = \" localhost \" Port = 1883 RequestCommandTopic = \" edgex / command / request / #\" # for subscribing to 3rd party command requests ResponseCommandTopicPrefix = \"edgex/command/response/\" # for publishing responses back to 3rd party systems /// will be added to this publish topic prefix RequestQueryTopic = \"edgex/commandquery/request\" ResponseQueryTopic = \"edgex/commandquery/response\" Note Core command contains no MessageQueue configuration today. This is all additive/new configuration and therefore backward compatible with EdgeX 2.x implementations. device service configuration Example device service configuration is provided below. [MessageQueue] ## already existing message queue configuration (for sending events/readings to the message bus) Protocol = \"redis\" Host = \"localhost\" Port = 6379 Type = \"redis\" AuthMode = \"usernamepassword\" # required for redis messagebus (secure or insecure). SecretName = \"redisdb\" PublishTopicPrefix = \"edgex/events/device\" # /// will be added to this Publish Topic prefix [MessageQueue.Optional] # Default MQTT Specific options that need to be here to enable environment variable overrides of them # Client Identifiers ClientId = \"device-rest\" # Connection information Qos = \"0\" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ## new configuration to allow device services to also communicate via message bus with core command CommandRequestTopic = \"edgex/command/request/#\" # subscribing for inbound command requests CommandResponseTopicPrefix = \"edgex/command/response/\" # publishing outbound command responses; /// will be added to this publish topic prefix Note Most of the device service configuration is existing based on its need to already communicate with the message bus for publishing events/readings. The last two lines are added to allow device services to subscribe and publish command messages from/to the message bus. EdgeX Service (Internal) Message Bus Requests Application services (or other EdgeX services in the future) may want to also use message communications to make command requests. Application services make command requests today via REST. In order to support this, the following need to be added: The command service will also need an internal request topic and internal response topic prefix configuration to allow internal EdgeX services to make command requests (and query requests). [MessageQueue] [InternalMessageQueue] Protocol = \"redis\" Host = \"localhost\" Port = 6379 Type = \"redis\" RequestTopicPrefix = \"edgex/command/request/\" # for publishing requests to the device service; /// will be added to this publish topic prefix ResponseTopic = \"edgex/command/response/#\u201d # for subscribing to device service responses InternalRequestCommandTopic = \" / command / request / #\" # for subscribing to internal command requests InternalResponseCommandTopicPrefix = \"/command/response/\" # for publishing responses back to internal service /// will be added to this publish topic prefix InternalRequestQueryTopic = \"/commandquery/request\" InternalResponseQueryTopic = \"/commandquery/response\" AuthMode = \"usernamepassword\" # required for redis messagebus (secure or insecure). SecretName = \"redisdb\" A new command message client will need to be created to allow internal services (app services in this instance) to conveniently use the message bus communications with core command. The client service's configuration will also be expanded to include the corresponding topic and UseMessageBus flag that enables the new messaging based CommandClient to be created. Example client configuration would look something like the following: [Clients] [Clients.core-command] UseMessageBus = true Protocol = \"redis\" Host = \"localhost\" Port = 6379 CommandRequestTopicPrefix = \"/command/request\" /< device-name >/< command-name >/< method > will be added to this publish topic prefix CommandResponseTopic = \"/command/response/#\" CommandQueryRequestTopic = \"/commandquery/request\" CommandQueryResponseTopic = \"/commandquery/response\" Questions Do we need separate topics for all the devices or would one on the device service suffice? Ans: we have defined the deviceName in the parameterized topic, so one topic should be sufficient for Device Service- edgex/command/request/ ... Would clients (non EdgeX services and applications) want to get a list of available commands via message (instead of calling REST)? Ans: this is a valid question and could be provided via later additions to the command service (or other service like metadata) in the future. It does not have to be tackled immediately. Dynamic configuration of the message subscription is not a user friendly operation today (requiring configuration changes). Ans: In the future, we might want to think about creating additional APIs for Adding/Updating/Deleting/Query the external subscription (and store them to the RedisDB). One could also use the Consul UI to change configuration, but this would require the configuration in question be added to the writable section. Is it acceptable for more than one response to be published by the device service on the same correlation ID? Eg, send back \"Acknowledged\", then \"Scheduled\", then \"Starting\", then \"Done\" statuses? Ans: No, the correlation id has a life span to/from the initial requester to the response back to the requester. Would it make sense to echo the command name into the response, as a reality check? Ans: solved via topic naming. Also, per @lenny-intel: \"not needed as we don't do this in the HTTP response. The response topic doesn't need the extra path info. The request ID or correlation ID is all that is needed to match the response to the request. No need to make it more complex.\" Would sending/receiving binary data (e.g. CBOR) be supported in this north-south message implementation? Ans: today, command service and device services support CBOR get operations but not set (C SDK suppports both). Suggest getting feature parity in place between the SDKs before exploring CBOR support messaging binary/CBOR payloads. Ans update per Monthly Architect's meeting of 2/28/22: support get with CBOR payload. Use of the message bus communications (by the non-EdgeX 3rd party service or application) would bypass the API Gateway. Ans: (per Monthly Architect's meeting of 3/28/22) Since the command service is serving as an external to internal message bus broker. While not an issue, it is worth calling out that the message bus security paradigm in use is not quite the same as what's provided by the API Gateway, which provides access control for EdgeX. When the API Gateway is used, the security configuration is defined by the EdgeX instance. When an EdgeX service acts as a bridge to an external message bus, if the external bus is properly configured, then any application on the bus can now interact with the EdgeX instance. Thus the security configuration is defined by the external broker, not EdgeX. Finally, note that most MQTT brokers support topic ACLs based on client username. Note a number of open questions in the Message Structure section that still need to be addressed. Alert Per TSC meeting of 4/27/22 - the discussion around error response was reopened. There is still some polite disagreement as to whether to keep the error response simple (as documented in this ADR) or to offer errorCode enumerations that are similar to HTTP response codes for common problems (such as ). As part of this discussion, the question is whether the error code enumerations should be exactly that of the HTTP response codes (400, 404, 423, 500, etc.) or more generic (i.e., non-HTTP) response error codes unique to this implementation. The resolution to this question was to explore some options at implementation time. The use of an enumeration (HTTP or other) can be explored during development and options brought forth via PR. Info This ADR does not handle securing the message bus communications between services. This need is to be covered universally in an upcoming ADR. Future Considerations If desired, the query commands could return information to make either a REST or message request. Presumably, the query responses would be the same then for both REST and message query requests so that information returned allows the 3rd party application to choose whether to use REST or message bus to make the command requests. As mentioned in this ADR, eKuiper is allowed access directly to the internal EdgeX message bus. This is a special circumstance of 3rd party external system communication. In future releases of EdgeX, even eKuiper may be routed through an external to internal message bus bridge for better decoupling and security. As noted, validation of all communications (REST or message bus) should be done in the future. In the future, we might want to think about creating additional APIs for Adding/Updating/Deleting/Query the external subscription (and store them to the RedisDB). As part of this consideration, it should be noted that one could also use the Consul UI to change configuration, but this would require the configuration in question be added to the writable section. Consequences References Core Command API IoTAAP MATT REST Bridge","title":"North-South Messaging"},{"location":"design/adr/0023-North-South-Messaging/#north-south-messaging","text":"","title":"North-South Messaging"},{"location":"design/adr/0023-North-South-Messaging/#status","text":"Approved by TSC Vote on 4/28/22","title":"Status"},{"location":"design/adr/0023-North-South-Messaging/#context-and-proposed-design","text":"Today, data flowing from sensors/devices (the \u201csouthside\u201d) through EdgeX to enterprise applications, databases and cloud-based systems (the \u201cnorthside\u201d) can be accomplished via REST or Message bus. That is, sensor or device data collected by a device service can be sent via REST or message bus to core data. Core data then relays the data to application services via message bus, but the sensor data can also be sent directly from device services to application services via message bus (bypassing core data). The message bus is implemented via Redis Pub/Sub (default) or via MQTT. From the application services, data can be sent to northside endpoints in any number of ways \u2013 including via MQTT. So, in summary, data can be collected from a sensor or device and be sent from the southside to the northside entirely using message bus technology when desired. Today, communications from a 3rd party system (enterprise application, cloud application, etc.) to EdgeX in order to acuate a device or get the latest information from a sensor is accomplished via REST. The 3rd party system makes a REST call of the command service which then relays a request to a device service also using REST. There is no built in means to make a message-based request of EdgeX or the devices/sensors it manages. Note, these REST calls are optionally made via the API Gateway in order to provide access control. In a future release of EdgeX, there is a desire to allow 3rd party systems to make requests of the southside via message bus. Specifically, a 3rd party system will send a command request to the command service via external message broker. The command service would then relay the request via message bus to the managing device service via one of the allowed internal message bus implementations (which could be MQTT or Redis Pub/Sub today). The device service would use the message to trigger action on the device/sensor as it does when it receives a REST request, and respond via message bus back to the command service. In turn, the command service would relay the response to the 3rd party system via external message bus. In summary, this ADR proposes that the core command service adds support for an external MQTT connection (in the same manner that app services provide an external MQTT connection), which will allow it to act as a bridge between the internal message bus (implemented via either MQTT or Redis Pub/Sub) and external MQTT message bus. Note For the purposes of this initial north-to-south message bus communications, external 3rd party communications to the command service will be limited to use of MQTT .","title":"Context and Proposed Design"},{"location":"design/adr/0023-North-South-Messaging/#core-command-as-message-bus-bridge","text":"The core command service will serve as the EdgeX entry point for external, north-to-south message bus requests to the south side. 3rd party systems should not be granted access to the EdgeX internal message bus. Therefore, in order to implement north to south communications via message bus (specifically MQTT), the command service needs to take messages from the 3rd party or external MQTT topics and pass them internally onto the EdgeX internal message bus where they can eventually be routed to the device services and then on to the devices/sensors (southside). In reverse, response messages from the southside will also be sent through the internal EdgeX message bus to the command service where they can then be bridged to the external MQTT topics and respond to the 3rd party system requester. Note Note that eKuiper is allowed access directly to the internal EdgeX message bus. This is a special circumstance of 3rd party external system communication as eKuiper is a sister project that is deemed the EdgeX reference implementation rules engine. In future releases of EdgeX, even eKuiper may be routed through an external to internal message bus bridge for better decoupling and security.","title":"Core Command as Message Bus Bridge"},{"location":"design/adr/0023-North-South-Messaging/#message-bus-subscriptions-and-publishing","text":"The command service will require the means to publish messages to device services via the EdgeX message bus ( internal message bus ). It would use the messaging client (go-mod-messaging) to create a new MessageClient, connect to the message bus, and publish to designated request message topics (see topic configuration below). The command service will also need to connect to the EdgeX message bus ( internal message bus ) in order to receive responses from the device services after a request by message bus has been made. Again, core command will use the go-mod-messaging MessageClient to subscribe and receive response messages from the device services. In a similar fashion, device services will need to both subscribe and publish to the EdgeX message bus ( internal message bus ) to get command requests and push back any responses to the command service. Go lang device services will, like the command service, use the go-mod-messaging module and MessagingClient to get command requests and send command responses to and from the EdgeX message bus. C based device services will use a C alternative to subscribe and publish to the EdgeX message bus ( internal message bus ). Note, device services already use go-mod-messaging when publishing events/readings to the message bus ( internal message bus ). The command service will also need to subscribe to 3rd party MQTT topics ( external message bus ) in order to get command requests from the 3rd party system. The command service will then relay command requests on to the appropriate device service via the internal message bus (forming the message bus to message bus bridge). Likewise, the command service will accept responses from the device services on the EdgeX message bus ( internal message bus ) and then publish responses to the 3rd party system via the 3rd party MQTT topics ( external message bus ).","title":"Message Bus Subscriptions and Publishing"},{"location":"design/adr/0023-North-South-Messaging/#command-queries-via-command-service","text":"Today, 3rd party systems can make a REST call of core command to get the possible commands that can be executed. There are two query REST API endpoints: /device/all (to get the commands for all devices) and device/name/{name} (to get the commands for a specific device by name). It stands to reason that if a 3rd party system wants to send commands via messaging that they would also want to get an understanding of what commands are available via messaging. For this reason, the core command service will also allow message requests to get all command or get all commands for a particular device name. In other words, the core command service must support command \"queries\" via messaging just as it supports command requests via messaging. In the case of command queries, the REST responses include the actual REST command endpoints. For example, the REST query would return core command paths, urls and parameters used to construct REST command requests (as shown in the example below). \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"get\" : true , \"path\" : \"/api/v2/device/name/testDevice1/command/coolingpoint1\" , \"url\" : \"http://localhost:59882\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] } ] When using messaging to make the \"queries\" the response message must return information about how to pass a message to the appropriate topic to make the command request. Therefore, the query response when using messaging would include something like the following: \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint/get\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] }, { \"name\" : \"coolingpoint1\" , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint1/set\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] } ] Note Per Core WG meeting of 4/7/22 - the JSON above serves as a general example. The implementation will have to address get/set (or read/write) differentiation, but this is considered an implementation detail to be resolved by the developers. Note The query response does not contain a URL since it is assumed that the broker address must already be known in order to make the query.","title":"Command Queries via Command Service"},{"location":"design/adr/0023-North-South-Messaging/#message-structure","text":"In REST based command requests (and responses), the HTTP request line contains important information such as the path or target of the request, and the HTTP method type (indicating a GET or PUT request). The HTTP status line provides the information such as the response code (ex: 200 for OK). The body or payload of the HTTP message contains the request details (such as parameters to a device PUT call) or response information (such as events and associated readings from a GET call). Since most message bus protocols lack a generic message header mechanism (as in HTTP), providing request/response metadata is accomplished by defining a message envelope object associated with each request/response. Therefore, messages described in this ADR must provide JSON envelope and payload objects for each request/response. The message topic names act like the HTTP paths and methods in REST requests. That is, the topic names specify the device receiver of any command request as paths do in the HTTP requests.","title":"Message Structure"},{"location":"design/adr/0023-North-South-Messaging/#message-envelope","text":"The messages defined in this ADR are JSON formatted requests and responses that share a common base structure. The outer most JSON object represents the message envelope , which is used to convey metadata about request/response (e.g. a correlation identifier which will be added to any relayed request message as well as the response message envelope so that the 3rd party system will know to associate the responses to the original request). Note A Correlation ID (see this article for a more detailed description) is a unique value that is added to every request and response involved in a transaction which could include multiple requests/responses between one or more microservices. It's not meant to correlate requests to responses, its meant to label every message involved in a potentially multi-request transaction. A Request ID should be an identifier returned on the response to a request (providing traceability between single request/response). The envelope will also contain the API version (something provided in the HTTP path when using REST). Command requests in HTTP may also contain ds-pushevent and ds-returnevent query parameters (for GET commands). These will be optionally provided key/value pairs represented in the message envelope 's query parameters (and optionally allows for other parameters in the future). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"API\" : \"V2\" , \"queryParams\" : { \"ds-pushevent\" : \"yes\" , \"ds-returnevent\" : \"yes\" , } ... } Note As with REST requests, if the ds-returnvent was no , then a message with envelope would be returned but with no payload as there would be no events to return.","title":"Message Envelope"},{"location":"design/adr/0023-North-South-Messaging/#command-message-payload","text":"The request message payload to the command service and those relayed to the device service would mimic their HTTP/REST request body alternatives. The payload provides details needed in executing the command at the south side. In the example GET and PUT messages below, note the envelope wraps or encases the message payload . The payload may be empty (as is typical of GET requests). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"queryParams\" : { \"ds-pushevent\" : \"yes\" , \"ds-returnevent\" : \"yes\" , } } { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"payload\" : { \"AHU-TargetTemperature\" : \"28.5\" , \"AHU-TargetBand\" : \"4.0\" , \"AHU-TargetHumidity\" : { \"Accuracy\" : \"0.2-0.3% RH\" , \"Value\" : 59 } } } Note Payload could be empty and therefore optional in the message structure - and exemplified in the top example here. The response message payload would contain the response from the south side, which is typically EdgeX event/reading objects (in the case of GET requests) but would also include any error message details. Example response messages for a GET and PUT request are shown below. Again, note that the message envelope wraps the response payload . { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"errorCode\" : 0 , \"payload\" : { \"event\" : { \"apiVersion\" : \"v2\" , \"id\" : \"3fa85f64-5717-4562-b3fc-2c963f66afa6\" , \"deviceName\" : \"string\" , \"profileName\" : \"string\" , \"created\" : 0 , \"origin\" : 0 , \"readings\" : [ \"string\" ], \"tags\" : { \"Gateway-id\" : \"HoustonStore-000123\" , \"Latitude\" : \"29.630771\" , \"Longitude\" : \"-95.377603\" } } } } { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"errorCode\" : 1 , \"payload\" : { \"message\" : \"string\" } } Note Get command responses may include CBOR data. The message envelope (which has a content type indicator) will indicate that the payload is either CBOR or JSON. The same message envelope content type indicator that is used in REST communications will be used in this message bus communications. Alert Open discussions per working group meetings and reviews... Should we be validating the messages for version (V2 in this case)? Per @lenny-intel, do we validate incoming REST requests for the particular version of the APIs? Ans (per monthly architect's meeting of 3/28/22): yes we should be validating, but this is not done in REST either. We should implement validation for this message communications. We should also add validation in REST communications in the future (Levski release??). Add this to future planning meeting topics. Should API version be in the response at all? Per @iain-anderson, would the API version be implied in that a V2 request would mean a V2 response? Ans (per monthly architect's meeting of 3/28/22): the DTO already has API version so it will be in the payload automatically. Additionally, it was decided to add it to the message envelope. The reason for this is that when using REST, the path of the request includes the version. But in message communications which lacks the URL/path, it will be beneficial to also include the version in the envelope (in addition to the payload as part of the DTO) so that it is easily determined from the envelope without having to dig around in the payload. Per core WG meeting discussion of 2/24/22 - do we really need the request id (is it redundant based on already having correlation id)? Ans (per monthly architect's meeting of 3/28/22): request id and correlation id are two different things. Request id should be returned on the response to a request (providing traceability between single request/response). Correlation id is used to track across an entire transaction of many service request/responses (providing traceability across a \"transaction\" which is often across many services). So both are needed and should be kept. If we have request id, should it be in the payload? Ans (per monthly architect's meeting of 3/28/22): This should be kept in the message envelope (not the message payaload). Per core WG meeting discussion of 2/24/22 - do we have status code? If so should it mimic the REST/HTTP status code responses? Do we really want to mimic HTTP in our message bus approach? As suggested by @farshidtz, maybe we should just have an error boolean and then have the message indicate the error condition. Ans (per monthly architect's meeting of 3/28/22): We will use errorCode to provide the indication of error. The errorCode will be 0 (no error) or 1 (indicating error) as the two enums for error conditions today. In the future, if we determine the need for additional errors, we will add more \"enums\" or error code numbers. When there is an error (with errorCode set to 1), then the payload contains a message string indicating more information about the error. When there is no error (errorCode 0) then there is no message string in the payload. If we have a status code or error code, where does it belong? In the payload or in the envelope (as it would be in the header in REST)? As a reference, the IoTAAP MQTT to REST bridge provides a status code to message string translation as an example means to handle this problem. Should we use something similar? Ans (per monthly architect's meeting of 3/28/22): errorCode should be in the message envelope not the payload. When there is an error, the payload contains a single message string.","title":"Command Message Payload"},{"location":"design/adr/0023-North-South-Messaging/#query-message-payload","text":"The request message payload to query the command service would mimic their HTTP/REST request body alternatives. The payload provides details needed in executing the command at the south side. In the example query to get all commands below, note the envelope wraps or encases the message payload . The payload will be empty. The query parameters will include the offset and limit (as per the REST counter parts). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"queryParams\" : { \"offset\" : 0 , \"limit\" : 20 , } } I n t he example query t o ge t comma n ds f or a speci f ic device by na me , t he device na me would be i n t he t opic , so t he query message would be wi t hou t i nf orma t io n (a n d removed fr om t he message as queryParams will be op t io nal ). { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , } The response message payload for queries would contain the information necessary to make a message-based command request. An example response message is shown below. Again, note that the message envelope wraps the response payload . { \"Correlation-ID\" : \"14a42ea6-c394-41c3-8bcd-a29b9f5e6835\" , \"apiVersion\" : \"v2\" , \"requestId\" : \"e6e8a2f4-eb14-4649-9e2b-175247911369\" , \"errorCode\" : 0 , \"payload\" : { \"apiVersion\" : \"v2\" , \"deviceCoreCommands\" : [ { \"deviceName\" : \"testDevice1\" , \"profileName\" : \"testProfile\" , \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"get\" : true , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint1/get\" , \"url\" : \"broker.address:1883\" , \"parameters\" : [ { \"resourceName\" : \"resource1\" , \"valueType\" : \"Int32\" } ] } ] }, { \"deviceName\" : \"testDevice1\" , \"profileName\" : \"testProfile\" , \"coreCommands\" : [ { \"name\" : \"coolingpoint1\" , \"set\" : true , \"topic\" : \"/edgex/command/request/testDevice1/coolingpoint1/set\" , \"url\" : \"broker.address:1883\" , \"parameters\" : [ { \"resourceName\" : \"resource5\" , \"valueType\" : \"String\" }, { \"resourceName\" : \"resource6\" , \"valueType\" : \"Bool\" } ] } ] } ] } }","title":"Query Message Payload"},{"location":"design/adr/0023-North-South-Messaging/#topic-naming","text":"","title":"Topic Naming"},{"location":"design/adr/0023-North-South-Messaging/#3rd-party-system-topics","text":"The 3rd party system or application must publish command requests messages to an EdgeX specified MQTT topic ( external message bus ) and subscribe to responses from the same. Messages topics should follow the following pattern: Publishing command request topic: /edgex/command/request/// Subscribing command response topic: /edgex/command/response/# For queries, the following topics are used - Publishing query command request topic: /edgex/commandquery/request - Subscribing query command response topic: /edgex/commandquery/response","title":"3rd party system topics"},{"location":"design/adr/0023-North-South-Messaging/#command-service-topics","text":"The command service must subscribe to the request topics of the 3rd party MQTT topic ( external message bus ) to get command requests, publish those to a topic to send them to a device service via the EdgeX message bus ( internal message bus ), subscribe to response messages on topics from device services ( internal ), and then publish response messages to a topic on the 3rd party MQTT broker ( external ). Message topics for the command service would follow the following standard: Subscribing to 3rd party command request topics: edgex/command/request/# Publishing to device service request topic: edgex/command/request//// Subscribing to device service command response topics: edgex/command/response/# Publishing to 3rd party command response topic: edgex/command/response/// For queries, the following topics are used: Subscribing to 3rd party command query request topics: edgex/commandquery/request Publishing to 3rd party command query response topic: edgex/commandquery/response","title":"command service topics"},{"location":"design/adr/0023-North-South-Messaging/#device-service-topics","text":"The device services must subscribe to the EdgeX command request topic ( internal message bus ) and publish response messages to an EdgeX command response topic. The following naming standard will be applied to these topic names: Subscribing to command request topic: edgex/command/request/# Publishing to command response topic: edgex/command/response////","title":"device service topics"},{"location":"design/adr/0023-North-South-Messaging/#configuration","text":"Both the EdgeX command service and the device services must contain configuration needed to connect to and publish/subscribe to messages from topics on the EdgeX message bus ( internal ). This includes configuration to access the message bus when secure or insecure. The command service must also be provided configuration to connect to the 3rd party MQTT broker's topics ( external ). Because the communications may be done in a secure or insecure fashion, the core command service will need to be provided access to the 3rd party MQTT broker ( external ) Similar to EdgeX application services, the command service will have access to an external MQTT broker to get command requests and send 3rd parties a response. This will require the command service to have two message queue configuration settings (internal and external).","title":"Configuration"},{"location":"design/adr/0023-North-South-Messaging/#command-service-configuration","text":"Example command service configuration is provided below. [MessageQueue] [InternalMessageQueue] Protocol = \"redis\" Host = \"localhost\" Port = 6379 Type = \"redis\" RequestTopicPrefix = \"edgex/command/request/\" # for publishing requests to the device service; /// will be added to this publish topic prefix ResponseTopic = \"edgex/command/response/#\u201d # for subscribing to device service responses AuthMode = \" usernamepassword \" # required for redis messagebus (secure or insecure). SecretName = \" redisdb \" [ExternalMQTT] Protocol = \" tcp \" Host = \" localhost \" Port = 1883 RequestCommandTopic = \" edgex / command / request / #\" # for subscribing to 3rd party command requests ResponseCommandTopicPrefix = \"edgex/command/response/\" # for publishing responses back to 3rd party systems /// will be added to this publish topic prefix RequestQueryTopic = \"edgex/commandquery/request\" ResponseQueryTopic = \"edgex/commandquery/response\" Note Core command contains no MessageQueue configuration today. This is all additive/new configuration and therefore backward compatible with EdgeX 2.x implementations.","title":"command service configuration"},{"location":"design/adr/0023-North-South-Messaging/#device-service-configuration","text":"Example device service configuration is provided below. [MessageQueue] ## already existing message queue configuration (for sending events/readings to the message bus) Protocol = \"redis\" Host = \"localhost\" Port = 6379 Type = \"redis\" AuthMode = \"usernamepassword\" # required for redis messagebus (secure or insecure). SecretName = \"redisdb\" PublishTopicPrefix = \"edgex/events/device\" # /// will be added to this Publish Topic prefix [MessageQueue.Optional] # Default MQTT Specific options that need to be here to enable environment variable overrides of them # Client Identifiers ClientId = \"device-rest\" # Connection information Qos = \"0\" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ## new configuration to allow device services to also communicate via message bus with core command CommandRequestTopic = \"edgex/command/request/#\" # subscribing for inbound command requests CommandResponseTopicPrefix = \"edgex/command/response/\" # publishing outbound command responses; /// will be added to this publish topic prefix Note Most of the device service configuration is existing based on its need to already communicate with the message bus for publishing events/readings. The last two lines are added to allow device services to subscribe and publish command messages from/to the message bus.","title":"device service configuration"},{"location":"design/adr/0023-North-South-Messaging/#edgex-service-internal-message-bus-requests","text":"Application services (or other EdgeX services in the future) may want to also use message communications to make command requests. Application services make command requests today via REST. In order to support this, the following need to be added: The command service will also need an internal request topic and internal response topic prefix configuration to allow internal EdgeX services to make command requests (and query requests). [MessageQueue] [InternalMessageQueue] Protocol = \"redis\" Host = \"localhost\" Port = 6379 Type = \"redis\" RequestTopicPrefix = \"edgex/command/request/\" # for publishing requests to the device service; /// will be added to this publish topic prefix ResponseTopic = \"edgex/command/response/#\u201d # for subscribing to device service responses InternalRequestCommandTopic = \" / command / request / #\" # for subscribing to internal command requests InternalResponseCommandTopicPrefix = \"/command/response/\" # for publishing responses back to internal service /// will be added to this publish topic prefix InternalRequestQueryTopic = \"/commandquery/request\" InternalResponseQueryTopic = \"/commandquery/response\" AuthMode = \"usernamepassword\" # required for redis messagebus (secure or insecure). SecretName = \"redisdb\" A new command message client will need to be created to allow internal services (app services in this instance) to conveniently use the message bus communications with core command. The client service's configuration will also be expanded to include the corresponding topic and UseMessageBus flag that enables the new messaging based CommandClient to be created. Example client configuration would look something like the following: [Clients] [Clients.core-command] UseMessageBus = true Protocol = \"redis\" Host = \"localhost\" Port = 6379 CommandRequestTopicPrefix = \"/command/request\" /< device-name >/< command-name >/< method > will be added to this publish topic prefix CommandResponseTopic = \"/command/response/#\" CommandQueryRequestTopic = \"/commandquery/request\" CommandQueryResponseTopic = \"/commandquery/response\"","title":"EdgeX Service (Internal) Message Bus Requests"},{"location":"design/adr/0023-North-South-Messaging/#questions","text":"Do we need separate topics for all the devices or would one on the device service suffice? Ans: we have defined the deviceName in the parameterized topic, so one topic should be sufficient for Device Service- edgex/command/request/ ... Would clients (non EdgeX services and applications) want to get a list of available commands via message (instead of calling REST)? Ans: this is a valid question and could be provided via later additions to the command service (or other service like metadata) in the future. It does not have to be tackled immediately. Dynamic configuration of the message subscription is not a user friendly operation today (requiring configuration changes). Ans: In the future, we might want to think about creating additional APIs for Adding/Updating/Deleting/Query the external subscription (and store them to the RedisDB). One could also use the Consul UI to change configuration, but this would require the configuration in question be added to the writable section. Is it acceptable for more than one response to be published by the device service on the same correlation ID? Eg, send back \"Acknowledged\", then \"Scheduled\", then \"Starting\", then \"Done\" statuses? Ans: No, the correlation id has a life span to/from the initial requester to the response back to the requester. Would it make sense to echo the command name into the response, as a reality check? Ans: solved via topic naming. Also, per @lenny-intel: \"not needed as we don't do this in the HTTP response. The response topic doesn't need the extra path info. The request ID or correlation ID is all that is needed to match the response to the request. No need to make it more complex.\" Would sending/receiving binary data (e.g. CBOR) be supported in this north-south message implementation? Ans: today, command service and device services support CBOR get operations but not set (C SDK suppports both). Suggest getting feature parity in place between the SDKs before exploring CBOR support messaging binary/CBOR payloads. Ans update per Monthly Architect's meeting of 2/28/22: support get with CBOR payload. Use of the message bus communications (by the non-EdgeX 3rd party service or application) would bypass the API Gateway. Ans: (per Monthly Architect's meeting of 3/28/22) Since the command service is serving as an external to internal message bus broker. While not an issue, it is worth calling out that the message bus security paradigm in use is not quite the same as what's provided by the API Gateway, which provides access control for EdgeX. When the API Gateway is used, the security configuration is defined by the EdgeX instance. When an EdgeX service acts as a bridge to an external message bus, if the external bus is properly configured, then any application on the bus can now interact with the EdgeX instance. Thus the security configuration is defined by the external broker, not EdgeX. Finally, note that most MQTT brokers support topic ACLs based on client username. Note a number of open questions in the Message Structure section that still need to be addressed. Alert Per TSC meeting of 4/27/22 - the discussion around error response was reopened. There is still some polite disagreement as to whether to keep the error response simple (as documented in this ADR) or to offer errorCode enumerations that are similar to HTTP response codes for common problems (such as ). As part of this discussion, the question is whether the error code enumerations should be exactly that of the HTTP response codes (400, 404, 423, 500, etc.) or more generic (i.e., non-HTTP) response error codes unique to this implementation. The resolution to this question was to explore some options at implementation time. The use of an enumeration (HTTP or other) can be explored during development and options brought forth via PR. Info This ADR does not handle securing the message bus communications between services. This need is to be covered universally in an upcoming ADR.","title":"Questions"},{"location":"design/adr/0023-North-South-Messaging/#future-considerations","text":"If desired, the query commands could return information to make either a REST or message request. Presumably, the query responses would be the same then for both REST and message query requests so that information returned allows the 3rd party application to choose whether to use REST or message bus to make the command requests. As mentioned in this ADR, eKuiper is allowed access directly to the internal EdgeX message bus. This is a special circumstance of 3rd party external system communication. In future releases of EdgeX, even eKuiper may be routed through an external to internal message bus bridge for better decoupling and security. As noted, validation of all communications (REST or message bus) should be done in the future. In the future, we might want to think about creating additional APIs for Adding/Updating/Deleting/Query the external subscription (and store them to the RedisDB). As part of this consideration, it should be noted that one could also use the Consul UI to change configuration, but this would require the configuration in question be added to the writable section.","title":"Future Considerations"},{"location":"design/adr/0023-North-South-Messaging/#consequences","text":"","title":"Consequences"},{"location":"design/adr/0023-North-South-Messaging/#references","text":"Core Command API IoTAAP MATT REST Bridge","title":"References"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/","text":"Device Services Send Events via Message Bus Status Context Decision Which Message Bus implementations? Go Device SDK C Device SDK Core Data and Persistence V2 Event DTO Validation Message Envelope Application Services MessageBus Topics Configuration Device Services [MessageQueue] Core Data [MessageQueue] Application Services [MessageBus] [Binding] Secure Connections Consequences Status Approved Context Currently EdgeX Events are sent from Device Services via HTTP to Core Data, which then puts the Events on the MessageBus after optionally persisting them to the database. This ADR details how Device Services will send EdgeX Events to other services via the EdgeX MessageBus. Note: Though this design is centered on device services, it does have cross cutting impacts with other EdgeX services and modules Note: This ADR is dependent on the Secret Provider for All to provide the secrets for secure Message Bus connections. Decision Which Message Bus implementations? Multiple Device Services may need to be publishing Events to the MessageBus concurrently. ZMQ will not be a valid option if multiple Device Services are configured to publish. This is because ZMQ only allows for a single publisher. ZMQ will still be valid if only one Device Service is publishing Events. The MQTT and Redis Streams are valid options to use when multiple Device Services are required, as they both support multiple publishers. These are the only other implementations currently available for Go services. The C base device services do not yet have a MessageBus implementation. See the C Device SDK below for details. Note: Documentation will need to be clear when ZMQ can be used and when it can not be used. Go Device SDK The Go Device SDK will take advantage of the existing go-mod-messaging module to enable use of the EdgeX MessageBus. A new bootstrap handler will be created which initializes the MessageBus client based on configuration. See Configuration section below for details. The Go Device SDK will be enhanced to optionally publish Events to the MessageBus anywhere it currently POSTs Events to Core Data. This publish vs POST option will be controlled by configuration with publish as the default. See Configuration section below for details. C Device SDK The C Device SDK will implement its own MessageBus abstraction similar to the one in go-mod-messaging . The first implementation type (MQTT or Redis Streams) is TBD. Using this abstraction allows for future implementations to be added when use cases warrant the additional implementations. As with the Go SDK, the C SDK will be enhanced to optionally publish Events to the MessageBus anywhere it currently POSTs Events to Core Data. This publish vs POST option will be controlled by configuration with publish as the default. See Configuration section below for details. Core Data and Persistence With this design, Events will be sent directly to Application Services w/o going through Core Data and thus will not be persisted unless changes are made to Core Data. To allow Events to optionally continue to be persisted, Core Data will become an additional or secondary (and optional) subscriber for the Events from the MessageBus. The Events will be persisted when they are received. Core Data will also retain the ability to receive Events via HTTP, persist them and publish them to the MessageBus as is done today. This allows for the flexibility to have some device services to be configured to POST Events and some to be configured to publish Events while we transition the Device Services to all have the capability to publishing Events. In the future, once this new Publish approach has been proven, we may decide to remove POSTing Events to Core Data from the Device SDKs. The existing PersistData setting will be ignored by the code path subscribing to Events since the only reason to do this is to persist the Events. There is a race condition for Marked As Pushed when Core Data is persisting Events received from the MessageBus. Core Data may not have finished persisting an Event before the Application Service has processed the Event and requested the Event be Marked As Pushed . It was decided to remove Mark as Pushed capability and just rely on time based scrubbing of old Events. V2 Event DTO As this development will be part of the Ireland release all Events published to the MessageBus will use the V2 Event DTO. This is already implemented in Core Data for the V2 AddEvent API. Validation Services receiving the Event DTO from the MessageBus will log validation errors and stop processing the Event. Message Envelope EdgeX Go Services currently uses a custom Message Envelope for all data that is published to the MessageBus. This envelope wraps the data with metadata, which is ContentType (JSON or CBOR), Correlation-Id and the obsolete Checksum . The Checksum is used when the data is CBOR encoded to identify the Event in V1 API to be mark it as pushed. This checksum is no longer needed as the V2 Event DTO requires the ID be set by the Device Services which will always be used in the V2 API to mark the Events as pushed. The Message Envelope will be updated to remove this property. The C SDK will recreate this Message Envelope. Application Services As part of the V2 API consumption work in Ireland the App Services SDK will be changed to expect to receive V2 Event DTOs rather than the V1 Event model. It will also be updated to no longer expect or use the Checksum currently on the Message Envelope. Note these changes must occur for the V2 consumption and are not directly tied to this effort. The App Service SDK will be enhanced for the secure MessageBus connection described below. See Secure Connections for details MessageBus Topics Note: The change recommended here is not required for this design, but it provides a good opportunity to adopt it. Currently Core Data publishes Events to the simple events topic. All Application Services running receive every Event published, whether they want them or not. The Events can be filtered out using the FilterByDeviceName or FilterByResourceName pipeline functions, but the Application Services still receives every Event and process all the Events to some extent. This could cause load issues in a deployment with many devices and large volume of Events from various devices or a very verbose device that the Application Services is not interested in. Note: The current FilterByDeviceName is only good if the device name is known statically and the only instance of the device defined by the DeviceProfileName . What we really need is FilterByDeviceProfileName which allows multiple instances of a device to be filtered for, rather than a single instance as it it now. The V2 API will be adding DeviceProfileName to the Events, so in Ireland this filter will be possible. Pub/Sub systems have advanced topic schema, which we can take advantage of from Application Services to filter for just the Events the Application Service actual wants. Publishers of Events must add the DeviceProfileName , DeviceName and SourceName to the topic in the form edgex/events/// . The SourceName is the Resource or Command name used to create the Event. This allows Application Services to filter for just the Events from the device(s) it wants by only subscribing to those DeviceProfileNames or the specific DeviceNames or just the specific SourceNames Example subscribe topics if above schema is used: edgex/events/# All Events Core Data will subscribe using this topic schema edgex/events/Random-Integer-Device/# Any Events from devices created from the Random-Integer-Device device profile edgex/events/Random-Integer-Device/Random-Integer-Device1 Only Events from the Random-Integer-Device1 Device edgex/events/Random-Integer-Device/#/Int16 Any Events with Readings from Int16 device resource from devices created from the Random-Integer-Device device profile. **edgex/events/Modbus-Device/#/HVACValues Any Events with Readings from HVACValues device command from devices created from the Modbus-Device device profile. The MessageBus abstraction allows for multiple subscriptions, so an Application Service could specify to receive data from multiple specific device profiles or devices by creating multiple subscriptions. i.e. edgex/Events/Random-Integer-Device/# and edgex/Events/Random-Boolean-Device/# . Currently the App SDK only allows for a single subscription topic to be configured, but that could easily be expanded to handle a list of subscriptions. See Configuration section below for details. Core Data's existing publishing of Events would also need to be changed to use this new topic schema. One challenge with this is Core Data doesn't currently know the DeviceProfileName or DeviceName when it receives a CBOR encoded event. This is because it doesn't decode the Event until after it has published it to the MessageBus. Also, Core Data doesn't know of SourceName at all. The V2 API will be enhanced to change the AddEvent endpoint from /event to /event/{profile}/{device}/{source} so that DeviceProfileName , DeviceName , and SourceName are always know no matter how the request is encoded. This new topic approach will be enabled via each publisher's PublishTopic having the DeviceProfileName , DeviceName and SourceName added to the configured PublishTopicPrefix PublishTopicPrefix = \"edgex/events\" # /// will be added to this Publish Topic prefix See Configuration section below for details. Configuration Device Services All Device services will have the following additional configuration to allow connecting and publishing to the MessageBus. As describe above in the MessageBus Topics section, the PublishTopic will include the DeviceProfileName and DeviceName . [MessageQueue] A MessageQueue section will be added, which is similar to that used in Core Data today, but with PublishTopicPrefix instead of Topic .To enable secure connections, the Username & Password have been replaced with ClientAuth & SecretPath , See Secure Connections section below for details. The added Enabled property controls whether the Device Service publishes to the MessageBus or POSTs to Core Data. [MessageQueue] Enabled = true Protocol = \"tcp\" Host = \"localhost\" Port = 1883 Type = \"mqtt\" PublishTopicPrefix = \"edgex/events\" # /// will be added to this Publish Topic prefix [MessageQueue.Optional] # Default MQTT Specific options that need to be here to enable environment variable overrides of them # Client Identifiers ClientId = \"\" # Connection information Qos = \"0\" # Quality of Sevice values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ClientAuth = \"none\" # Valid values are: `none`, `usernamepassword` or `clientcert` Secretpath = \"messagebus\" # Path in secret store used if ClientAuth not `none` Core Data Core data will also require additional configuration to be able to subscribe to receive Events from the MessageBus. As describe above in the MessageBus Topics section, the PublishTopicPrefix will have DeviceProfileName and DeviceName added to create the actual Public Topic. [MessageQueue] The MessageQueue section will be changed so that the Topic property changes to PublishTopicPrefix and SubscribeEnabled and SubscribeTopic will be added. As with device services configuration, the Username & Password have been replaced with ClientAuth & SecretPath for secure connections. See Secure Connections section below for details. In addition, the Boolean SubscribeEnabled property will be used to control if the service subscribes to Events from the MessageBus or not. [MessageQueue] Protocol = \"tcp\" Host = \"localhost\" Port = 1883 Type = \"mqtt\" PublishTopicPrefix = \"edgex/events\" # /// will be added to this Publish Topic prefix SubscribeEnabled = true SubscribeTopic = \"edgex/events/#\" [MessageQueue.Optional] # Default MQTT Specific options that need to be here to enable evnironment variable overrides of them # Client Identifiers ClientId = \"edgex-core-data\" # Connection information Qos = \"0\" # Quality of Sevice values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ClientAuth = \"none\" # Valid values are: `none`, `usernamepassword` or `clientcert` Secretpath = \"messagebus\" # Path in secret store used if ClientAuth not `none` Application Services [MessageBus] Similar to above, the Application Services MessageBus configuration will change to allow for secure connection to the MessageBus. The Username & Password have been replaced with ClientAuth & SecretPath for secure connections. See Secure Connections section below for details. [MessageBus.Optional] # MQTT Specific options # Client Identifiers ClientId = \"\" # Connection information Qos = \"0\" # Quality of Sevice values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ClientAuth = \"none\" # Valid values are: `none`, `usernamepassword` or `clientcert` Secretpath = \"messagebus\" # Path in secret store used if ClientAuth not `none` [Binding] The Binding configuration section will require changes for the subscribe topics scheme described in the MessageBus Topics section above to filter for Events from specific device profiles or devices. SubscribeTopic will change from a string property containing a single topic to the SubscribeTopics string property containing a comma separated list of topics. This allows for the flexibility for the property to be a single topic with the # wild card so the Application Service receives all Events as it does today. Receive only Events from the Random-Integer-Device and Random-Boolean-Device profiles [Binding] Type = \"messagebus\" SubscribeTopics = \"edgex/events/Random-Integer-Device, edgex/events/Random-Boolean-Device\" Receive only Events from the Random-Integer-Device1 from the Random-Integer-Device profile [Binding] Type = \"messagebus\" SubscribeTopics = \"edgex/events/Random-Integer-Device/Random-Integer-Device1\" or receives all Events: [Binding] Type = \"messagebus\" SubscribeTopics = \"edgex/events/#\" Secure Connections As stated earlier, this ADR is dependent on the Secret Provider for All ADR to provide a common Secret Provider for all Edgex Services to access their secrets. Once this is available, the MessageBus connection can be secured via the following configurable client authentications modes which follows similar implementation for secure MQTT Export and secure MQTT Trigger used in Application Services. none - No authentication usernamepassword - Username & password authentication. clientcert - Client certificate and key for authentication. The secrets specified for the above options are pulled from the Secret Provider using the configured SecretPath . How the secrets are injected into the Secret Provider is out of scope for this ADR and covered in the Secret Provider for All ADR. Consequences If C SDK doesn't support ZMQ or Redis Streams then there must be a MQTT Broker running when a C Device service is in use and configured to publish to MessageBus. Since we've adopted the publish topic scheme with DeviceProfileName and DeviceName the V2 API must restrict the characters used in device names to those allowed in a topic. An issue for V2 API already exists for restricting the allowable characters to RFC 3986 , which will suffice. Newer ZMQ may allow for multiple publishers. Requires investigation and very likely rework of the ZMQ implementation in go-mod-messaging. No alternative has been found . Mark as Push V2 Api will be removed from Core Data, Core Data Client and the App SDK Consider moving App Service Binding to Writable. (out of scope for this ADR)","title":"Device Services Send Events via Message Bus"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#device-services-send-events-via-message-bus","text":"Status Context Decision Which Message Bus implementations? Go Device SDK C Device SDK Core Data and Persistence V2 Event DTO Validation Message Envelope Application Services MessageBus Topics Configuration Device Services [MessageQueue] Core Data [MessageQueue] Application Services [MessageBus] [Binding] Secure Connections Consequences","title":"Device Services Send Events via Message Bus"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#status","text":"Approved","title":"Status"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#context","text":"Currently EdgeX Events are sent from Device Services via HTTP to Core Data, which then puts the Events on the MessageBus after optionally persisting them to the database. This ADR details how Device Services will send EdgeX Events to other services via the EdgeX MessageBus. Note: Though this design is centered on device services, it does have cross cutting impacts with other EdgeX services and modules Note: This ADR is dependent on the Secret Provider for All to provide the secrets for secure Message Bus connections.","title":"Context"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#decision","text":"","title":"Decision"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#which-message-bus-implementations","text":"Multiple Device Services may need to be publishing Events to the MessageBus concurrently. ZMQ will not be a valid option if multiple Device Services are configured to publish. This is because ZMQ only allows for a single publisher. ZMQ will still be valid if only one Device Service is publishing Events. The MQTT and Redis Streams are valid options to use when multiple Device Services are required, as they both support multiple publishers. These are the only other implementations currently available for Go services. The C base device services do not yet have a MessageBus implementation. See the C Device SDK below for details. Note: Documentation will need to be clear when ZMQ can be used and when it can not be used.","title":"Which Message Bus implementations?"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#go-device-sdk","text":"The Go Device SDK will take advantage of the existing go-mod-messaging module to enable use of the EdgeX MessageBus. A new bootstrap handler will be created which initializes the MessageBus client based on configuration. See Configuration section below for details. The Go Device SDK will be enhanced to optionally publish Events to the MessageBus anywhere it currently POSTs Events to Core Data. This publish vs POST option will be controlled by configuration with publish as the default. See Configuration section below for details.","title":"Go Device SDK"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#c-device-sdk","text":"The C Device SDK will implement its own MessageBus abstraction similar to the one in go-mod-messaging . The first implementation type (MQTT or Redis Streams) is TBD. Using this abstraction allows for future implementations to be added when use cases warrant the additional implementations. As with the Go SDK, the C SDK will be enhanced to optionally publish Events to the MessageBus anywhere it currently POSTs Events to Core Data. This publish vs POST option will be controlled by configuration with publish as the default. See Configuration section below for details.","title":"C Device SDK"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#core-data-and-persistence","text":"With this design, Events will be sent directly to Application Services w/o going through Core Data and thus will not be persisted unless changes are made to Core Data. To allow Events to optionally continue to be persisted, Core Data will become an additional or secondary (and optional) subscriber for the Events from the MessageBus. The Events will be persisted when they are received. Core Data will also retain the ability to receive Events via HTTP, persist them and publish them to the MessageBus as is done today. This allows for the flexibility to have some device services to be configured to POST Events and some to be configured to publish Events while we transition the Device Services to all have the capability to publishing Events. In the future, once this new Publish approach has been proven, we may decide to remove POSTing Events to Core Data from the Device SDKs. The existing PersistData setting will be ignored by the code path subscribing to Events since the only reason to do this is to persist the Events. There is a race condition for Marked As Pushed when Core Data is persisting Events received from the MessageBus. Core Data may not have finished persisting an Event before the Application Service has processed the Event and requested the Event be Marked As Pushed . It was decided to remove Mark as Pushed capability and just rely on time based scrubbing of old Events.","title":"Core Data and Persistence"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#v2-event-dto","text":"As this development will be part of the Ireland release all Events published to the MessageBus will use the V2 Event DTO. This is already implemented in Core Data for the V2 AddEvent API.","title":"V2 Event DTO"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#validation","text":"Services receiving the Event DTO from the MessageBus will log validation errors and stop processing the Event.","title":"Validation"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#message-envelope","text":"EdgeX Go Services currently uses a custom Message Envelope for all data that is published to the MessageBus. This envelope wraps the data with metadata, which is ContentType (JSON or CBOR), Correlation-Id and the obsolete Checksum . The Checksum is used when the data is CBOR encoded to identify the Event in V1 API to be mark it as pushed. This checksum is no longer needed as the V2 Event DTO requires the ID be set by the Device Services which will always be used in the V2 API to mark the Events as pushed. The Message Envelope will be updated to remove this property. The C SDK will recreate this Message Envelope.","title":"Message Envelope"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#application-services","text":"As part of the V2 API consumption work in Ireland the App Services SDK will be changed to expect to receive V2 Event DTOs rather than the V1 Event model. It will also be updated to no longer expect or use the Checksum currently on the Message Envelope. Note these changes must occur for the V2 consumption and are not directly tied to this effort. The App Service SDK will be enhanced for the secure MessageBus connection described below. See Secure Connections for details","title":"Application Services"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#messagebus-topics","text":"Note: The change recommended here is not required for this design, but it provides a good opportunity to adopt it. Currently Core Data publishes Events to the simple events topic. All Application Services running receive every Event published, whether they want them or not. The Events can be filtered out using the FilterByDeviceName or FilterByResourceName pipeline functions, but the Application Services still receives every Event and process all the Events to some extent. This could cause load issues in a deployment with many devices and large volume of Events from various devices or a very verbose device that the Application Services is not interested in. Note: The current FilterByDeviceName is only good if the device name is known statically and the only instance of the device defined by the DeviceProfileName . What we really need is FilterByDeviceProfileName which allows multiple instances of a device to be filtered for, rather than a single instance as it it now. The V2 API will be adding DeviceProfileName to the Events, so in Ireland this filter will be possible. Pub/Sub systems have advanced topic schema, which we can take advantage of from Application Services to filter for just the Events the Application Service actual wants. Publishers of Events must add the DeviceProfileName , DeviceName and SourceName to the topic in the form edgex/events/// . The SourceName is the Resource or Command name used to create the Event. This allows Application Services to filter for just the Events from the device(s) it wants by only subscribing to those DeviceProfileNames or the specific DeviceNames or just the specific SourceNames Example subscribe topics if above schema is used: edgex/events/# All Events Core Data will subscribe using this topic schema edgex/events/Random-Integer-Device/# Any Events from devices created from the Random-Integer-Device device profile edgex/events/Random-Integer-Device/Random-Integer-Device1 Only Events from the Random-Integer-Device1 Device edgex/events/Random-Integer-Device/#/Int16 Any Events with Readings from Int16 device resource from devices created from the Random-Integer-Device device profile. **edgex/events/Modbus-Device/#/HVACValues Any Events with Readings from HVACValues device command from devices created from the Modbus-Device device profile. The MessageBus abstraction allows for multiple subscriptions, so an Application Service could specify to receive data from multiple specific device profiles or devices by creating multiple subscriptions. i.e. edgex/Events/Random-Integer-Device/# and edgex/Events/Random-Boolean-Device/# . Currently the App SDK only allows for a single subscription topic to be configured, but that could easily be expanded to handle a list of subscriptions. See Configuration section below for details. Core Data's existing publishing of Events would also need to be changed to use this new topic schema. One challenge with this is Core Data doesn't currently know the DeviceProfileName or DeviceName when it receives a CBOR encoded event. This is because it doesn't decode the Event until after it has published it to the MessageBus. Also, Core Data doesn't know of SourceName at all. The V2 API will be enhanced to change the AddEvent endpoint from /event to /event/{profile}/{device}/{source} so that DeviceProfileName , DeviceName , and SourceName are always know no matter how the request is encoded. This new topic approach will be enabled via each publisher's PublishTopic having the DeviceProfileName , DeviceName and SourceName added to the configured PublishTopicPrefix PublishTopicPrefix = \"edgex/events\" # /// will be added to this Publish Topic prefix See Configuration section below for details.","title":"MessageBus Topics"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#configuration","text":"","title":"Configuration"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#device-services","text":"All Device services will have the following additional configuration to allow connecting and publishing to the MessageBus. As describe above in the MessageBus Topics section, the PublishTopic will include the DeviceProfileName and DeviceName .","title":"Device Services"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#messagequeue","text":"A MessageQueue section will be added, which is similar to that used in Core Data today, but with PublishTopicPrefix instead of Topic .To enable secure connections, the Username & Password have been replaced with ClientAuth & SecretPath , See Secure Connections section below for details. The added Enabled property controls whether the Device Service publishes to the MessageBus or POSTs to Core Data. [MessageQueue] Enabled = true Protocol = \"tcp\" Host = \"localhost\" Port = 1883 Type = \"mqtt\" PublishTopicPrefix = \"edgex/events\" # /// will be added to this Publish Topic prefix [MessageQueue.Optional] # Default MQTT Specific options that need to be here to enable environment variable overrides of them # Client Identifiers ClientId = \"\" # Connection information Qos = \"0\" # Quality of Sevice values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ClientAuth = \"none\" # Valid values are: `none`, `usernamepassword` or `clientcert` Secretpath = \"messagebus\" # Path in secret store used if ClientAuth not `none`","title":"[MessageQueue]"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#core-data","text":"Core data will also require additional configuration to be able to subscribe to receive Events from the MessageBus. As describe above in the MessageBus Topics section, the PublishTopicPrefix will have DeviceProfileName and DeviceName added to create the actual Public Topic.","title":"Core Data"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#messagequeue_1","text":"The MessageQueue section will be changed so that the Topic property changes to PublishTopicPrefix and SubscribeEnabled and SubscribeTopic will be added. As with device services configuration, the Username & Password have been replaced with ClientAuth & SecretPath for secure connections. See Secure Connections section below for details. In addition, the Boolean SubscribeEnabled property will be used to control if the service subscribes to Events from the MessageBus or not. [MessageQueue] Protocol = \"tcp\" Host = \"localhost\" Port = 1883 Type = \"mqtt\" PublishTopicPrefix = \"edgex/events\" # /// will be added to this Publish Topic prefix SubscribeEnabled = true SubscribeTopic = \"edgex/events/#\" [MessageQueue.Optional] # Default MQTT Specific options that need to be here to enable evnironment variable overrides of them # Client Identifiers ClientId = \"edgex-core-data\" # Connection information Qos = \"0\" # Quality of Sevice values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ClientAuth = \"none\" # Valid values are: `none`, `usernamepassword` or `clientcert` Secretpath = \"messagebus\" # Path in secret store used if ClientAuth not `none`","title":"[MessageQueue]"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#application-services_1","text":"","title":"Application Services"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#messagebus","text":"Similar to above, the Application Services MessageBus configuration will change to allow for secure connection to the MessageBus. The Username & Password have been replaced with ClientAuth & SecretPath for secure connections. See Secure Connections section below for details. [MessageBus.Optional] # MQTT Specific options # Client Identifiers ClientId = \"\" # Connection information Qos = \"0\" # Quality of Sevice values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"5\" # Seconds SkipCertVerify = \"false\" # Only used if Cert/Key file or Cert/Key PEMblock are specified ClientAuth = \"none\" # Valid values are: `none`, `usernamepassword` or `clientcert` Secretpath = \"messagebus\" # Path in secret store used if ClientAuth not `none`","title":"[MessageBus]"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#binding","text":"The Binding configuration section will require changes for the subscribe topics scheme described in the MessageBus Topics section above to filter for Events from specific device profiles or devices. SubscribeTopic will change from a string property containing a single topic to the SubscribeTopics string property containing a comma separated list of topics. This allows for the flexibility for the property to be a single topic with the # wild card so the Application Service receives all Events as it does today. Receive only Events from the Random-Integer-Device and Random-Boolean-Device profiles [Binding] Type = \"messagebus\" SubscribeTopics = \"edgex/events/Random-Integer-Device, edgex/events/Random-Boolean-Device\" Receive only Events from the Random-Integer-Device1 from the Random-Integer-Device profile [Binding] Type = \"messagebus\" SubscribeTopics = \"edgex/events/Random-Integer-Device/Random-Integer-Device1\" or receives all Events: [Binding] Type = \"messagebus\" SubscribeTopics = \"edgex/events/#\"","title":"[Binding]"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#secure-connections","text":"As stated earlier, this ADR is dependent on the Secret Provider for All ADR to provide a common Secret Provider for all Edgex Services to access their secrets. Once this is available, the MessageBus connection can be secured via the following configurable client authentications modes which follows similar implementation for secure MQTT Export and secure MQTT Trigger used in Application Services. none - No authentication usernamepassword - Username & password authentication. clientcert - Client certificate and key for authentication. The secrets specified for the above options are pulled from the Secret Provider using the configured SecretPath . How the secrets are injected into the Secret Provider is out of scope for this ADR and covered in the Secret Provider for All ADR.","title":"Secure Connections"},{"location":"design/adr/013-Device-Service-Events-Message-Bus/#consequences","text":"If C SDK doesn't support ZMQ or Redis Streams then there must be a MQTT Broker running when a C Device service is in use and configured to publish to MessageBus. Since we've adopted the publish topic scheme with DeviceProfileName and DeviceName the V2 API must restrict the characters used in device names to those allowed in a topic. An issue for V2 API already exists for restricting the allowable characters to RFC 3986 , which will suffice. Newer ZMQ may allow for multiple publishers. Requires investigation and very likely rework of the ZMQ implementation in go-mod-messaging. No alternative has been found . Mark as Push V2 Api will be removed from Core Data, Core Data Client and the App SDK Consider moving App Service Binding to Writable. (out of scope for this ADR)","title":"Consequences"},{"location":"design/adr/014-Secret-Provider-For-All/","text":"Secret Provider for All Status Context Existing Implementations What is a Secret? Service Exclusive vs Service Shared Secrets Known and Unknown Services Static Secrets and Runtime Secrets Interfaces and factory methods Bootstrap's current implementation Interfaces Factory and bootstrap handler methods App SDK's current implementation Interface Factory and bootstrap handler methods Secret Store for non-secure mode InsecureSecrets Configuration Decision Only Exclusive Secret Stores Abstraction Interface Implementation Factory Method and Bootstrap Handler Caching of Secrets Insecure Secrets Handling on-the-fly changes to InsecureSecrets Mocks Where will SecretProvider reside? Go Services C Device Service Consequences Status Approved Context This ADR defines the new SecretProvider abstraction that will be used by all EdgeX services, including Device Services. The Secret Provider is used by services to retrieve secrets from the Secret Store. The Secret Store, in secure mode, is currently Vault. In non-secure mode it is configuration in some form, i.e. DatabaseInfo configuration or InsecureSecrets configuration for Application Services. Existing Implementations The Secret Provider abstraction defined in this ADR is based on the Secret Provider abstraction implementations in the Application Functions SDK (App SDK) for Application Services and the one in go-mod-bootstrap (Bootstrap) used by the Core, Support & Security services in edgex-go. Device Services do not currently use secure secrets. The App SDK implementation was initially based on the Bootstrap implementation. The similarities and differences between these implementations are: Both wrap the SecretClient from go-mod-secrets Both initialize the SecretClient based on the SecretStore configuration(s) Both have factory methods, but they differ greatly Both implement the GetDatabaseCredentials API Bootstrap's uses split interfaces definitions ( CredentialsProvider & CertificateProvider ) while the App SDK's use a single interface ( SecretProvider ) for the abstraction Bootstrap's includes the bootstrap handler while the App SDK's has the bootstrap handler separated out Bootstrap's implements the GetCertificateKeyPair API, which the App SDK's does not App SDK's implements the following, which the Bootstrap's does not Initialize API (Bootstrap's initialization is done by the bootstrap handler) StoreSecrets API GetSecrets API InsecureSecretsUpdated API SecretsLastUpdated API Wraps a second SecretClient for the Application Service instance's exclusive secrets. Used by the StoreSecrets & GetSecrets APIs The standard SecretClient is considered the shared client for secrets that all Application Service instances share. It is only used by the GetDatabaseCredentials API Configuration based secret store for non-secure mode called InsecureSecrets Caching of secrets Needed so that secrets used by pipeline functions do not cause call out to Vault for every Event processed What is a Secret? A secret is a collection of key/value pairs stored in a SecretStore at specified path whose values are sensitive in nature. Redis database credentials are an example of a Secret which contains the username and password key/values stored at the redisdb path. Service Exclusive vs Service Shared Secrets Service Exclusive secrets are those that are exclusive to the instance of the running service. An example of exclusive secrets are the HTTP Auth tokens used by two running instances of app-service-configurable (http-export) which export different device Events to different endpoints with different Auth tokens in the HTTP headers. Service Exclusive secrets are seeded by POSTing the secrets to the /api/vX/secrets endpoint on the running instance of each Application Service. Service Shared secrets are those that all instances of a class of service, such a Application Services, share. Think of Core Data as it own class of service. An example of shared secrets are the database credentials for the single database instance for Store and Forward data that all Application Services may need to access. Another example is the database credentials for each of instance the Core Data. It is shared, but only one instance of Core Data is currently ever run. Service Shared secrets are seeded by security-secretstore-setup using static configuration for static secrets for known services. Currently database credentials are the only shared secrets. In the future we may have Message Bus credentials as shared secrets, but these will be truly shared secrets for all services to securely connect to the Message Bus, not just shared between instances of a service. Application Services currently have the ability to configure SecretStores for Service Exclusive and/or Service Shared secrets depending on their needs. Known and Unknown Services Known Services are those identified in the static configuration by security-secretstore-setup These currently are Core Data, Core Metadata, Support Notifications, Support Scheduler and Application Service (class) Unknown Services are those not known in the static configuration that become known when added to the Docker compose file or Snap. Application Service (instance) are examples of these services. Service exclusive SecretStore can be created for these services by adding the services' unique name , i.e. appservice-http-export, to the ADD_SECRETSTORE_TOKENS environment variable for security-secretstore-setup ADD_SECRETSTORE_TOKENS: \"appservice-http-export, appservice-mqtt-export\" This creates an exclusive secret store token for each service listed. The name provided for each service must be used in the service's SecretStore configuration and Docker volume mount (if applicable). Typically the configuration is set via environment overrides or is already in an existing configuration profile ( http-export profile for app-service-configurable). Example docker-compose file entries: environment : ... SecretStoreExclusive_Path : \"/v1/secret/edgex/appservice-http-export/\" TokenFile : \"/tmp/edgex/secrets/appservice-http-export/secrets-token.json\" volumes : ... - /tmp/edgex/secrets/appservice-http-export:/tmp/edgex/secrets/appservice-http-export:ro,z Static Secrets and Runtime Secrets Static Secrets are those identified by name in the static configuration whose values are randomly generated at seed time. These secrets are seeded on start-up of EdgeX. Database credentials are currently the only secrets of this type Runtime Secrets are those not known in the static configuration and that become known during run time. These secrets are seeded at run time via the Application Services /api/vX/secrets endpoint HTTP header authorization credentials for HTTP Export are types of these secrets Interfaces and factory methods Bootstrap's current implementation Interfaces type CredentialsProvider interface { GetDatabaseCredentials ( database config . Database ) ( config . Credentials , error ) } and type CertificateProvider interface { GetCertificateKeyPair ( path string ) ( config . CertKeyPair , error ) } Factory and bootstrap handler methods type SecretProvider struct { secretClient pkg . SecretClient } func NewSecret () * SecretProvider { return & SecretProvider {} } func ( s * SecretProvider ) BootstrapHandler ( ctx context . Context , _ * sync . WaitGroup , startupTimer startup . Timer , dic * di . Container ) bool { ... Intializes the SecretClient and adds it to the DIC for both interfaces . ... } App SDK's current implementation Interface type SecretProvider interface { Initialize ( _ context . Context ) bool StoreSecrets ( path string , secrets map [ string ] string ) error GetSecrets ( path string , _ ... string ) ( map [ string ] string , error ) GetDatabaseCredentials ( database db . DatabaseInfo ) ( common . Credentials , error ) InsecureSecretsUpdated () SecretsLastUpdated () time . Time } Factory and bootstrap handler methods type SecretProviderImpl struct { SharedSecretClient pkg . SecretClient ExclusiveSecretClient pkg . SecretClient secretsCache map [ string ] map [ string ] string // secret's path, key, value configuration * common . ConfigurationStruct cacheMuxtex * sync . Mutex loggingClient logger . LoggingClient //used to track when secrets have last been retrieved LastUpdated time . Time } func NewSecretProvider ( loggingClient logger . LoggingClient , configuration * common . ConfigurationStruct ) * SecretProviderImpl { sp := & SecretProviderImpl { secretsCache : make ( map [ string ] map [ string ] string ), cacheMuxtex : & sync . Mutex {}, configuration : configuration , loggingClient : loggingClient , LastUpdated : time . Now (), } return sp } type Secrets struct { } func NewSecrets () * Secrets { return & Secrets {} } func ( _ * Secrets ) BootstrapHandler ( ctx context . Context , _ * sync . WaitGroup , startupTimer startup . Timer , dic * di . Container ) bool { ... Creates NewNewSecretProvider , calls Initailizes () and adds it to the DIC ... } Secret Store for non-secure mode Both Bootstrap's and App SDK's implementation use the DatabaseInfo configuration for GetDatabaseCredentials API in non-secure mode. The App SDK only uses it, for backward compatibility, if the database credentials are not found in the new InsecureSecrets configuration section. For Ireland it was planned to only use the new InsecureSecrets configuration section in non-secure mode. Note: Redis credentials are blank in non-secure mode Core Data [Databases] [Databases.Primary] Host = \"localhost\" Name = \"coredata\" Username = \"\" Password = \"\" Port = 6379 Timeout = 5000 Type = \"redisdb\" Application Services [Database] Type = \"redisdb\" Host = \"localhost\" Port = 6379 Username = \"\" Password = \"\" Timeout = \"30s\" InsecureSecrets Configuration The App SDK defines a new Writable configuration section called InsecureSecrets . This structure mimics that of the secure SecretStore when EDGEX_SECURITY_SECRET_STORE environment variable is set to false . Having the InsecureSecrets in the Writable section allows for the secrets to be updated without restarting the service. Some minor processing must occur when the InsecureSecrets section is updated. This is to call the InsecureSecretsUpdated API. This API simply sets the time the secrets were last updated. The SecretsLastUpdated API returns this timestamp so pipeline functions that use credentials for exporting know if their client needs to be recreated with new credentials, i.e MQTT export. type WritableInfo struct { LogLevel string ... InsecureSecrets InsecureSecrets } type InsecureSecrets map [ string ] InsecureSecretsInfo type InsecureSecretsInfo struct { Path string Secrets map [ string ] string } [Writable.InsecureSecrets] [Writable.InsecureSecrets.DB] path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\" [Writable.InsecureSecrets.mqtt] path = \"mqtt\" [Writable.InsecureSecrets.mqtt.Secrets] username = \"\" password = \"\" cacert = \"\" clientcert = \"\" clientkey = \"\" Decision The new SecretProvider abstraction defined by this ADR is a combination of the two implementations described above in the Existing Implementations section. Only Exclusive Secret Stores To simplify the SecretProvider abstraction, we need to reduce to using only exclusive SecretStores . This allows all the APIs to deal with a single SecretClient , rather than the split up way we currently have in Application Services. This requires that the current Application Service shared secrets (database credentials) must be copied into each Application Service's exclusive SecretStore when it is created. The challenge is how do we seed static secrets for unknown services when they become known. As described above in the Known and Unknown Services section above, services currently identify themselves for exclusive SecretStore creation via the ADD_SECRETSTORE_TOKENS environment variable on security-secretstore-setup. This environment variable simply takes a comma separated list of service names. ADD_SECRETSTORE_TOKENS : \",\" If we expanded this to add an optional list of static secret identifiers for each service, i.e. appservice/redisdb , the exclusive store could also be seeded with a copy of static shared secrets. In this case the Redis database credentials for the Application Services' shared database. The environment variable name will change to ADD_SECRETSTORE now that it is more than just tokens. ADD_SECRETSTORE : \"app-service-xyz[appservice/redisdb]\" Note: The secret identifier here is the short path to the secret in the existing appservice SecretStore . In the above example this expands to the full path of /secret/edgex/appservice/redisdb The above example results in the Redis credentials being copied into app-service-xyz's SecretStore at /secret/edgex/app-service-xyz/redis . Similar approach could be taken for Message Bus credentials where a common SecretStore is created with the Message Bus credentials saved. The services request the credentials are copied into their exclusive SecretStore using common/messagebus as the secret identifier. Full specification for the environment variable's value is a comma separated list of service entries defined as: [optional list of static secret IDs sperated by ;],[optional list of static secret IDs sperated by ;],... Example with one service specifying IDs for static secrets and one without static secrets ADD_SECRETSTORE : \"appservice-xyz[appservice/redisdb; common/messagebus], appservice-http-export\" When the ADD_SECRETSTORE environment variable is processed to create these SecretStores , it will copy the specified saved secrets from the initial SecretStore into the service's SecretStore . This all depends on the completion of database or other credential bootstrapping and the secrets having been stored prior to the environment variable being processed. security-secretstore-setup will need to be refactored to ensure this sequencing. Abstraction Interface The following will be the new SecretProvider abstraction interface used by all Edgex services type SecretProvider interface { // Stores new secrets into the service's exclusive SecretStore at the specified path. StoreSecrets ( path string , secrets map [ string ] string ) error // Retrieves secrets from the service's exclusive SecretStore at the specified path. GetSecrets ( path string , _ ... string ) ( map [ string ] string , error ) // Sets the secrets lastupdated time to current time. SecretsUpdated () // Returns the secrets last updated time SecretsLastUpdated () time . Time } Note: The GetDatabaseCredentials and GetCertificateKeyPair APIs have been removed. These are no longer needed since insecure database credentials will no longer be stored in the DatabaseInfo configuration and certificate key pairs are secrets like any others. This allows these secrets to be retrieved via the GetSecrets API. Implementation Factory Method and Bootstrap Handler The factory method and bootstrap handler will follow that currently in the Bootstrap implementation with some tweaks. Rather than putting the two split interfaces into the DIC, it will put just the single interface instance into the DIC. See details in the Interfaces and factory methods section above under Existing Implementations . Caching of Secrets Secrets will be cached as they are currently in the Application Service implementation Insecure Secrets Insecure Secrets will be handled as they are currently in the Application Service implementation. DatabaseInfo configuration will no longer be an option for storing the insecure database credentials. They will be stored in the InsecureSecrets configuration only. [Writable.InsecureSecrets] [Writable.InsecureSecrets.DB] path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\" Handling on-the-fly changes to InsecureSecrets All services will need to handle the special processing when InsecureSecrets are changed on-the-fly via Consul. Since this will now be a common configuration item in Writable it can be handled in go-mod-bootstrap along with existing log level processing. This special processing will be taken from App SDK. Mocks Proper mock of the SecretProvider interface will be created with Mockery to be used in unit tests. Current mock in App SDK is hand written rather then generated with Mockery . Where will SecretProvider reside? Go Services The final decision to make is where will this new SecretProvider abstraction reside? Originally is was assumed that it would reside in go-mod-secrets , which seems logical. If we were to attempt this with the implementation including the bootstrap handler, go-mod-secrets would have a dependency on go-mod-bootstrap which will likely create a circular dependency. Refactoring the existing implementation in go-mod-bootstrap and have it reside there now seems to be the best choice. C Device Service The C Device SDK will implement the same SecretProvider abstraction, InsecureSercets configuration and the underling SecretStore client. Consequences All service's will have Writable.InsecureSecrets section added to their configuration InsecureSecrets definition will be moved from App SDK to go-mod-bootstrap Go Device SDK will add the SecretProvider to it's bootstrapping C Device SDK implementation could be big lift? SecretStore configuration section will be added to all Device Services edgex-go services will be modified to use the single SecretProvider interface from the DIC in place of current usage of the GetDatabaseCredentials and GetCertificateKeyPair interfaces. Calls to GetDatabaseCredentials and GetCertificateKeyPair will be replaced with calls to GetSecrets API and appropriate processing of the returned secrets will be added. App SDK will be modified to use GetSecrets API in place of the GetDatabaseCredentials API App SDK will be modified to use the new SecretProvider bootstrap handler app-service-configurable's configuration profiles as well as all the Application Service examples configurations will be updated to remove the SecretStoreExclusive configuration and just use the existing SecretStore configuration security-secretstore-setup will be enhanced as described in the Only Exclusive Secret Stores section above Adding new services that need static secrets added to their SecretStore requires stopping and restarting all the services. The is because security-secretstore-setup has completed but not stopped. If it is rerun without stopping the other services, there tokens and static secrets will have changed. The planned refactor of security-secretstore-setup will attempt to resolve this. Snaps do not yet support setting the environment variable for adding SecretStore. It is planned for Ireland release.","title":"Secret Provider for All"},{"location":"design/adr/014-Secret-Provider-For-All/#secret-provider-for-all","text":"Status Context Existing Implementations What is a Secret? Service Exclusive vs Service Shared Secrets Known and Unknown Services Static Secrets and Runtime Secrets Interfaces and factory methods Bootstrap's current implementation Interfaces Factory and bootstrap handler methods App SDK's current implementation Interface Factory and bootstrap handler methods Secret Store for non-secure mode InsecureSecrets Configuration Decision Only Exclusive Secret Stores Abstraction Interface Implementation Factory Method and Bootstrap Handler Caching of Secrets Insecure Secrets Handling on-the-fly changes to InsecureSecrets Mocks Where will SecretProvider reside? Go Services C Device Service Consequences","title":"Secret Provider for All"},{"location":"design/adr/014-Secret-Provider-For-All/#status","text":"Approved","title":"Status"},{"location":"design/adr/014-Secret-Provider-For-All/#context","text":"This ADR defines the new SecretProvider abstraction that will be used by all EdgeX services, including Device Services. The Secret Provider is used by services to retrieve secrets from the Secret Store. The Secret Store, in secure mode, is currently Vault. In non-secure mode it is configuration in some form, i.e. DatabaseInfo configuration or InsecureSecrets configuration for Application Services.","title":"Context"},{"location":"design/adr/014-Secret-Provider-For-All/#existing-implementations","text":"The Secret Provider abstraction defined in this ADR is based on the Secret Provider abstraction implementations in the Application Functions SDK (App SDK) for Application Services and the one in go-mod-bootstrap (Bootstrap) used by the Core, Support & Security services in edgex-go. Device Services do not currently use secure secrets. The App SDK implementation was initially based on the Bootstrap implementation. The similarities and differences between these implementations are: Both wrap the SecretClient from go-mod-secrets Both initialize the SecretClient based on the SecretStore configuration(s) Both have factory methods, but they differ greatly Both implement the GetDatabaseCredentials API Bootstrap's uses split interfaces definitions ( CredentialsProvider & CertificateProvider ) while the App SDK's use a single interface ( SecretProvider ) for the abstraction Bootstrap's includes the bootstrap handler while the App SDK's has the bootstrap handler separated out Bootstrap's implements the GetCertificateKeyPair API, which the App SDK's does not App SDK's implements the following, which the Bootstrap's does not Initialize API (Bootstrap's initialization is done by the bootstrap handler) StoreSecrets API GetSecrets API InsecureSecretsUpdated API SecretsLastUpdated API Wraps a second SecretClient for the Application Service instance's exclusive secrets. Used by the StoreSecrets & GetSecrets APIs The standard SecretClient is considered the shared client for secrets that all Application Service instances share. It is only used by the GetDatabaseCredentials API Configuration based secret store for non-secure mode called InsecureSecrets Caching of secrets Needed so that secrets used by pipeline functions do not cause call out to Vault for every Event processed","title":"Existing Implementations"},{"location":"design/adr/014-Secret-Provider-For-All/#what-is-a-secret","text":"A secret is a collection of key/value pairs stored in a SecretStore at specified path whose values are sensitive in nature. Redis database credentials are an example of a Secret which contains the username and password key/values stored at the redisdb path.","title":"What is a Secret?"},{"location":"design/adr/014-Secret-Provider-For-All/#service-exclusive-vs-service-shared-secrets","text":"Service Exclusive secrets are those that are exclusive to the instance of the running service. An example of exclusive secrets are the HTTP Auth tokens used by two running instances of app-service-configurable (http-export) which export different device Events to different endpoints with different Auth tokens in the HTTP headers. Service Exclusive secrets are seeded by POSTing the secrets to the /api/vX/secrets endpoint on the running instance of each Application Service. Service Shared secrets are those that all instances of a class of service, such a Application Services, share. Think of Core Data as it own class of service. An example of shared secrets are the database credentials for the single database instance for Store and Forward data that all Application Services may need to access. Another example is the database credentials for each of instance the Core Data. It is shared, but only one instance of Core Data is currently ever run. Service Shared secrets are seeded by security-secretstore-setup using static configuration for static secrets for known services. Currently database credentials are the only shared secrets. In the future we may have Message Bus credentials as shared secrets, but these will be truly shared secrets for all services to securely connect to the Message Bus, not just shared between instances of a service. Application Services currently have the ability to configure SecretStores for Service Exclusive and/or Service Shared secrets depending on their needs.","title":"Service Exclusive vs Service Shared Secrets"},{"location":"design/adr/014-Secret-Provider-For-All/#known-and-unknown-services","text":"Known Services are those identified in the static configuration by security-secretstore-setup These currently are Core Data, Core Metadata, Support Notifications, Support Scheduler and Application Service (class) Unknown Services are those not known in the static configuration that become known when added to the Docker compose file or Snap. Application Service (instance) are examples of these services. Service exclusive SecretStore can be created for these services by adding the services' unique name , i.e. appservice-http-export, to the ADD_SECRETSTORE_TOKENS environment variable for security-secretstore-setup ADD_SECRETSTORE_TOKENS: \"appservice-http-export, appservice-mqtt-export\" This creates an exclusive secret store token for each service listed. The name provided for each service must be used in the service's SecretStore configuration and Docker volume mount (if applicable). Typically the configuration is set via environment overrides or is already in an existing configuration profile ( http-export profile for app-service-configurable). Example docker-compose file entries: environment : ... SecretStoreExclusive_Path : \"/v1/secret/edgex/appservice-http-export/\" TokenFile : \"/tmp/edgex/secrets/appservice-http-export/secrets-token.json\" volumes : ... - /tmp/edgex/secrets/appservice-http-export:/tmp/edgex/secrets/appservice-http-export:ro,z","title":"Known and Unknown Services"},{"location":"design/adr/014-Secret-Provider-For-All/#static-secrets-and-runtime-secrets","text":"Static Secrets are those identified by name in the static configuration whose values are randomly generated at seed time. These secrets are seeded on start-up of EdgeX. Database credentials are currently the only secrets of this type Runtime Secrets are those not known in the static configuration and that become known during run time. These secrets are seeded at run time via the Application Services /api/vX/secrets endpoint HTTP header authorization credentials for HTTP Export are types of these secrets","title":"Static Secrets and Runtime Secrets"},{"location":"design/adr/014-Secret-Provider-For-All/#interfaces-and-factory-methods","text":"","title":"Interfaces and factory methods"},{"location":"design/adr/014-Secret-Provider-For-All/#bootstraps-current-implementation","text":"","title":"Bootstrap's current implementation"},{"location":"design/adr/014-Secret-Provider-For-All/#interfaces","text":"type CredentialsProvider interface { GetDatabaseCredentials ( database config . Database ) ( config . Credentials , error ) } and type CertificateProvider interface { GetCertificateKeyPair ( path string ) ( config . CertKeyPair , error ) }","title":"Interfaces"},{"location":"design/adr/014-Secret-Provider-For-All/#factory-and-bootstrap-handler-methods","text":"type SecretProvider struct { secretClient pkg . SecretClient } func NewSecret () * SecretProvider { return & SecretProvider {} } func ( s * SecretProvider ) BootstrapHandler ( ctx context . Context , _ * sync . WaitGroup , startupTimer startup . Timer , dic * di . Container ) bool { ... Intializes the SecretClient and adds it to the DIC for both interfaces . ... }","title":"Factory and bootstrap handler methods"},{"location":"design/adr/014-Secret-Provider-For-All/#app-sdks-current-implementation","text":"","title":"App SDK's current implementation"},{"location":"design/adr/014-Secret-Provider-For-All/#interface","text":"type SecretProvider interface { Initialize ( _ context . Context ) bool StoreSecrets ( path string , secrets map [ string ] string ) error GetSecrets ( path string , _ ... string ) ( map [ string ] string , error ) GetDatabaseCredentials ( database db . DatabaseInfo ) ( common . Credentials , error ) InsecureSecretsUpdated () SecretsLastUpdated () time . Time }","title":"Interface"},{"location":"design/adr/014-Secret-Provider-For-All/#factory-and-bootstrap-handler-methods_1","text":"type SecretProviderImpl struct { SharedSecretClient pkg . SecretClient ExclusiveSecretClient pkg . SecretClient secretsCache map [ string ] map [ string ] string // secret's path, key, value configuration * common . ConfigurationStruct cacheMuxtex * sync . Mutex loggingClient logger . LoggingClient //used to track when secrets have last been retrieved LastUpdated time . Time } func NewSecretProvider ( loggingClient logger . LoggingClient , configuration * common . ConfigurationStruct ) * SecretProviderImpl { sp := & SecretProviderImpl { secretsCache : make ( map [ string ] map [ string ] string ), cacheMuxtex : & sync . Mutex {}, configuration : configuration , loggingClient : loggingClient , LastUpdated : time . Now (), } return sp } type Secrets struct { } func NewSecrets () * Secrets { return & Secrets {} } func ( _ * Secrets ) BootstrapHandler ( ctx context . Context , _ * sync . WaitGroup , startupTimer startup . Timer , dic * di . Container ) bool { ... Creates NewNewSecretProvider , calls Initailizes () and adds it to the DIC ... }","title":"Factory and bootstrap handler methods"},{"location":"design/adr/014-Secret-Provider-For-All/#secret-store-for-non-secure-mode","text":"Both Bootstrap's and App SDK's implementation use the DatabaseInfo configuration for GetDatabaseCredentials API in non-secure mode. The App SDK only uses it, for backward compatibility, if the database credentials are not found in the new InsecureSecrets configuration section. For Ireland it was planned to only use the new InsecureSecrets configuration section in non-secure mode. Note: Redis credentials are blank in non-secure mode Core Data [Databases] [Databases.Primary] Host = \"localhost\" Name = \"coredata\" Username = \"\" Password = \"\" Port = 6379 Timeout = 5000 Type = \"redisdb\" Application Services [Database] Type = \"redisdb\" Host = \"localhost\" Port = 6379 Username = \"\" Password = \"\" Timeout = \"30s\"","title":"Secret Store for non-secure mode"},{"location":"design/adr/014-Secret-Provider-For-All/#insecuresecrets-configuration","text":"The App SDK defines a new Writable configuration section called InsecureSecrets . This structure mimics that of the secure SecretStore when EDGEX_SECURITY_SECRET_STORE environment variable is set to false . Having the InsecureSecrets in the Writable section allows for the secrets to be updated without restarting the service. Some minor processing must occur when the InsecureSecrets section is updated. This is to call the InsecureSecretsUpdated API. This API simply sets the time the secrets were last updated. The SecretsLastUpdated API returns this timestamp so pipeline functions that use credentials for exporting know if their client needs to be recreated with new credentials, i.e MQTT export. type WritableInfo struct { LogLevel string ... InsecureSecrets InsecureSecrets } type InsecureSecrets map [ string ] InsecureSecretsInfo type InsecureSecretsInfo struct { Path string Secrets map [ string ] string } [Writable.InsecureSecrets] [Writable.InsecureSecrets.DB] path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\" [Writable.InsecureSecrets.mqtt] path = \"mqtt\" [Writable.InsecureSecrets.mqtt.Secrets] username = \"\" password = \"\" cacert = \"\" clientcert = \"\" clientkey = \"\"","title":"InsecureSecrets Configuration"},{"location":"design/adr/014-Secret-Provider-For-All/#decision","text":"The new SecretProvider abstraction defined by this ADR is a combination of the two implementations described above in the Existing Implementations section.","title":"Decision"},{"location":"design/adr/014-Secret-Provider-For-All/#only-exclusive-secret-stores","text":"To simplify the SecretProvider abstraction, we need to reduce to using only exclusive SecretStores . This allows all the APIs to deal with a single SecretClient , rather than the split up way we currently have in Application Services. This requires that the current Application Service shared secrets (database credentials) must be copied into each Application Service's exclusive SecretStore when it is created. The challenge is how do we seed static secrets for unknown services when they become known. As described above in the Known and Unknown Services section above, services currently identify themselves for exclusive SecretStore creation via the ADD_SECRETSTORE_TOKENS environment variable on security-secretstore-setup. This environment variable simply takes a comma separated list of service names. ADD_SECRETSTORE_TOKENS : \",\" If we expanded this to add an optional list of static secret identifiers for each service, i.e. appservice/redisdb , the exclusive store could also be seeded with a copy of static shared secrets. In this case the Redis database credentials for the Application Services' shared database. The environment variable name will change to ADD_SECRETSTORE now that it is more than just tokens. ADD_SECRETSTORE : \"app-service-xyz[appservice/redisdb]\" Note: The secret identifier here is the short path to the secret in the existing appservice SecretStore . In the above example this expands to the full path of /secret/edgex/appservice/redisdb The above example results in the Redis credentials being copied into app-service-xyz's SecretStore at /secret/edgex/app-service-xyz/redis . Similar approach could be taken for Message Bus credentials where a common SecretStore is created with the Message Bus credentials saved. The services request the credentials are copied into their exclusive SecretStore using common/messagebus as the secret identifier. Full specification for the environment variable's value is a comma separated list of service entries defined as: [optional list of static secret IDs sperated by ;],[optional list of static secret IDs sperated by ;],... Example with one service specifying IDs for static secrets and one without static secrets ADD_SECRETSTORE : \"appservice-xyz[appservice/redisdb; common/messagebus], appservice-http-export\" When the ADD_SECRETSTORE environment variable is processed to create these SecretStores , it will copy the specified saved secrets from the initial SecretStore into the service's SecretStore . This all depends on the completion of database or other credential bootstrapping and the secrets having been stored prior to the environment variable being processed. security-secretstore-setup will need to be refactored to ensure this sequencing.","title":"Only Exclusive Secret Stores"},{"location":"design/adr/014-Secret-Provider-For-All/#abstraction-interface","text":"The following will be the new SecretProvider abstraction interface used by all Edgex services type SecretProvider interface { // Stores new secrets into the service's exclusive SecretStore at the specified path. StoreSecrets ( path string , secrets map [ string ] string ) error // Retrieves secrets from the service's exclusive SecretStore at the specified path. GetSecrets ( path string , _ ... string ) ( map [ string ] string , error ) // Sets the secrets lastupdated time to current time. SecretsUpdated () // Returns the secrets last updated time SecretsLastUpdated () time . Time } Note: The GetDatabaseCredentials and GetCertificateKeyPair APIs have been removed. These are no longer needed since insecure database credentials will no longer be stored in the DatabaseInfo configuration and certificate key pairs are secrets like any others. This allows these secrets to be retrieved via the GetSecrets API.","title":"Abstraction Interface"},{"location":"design/adr/014-Secret-Provider-For-All/#implementation","text":"","title":"Implementation"},{"location":"design/adr/014-Secret-Provider-For-All/#factory-method-and-bootstrap-handler","text":"The factory method and bootstrap handler will follow that currently in the Bootstrap implementation with some tweaks. Rather than putting the two split interfaces into the DIC, it will put just the single interface instance into the DIC. See details in the Interfaces and factory methods section above under Existing Implementations .","title":"Factory Method and Bootstrap Handler"},{"location":"design/adr/014-Secret-Provider-For-All/#caching-of-secrets","text":"Secrets will be cached as they are currently in the Application Service implementation","title":"Caching of Secrets"},{"location":"design/adr/014-Secret-Provider-For-All/#insecure-secrets","text":"Insecure Secrets will be handled as they are currently in the Application Service implementation. DatabaseInfo configuration will no longer be an option for storing the insecure database credentials. They will be stored in the InsecureSecrets configuration only. [Writable.InsecureSecrets] [Writable.InsecureSecrets.DB] path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\"","title":"Insecure Secrets"},{"location":"design/adr/014-Secret-Provider-For-All/#handling-on-the-fly-changes-to-insecuresecrets","text":"All services will need to handle the special processing when InsecureSecrets are changed on-the-fly via Consul. Since this will now be a common configuration item in Writable it can be handled in go-mod-bootstrap along with existing log level processing. This special processing will be taken from App SDK.","title":"Handling on-the-fly changes to InsecureSecrets"},{"location":"design/adr/014-Secret-Provider-For-All/#mocks","text":"Proper mock of the SecretProvider interface will be created with Mockery to be used in unit tests. Current mock in App SDK is hand written rather then generated with Mockery .","title":"Mocks"},{"location":"design/adr/014-Secret-Provider-For-All/#where-will-secretprovider-reside","text":"","title":"Where will SecretProvider reside?"},{"location":"design/adr/014-Secret-Provider-For-All/#go-services","text":"The final decision to make is where will this new SecretProvider abstraction reside? Originally is was assumed that it would reside in go-mod-secrets , which seems logical. If we were to attempt this with the implementation including the bootstrap handler, go-mod-secrets would have a dependency on go-mod-bootstrap which will likely create a circular dependency. Refactoring the existing implementation in go-mod-bootstrap and have it reside there now seems to be the best choice.","title":"Go Services"},{"location":"design/adr/014-Secret-Provider-For-All/#c-device-service","text":"The C Device SDK will implement the same SecretProvider abstraction, InsecureSercets configuration and the underling SecretStore client.","title":"C Device Service"},{"location":"design/adr/014-Secret-Provider-For-All/#consequences","text":"All service's will have Writable.InsecureSecrets section added to their configuration InsecureSecrets definition will be moved from App SDK to go-mod-bootstrap Go Device SDK will add the SecretProvider to it's bootstrapping C Device SDK implementation could be big lift? SecretStore configuration section will be added to all Device Services edgex-go services will be modified to use the single SecretProvider interface from the DIC in place of current usage of the GetDatabaseCredentials and GetCertificateKeyPair interfaces. Calls to GetDatabaseCredentials and GetCertificateKeyPair will be replaced with calls to GetSecrets API and appropriate processing of the returned secrets will be added. App SDK will be modified to use GetSecrets API in place of the GetDatabaseCredentials API App SDK will be modified to use the new SecretProvider bootstrap handler app-service-configurable's configuration profiles as well as all the Application Service examples configurations will be updated to remove the SecretStoreExclusive configuration and just use the existing SecretStore configuration security-secretstore-setup will be enhanced as described in the Only Exclusive Secret Stores section above Adding new services that need static secrets added to their SecretStore requires stopping and restarting all the services. The is because security-secretstore-setup has completed but not stopped. If it is rerun without stopping the other services, there tokens and static secrets will have changed. The planned refactor of security-secretstore-setup will attempt to resolve this. Snaps do not yet support setting the environment variable for adding SecretStore. It is planned for Ireland release.","title":"Consequences"},{"location":"design/adr/core/0003-V2-API-Principles/","text":"Geneva API Guiding Principles Status Accepted by EdgeX Foundry working groups as of Core Working Group meeting 16-Jan-2020 Note This ADR was written pre-Geneva with an assumption that the V2 APIs would be available in Geneva. In actuality, the full V2 APIs will be delivered in the Ireland release (Spring 2020) Context A redesign of the EdgeX Foundry API is proposed for the Geneva release. This is understood by the community to warrant a 2.0 release that will not be backward compatible. The goal is to rework the API using solid principles that will allow for extension over the course of several release cycles, avoiding the necessity of yet another major release version in a short period of time. Briefly, this effort grew from the acknowledgement that the current models used to facilitate requests and responses via the EdgeX Foundry API were legacy definitions that were once used as internal representations of state within the EdgeX services themselves. Thus if you want to add or update a device, you populate a full device model rather than a specific Add/UpdateDeviceRequest. Currently, your request model has the same definition, and thus validation constraints, as the response model because they are one and the same! It is desirable to separate and be specific about what is required for a given request, as well as its state validity, and the bare minimum that must be returned within a response. Following from that central need, other considerations have been used when designing this proposed API. These will be enumerated and briefly explained below. 1.) Transport-agnostic Define the request/response data transfer objects (DTO) in a manner whereby they can be used independent of transport. For example, although an OpenAPI doc is implicitly coupled to HTTP/REST, define the DTOs in such a way that they could also be used if the platform were to evolve to a pub/sub architecture. 2.) Support partial updates via PATCH Given a request to, for example, update a device the user should be able to update only some properties of the device. Previously this would require an endpoint for each individual property to be updated since the \"update device\" endpoint, facilitated by a PUT, would perform a complete replacement of the device's data. If you only wanted to update the LastConnected timestamp, then a separate endpoint for that property was required. We will leverage PATCH in order to update an entity and only those properties populated on the request will be considered. Properties that are missing or left blank will not be touched. 3.) Support multiple requests at once Endpoints for the addition or updating of data (POST/PATCH) should accept multiple requests at once. If it were desirable to add or update multiple devices with one request, for example, the API should facilitate this. 4.) Support multiple correlated responses at once Following from #3 above, each request sent to the endpoint must result in a corresponding response. In the case of HTTP/REST, this means if four requests are sent to a POST operation, the return payload will have four responses. Each response must expose a \"code\" property containing a numeric result for what occurred. These could be equivalent to HTTP status codes, for example. So while the overall call might succeed, one or more of the child requests may not have. It is up to the caller to examine each response and handle accordingly. In order to correlate each response to its original request, each request must be assigned its own ID (in GUID format). The caller can then tie a response to an individual request and handle the result accordingly, or otherwise track that a response to a given request was not received. 5.) Use of 207 HTTP Status (Multi-Result) In the case where an endpoint can support multiple responses, the returned HTTP code from a REST API will be 207 (Multi-status) 6.) Each service should provide a \"batch\" request endpoint In addition to use-case specific endpoints that you'd find in any REST API, each service should provide a \"batch\" endpoint that can take any kind of request. This is a generic endpoint that allows you to group requests of different types within a single call. For example, instead of having to call two endpoints to get two jobs done, you can call a single endpoint passing the specific requests and have them routed appropriately within the service. Also, when considering agnostic transport, the batch endpoint would allow for the definition and handling of \"GET\" equivalent DTOs which are now implicit in the format of a URL. 7.) GET endpoints returning a list of items must support pagination URL parameters must be supported for every GET endpoint to support pagination. These parameters should indicate the current page of results and the number of results on a page. Decision Commnunity has accepted the reasoning for the new API and the design principles outlined above. The approach will be to gradually implement the V2 API side-by-side with the current V1 APIs. We believe it will take more than a single release cycle to implement the new specification. Releases of that occur prior to the V2 API implementation completion will continue to be major versioned as 1.x. Subsequent to completion, releases will be major versioned as 2.x. Consequences Backward incompatibility with EdgeX Foundry's V1 API requires a major version increment (e.g. v2.x). Service-level testing (e.g. blackbox tests) needs to be rewritten. Specification-first development allows for different implementations of EdgeX services to be certified as \"EdgeX Compliant\" in reference to an objective standard. Transport-agnostic focus enables different architectural patterns (pub/sub versus REST) using the same data representation.","title":"Geneva API Guiding Principles"},{"location":"design/adr/core/0003-V2-API-Principles/#geneva-api-guiding-principles","text":"","title":"Geneva API Guiding Principles"},{"location":"design/adr/core/0003-V2-API-Principles/#status","text":"Accepted by EdgeX Foundry working groups as of Core Working Group meeting 16-Jan-2020 Note This ADR was written pre-Geneva with an assumption that the V2 APIs would be available in Geneva. In actuality, the full V2 APIs will be delivered in the Ireland release (Spring 2020)","title":"Status"},{"location":"design/adr/core/0003-V2-API-Principles/#context","text":"A redesign of the EdgeX Foundry API is proposed for the Geneva release. This is understood by the community to warrant a 2.0 release that will not be backward compatible. The goal is to rework the API using solid principles that will allow for extension over the course of several release cycles, avoiding the necessity of yet another major release version in a short period of time. Briefly, this effort grew from the acknowledgement that the current models used to facilitate requests and responses via the EdgeX Foundry API were legacy definitions that were once used as internal representations of state within the EdgeX services themselves. Thus if you want to add or update a device, you populate a full device model rather than a specific Add/UpdateDeviceRequest. Currently, your request model has the same definition, and thus validation constraints, as the response model because they are one and the same! It is desirable to separate and be specific about what is required for a given request, as well as its state validity, and the bare minimum that must be returned within a response. Following from that central need, other considerations have been used when designing this proposed API. These will be enumerated and briefly explained below. 1.) Transport-agnostic Define the request/response data transfer objects (DTO) in a manner whereby they can be used independent of transport. For example, although an OpenAPI doc is implicitly coupled to HTTP/REST, define the DTOs in such a way that they could also be used if the platform were to evolve to a pub/sub architecture. 2.) Support partial updates via PATCH Given a request to, for example, update a device the user should be able to update only some properties of the device. Previously this would require an endpoint for each individual property to be updated since the \"update device\" endpoint, facilitated by a PUT, would perform a complete replacement of the device's data. If you only wanted to update the LastConnected timestamp, then a separate endpoint for that property was required. We will leverage PATCH in order to update an entity and only those properties populated on the request will be considered. Properties that are missing or left blank will not be touched. 3.) Support multiple requests at once Endpoints for the addition or updating of data (POST/PATCH) should accept multiple requests at once. If it were desirable to add or update multiple devices with one request, for example, the API should facilitate this. 4.) Support multiple correlated responses at once Following from #3 above, each request sent to the endpoint must result in a corresponding response. In the case of HTTP/REST, this means if four requests are sent to a POST operation, the return payload will have four responses. Each response must expose a \"code\" property containing a numeric result for what occurred. These could be equivalent to HTTP status codes, for example. So while the overall call might succeed, one or more of the child requests may not have. It is up to the caller to examine each response and handle accordingly. In order to correlate each response to its original request, each request must be assigned its own ID (in GUID format). The caller can then tie a response to an individual request and handle the result accordingly, or otherwise track that a response to a given request was not received. 5.) Use of 207 HTTP Status (Multi-Result) In the case where an endpoint can support multiple responses, the returned HTTP code from a REST API will be 207 (Multi-status) 6.) Each service should provide a \"batch\" request endpoint In addition to use-case specific endpoints that you'd find in any REST API, each service should provide a \"batch\" endpoint that can take any kind of request. This is a generic endpoint that allows you to group requests of different types within a single call. For example, instead of having to call two endpoints to get two jobs done, you can call a single endpoint passing the specific requests and have them routed appropriately within the service. Also, when considering agnostic transport, the batch endpoint would allow for the definition and handling of \"GET\" equivalent DTOs which are now implicit in the format of a URL. 7.) GET endpoints returning a list of items must support pagination URL parameters must be supported for every GET endpoint to support pagination. These parameters should indicate the current page of results and the number of results on a page.","title":"Context"},{"location":"design/adr/core/0003-V2-API-Principles/#decision","text":"Commnunity has accepted the reasoning for the new API and the design principles outlined above. The approach will be to gradually implement the V2 API side-by-side with the current V1 APIs. We believe it will take more than a single release cycle to implement the new specification. Releases of that occur prior to the V2 API implementation completion will continue to be major versioned as 1.x. Subsequent to completion, releases will be major versioned as 2.x.","title":"Decision"},{"location":"design/adr/core/0003-V2-API-Principles/#consequences","text":"Backward incompatibility with EdgeX Foundry's V1 API requires a major version increment (e.g. v2.x). Service-level testing (e.g. blackbox tests) needs to be rewritten. Specification-first development allows for different implementations of EdgeX services to be certified as \"EdgeX Compliant\" in reference to an objective standard. Transport-agnostic focus enables different architectural patterns (pub/sub versus REST) using the same data representation.","title":"Consequences"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/","text":"EdgeX-CLI V2 Design Status Approved (by TSC vote on 10/6/21) Context This ADR presents a technical plan for creation of a 2.0 version of edgex-cli which supports the new V2 REST APIs developed as part of the Ireland release of EdgeX. Existing Behavior The latest version of edgex-cli (1.0.1) only supports the V1 REST APIs and thus cannot be used with V2 releases of EdgeX. As the edgex-cli was developed organically over time, the current implementation has a number of bugs mostly involving a lack of consistent behavior, especially with respect to formatting of output. Other issues with the existing client include: lack of tab completion default output of commands is too verbose verbose output sometime prevents use of jq static configuration file required (i.e. no registry support) project hierarchy not conforming to best practice guidelines History The original Hanoi V1 client was created by a team at VMWare which is no longer participating in the project. Canonical will lead the development of the Ireland/Jakarta V2 client. Decision Use standardized command-line args/flags Argument/Flag Description -d , --debug show additional output for debugging purposes (e.g. REST URL, request JSON, \u2026). This command-line arg will replace -v, --verbose and will no longer trigger output of the response JSON (see -j, --json). -j , --json output the raw JSON response returned by the EdgeX REST API and nothing else. This output mode is used for script-based usage of the client. --version output the version of the client and if available, the version of EdgeX installed on the system (using the version of the metadata data service) Restructure the Go code hierarchy to follow the most recent recommended guidelines . For instance /cmd should just contain the main application for the project, not an implementation for each command - that should be in /internal/cmd Take full advantage of the features of the underlying command-line library, Cobra , such as tab-completion of commands. Allow overlap of command names across services by supporting an argument to specify the service to use: -m/--metadata , -c/--command , -n/--notification , -s/--scheduler or --data (which is the default). Examples: edgex-cli ping --data edgex-cli ping -m edgex-cli version -c Implement all required V2 endpoints for core services Core Command - edgex-cli command read | write | list Core Data - edgex-cli event add | count | list | rm | scrub** - edgex-cli reading count | list Metadata - edgex-cli device add | adminstate | list | operstate | rm | update - edgex-cli deviceprofile add | list | rm | update - edgex-cli deviceservice add | list | rm | update - edgex-cli provisionwatcher add | list | rm | update Support Notifications - edgex-cli notification add | list | rm - edgex-cli subscription add | list | rm Support Scheduler - edgex-cli interval add | list | rm | update ** Common endpoints in all services ** - ** ` edgex - cli version ` ** - ** ` edgex - cli ping ` ** - ** ` edgex - cli metrics ` ** - ** ` edgex - cli status ` ** The commands will support arguments as appropriate . For instance : - ` event list ` using ` / event / all ` to return all events - ` event list -- device { name }` using ` / event / device / name / { name }` to return the events sourced from the specified device . Currently, some commands default to always displaying GUIDs in objects when they're not really needed. Change this so that by default GUIDs aren't displayed, but add a flag which causes them to be displayed. scrub may not work with Redis being secured by default. That might also apply to the top-level db command (used to wipe the entire db). If so, then the commands will be disabled in secure mode, but permitted in non-secure mode. Have built-in defaults with port numbers for all core services and allow overrides, avoiding the need for static configuration file or configuration provider. (Stretch) implement a -o / --output argument which could be used to customize the pretty-printed objects (i.e. non-JSON). (Stretch) Implement support for use of the client via the API Gateway, including being able to connect to a remote EdgeX instance. This might require updates in go-mod-core-contracts. References Command Line Interface Guidelines The Unix Programming Environment, Brian W. Kernighan and Rob Pike POSIX Utility Conventions Program Behavior for All Programs, GNU Coding Standards 12 Factor CLI Apps, Jeff Dickey CLI Style Guide, Heroku Standard Go Project Layout","title":"EdgeX-CLI V2 Design"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#edgex-cli-v2-design","text":"","title":"EdgeX-CLI V2 Design"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#status","text":"Approved (by TSC vote on 10/6/21)","title":"Status"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#context","text":"This ADR presents a technical plan for creation of a 2.0 version of edgex-cli which supports the new V2 REST APIs developed as part of the Ireland release of EdgeX.","title":"Context"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#existing-behavior","text":"The latest version of edgex-cli (1.0.1) only supports the V1 REST APIs and thus cannot be used with V2 releases of EdgeX. As the edgex-cli was developed organically over time, the current implementation has a number of bugs mostly involving a lack of consistent behavior, especially with respect to formatting of output. Other issues with the existing client include: lack of tab completion default output of commands is too verbose verbose output sometime prevents use of jq static configuration file required (i.e. no registry support) project hierarchy not conforming to best practice guidelines","title":"Existing Behavior"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#history","text":"The original Hanoi V1 client was created by a team at VMWare which is no longer participating in the project. Canonical will lead the development of the Ireland/Jakarta V2 client.","title":"History"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#decision","text":"Use standardized command-line args/flags Argument/Flag Description -d , --debug show additional output for debugging purposes (e.g. REST URL, request JSON, \u2026). This command-line arg will replace -v, --verbose and will no longer trigger output of the response JSON (see -j, --json). -j , --json output the raw JSON response returned by the EdgeX REST API and nothing else. This output mode is used for script-based usage of the client. --version output the version of the client and if available, the version of EdgeX installed on the system (using the version of the metadata data service) Restructure the Go code hierarchy to follow the most recent recommended guidelines . For instance /cmd should just contain the main application for the project, not an implementation for each command - that should be in /internal/cmd Take full advantage of the features of the underlying command-line library, Cobra , such as tab-completion of commands. Allow overlap of command names across services by supporting an argument to specify the service to use: -m/--metadata , -c/--command , -n/--notification , -s/--scheduler or --data (which is the default). Examples: edgex-cli ping --data edgex-cli ping -m edgex-cli version -c Implement all required V2 endpoints for core services Core Command - edgex-cli command read | write | list Core Data - edgex-cli event add | count | list | rm | scrub** - edgex-cli reading count | list Metadata - edgex-cli device add | adminstate | list | operstate | rm | update - edgex-cli deviceprofile add | list | rm | update - edgex-cli deviceservice add | list | rm | update - edgex-cli provisionwatcher add | list | rm | update Support Notifications - edgex-cli notification add | list | rm - edgex-cli subscription add | list | rm Support Scheduler - edgex-cli interval add | list | rm | update ** Common endpoints in all services ** - ** ` edgex - cli version ` ** - ** ` edgex - cli ping ` ** - ** ` edgex - cli metrics ` ** - ** ` edgex - cli status ` ** The commands will support arguments as appropriate . For instance : - ` event list ` using ` / event / all ` to return all events - ` event list -- device { name }` using ` / event / device / name / { name }` to return the events sourced from the specified device . Currently, some commands default to always displaying GUIDs in objects when they're not really needed. Change this so that by default GUIDs aren't displayed, but add a flag which causes them to be displayed. scrub may not work with Redis being secured by default. That might also apply to the top-level db command (used to wipe the entire db). If so, then the commands will be disabled in secure mode, but permitted in non-secure mode. Have built-in defaults with port numbers for all core services and allow overrides, avoiding the need for static configuration file or configuration provider. (Stretch) implement a -o / --output argument which could be used to customize the pretty-printed objects (i.e. non-JSON). (Stretch) Implement support for use of the client via the API Gateway, including being able to connect to a remote EdgeX instance. This might require updates in go-mod-core-contracts.","title":"Decision"},{"location":"design/adr/core/0019-EdgeX-CLI-V2/#references","text":"Command Line Interface Guidelines The Unix Programming Environment, Brian W. Kernighan and Rob Pike POSIX Utility Conventions Program Behavior for All Programs, GNU Coding Standards 12 Factor CLI Apps, Jeff Dickey CLI Style Guide, Heroku Standard Go Project Layout","title":"References"},{"location":"design/adr/core/0021-Device-Profile-Changes/","text":"Changes to Device Profiles Status Approved By TSC Vote on 2/14/22 Please see a prior PR on this topic that detailed much of the debate and context on this issue. For clarity and simplicity, that PR was closed in favor of this simpler ADR. Context While the device profile has always been the way to describe a device/sensor and template its communications to the rest of the EdgeX platform, over the course of EdgeX evolution there have been changes in what could change in a profile (often based on its associations to other EdgeX objects). This document is meant to address the issue of change surrounding device profiles in EdgeX going forward \u2013 specifically when can a device profile (or its sub-elements such as device resources) be added, modified or removed. Summary of Device Profile Rules These rules will be implemented in core metadata on device profile API calls. A device profile can be added anytime Device resources or device commands can be added to a device profile anytime Attributes can be added to a device profile anytime A device profile can be removed or modified when the device profile is not associated to a device or provision watcher this includes modifying any field (except identifiers like names and ids) this includes changes to the array of device resources, device commands this includes changes to attributes (of device resources) even when a device profile is associated to a device or provision watcher, fields of the device profile or device resource can be modified when the field change will not affect the behavior of the system. on profile, the following fields do not affect the behavior: description, manufacturer, model, labels. on device resource, the following fields do not affect the behavior: description and tag A device profile cannot be removed when it is associated to a device or provision watcher. A device profile can be removed or modified even when associated to an event or reading. However, configuration options (see New Configuration Settings below) are available to block the change or removal of a device profile for any reason. the rationale behind the new configuraton settings was specifically to protect the event/reading association to device profiles. Events and readings are generally considered short lived (ephemeral) objects and already contain the necessary device profile information that are needed by the system during their short life without having to refer to and keep the device profile. But if an adopter wants to make sure the device profile is unmodified and still exists for any event/readings association (or for any reason), then the new config setting will block device profile changes or removals. see note below in Consequences that a new Units property must be added to the Reading object in order to support this rule and the need for all relevant profile data to be in the reading. Ancillary Rules associated to Device Profiles Name and ID fields (identifying fields) for device profiles, device resources, etc. cannot be modified and can never be null. A device profile can begin life \u201cempty\u201d - meaning that it has no device resources or device commands. New APIs The following APIs would be added to the metadata REST service in order to meet the design specified above. Add Profile General Property PATCH API (allow to modify profile's description, manufacturer, model and label fields) Add Profile Device Resource POST API Add Profile Device Resource PATCH API (allow to modify Description and IsHidden only) Add Profile Device Resource DELETE API (allow as described above) Add Profile Device Command POST API Add Profile Device Command PATCH API (allow as described above) Add Profile Device Command DELETE API (allow as described above) New Configuration Settings Some adopters may not view event/reading data as ephemeral or short lived. These adopters may choose not to allow device profiles to be modified or removed when associated to an event or reading. For this reason, two new configuration options, in the [Writable.ProfileChange] section, will be added to metadata configuration that are used to reject modifications or deletions. StrictDeviceProfileChanges (set to false by default) StrictDeviceProfileDeletes (set to false by default) When either of these config settings are set to true, metadata would accordingly reject changes to or removal of profiles (note: metadata will not check that there are actually events or readings - or any object - associated to the device profile when these are set to true. It simply rejects all modification or deletes to device profiles with the assumption that there could be events, readings or other objects associated and which need to be preserved). Consequences/Considerations In order to allow device profiles to be updated or removed even when associated to an EdgeX event/reading, a new property needs to be added to the reading object. Readings will now contain a \u201cUnits\u201d (string) property. This property will indicate the units of measure for the Value in the Reading and will be populated based on the Units for the device resource. A new device service configuration property, ReadingUnits (set true by default) will allow adopters to indicate they do not want units to be added to the readings (for cases where there is a concern about the number of readings and the extra data of adding units). The ReadingUnits configuration option will be added to the [Writable.Reading] section of device services (and addressed in the device service SDKs). This allows the event/reading to contain all relevant information from the device profile that is needed by the system during the course of the event/reading\u2019s life. This allows the device profile to be modified or even removed even when there are events/readings in the system that were created from information in the device profile. References Metadata API Device Service SDK Required Functionality","title":"Changes to Device Profiles"},{"location":"design/adr/core/0021-Device-Profile-Changes/#changes-to-device-profiles","text":"","title":"Changes to Device Profiles"},{"location":"design/adr/core/0021-Device-Profile-Changes/#status","text":"Approved By TSC Vote on 2/14/22 Please see a prior PR on this topic that detailed much of the debate and context on this issue. For clarity and simplicity, that PR was closed in favor of this simpler ADR.","title":"Status"},{"location":"design/adr/core/0021-Device-Profile-Changes/#context","text":"While the device profile has always been the way to describe a device/sensor and template its communications to the rest of the EdgeX platform, over the course of EdgeX evolution there have been changes in what could change in a profile (often based on its associations to other EdgeX objects). This document is meant to address the issue of change surrounding device profiles in EdgeX going forward \u2013 specifically when can a device profile (or its sub-elements such as device resources) be added, modified or removed.","title":"Context"},{"location":"design/adr/core/0021-Device-Profile-Changes/#summary-of-device-profile-rules","text":"These rules will be implemented in core metadata on device profile API calls. A device profile can be added anytime Device resources or device commands can be added to a device profile anytime Attributes can be added to a device profile anytime A device profile can be removed or modified when the device profile is not associated to a device or provision watcher this includes modifying any field (except identifiers like names and ids) this includes changes to the array of device resources, device commands this includes changes to attributes (of device resources) even when a device profile is associated to a device or provision watcher, fields of the device profile or device resource can be modified when the field change will not affect the behavior of the system. on profile, the following fields do not affect the behavior: description, manufacturer, model, labels. on device resource, the following fields do not affect the behavior: description and tag A device profile cannot be removed when it is associated to a device or provision watcher. A device profile can be removed or modified even when associated to an event or reading. However, configuration options (see New Configuration Settings below) are available to block the change or removal of a device profile for any reason. the rationale behind the new configuraton settings was specifically to protect the event/reading association to device profiles. Events and readings are generally considered short lived (ephemeral) objects and already contain the necessary device profile information that are needed by the system during their short life without having to refer to and keep the device profile. But if an adopter wants to make sure the device profile is unmodified and still exists for any event/readings association (or for any reason), then the new config setting will block device profile changes or removals. see note below in Consequences that a new Units property must be added to the Reading object in order to support this rule and the need for all relevant profile data to be in the reading.","title":"Summary of Device Profile Rules"},{"location":"design/adr/core/0021-Device-Profile-Changes/#ancillary-rules-associated-to-device-profiles","text":"Name and ID fields (identifying fields) for device profiles, device resources, etc. cannot be modified and can never be null. A device profile can begin life \u201cempty\u201d - meaning that it has no device resources or device commands.","title":"Ancillary Rules associated to Device Profiles"},{"location":"design/adr/core/0021-Device-Profile-Changes/#new-apis","text":"The following APIs would be added to the metadata REST service in order to meet the design specified above. Add Profile General Property PATCH API (allow to modify profile's description, manufacturer, model and label fields) Add Profile Device Resource POST API Add Profile Device Resource PATCH API (allow to modify Description and IsHidden only) Add Profile Device Resource DELETE API (allow as described above) Add Profile Device Command POST API Add Profile Device Command PATCH API (allow as described above) Add Profile Device Command DELETE API (allow as described above)","title":"New APIs"},{"location":"design/adr/core/0021-Device-Profile-Changes/#new-configuration-settings","text":"Some adopters may not view event/reading data as ephemeral or short lived. These adopters may choose not to allow device profiles to be modified or removed when associated to an event or reading. For this reason, two new configuration options, in the [Writable.ProfileChange] section, will be added to metadata configuration that are used to reject modifications or deletions. StrictDeviceProfileChanges (set to false by default) StrictDeviceProfileDeletes (set to false by default) When either of these config settings are set to true, metadata would accordingly reject changes to or removal of profiles (note: metadata will not check that there are actually events or readings - or any object - associated to the device profile when these are set to true. It simply rejects all modification or deletes to device profiles with the assumption that there could be events, readings or other objects associated and which need to be preserved).","title":"New Configuration Settings"},{"location":"design/adr/core/0021-Device-Profile-Changes/#consequencesconsiderations","text":"In order to allow device profiles to be updated or removed even when associated to an EdgeX event/reading, a new property needs to be added to the reading object. Readings will now contain a \u201cUnits\u201d (string) property. This property will indicate the units of measure for the Value in the Reading and will be populated based on the Units for the device resource. A new device service configuration property, ReadingUnits (set true by default) will allow adopters to indicate they do not want units to be added to the readings (for cases where there is a concern about the number of readings and the extra data of adding units). The ReadingUnits configuration option will be added to the [Writable.Reading] section of device services (and addressed in the device service SDKs). This allows the event/reading to contain all relevant information from the device profile that is needed by the system during the course of the event/reading\u2019s life. This allows the device profile to be modified or even removed even when there are events/readings in the system that were created from information in the device profile.","title":"Consequences/Considerations"},{"location":"design/adr/core/0021-Device-Profile-Changes/#references","text":"Metadata API Device Service SDK Required Functionality","title":"References"},{"location":"design/adr/core/0022-UoM/","text":"EdgeX Unit of Measure (UoM) Status Approved by TSC Vote on 3/16/2022 This ADR began under a different ADR pull request. The prior ADR recommended a UoM per device resource and just allowed for the association of an arbitrary set of unit of measure references against the resource. However, it did not include any specific units of measure or validation of those units against the actual profiles (and ultimately the associated readings). See the previous UoM ADR for details and prior debate. Implementation: to be determined, but could be as soon as Kamakura (Spring 2022). Context Unit of measurement (UoM) is defined as \"a standard amount of a physical quantity, such as length, mass, energy, etc, specified multiples of which are used to express magnitudes of that physical quantity\". In EdgeX, data collected from sensors are physical quantities which should be associated to some unit of measure to express magnitude of that physical quantity. For example, if EdgeX collected a temperature reading from a thermostat as 45 , the user of that sensor reading would want to know if the unit of measure for the 45 quantity was expressed in Celsius, Fahrenheit or even the Kelvin scale. Since the founding of the project, there has been consensus that a unit of measure should be associated to any sensor or metric quantity collected by EdgeX. Also since the founding of the project, a unit of measure has therefore been specified (directly or indirectly) to each device resource (found in device profiles) and associated values collected as part of readings. The unit of measure was, however, in all cases just a string reference to some arbitrary unit (which may or may not be in a UoM standard) to be interpreted by the consumer of EdgeX data. The reporting sensor/device or programmer of the device service could choose what UoM string was associated to the device resources (and readings produced by the device service) as the unit of measure for any piece of data. Per the temperature example above, the unit of measure could have been \"F\" or \"C\", \"Celsius\" or \"Fahrenheit\", or any other representation. In other words, the associated unit of measure for all data in EdgeX was left to agreement and interpretation by the data provider/producer and EdgeX data consumer. There are various specifications and standards around unit of measure. Specifically, there are several options to choose from as it relates to the exchange of data in electronic communications - and units of measure associated in that exchange. As examples, two big competing standards around EDI (electronic data exchange) that both have associated unit of measure codes are: ANSI X12: EDI standard used mostly in the US EDIFACT: UN EDI standard used mostly in Europe and Asia The Unified Code for Units of Measure provides an alternative list (not a standard) that is used by various organizations like OSGI and the Eclipse Foundation. While standards exist, use by various open source projects (especially IoT/edge projects) is inconsistent and haphazard. Groups like oneM2M seem to define their own selection of units in specifications per vertical (home for example) while Kura doesn't even appear to use the UoM JSR (a Java related unit of measure specification for Java applications like Kura). Decision It would be speculative and inappropriate for EdgeX to select a unit of measure standard which is not widely adopted in the industry or choose a static unit of measure list that is incomplete with regard to possible IoT / edge use case needs. At this time, there does not appear to be a single and unequivocal standard for units of measure that encompasses all EdgeX related use cases (now and in the future). Therefore, EdgeX chooses not to select or adopt a unit of measure specification, standard, or code list to apply across the platform. Instead, EdgeX adopters will be allowed to optionally specify which unit of measure specification, standard, or unit of measure code list they would like used in their instance(s) of EdgeX. Specifying the Units of Measure Units of measure allowed by the instance of EdgeX will be specified in a configuration file (in TOML format called uom.toml by default). Note: the UoM configuration is a separate configuration TOML file (separate from the metadata service configuration file - configuration.toml ). The units of measure in the configuration file can be attributed, optionally, to a specification, document, or other UoM definition source. The source only helps provide the location of documentation about the origins and details of the units specified for the reader, but it will not be used or checked by EdgeX. An optional default source can be provided at the top level configuration (as shown in the examples below) so that other sources are only needed when there are specific units used that are not found in the default source. The units of measure can be categorized for better organization and to allow for different sources to be specified for different units. The categories are defined by the TOML section names (the UoM dot labels). Sample TOML unit of measure configuration [Uom] Source = \"reference to source for all UoM if not specified below\" [Uom.temperature] Source = \"www.weather.com\" Values = \"C,F,K\" [Uom.weights] Source = \"www.usa.gov/federal-agencies/weights-and-measures-division\" Values = \"lbs,ounces,kilos,grams\" Specifying the UoM File Location The location of the UoM file will be specified in core metadata's configuration (currently in res/configuration.toml ) - see example A below. Example Metadata Configuration - location of of the UoM configuration file [Writable] [Writable.UoM] Validation = false ## false (meaning off) by default ## in the non-writable area - example file specified to units of measure [UoM] UoMFile = \"./res/uom.toml\" # the UoMFile location can be either absolute or relative path location The location of the UoM file should point to an accessible file (relative to application executable or absolute path). The file must be something that the service can reach (ex: in shared volume, volume mount, etc.) in order to allow for the adopter to provide the units of measure independently during configuration/setup of the EdgeX instance without requiring a build of the metadata service or a reconstruction of the Docker image/container. Info In future versions, multiple UoM definition files might be specified. This may help the organization of the units in the future. Note The environmental overrides can be used to specify and override the location of the UoM configuration file . Info It was discussed that the file location could be done via URI and even allow for HTTP, HTTPS or other protocol access of the file. For this first implementation, it was decided (per Monthly Architect's meeting of 2/28/22) to only allow for a simple file path reference (relative or absolute). Future implementation can consider URI use. Specifying Validation on or off Additionally, in metadata's configuration, a configuration option for unit of measure validation being on or off will be provided (note Validation in both example above). The location of the UoM file is static, but the ability to turn validation on/off is dynamic and therefore in the writable area of configuration. For backward compatibility, validation will be off by default. Note on and off are specified by boolean values true and false in the configuration file. Validation of the Units of Measure Core metadata will read the units of measure from its configuration file. Like all configuration information, this data will be stored in the configuration service (Consul today) on initial startup of the core metadata service. When validation is turned on (Writable.UoM.validation is set to true), all device profile units (in device resource, device properties) will be validated against the list of units of measure by core metadata. In other words, when a device profile is created or updated or when a device resource is added or updated via the core metadata API, the units specified in the device resource's units field (see resource example below) will be checked against the valid list of UoM provided via core metadata configuration. If the units value matches any one of the configuration units of measure, then the device resource is considered valid - allowing the create or update operation to continue. If the units value does not match any one of the configuration units of measure, then the device profile or device resource operation (create or update) is rejected (error code 500 is returned) and an appropriate error message is returned in the response to the caller of the core metadata API. Note Importantly (as discussed in Core WG 2/17/22), the units field on a profile is and shall remain optional. If the units field is not specified in the device profile, then it is assumed that the device resource does not have well defined units of measure. In other words, core metadata will not fail a profile with no units field specified on a device resource. In the example device resource below, core metadata would check that C is in the list of units of measure in the configuration. deviceResources : - name : \"RoomTemperature\" isHidden : false description : \"Room Temperature x10 \u00b0C (Read Only)\" attributes : { primaryTable : \"INPUT_REGISTERS\" , startingAddress : 3 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.1\" units : \"C\" ## core metadata checks this value against its list of valid units of measure By checking the units property of the device resources (on creation or updates of the device profile or create/update of the device resources), and rejecting any additions or changes that include non-valid units of measure, then we can be assured that all readings created by the device service will contain valid units by default (assuming that validation of the units of measure is always on) or that the units are inconsequential (when the units field is not specified for a device resource). This means, the units in a reading do not need to be validated separately. Considerations Validation important and architecturally significant Based on discussion in the Core WG meeting of 2/3/22, it was decided that without validation and some valid list of actual UoM, the ADR was just adding metadata to the profile and thus did not even rise to the level of \"significant\" architectural decision. It was further felt that in order to really provide any value to adopters and to get adherence to their chosen units of measure, EdgeX had to allow for a valid list of units of measure to be specified and be used to check profile units - but in a way that is easy to configure/provide without having to rebuild a service for example. If the units of measure were defined just in the standard configuration file, it would make it hard to change this list in deployments. This new UoM ADR is the result of that discussion. In general, it specifies, through adopter provided configuration, the exact unit of measures that are allowed for the EdgeX instance and any optional reference (such as a specification) where those units are defined. It does so through a separate core metadata configuration file making it easier to change. Use of SenML SenML was suggested as a specification (currently a proposed standard) from which EdgeX may draw some guidance or inspiration with regard to unit of measure representation in \"simple sensor measurements and device parameters.\" In fact, SenML defines a simple data model (in JSON, CBOR, XML, EXI) for the exchange of what EdgeX would call readings. A JSON example is below: [{ \"n\" : \"urn:dev:ow:10e2073a01080063\" , \"u\" : \"Cel\" , \"v\" : 23.1 }] In the example above, the array (what EdgeX would consider a collection of readings) has a single SenML Record with a measurement for a sensor named \"urn:dev:ow:10e2073a01080063\" with a current value of 23.1 for degrees measured in Celsius (Cel) unit of measure. However, SenML suggests the use of short names for the keys in most cases, but long names could be used. In which case, the JSON SenML reading would look like the following: [{ \"Name\" : \"urn:dev:ow:10e2073a01080063\" , \"Unit\" : \"Cel\" , \"Value\" : 23.1 }] In this way, the parallels to EdgeX model are, by accident, uncanny - at least in the JSON instance. SenML goes to much more depth to provide extensions and more definitions around measurements. But at its base, the EdgeX format is not unlike SenML and could easily be aligned with SenML in the future (or allow for an application service to export in SenML with an additional function fairly easily and if there were demand). However, on the basis of \"unit of measure\", SenML is actually light on details. With regard to UoM, the SenML specification only says: Quote If the Record has no Unit, the Base Unit is used as the Unit. Having no Unit and no Base Unit is allowed; any information that may be required about units applicable to the value then needs to be provided by the application context. A SenML Units Registry provides for a list of unit symbols (the \"SenML Units registry\" ). This list could be used as one of the sources for EdgeX UoM definition. SenML should be examined for future versions of EdgeX with regard to data model, but its relevance to unit of measure is believed to be minimal at this time. Future Considerations/Additions/Improvements In the future, validation may be turned on or off per device service; allowing the decision to validate units of measure to be accomplished on a service or even allow the device service to validate/not validate based on particular devices. In the future, additional criteria may be added to the unit of measure information to all for more specific (or allowing more granularity) validation. For example, the category of units of measure could be specified in a device resource so that a profile's units are validated against specific sources or collections of unit of measure. Use of URI to specify the unit of measures file was discussed. This would be novel with regard to providing EdgeX information. Per core working group of 2/17/22 and then again at the monthly architect's meeting of 2/28/22, we may look to use a URI to specify a configuration file to specify UoM in the future. Indeed, URIs may be used (an EdgeX 3.0 consideration) to point to device profiles, configuration files, and other information in the future. This would even allow multiple EdgeX instances to use the same configuration or profile (multiple EdgeX instances using the same URI to use a shared profile for example). However, it was deemed scope creep and too much to do for this first iteration. Initially, this ADR allowed for the UoM to also or alternately to be defined in the standard metadata service configuration file (`configuration.toml'). During the Core WG meeting of 3/3/22, it was decided to simplify the design and strictly limit UoM to a separate configuration file. If future use cases or adopters request inline definition, this can be implemented in a future release. Consequences Validation could impact performance. Therefore allowing it to be turned on or off is critical to some use cases. However, it will only impact performance of profile creation/updates (and associated device resources) in core metadata. References UoM Standards https://ediacademy.com/blog/x12-unit-of-measurement-codes/ https://unece.org/fileadmin/DAM/cefact/recommendations/rec20/rec20_rev3_Annex2e.pdf https://en.wikipedia.org/wiki/Unified_Code_for_Units_of_Measure https://www.ogc.org/standards/sensorthings https://datatracker.ietf.org/doc/html/rfc8428 https://datatracker.ietf.org/doc/html/rfc8428#section-12.1 UoM Tools and Databases https://ucum.nlm.nih.gov/ucum-lhc/demo.html https://project-haystack.org/doc/Units https://github.com/fantom-lang/fantom/blob/master/etc/sys/units.txt https://gs1.github.io/UnitConverterUNECERec20/","title":"EdgeX Unit of Measure (UoM)"},{"location":"design/adr/core/0022-UoM/#edgex-unit-of-measure-uom","text":"","title":"EdgeX Unit of Measure (UoM)"},{"location":"design/adr/core/0022-UoM/#status","text":"Approved by TSC Vote on 3/16/2022 This ADR began under a different ADR pull request. The prior ADR recommended a UoM per device resource and just allowed for the association of an arbitrary set of unit of measure references against the resource. However, it did not include any specific units of measure or validation of those units against the actual profiles (and ultimately the associated readings). See the previous UoM ADR for details and prior debate. Implementation: to be determined, but could be as soon as Kamakura (Spring 2022).","title":"Status"},{"location":"design/adr/core/0022-UoM/#context","text":"Unit of measurement (UoM) is defined as \"a standard amount of a physical quantity, such as length, mass, energy, etc, specified multiples of which are used to express magnitudes of that physical quantity\". In EdgeX, data collected from sensors are physical quantities which should be associated to some unit of measure to express magnitude of that physical quantity. For example, if EdgeX collected a temperature reading from a thermostat as 45 , the user of that sensor reading would want to know if the unit of measure for the 45 quantity was expressed in Celsius, Fahrenheit or even the Kelvin scale. Since the founding of the project, there has been consensus that a unit of measure should be associated to any sensor or metric quantity collected by EdgeX. Also since the founding of the project, a unit of measure has therefore been specified (directly or indirectly) to each device resource (found in device profiles) and associated values collected as part of readings. The unit of measure was, however, in all cases just a string reference to some arbitrary unit (which may or may not be in a UoM standard) to be interpreted by the consumer of EdgeX data. The reporting sensor/device or programmer of the device service could choose what UoM string was associated to the device resources (and readings produced by the device service) as the unit of measure for any piece of data. Per the temperature example above, the unit of measure could have been \"F\" or \"C\", \"Celsius\" or \"Fahrenheit\", or any other representation. In other words, the associated unit of measure for all data in EdgeX was left to agreement and interpretation by the data provider/producer and EdgeX data consumer. There are various specifications and standards around unit of measure. Specifically, there are several options to choose from as it relates to the exchange of data in electronic communications - and units of measure associated in that exchange. As examples, two big competing standards around EDI (electronic data exchange) that both have associated unit of measure codes are: ANSI X12: EDI standard used mostly in the US EDIFACT: UN EDI standard used mostly in Europe and Asia The Unified Code for Units of Measure provides an alternative list (not a standard) that is used by various organizations like OSGI and the Eclipse Foundation. While standards exist, use by various open source projects (especially IoT/edge projects) is inconsistent and haphazard. Groups like oneM2M seem to define their own selection of units in specifications per vertical (home for example) while Kura doesn't even appear to use the UoM JSR (a Java related unit of measure specification for Java applications like Kura).","title":"Context"},{"location":"design/adr/core/0022-UoM/#decision","text":"It would be speculative and inappropriate for EdgeX to select a unit of measure standard which is not widely adopted in the industry or choose a static unit of measure list that is incomplete with regard to possible IoT / edge use case needs. At this time, there does not appear to be a single and unequivocal standard for units of measure that encompasses all EdgeX related use cases (now and in the future). Therefore, EdgeX chooses not to select or adopt a unit of measure specification, standard, or code list to apply across the platform. Instead, EdgeX adopters will be allowed to optionally specify which unit of measure specification, standard, or unit of measure code list they would like used in their instance(s) of EdgeX.","title":"Decision"},{"location":"design/adr/core/0022-UoM/#specifying-the-units-of-measure","text":"Units of measure allowed by the instance of EdgeX will be specified in a configuration file (in TOML format called uom.toml by default). Note: the UoM configuration is a separate configuration TOML file (separate from the metadata service configuration file - configuration.toml ). The units of measure in the configuration file can be attributed, optionally, to a specification, document, or other UoM definition source. The source only helps provide the location of documentation about the origins and details of the units specified for the reader, but it will not be used or checked by EdgeX. An optional default source can be provided at the top level configuration (as shown in the examples below) so that other sources are only needed when there are specific units used that are not found in the default source. The units of measure can be categorized for better organization and to allow for different sources to be specified for different units. The categories are defined by the TOML section names (the UoM dot labels). Sample TOML unit of measure configuration [Uom] Source = \"reference to source for all UoM if not specified below\" [Uom.temperature] Source = \"www.weather.com\" Values = \"C,F,K\" [Uom.weights] Source = \"www.usa.gov/federal-agencies/weights-and-measures-division\" Values = \"lbs,ounces,kilos,grams\"","title":"Specifying the Units of Measure"},{"location":"design/adr/core/0022-UoM/#specifying-the-uom-file-location","text":"The location of the UoM file will be specified in core metadata's configuration (currently in res/configuration.toml ) - see example A below. Example Metadata Configuration - location of of the UoM configuration file [Writable] [Writable.UoM] Validation = false ## false (meaning off) by default ## in the non-writable area - example file specified to units of measure [UoM] UoMFile = \"./res/uom.toml\" # the UoMFile location can be either absolute or relative path location The location of the UoM file should point to an accessible file (relative to application executable or absolute path). The file must be something that the service can reach (ex: in shared volume, volume mount, etc.) in order to allow for the adopter to provide the units of measure independently during configuration/setup of the EdgeX instance without requiring a build of the metadata service or a reconstruction of the Docker image/container. Info In future versions, multiple UoM definition files might be specified. This may help the organization of the units in the future. Note The environmental overrides can be used to specify and override the location of the UoM configuration file . Info It was discussed that the file location could be done via URI and even allow for HTTP, HTTPS or other protocol access of the file. For this first implementation, it was decided (per Monthly Architect's meeting of 2/28/22) to only allow for a simple file path reference (relative or absolute). Future implementation can consider URI use.","title":"Specifying the UoM File Location"},{"location":"design/adr/core/0022-UoM/#specifying-validation-on-or-off","text":"Additionally, in metadata's configuration, a configuration option for unit of measure validation being on or off will be provided (note Validation in both example above). The location of the UoM file is static, but the ability to turn validation on/off is dynamic and therefore in the writable area of configuration. For backward compatibility, validation will be off by default. Note on and off are specified by boolean values true and false in the configuration file.","title":"Specifying Validation on or off"},{"location":"design/adr/core/0022-UoM/#validation-of-the-units-of-measure","text":"Core metadata will read the units of measure from its configuration file. Like all configuration information, this data will be stored in the configuration service (Consul today) on initial startup of the core metadata service. When validation is turned on (Writable.UoM.validation is set to true), all device profile units (in device resource, device properties) will be validated against the list of units of measure by core metadata. In other words, when a device profile is created or updated or when a device resource is added or updated via the core metadata API, the units specified in the device resource's units field (see resource example below) will be checked against the valid list of UoM provided via core metadata configuration. If the units value matches any one of the configuration units of measure, then the device resource is considered valid - allowing the create or update operation to continue. If the units value does not match any one of the configuration units of measure, then the device profile or device resource operation (create or update) is rejected (error code 500 is returned) and an appropriate error message is returned in the response to the caller of the core metadata API. Note Importantly (as discussed in Core WG 2/17/22), the units field on a profile is and shall remain optional. If the units field is not specified in the device profile, then it is assumed that the device resource does not have well defined units of measure. In other words, core metadata will not fail a profile with no units field specified on a device resource. In the example device resource below, core metadata would check that C is in the list of units of measure in the configuration. deviceResources : - name : \"RoomTemperature\" isHidden : false description : \"Room Temperature x10 \u00b0C (Read Only)\" attributes : { primaryTable : \"INPUT_REGISTERS\" , startingAddress : 3 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.1\" units : \"C\" ## core metadata checks this value against its list of valid units of measure By checking the units property of the device resources (on creation or updates of the device profile or create/update of the device resources), and rejecting any additions or changes that include non-valid units of measure, then we can be assured that all readings created by the device service will contain valid units by default (assuming that validation of the units of measure is always on) or that the units are inconsequential (when the units field is not specified for a device resource). This means, the units in a reading do not need to be validated separately.","title":"Validation of the Units of Measure"},{"location":"design/adr/core/0022-UoM/#considerations","text":"","title":"Considerations"},{"location":"design/adr/core/0022-UoM/#validation-important-and-architecturally-significant","text":"Based on discussion in the Core WG meeting of 2/3/22, it was decided that without validation and some valid list of actual UoM, the ADR was just adding metadata to the profile and thus did not even rise to the level of \"significant\" architectural decision. It was further felt that in order to really provide any value to adopters and to get adherence to their chosen units of measure, EdgeX had to allow for a valid list of units of measure to be specified and be used to check profile units - but in a way that is easy to configure/provide without having to rebuild a service for example. If the units of measure were defined just in the standard configuration file, it would make it hard to change this list in deployments. This new UoM ADR is the result of that discussion. In general, it specifies, through adopter provided configuration, the exact unit of measures that are allowed for the EdgeX instance and any optional reference (such as a specification) where those units are defined. It does so through a separate core metadata configuration file making it easier to change.","title":"Validation important and architecturally significant"},{"location":"design/adr/core/0022-UoM/#use-of-senml","text":"SenML was suggested as a specification (currently a proposed standard) from which EdgeX may draw some guidance or inspiration with regard to unit of measure representation in \"simple sensor measurements and device parameters.\" In fact, SenML defines a simple data model (in JSON, CBOR, XML, EXI) for the exchange of what EdgeX would call readings. A JSON example is below: [{ \"n\" : \"urn:dev:ow:10e2073a01080063\" , \"u\" : \"Cel\" , \"v\" : 23.1 }] In the example above, the array (what EdgeX would consider a collection of readings) has a single SenML Record with a measurement for a sensor named \"urn:dev:ow:10e2073a01080063\" with a current value of 23.1 for degrees measured in Celsius (Cel) unit of measure. However, SenML suggests the use of short names for the keys in most cases, but long names could be used. In which case, the JSON SenML reading would look like the following: [{ \"Name\" : \"urn:dev:ow:10e2073a01080063\" , \"Unit\" : \"Cel\" , \"Value\" : 23.1 }] In this way, the parallels to EdgeX model are, by accident, uncanny - at least in the JSON instance. SenML goes to much more depth to provide extensions and more definitions around measurements. But at its base, the EdgeX format is not unlike SenML and could easily be aligned with SenML in the future (or allow for an application service to export in SenML with an additional function fairly easily and if there were demand). However, on the basis of \"unit of measure\", SenML is actually light on details. With regard to UoM, the SenML specification only says: Quote If the Record has no Unit, the Base Unit is used as the Unit. Having no Unit and no Base Unit is allowed; any information that may be required about units applicable to the value then needs to be provided by the application context. A SenML Units Registry provides for a list of unit symbols (the \"SenML Units registry\" ). This list could be used as one of the sources for EdgeX UoM definition. SenML should be examined for future versions of EdgeX with regard to data model, but its relevance to unit of measure is believed to be minimal at this time.","title":"Use of SenML"},{"location":"design/adr/core/0022-UoM/#future-considerationsadditionsimprovements","text":"In the future, validation may be turned on or off per device service; allowing the decision to validate units of measure to be accomplished on a service or even allow the device service to validate/not validate based on particular devices. In the future, additional criteria may be added to the unit of measure information to all for more specific (or allowing more granularity) validation. For example, the category of units of measure could be specified in a device resource so that a profile's units are validated against specific sources or collections of unit of measure. Use of URI to specify the unit of measures file was discussed. This would be novel with regard to providing EdgeX information. Per core working group of 2/17/22 and then again at the monthly architect's meeting of 2/28/22, we may look to use a URI to specify a configuration file to specify UoM in the future. Indeed, URIs may be used (an EdgeX 3.0 consideration) to point to device profiles, configuration files, and other information in the future. This would even allow multiple EdgeX instances to use the same configuration or profile (multiple EdgeX instances using the same URI to use a shared profile for example). However, it was deemed scope creep and too much to do for this first iteration. Initially, this ADR allowed for the UoM to also or alternately to be defined in the standard metadata service configuration file (`configuration.toml'). During the Core WG meeting of 3/3/22, it was decided to simplify the design and strictly limit UoM to a separate configuration file. If future use cases or adopters request inline definition, this can be implemented in a future release.","title":"Future Considerations/Additions/Improvements"},{"location":"design/adr/core/0022-UoM/#consequences","text":"Validation could impact performance. Therefore allowing it to be turned on or off is critical to some use cases. However, it will only impact performance of profile creation/updates (and associated device resources) in core metadata.","title":"Consequences"},{"location":"design/adr/core/0022-UoM/#references","text":"","title":"References"},{"location":"design/adr/core/0022-UoM/#uom-standards","text":"https://ediacademy.com/blog/x12-unit-of-measurement-codes/ https://unece.org/fileadmin/DAM/cefact/recommendations/rec20/rec20_rev3_Annex2e.pdf https://en.wikipedia.org/wiki/Unified_Code_for_Units_of_Measure https://www.ogc.org/standards/sensorthings https://datatracker.ietf.org/doc/html/rfc8428 https://datatracker.ietf.org/doc/html/rfc8428#section-12.1","title":"UoM Standards"},{"location":"design/adr/core/0022-UoM/#uom-tools-and-databases","text":"https://ucum.nlm.nih.gov/ucum-lhc/demo.html https://project-haystack.org/doc/Units https://github.com/fantom-lang/fantom/blob/master/etc/sys/units.txt https://gs1.github.io/UnitConverterUNECERec20/","title":"UoM Tools and Databases"},{"location":"design/adr/device-service/0002-Array-Datatypes/","text":"Array Datatypes Design Status Context Decision Consequences Status Approved Context The current data model does not directly provide for devices which provide array data. Small fixed-length arrays may be handled by defining multiple device resources - one for each element - and aggregating them via a resource command. Other array data may be passed using the Binary type. Neither of these approaches is ideal: the binary data is opaque and any service processing it would need specific knowledge to do so, and aggregation presents the device service implementation with a multiple-read request that could in many cases be better handled by a single request. This design adds arrays of primitives to the range of supported types in EdgeX. It comprises an extension of the DeviceProfile model, and an update to the definition of Reading. Decision DeviceProfile extension The permitted values of the Type field in PropertyValue are extended to include: \"BoolArray\", \"Uint8Array\", \"Uint16Array\", \"Uint32Array\", \"Uint64Array\", \"Int8Array\", Int16Array\", \"Int32Array\", \"Int64Array\", \"Float32Array\", \"Float64Array\" Readings In the API (v1 and v2), Reading.Value is a string representation of the data. If this is maintained, the representation for Array types will follow the JSON array syntax, ie [\"value1\", \"value2\", ...] Consequences Any service which processes Readings will need to be reworked to account for the new Reading type. Device Service considerations The API used for interfacing between device SDKs and devices service implementations contains a local representation of reading values. This will need to be updated in line with the changes outlined here. For C, this will involve an extension of the existing union type. For Go, additional fields may be added to the CommandValue structure. Processing of numeric data in the device service, ie offset , scale etc will not be applied to the values in an array.","title":"Array Datatypes Design"},{"location":"design/adr/device-service/0002-Array-Datatypes/#array-datatypes-design","text":"Status Context Decision Consequences","title":"Array Datatypes Design"},{"location":"design/adr/device-service/0002-Array-Datatypes/#status","text":"Approved","title":"Status"},{"location":"design/adr/device-service/0002-Array-Datatypes/#context","text":"The current data model does not directly provide for devices which provide array data. Small fixed-length arrays may be handled by defining multiple device resources - one for each element - and aggregating them via a resource command. Other array data may be passed using the Binary type. Neither of these approaches is ideal: the binary data is opaque and any service processing it would need specific knowledge to do so, and aggregation presents the device service implementation with a multiple-read request that could in many cases be better handled by a single request. This design adds arrays of primitives to the range of supported types in EdgeX. It comprises an extension of the DeviceProfile model, and an update to the definition of Reading.","title":"Context"},{"location":"design/adr/device-service/0002-Array-Datatypes/#decision","text":"","title":"Decision"},{"location":"design/adr/device-service/0002-Array-Datatypes/#deviceprofile-extension","text":"The permitted values of the Type field in PropertyValue are extended to include: \"BoolArray\", \"Uint8Array\", \"Uint16Array\", \"Uint32Array\", \"Uint64Array\", \"Int8Array\", Int16Array\", \"Int32Array\", \"Int64Array\", \"Float32Array\", \"Float64Array\"","title":"DeviceProfile extension"},{"location":"design/adr/device-service/0002-Array-Datatypes/#readings","text":"In the API (v1 and v2), Reading.Value is a string representation of the data. If this is maintained, the representation for Array types will follow the JSON array syntax, ie [\"value1\", \"value2\", ...]","title":"Readings"},{"location":"design/adr/device-service/0002-Array-Datatypes/#consequences","text":"Any service which processes Readings will need to be reworked to account for the new Reading type.","title":"Consequences"},{"location":"design/adr/device-service/0002-Array-Datatypes/#device-service-considerations","text":"The API used for interfacing between device SDKs and devices service implementations contains a local representation of reading values. This will need to be updated in line with the changes outlined here. For C, this will involve an extension of the existing union type. For Go, additional fields may be added to the CommandValue structure. Processing of numeric data in the device service, ie offset , scale etc will not be applied to the values in an array.","title":"Device Service considerations"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/","text":"Device Service REST API Status Approved Context This ADR details the REST API to be provided by Device Service implementations in EdgeX version 2.x. As such, it supercedes the equivalent sections of the earlier \"Device Service Functional Requirements\" document. These requirements should be implemented as far as possible within the Device Service SDKs, but they also apply to any Device Service implementation. Decision Common endpoints The DS should provide the REST endpoints that are expected of all EdgeX microservices, specifically: config metrics ping version Callback Endpoint Methods callback/device PUT and POST callback/device/name/{name} DELETE callback/profile PUT callback/watcher PUT and POST callback/watcher/name/{name} DELETE parameter meaning {name} the name of the device or watcher These endpoints are used by the Core Metadata service to inform the device service of metadata updates. Endpoints are defined for each of the objects of interest to a device service, ie Devices, Device Profiles and Provision Watchers. On receipt of calls to these endpoints the device service should update its internal state accordingly. Note that the device service does not need to be informed of the creation or deletion of device profiles, as these operations may only occur where no devices are associated with the profile. To avoid stale profile entries the device service should delete a profile from its cache when the last device using it is deleted. Object deletion When an object is deleted, the Metadata service makes a DELETE request to the relevant callback/{type}/name/{name} endpoint. Object creation and updates When an object is created or updated, the Metadata service makes a POST or PUT request respectively to the relevant callback/{type} endpoint. The payload of the request is the new or updated object, ie one of the Device, DeviceProfile or ProvisionWatcher DTOs. Device Endpoint Methods device/name/{name}/{command} GET and PUT parameter meaning {name} the name of the device {command} the command name The command specified must match a deviceCommand or deviceResource name in the device's profile body (for PUT ): An application/json SettingRequest, which is a set of key/value pairs where the keys are valid deviceResource names, and the values provide the command argument for that resource. Example: {\"AHU-TargetTemperature\": \"28.5\", \"AHU-TargetBand\": \"4.0\"} Return code Meaning 200 the command was successful 404 the specified device does not exist, or the command/resource is unknown 405 attempted write to a read-only resource 423 the specified device is locked (admin state) or disabled (operating state) 500 the device driver is unable to process the request response body : A successful GET operation will return a JSON-encoded EventResponse object, which contains one or more Readings. Example: {\"apiVersion\":\"v2\",\"deviceName\":\"Gyro\",\"origin\":1592405201763915855,\"readings\":[{\"deviceName\":\"Gyro\",\"name\":\"Xrotation\",\"value\":\"124\",\"origin\":1592405201763915855,\"valueType\":\"int32\"},{\"deviceName\":\"Gyro\",\"name\":\"Yrotation\",\"value\":\"-54\",\"origin\":1592405201763915855,\"valueType\":\"int32\"},{\"deviceName\":\"Gyro\",\"name\":\"Zrotation\",\"value\":\"122\",\"origin\":1592405201763915855,\"valueType\":\"int32\"}]} This endpoint is used for obtaining readings from a device, and for writing settings to a device. Data formats The values obtained when readings are taken, or used to make settings, are expressed as strings. Type EdgeX types Representation Boolean Bool \"true\" or \"false\" Integer Uint8-Uint64 , Int8-Int64 Numeric string, eg \"-132\" Float Float32 , Float64 Decimal with exponent, eg \"1.234e-5\" String String string Binary Bytes octet array Array BoolArray , Uint8Array-Uint64Array , Int8Array-Int64Array , Float32Array , Float64Array JSON Array, eg \"[\"1\", \"34\", \"-5\"]\" Notes: - The presence of a Binary reading will cause the entire Event to be encoded using CBOR rather than JSON - Arrays of String and Binary data are not supported Readings and Events A Reading represents a value obtained from a deviceResource. It contains the following fields Field name Description deviceName The name of the device profileName The name of the Profile describing the Device resourceName The name of the deviceResource origin A timestamp indicating when the reading was taken value The reading value valueType The type of the data Or for binary Readings, the following fields Field name Description deviceName The name of the device profileName The name of the Profile describing the Device resourceName The name of the deviceResource origin A timestamp indicating when the reading was taken binaryValue The reading value mediaType The MIME type of the data An Event represents the result of a GET command. If the command names a deviceResource, the Event will contain a single Reading. If the command names a deviceCommand, the Event will contain as many Readings as there are deviceResources listed in the deviceCommand. The fields of an Event are as follows: Field name Description deviceName The name of the Device from which the Readings are taken profileName The name of the Profile describing the Device origin The time at which the Event was created readings An array of Readings Query Parameters Calls to the device endpoints may include a Query String in the URL. This may be used to pass parameters relating to the request to the device service. Individual device services may define their own parameters to control specific behaviors. Parameters beginning with the prefix ds- are reserved to the Device SDKs and the following parameters are defined for GET requests: Parameter Valid Values Default Meaning ds-pushevent \"yes\" or \"no\" \"no\" If set to yes, a successful GET will result in an event being pushed to the EdgeX system ds-returnevent \"yes\" or \"no\" \"yes\" If set to no, there will be no Event returned in the http response Device States A Device in EdgeX has two states associated with it: the Administrative state and the Operational state. The Administrative state may be set to LOCKED (normally UNLOCKED ) to block access to the device for administrative reasons. The Operational state may be set to DOWN (normally UP ) to indicate that the device is not currently working. In either case access to the device via this endpoint will be denied and HTTP 423 (\"Locked\") will be returned. Data Transformations A number of simple data transformations may be defined in the deviceResource. The table below shows these transformations in the order in which they are applied to outgoing data, ie Readings. The transformations are inverted and applied in reverse order for incoming data. Transform Applicable reading types Effect mask Integers The reading is masked (bitwise-and operation) with the specified value. shift Integers The reading is bit-shifted by the specified value. Positive values indicate right-shift, negative for left. base Integers and Floats The reading is replaced by the specified value raised to the power of the reading. scale Integers and Floats The reading is multiplied by the specified value. offset Integers and Floats The reading is increased by the specified value. The operation of the mask transform on incoming data (a setting) is that the value to be set on the resource is the existing value bitwise-anded with the complement of the mask, bitwise-ored with the value specified in the request. ie, new-value = (current-value & !mask) | request-value The combination of mask and shift can therefore be used to access data contained in a subdivision of an octet. It is possible that following the application of the specified transformations, a value may exceed the range that may be represented by its type. Should this occur on a set operation, a suitable error should be logged and returned, along with the Bad Request http code 400. If it occurs as part of a get operation, the Reading's value should be set to the String \"overflow\" and its valueType to String . Assertions and Mappings Assertions are another attribute in a device resource's PropertyValue, which specify a string which the reading value is compared against. If the comparison fails, then the http request returns a string of the form \"Assertion failed for device resource: \\ , with value: \\ \" , this also has a side-effect of setting the device operatingstate to DISABLED . A 500 status code is also returned. Note that the error response and status code should be returned regardless of the ds-returnevent setting. Assertions are also checked where an event is being generated due to an AutoEvent, or asynchronous readings are pushed. In these cases if the assertion is triggered, an error should be logged and the operating state should be set as above. Assertions are not checked for settings, only for readings. Mappings may be defined in a deviceCommand. These allow Readings of string type to be remapped. Mappings are applied after assertions are checked, and are the final transformation before Readings are created. Mappings are also applied, but in reverse, to settings ( PUT request data). lastConnected timestamp Each Device has as part of its metadata a timestamp named lastConnected , this indicates the most recent occasion when the device was successfully interacted with. The device service should update this timestamp every time a GET or PUT operation succeeds, unless it has been configured not to do so (eg for performance reasons). Discovery Endpoint Methods discovery POST A call to this endpoint triggers the device discovery process, if enabled. See Discovery Design for details. Consequences Changes from v1.x API The callback endpoint is split according to the type of object being updated Callbacks for new and updated objects take the object in the request body The device/all form is removed GET requests take parameters controlling what is to be done with resulting Events, and the default behavior does not send the Event to core-data References OpenAPI definition of v2 API : https://github.com/edgexfoundry/device-sdk-go/blob/master/openapi/v2/device-sdk.yaml Device Service Functional Requirements (Geneva) : https://wiki.edgexfoundry.org/download/attachments/329488/edgex-device-service-requirements-v11.pdf?version=1&modificationDate=1591621033000&api=v2","title":"Device Service REST API"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#device-service-rest-api","text":"","title":"Device Service REST API"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#status","text":"Approved","title":"Status"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#context","text":"This ADR details the REST API to be provided by Device Service implementations in EdgeX version 2.x. As such, it supercedes the equivalent sections of the earlier \"Device Service Functional Requirements\" document. These requirements should be implemented as far as possible within the Device Service SDKs, but they also apply to any Device Service implementation.","title":"Context"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#decision","text":"","title":"Decision"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#common-endpoints","text":"The DS should provide the REST endpoints that are expected of all EdgeX microservices, specifically: config metrics ping version","title":"Common endpoints"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#callback","text":"Endpoint Methods callback/device PUT and POST callback/device/name/{name} DELETE callback/profile PUT callback/watcher PUT and POST callback/watcher/name/{name} DELETE parameter meaning {name} the name of the device or watcher These endpoints are used by the Core Metadata service to inform the device service of metadata updates. Endpoints are defined for each of the objects of interest to a device service, ie Devices, Device Profiles and Provision Watchers. On receipt of calls to these endpoints the device service should update its internal state accordingly. Note that the device service does not need to be informed of the creation or deletion of device profiles, as these operations may only occur where no devices are associated with the profile. To avoid stale profile entries the device service should delete a profile from its cache when the last device using it is deleted.","title":"Callback"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#object-deletion","text":"When an object is deleted, the Metadata service makes a DELETE request to the relevant callback/{type}/name/{name} endpoint.","title":"Object deletion"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#object-creation-and-updates","text":"When an object is created or updated, the Metadata service makes a POST or PUT request respectively to the relevant callback/{type} endpoint. The payload of the request is the new or updated object, ie one of the Device, DeviceProfile or ProvisionWatcher DTOs.","title":"Object creation and updates"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#device","text":"Endpoint Methods device/name/{name}/{command} GET and PUT parameter meaning {name} the name of the device {command} the command name The command specified must match a deviceCommand or deviceResource name in the device's profile body (for PUT ): An application/json SettingRequest, which is a set of key/value pairs where the keys are valid deviceResource names, and the values provide the command argument for that resource. Example: {\"AHU-TargetTemperature\": \"28.5\", \"AHU-TargetBand\": \"4.0\"} Return code Meaning 200 the command was successful 404 the specified device does not exist, or the command/resource is unknown 405 attempted write to a read-only resource 423 the specified device is locked (admin state) or disabled (operating state) 500 the device driver is unable to process the request response body : A successful GET operation will return a JSON-encoded EventResponse object, which contains one or more Readings. Example: {\"apiVersion\":\"v2\",\"deviceName\":\"Gyro\",\"origin\":1592405201763915855,\"readings\":[{\"deviceName\":\"Gyro\",\"name\":\"Xrotation\",\"value\":\"124\",\"origin\":1592405201763915855,\"valueType\":\"int32\"},{\"deviceName\":\"Gyro\",\"name\":\"Yrotation\",\"value\":\"-54\",\"origin\":1592405201763915855,\"valueType\":\"int32\"},{\"deviceName\":\"Gyro\",\"name\":\"Zrotation\",\"value\":\"122\",\"origin\":1592405201763915855,\"valueType\":\"int32\"}]} This endpoint is used for obtaining readings from a device, and for writing settings to a device.","title":"Device"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#data-formats","text":"The values obtained when readings are taken, or used to make settings, are expressed as strings. Type EdgeX types Representation Boolean Bool \"true\" or \"false\" Integer Uint8-Uint64 , Int8-Int64 Numeric string, eg \"-132\" Float Float32 , Float64 Decimal with exponent, eg \"1.234e-5\" String String string Binary Bytes octet array Array BoolArray , Uint8Array-Uint64Array , Int8Array-Int64Array , Float32Array , Float64Array JSON Array, eg \"[\"1\", \"34\", \"-5\"]\" Notes: - The presence of a Binary reading will cause the entire Event to be encoded using CBOR rather than JSON - Arrays of String and Binary data are not supported","title":"Data formats"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#readings-and-events","text":"A Reading represents a value obtained from a deviceResource. It contains the following fields Field name Description deviceName The name of the device profileName The name of the Profile describing the Device resourceName The name of the deviceResource origin A timestamp indicating when the reading was taken value The reading value valueType The type of the data Or for binary Readings, the following fields Field name Description deviceName The name of the device profileName The name of the Profile describing the Device resourceName The name of the deviceResource origin A timestamp indicating when the reading was taken binaryValue The reading value mediaType The MIME type of the data An Event represents the result of a GET command. If the command names a deviceResource, the Event will contain a single Reading. If the command names a deviceCommand, the Event will contain as many Readings as there are deviceResources listed in the deviceCommand. The fields of an Event are as follows: Field name Description deviceName The name of the Device from which the Readings are taken profileName The name of the Profile describing the Device origin The time at which the Event was created readings An array of Readings","title":"Readings and Events"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#query-parameters","text":"Calls to the device endpoints may include a Query String in the URL. This may be used to pass parameters relating to the request to the device service. Individual device services may define their own parameters to control specific behaviors. Parameters beginning with the prefix ds- are reserved to the Device SDKs and the following parameters are defined for GET requests: Parameter Valid Values Default Meaning ds-pushevent \"yes\" or \"no\" \"no\" If set to yes, a successful GET will result in an event being pushed to the EdgeX system ds-returnevent \"yes\" or \"no\" \"yes\" If set to no, there will be no Event returned in the http response","title":"Query Parameters"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#device-states","text":"A Device in EdgeX has two states associated with it: the Administrative state and the Operational state. The Administrative state may be set to LOCKED (normally UNLOCKED ) to block access to the device for administrative reasons. The Operational state may be set to DOWN (normally UP ) to indicate that the device is not currently working. In either case access to the device via this endpoint will be denied and HTTP 423 (\"Locked\") will be returned.","title":"Device States"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#data-transformations","text":"A number of simple data transformations may be defined in the deviceResource. The table below shows these transformations in the order in which they are applied to outgoing data, ie Readings. The transformations are inverted and applied in reverse order for incoming data. Transform Applicable reading types Effect mask Integers The reading is masked (bitwise-and operation) with the specified value. shift Integers The reading is bit-shifted by the specified value. Positive values indicate right-shift, negative for left. base Integers and Floats The reading is replaced by the specified value raised to the power of the reading. scale Integers and Floats The reading is multiplied by the specified value. offset Integers and Floats The reading is increased by the specified value. The operation of the mask transform on incoming data (a setting) is that the value to be set on the resource is the existing value bitwise-anded with the complement of the mask, bitwise-ored with the value specified in the request. ie, new-value = (current-value & !mask) | request-value The combination of mask and shift can therefore be used to access data contained in a subdivision of an octet. It is possible that following the application of the specified transformations, a value may exceed the range that may be represented by its type. Should this occur on a set operation, a suitable error should be logged and returned, along with the Bad Request http code 400. If it occurs as part of a get operation, the Reading's value should be set to the String \"overflow\" and its valueType to String .","title":"Data Transformations"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#assertions-and-mappings","text":"Assertions are another attribute in a device resource's PropertyValue, which specify a string which the reading value is compared against. If the comparison fails, then the http request returns a string of the form \"Assertion failed for device resource: \\ , with value: \\ \" , this also has a side-effect of setting the device operatingstate to DISABLED . A 500 status code is also returned. Note that the error response and status code should be returned regardless of the ds-returnevent setting. Assertions are also checked where an event is being generated due to an AutoEvent, or asynchronous readings are pushed. In these cases if the assertion is triggered, an error should be logged and the operating state should be set as above. Assertions are not checked for settings, only for readings. Mappings may be defined in a deviceCommand. These allow Readings of string type to be remapped. Mappings are applied after assertions are checked, and are the final transformation before Readings are created. Mappings are also applied, but in reverse, to settings ( PUT request data).","title":"Assertions and Mappings"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#lastconnected-timestamp","text":"Each Device has as part of its metadata a timestamp named lastConnected , this indicates the most recent occasion when the device was successfully interacted with. The device service should update this timestamp every time a GET or PUT operation succeeds, unless it has been configured not to do so (eg for performance reasons).","title":"lastConnected timestamp"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#discovery","text":"Endpoint Methods discovery POST A call to this endpoint triggers the device discovery process, if enabled. See Discovery Design for details.","title":"Discovery"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#consequences","text":"","title":"Consequences"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#changes-from-v1x-api","text":"The callback endpoint is split according to the type of object being updated Callbacks for new and updated objects take the object in the request body The device/all form is removed GET requests take parameters controlling what is to be done with resulting Events, and the default behavior does not send the Event to core-data","title":"Changes from v1.x API"},{"location":"design/adr/device-service/0011-DeviceService-Rest-API/#references","text":"OpenAPI definition of v2 API : https://github.com/edgexfoundry/device-sdk-go/blob/master/openapi/v2/device-sdk.yaml Device Service Functional Requirements (Geneva) : https://wiki.edgexfoundry.org/download/attachments/329488/edgex-device-service-requirements-v11.pdf?version=1&modificationDate=1591621033000&api=v2","title":"References"},{"location":"design/adr/device-service/0012-DeviceService-Filters/","text":"Device Service Filters Status Approved (by TSC vote on 3/15/21) design (initially) for Hanoi - but now being considered for Ireland implementation TBD (desired feature targeted for Ireland or Jakarata) Context In EdgeX today, sensor/device data collected can be \"filtered\" by application services before being exported or sent to some north side application or system. Built-in application service functions (available through the app services SDK) allow EdgeX event/reading objects to be filtered by device name or by device ResourceName. That is, event/readings can be filtered by: which device sent the event/reading (as determined by the Event device property). the classification or origin (such as temperature or humidity) of data produced by the device as determined by the Reading's name property (which used to be the value descriptor and now refers to the device ResourceName). Two Levels of Device Service Filtering There are potentially two places where \"filtering\" in a device service could be useful. One (Sensor Data Filter) - after the device service has communicated with the sensor or device to get sensor values (but before the service creates Event/Reading objects and pushes those to core data). A sensor data filter would allow the device service to essentially ignore some of the raw sensed data. This would allow for some device service optimization in that the device service would not have perform type transformations and creation of event/reading objects if the data can be eliminated at this early stage. This first level filtering would, if put in place , likely occur in code associated with the read command gets done by the ProtocolDriver . Two (Reading Filter) - after the sensor data has been collected and read and put into Event/Reading objects, there is a desire to filter some of the Readings based on the Reading values or Reading name (which is the device ResourceName) or some combination of value and name. At this time, this design only addresses the need for the second filter (Reading Filter) . At the time of this writing, no applicable use case has yet to be defined to warrant the Sensor Data Filter. Reading Filters Reading filters will allow, not unlike application service filter functions today, to have Readings in an Event to be removed if: the value was outside or inside some range, or the value was greater than, less than or equal to some value based on the Reading value (numeric) of a Reading outside a specified range (min/max) described in the service configuration. Thus avoiding sending in outlier or jittery data Readings that could negatively effect analytics. Future scope: based on the Reading value (numeric) equal to or near (with in some specified range) the last reading. This allows a device service to reduce sending in Event/Readings that do not represent any significant change. This differs from the already implemented onChangeOnly in that it is filtering Readings within a specified degree of change. Note: this feature would require caching of readings which has not fully been implemented in the SDK. The existing mechanism for autoevents provides a partial cache. Added for future reference, but this feature would not be accomplished in the initial implementation; requiring extra design work on caching to be implemented. the value was the same as some or not the same as some specified value or values (for strings, boolean and other non-numeric values) the value matches a pattern (glob and/or regex) when the value is a string. the name (the device ResourceName) matched a particular value; in other words match temperature or humidity as example device resources. Unlike application services, there is not a need to filter on a device name (or identifier). Simply disable the device in the device service if all Event/Readings are to be stopped for the device. In the case that all Readings of an Event are filtered, it is assumed the entire Event is deemed to be worthless and not sent to core data by the device service. If only some Readings from and Event are filtered, the Event minus the filtered Readings would be sent to core data. The filter behaves the same whether the collection of Readings and Events is triggered by a scheduled collection of data from the underlying sensor/device or triggered by a command request (as from the command service). Therefore, the call for a command request still results in a successful status code and a return of no results (or partial results) if the filter causes all or some of the readings to be removed. Design / Architecture A new function interface shall be defined that, when implemented, performs a Reading Filter operation. A ReadingFilter function would take a parameter (an Event containing readings), check whether the Readings of the Event match on the filtering configuration (see below) and if they do then remove them from the Event . The ReadingFilter function would return the Event object (minus filtered Readings ) or nil if the Event held no more Readings . Pseudo code for the generic function is provided below. The results returned will include a boolean to indicate whether any Reading objects were removed from the Event (allowing the receiver to know if some were filtered from the original list). func ( f Filter ) ReadingFilter ( lc logger . LoggingClient , event * models . Event ) ( * models . Event , error , boolean ) { // depending on impl; filtering for values in/out of a range, >, <, =, same, not same, from a particular name (device resource), etc. // The boolean will indicate whether any Readings were filtered from the Event. if ( len ( event . Reading )) > 0 ) if ( len filteredReadings > 0 ) return event , true else return event , false else return nil , true } Based on current needs/use cases, implementations of the function interface could include the following filter functions: func ( f Filter ) FilterByValue ( lc logger . LoggingClient , event * models . Event ) ( * models . Event , error , boolean ) {} func ( f Filter ) FilterByResourceNamesMatch ( lc logger . LoggingClient , event * models . Event ) ( * models . Event , error , boolean ) {} Note The app functions SDK comes with FilterByDeviceName and FilterByResourceName functions today. The FilterByResourceName would behave similarly to FilterByResourceNameMatch. The Filter structure houses the configuration parameters for which the filter functions work and filter on. Note The app functions SDK uses a fairly simple Filter structure. type Filter struct { FilterValues [] string FilterOut bool } Given the collection of filter operations (in range, out of range, equal or not equal), the following structure is proposed: type Filter struct { FilterValues [] string TargetResourceName string FilterOp string // enum of in (in range inclusive), out (outside a range exclusive), eq (equal) or ne (not equal) } Examples use of the Filter structure to specify filtering: Filter { FilterValues : { 10 , 20 }, \"Int64\" , FilterOp : \"in\" } // filter for those Int64 readings with values between 10-20 inclusive Filter { FilterValues : { 10 , 20 }, \"Int64\" , FilterOp : \"out\" } // filter for those Int64 readings with values outside of 10-20. Filter { FilterValues : { 8 , 10 , 12 }, \"Int64\" , FilterOp : \"eq\" } //filter for those Int64 readings with values of 8, 10, or 12. Filter { FilterValues : { 8 , 10 }, \"Int64\" , FilterOp : \"ne\" } //filter for those Int64 readings with values not equal to 8 or 10 Filter { FilterValues : { \"Int32\" , \"Int64\" }, nil , FilterOp : \"eq\" } //filter to be used with FilterByResourceNameMatch. Filter for resource names of Int32 or Int64. Filter { FilterValues : { \"Int32\" }, nil , FilterOp : \"ne\" } //filter to be used with FilterByResourceNameMatch. Filter for resource names not equal to (excluding) Int32. A NewFilter function creates, initializes and returns a new instance of the filter based on the configuration provided. func NewReadingNameFilter ( filterValues [] string , filterOp string ) Filter { return Filter { FilterValues : filterValues , TargetResourceName string , FilterOp : filterOp } } Sharing filter functions If one were to explore the filtering functions in the app functions SDK filter.go (both FilterByDeviceName and FilterByValueDescriptor ), the filters operate on the Event model object and return the same objects ( Event or nil). Ideally, since both app services and device services generally share the same interface model (from go-mod-core-contracts ), it would be the desire to share the same filter functions functions between SDKs and associated services. Decisions on how to do this in Go - whether by shared module for example - is left as a future release design and implementation task - and as the need for common filter functions across device services and application services are identified in use cases. C needs are likely to be handled in the SDK directly. Additional Design Considerations As Device Services do not have the concept of a functions pipeline like application services do, consideration must be given as to how and where to: provide configuration to specify which filter functions to invoke create the filter invoke the filtering functions At this time, custom filters will not be supported as the custom filters would not be known by the SDK and therefore could not be specified in configuration. This is consistent with the app functions SDK and filtering. Function Inflection Point It is precisely after the convert to Event/Reading objects (after the async readings are assembled into events) and before returning that result in common.SendEvent (in utils.go) function that the device service should invoke the required filter functions. In the existing V1 implementation of the device-sdk-go, commands, async readings, and auto-events all call the function common.SendEvent() . Note: V2 implementation will require some re-evaluation of this inflection point. Where possible, the implementation should locate a single point of inflection if possible. In the C SDK, it is likely that the filters will be called before conversion to Event/Reading objects - they will operate on commandresult objects (equivalent to CommandValues). The order in which functions are called is important when more than one filter is provided. The order that functions are called should be reflected in the order listed in the configuration of the filters. Events containing binary values (event.HasBinaryValue), will not be filtered. Future releases may include binary value filters. Setting Filter Function and Configuration When filter functions are shared (or appear to be doing the same type of work) between SDKs, the configuration of the similar filter functions should also look similar. The app functions SDK configuration model for filters should therefore be followed. While device services do not have pipelines, the inclusion and configuration of filters for device services should have a similar look (to provide symmetry with app services). The configuration has to provide the functions required and parameters to make the functions work - even though the association to a pipeline is not required. Below is the common app service configuration as it relates to filters: [Writable.Pipeline] ExecutionOrder = \"FilterByDeviceName, TransformToXML, SetOutputData\" [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] DeviceNames = \"Random-Float-Device,Random-Integer-Device\" FilterOut = \"false\" Suggested and hypothetical configuration for the device service reading filters should look something like that below. [Writable.Filters] # filter readings where resource name equals Int32 ExecutionOrder = \"FilterByResourceNamesMatch, FilterByValue\" [Writable.Filter.Functions.FilterByResourceNamesMatch] [Writable.Filter.Functions.FilterByResourceNamesMatch.Parameters] FilterValues = \"Int32\" FilterOps = \"eq\" # filter readings where the Int64 readings (resource name) is Int64 and the values are between 10 and 20 [Writable.Filter.Functions.FilterByValue] [Writable.Filter.Functions.FilterByValue.Parameters] TargetResourceName = \"Int64\" FilterValues = { 10 , 20 } FilterOp = \"in\" Decision To be determined Consequences This design does not take into account potential changes found with the V2 API. References","title":"Device Service Filters"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#device-service-filters","text":"","title":"Device Service Filters"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#status","text":"Approved (by TSC vote on 3/15/21) design (initially) for Hanoi - but now being considered for Ireland implementation TBD (desired feature targeted for Ireland or Jakarata)","title":"Status"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#context","text":"In EdgeX today, sensor/device data collected can be \"filtered\" by application services before being exported or sent to some north side application or system. Built-in application service functions (available through the app services SDK) allow EdgeX event/reading objects to be filtered by device name or by device ResourceName. That is, event/readings can be filtered by: which device sent the event/reading (as determined by the Event device property). the classification or origin (such as temperature or humidity) of data produced by the device as determined by the Reading's name property (which used to be the value descriptor and now refers to the device ResourceName).","title":"Context"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#two-levels-of-device-service-filtering","text":"There are potentially two places where \"filtering\" in a device service could be useful. One (Sensor Data Filter) - after the device service has communicated with the sensor or device to get sensor values (but before the service creates Event/Reading objects and pushes those to core data). A sensor data filter would allow the device service to essentially ignore some of the raw sensed data. This would allow for some device service optimization in that the device service would not have perform type transformations and creation of event/reading objects if the data can be eliminated at this early stage. This first level filtering would, if put in place , likely occur in code associated with the read command gets done by the ProtocolDriver . Two (Reading Filter) - after the sensor data has been collected and read and put into Event/Reading objects, there is a desire to filter some of the Readings based on the Reading values or Reading name (which is the device ResourceName) or some combination of value and name. At this time, this design only addresses the need for the second filter (Reading Filter) . At the time of this writing, no applicable use case has yet to be defined to warrant the Sensor Data Filter.","title":"Two Levels of Device Service Filtering"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#reading-filters","text":"Reading filters will allow, not unlike application service filter functions today, to have Readings in an Event to be removed if: the value was outside or inside some range, or the value was greater than, less than or equal to some value based on the Reading value (numeric) of a Reading outside a specified range (min/max) described in the service configuration. Thus avoiding sending in outlier or jittery data Readings that could negatively effect analytics. Future scope: based on the Reading value (numeric) equal to or near (with in some specified range) the last reading. This allows a device service to reduce sending in Event/Readings that do not represent any significant change. This differs from the already implemented onChangeOnly in that it is filtering Readings within a specified degree of change. Note: this feature would require caching of readings which has not fully been implemented in the SDK. The existing mechanism for autoevents provides a partial cache. Added for future reference, but this feature would not be accomplished in the initial implementation; requiring extra design work on caching to be implemented. the value was the same as some or not the same as some specified value or values (for strings, boolean and other non-numeric values) the value matches a pattern (glob and/or regex) when the value is a string. the name (the device ResourceName) matched a particular value; in other words match temperature or humidity as example device resources. Unlike application services, there is not a need to filter on a device name (or identifier). Simply disable the device in the device service if all Event/Readings are to be stopped for the device. In the case that all Readings of an Event are filtered, it is assumed the entire Event is deemed to be worthless and not sent to core data by the device service. If only some Readings from and Event are filtered, the Event minus the filtered Readings would be sent to core data. The filter behaves the same whether the collection of Readings and Events is triggered by a scheduled collection of data from the underlying sensor/device or triggered by a command request (as from the command service). Therefore, the call for a command request still results in a successful status code and a return of no results (or partial results) if the filter causes all or some of the readings to be removed.","title":"Reading Filters"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#design-architecture","text":"A new function interface shall be defined that, when implemented, performs a Reading Filter operation. A ReadingFilter function would take a parameter (an Event containing readings), check whether the Readings of the Event match on the filtering configuration (see below) and if they do then remove them from the Event . The ReadingFilter function would return the Event object (minus filtered Readings ) or nil if the Event held no more Readings . Pseudo code for the generic function is provided below. The results returned will include a boolean to indicate whether any Reading objects were removed from the Event (allowing the receiver to know if some were filtered from the original list). func ( f Filter ) ReadingFilter ( lc logger . LoggingClient , event * models . Event ) ( * models . Event , error , boolean ) { // depending on impl; filtering for values in/out of a range, >, <, =, same, not same, from a particular name (device resource), etc. // The boolean will indicate whether any Readings were filtered from the Event. if ( len ( event . Reading )) > 0 ) if ( len filteredReadings > 0 ) return event , true else return event , false else return nil , true } Based on current needs/use cases, implementations of the function interface could include the following filter functions: func ( f Filter ) FilterByValue ( lc logger . LoggingClient , event * models . Event ) ( * models . Event , error , boolean ) {} func ( f Filter ) FilterByResourceNamesMatch ( lc logger . LoggingClient , event * models . Event ) ( * models . Event , error , boolean ) {} Note The app functions SDK comes with FilterByDeviceName and FilterByResourceName functions today. The FilterByResourceName would behave similarly to FilterByResourceNameMatch. The Filter structure houses the configuration parameters for which the filter functions work and filter on. Note The app functions SDK uses a fairly simple Filter structure. type Filter struct { FilterValues [] string FilterOut bool } Given the collection of filter operations (in range, out of range, equal or not equal), the following structure is proposed: type Filter struct { FilterValues [] string TargetResourceName string FilterOp string // enum of in (in range inclusive), out (outside a range exclusive), eq (equal) or ne (not equal) } Examples use of the Filter structure to specify filtering: Filter { FilterValues : { 10 , 20 }, \"Int64\" , FilterOp : \"in\" } // filter for those Int64 readings with values between 10-20 inclusive Filter { FilterValues : { 10 , 20 }, \"Int64\" , FilterOp : \"out\" } // filter for those Int64 readings with values outside of 10-20. Filter { FilterValues : { 8 , 10 , 12 }, \"Int64\" , FilterOp : \"eq\" } //filter for those Int64 readings with values of 8, 10, or 12. Filter { FilterValues : { 8 , 10 }, \"Int64\" , FilterOp : \"ne\" } //filter for those Int64 readings with values not equal to 8 or 10 Filter { FilterValues : { \"Int32\" , \"Int64\" }, nil , FilterOp : \"eq\" } //filter to be used with FilterByResourceNameMatch. Filter for resource names of Int32 or Int64. Filter { FilterValues : { \"Int32\" }, nil , FilterOp : \"ne\" } //filter to be used with FilterByResourceNameMatch. Filter for resource names not equal to (excluding) Int32. A NewFilter function creates, initializes and returns a new instance of the filter based on the configuration provided. func NewReadingNameFilter ( filterValues [] string , filterOp string ) Filter { return Filter { FilterValues : filterValues , TargetResourceName string , FilterOp : filterOp } }","title":"Design / Architecture"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#sharing-filter-functions","text":"If one were to explore the filtering functions in the app functions SDK filter.go (both FilterByDeviceName and FilterByValueDescriptor ), the filters operate on the Event model object and return the same objects ( Event or nil). Ideally, since both app services and device services generally share the same interface model (from go-mod-core-contracts ), it would be the desire to share the same filter functions functions between SDKs and associated services. Decisions on how to do this in Go - whether by shared module for example - is left as a future release design and implementation task - and as the need for common filter functions across device services and application services are identified in use cases. C needs are likely to be handled in the SDK directly.","title":"Sharing filter functions"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#additional-design-considerations","text":"As Device Services do not have the concept of a functions pipeline like application services do, consideration must be given as to how and where to: provide configuration to specify which filter functions to invoke create the filter invoke the filtering functions At this time, custom filters will not be supported as the custom filters would not be known by the SDK and therefore could not be specified in configuration. This is consistent with the app functions SDK and filtering.","title":"Additional Design Considerations"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#function-inflection-point","text":"It is precisely after the convert to Event/Reading objects (after the async readings are assembled into events) and before returning that result in common.SendEvent (in utils.go) function that the device service should invoke the required filter functions. In the existing V1 implementation of the device-sdk-go, commands, async readings, and auto-events all call the function common.SendEvent() . Note: V2 implementation will require some re-evaluation of this inflection point. Where possible, the implementation should locate a single point of inflection if possible. In the C SDK, it is likely that the filters will be called before conversion to Event/Reading objects - they will operate on commandresult objects (equivalent to CommandValues). The order in which functions are called is important when more than one filter is provided. The order that functions are called should be reflected in the order listed in the configuration of the filters. Events containing binary values (event.HasBinaryValue), will not be filtered. Future releases may include binary value filters.","title":"Function Inflection Point"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#setting-filter-function-and-configuration","text":"When filter functions are shared (or appear to be doing the same type of work) between SDKs, the configuration of the similar filter functions should also look similar. The app functions SDK configuration model for filters should therefore be followed. While device services do not have pipelines, the inclusion and configuration of filters for device services should have a similar look (to provide symmetry with app services). The configuration has to provide the functions required and parameters to make the functions work - even though the association to a pipeline is not required. Below is the common app service configuration as it relates to filters: [Writable.Pipeline] ExecutionOrder = \"FilterByDeviceName, TransformToXML, SetOutputData\" [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] DeviceNames = \"Random-Float-Device,Random-Integer-Device\" FilterOut = \"false\" Suggested and hypothetical configuration for the device service reading filters should look something like that below. [Writable.Filters] # filter readings where resource name equals Int32 ExecutionOrder = \"FilterByResourceNamesMatch, FilterByValue\" [Writable.Filter.Functions.FilterByResourceNamesMatch] [Writable.Filter.Functions.FilterByResourceNamesMatch.Parameters] FilterValues = \"Int32\" FilterOps = \"eq\" # filter readings where the Int64 readings (resource name) is Int64 and the values are between 10 and 20 [Writable.Filter.Functions.FilterByValue] [Writable.Filter.Functions.FilterByValue.Parameters] TargetResourceName = \"Int64\" FilterValues = { 10 , 20 } FilterOp = \"in\"","title":"Setting Filter Function and Configuration"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#decision","text":"To be determined","title":"Decision"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#consequences","text":"This design does not take into account potential changes found with the V2 API.","title":"Consequences"},{"location":"design/adr/device-service/0012-DeviceService-Filters/#references","text":"","title":"References"},{"location":"design/adr/devops/0007-Release-Automation/","text":"Release Automation Status Approved by TSC 04/08/2020 Context EdgeX Foundry is a framework composed of microservices to ease development of IoT/Edge solutions. With the framework getting richer, project growth, the number of artifacts to be released has increased. This proposal outlines a method for automating the release process for the base artifacts. Requirements Release Artifact Definition For the scope of Hanoi release artifact types are defined as: GitHub tags in the repositories. Docker images in our Nexus repository and Docker hub. *Snaps in the Snapcraft store. This list is likely to expand in future releases. *The building and publishing of snaps was removed from community scope in September 2020 and is managed outside the community by Canonical. General Requirements As the EdgeX Release Czar I gathered the following requirements for automating this part of the release. The release automation needs a manual trigger to be triggered by the EdgeX Release Czar or the Linux Foundation Release Engineers. The goal of this automation is to have a \"push button\" release mechanism to reduce human error in our release process. Release artifacts can come from one or more GitHub repositories at a time. GitHub repositories can have one or more release artifact types to release. GitHub repositories can have one or more artifacts of a specific type to release. (For example: The mono repository, edgex-go, has more than 20 docker images to release.) GitHub repositories may be released at different times. (For example: Application and Device service repositories can be released on a different day than the Core services in the mono repository.) Ability to track multiple release streams for the project. An audit trail history for releases. Location The code that will manage the release automation for EdgeX Foundry will live in a repository called cd-management . This repository will have a branch named release that will track the releases of artifacts off the main branch of the EdgeX Foundry repositories. Multiple Release Streams EdgeX Foundry has this idea of multple release streams that basically coincides with different named branches in GitHub. For the majority of the main releases we will be targeting those off the main branch. In our cd-management repository we will have a release branch that will track the main branches EdgeX repositories. In the future we will mark a specific release for long term support (LTS). When this happens we will have to branch off main in the EdgeX repositories and create a separate release stream for the LTS. The suggestion at that point will be to branch off the release branch in cd-management as well and use this new release branch to track the LTS branches in the EdgeX repositories. Release Flow Go Modules, Device and Application SDKs During Development Go modules, Application and Device SDKs only release a GitHub tag as their release. Go modules, Application and Device SDKs are set up to automatically increment a developmental version tag on each merge to main . (IE: 1.0.0-dev.1 -> 1.0.0-dev.2) Release The release automation for Go Modules, Device and Application SDKs is used to set the final release version git tag. (IE: 1.0.0-dev.X -> 1.0.0) For each release, the Go Modules, Device and Application SDK repositories will be tagged with the release version. Core Services (Including Security and System Management services), Application Services, Device Services and Supporting Docker Images During Development For the Core Services, Application Services, Device Services and Supporting Docker Images we release Github tags and docker images. On every merge to the main branch we will do the following; increment a developmental version tag on GitHub, (IE: 1.0.0-dev.1 -> 1.0.0-dev.2), stage docker images in our Nexus repository (docker.staging). Release The release automation will need to do the following: Set version tag on GitHub. (IE: 1.0.0-dev.X -> 1.0.0) Promote docker images in our Nexus repository from docker.staging to docker.release and public Docker hub. Supporting Assets (e.g. edgex-cli) During Development For supporting release assets (e.g. edgex-cli) we release GitHub tags on every merge to the main branch. For every merge to main we will do the following; increment a developmental version tag on GitHub, (IE: 1.0.0-dev.1 -> 1.0.0-dev.2) and store the build artifacts in our Nexus repository. Release For EdgeX releases the release automation will set the final release version by creating a git tag (e.g. 1.0.0-dev.X -> 1.0.0) and produce a Github Release containing the binary assets targeted for release.","title":"Release Automation"},{"location":"design/adr/devops/0007-Release-Automation/#release-automation","text":"","title":"Release Automation"},{"location":"design/adr/devops/0007-Release-Automation/#status","text":"Approved by TSC 04/08/2020","title":"Status"},{"location":"design/adr/devops/0007-Release-Automation/#context","text":"EdgeX Foundry is a framework composed of microservices to ease development of IoT/Edge solutions. With the framework getting richer, project growth, the number of artifacts to be released has increased. This proposal outlines a method for automating the release process for the base artifacts.","title":"Context"},{"location":"design/adr/devops/0007-Release-Automation/#requirements","text":"","title":"Requirements"},{"location":"design/adr/devops/0007-Release-Automation/#release-artifact-definition","text":"For the scope of Hanoi release artifact types are defined as: GitHub tags in the repositories. Docker images in our Nexus repository and Docker hub. *Snaps in the Snapcraft store. This list is likely to expand in future releases. *The building and publishing of snaps was removed from community scope in September 2020 and is managed outside the community by Canonical.","title":"Release Artifact Definition"},{"location":"design/adr/devops/0007-Release-Automation/#general-requirements","text":"As the EdgeX Release Czar I gathered the following requirements for automating this part of the release. The release automation needs a manual trigger to be triggered by the EdgeX Release Czar or the Linux Foundation Release Engineers. The goal of this automation is to have a \"push button\" release mechanism to reduce human error in our release process. Release artifacts can come from one or more GitHub repositories at a time. GitHub repositories can have one or more release artifact types to release. GitHub repositories can have one or more artifacts of a specific type to release. (For example: The mono repository, edgex-go, has more than 20 docker images to release.) GitHub repositories may be released at different times. (For example: Application and Device service repositories can be released on a different day than the Core services in the mono repository.) Ability to track multiple release streams for the project. An audit trail history for releases.","title":"General Requirements"},{"location":"design/adr/devops/0007-Release-Automation/#location","text":"The code that will manage the release automation for EdgeX Foundry will live in a repository called cd-management . This repository will have a branch named release that will track the releases of artifacts off the main branch of the EdgeX Foundry repositories.","title":"Location"},{"location":"design/adr/devops/0007-Release-Automation/#multiple-release-streams","text":"EdgeX Foundry has this idea of multple release streams that basically coincides with different named branches in GitHub. For the majority of the main releases we will be targeting those off the main branch. In our cd-management repository we will have a release branch that will track the main branches EdgeX repositories. In the future we will mark a specific release for long term support (LTS). When this happens we will have to branch off main in the EdgeX repositories and create a separate release stream for the LTS. The suggestion at that point will be to branch off the release branch in cd-management as well and use this new release branch to track the LTS branches in the EdgeX repositories.","title":"Multiple Release Streams"},{"location":"design/adr/devops/0007-Release-Automation/#release-flow","text":"","title":"Release Flow"},{"location":"design/adr/devops/0007-Release-Automation/#go-modules-device-and-application-sdks","text":"","title":"Go Modules, Device and Application SDKs"},{"location":"design/adr/devops/0007-Release-Automation/#during-development","text":"Go modules, Application and Device SDKs only release a GitHub tag as their release. Go modules, Application and Device SDKs are set up to automatically increment a developmental version tag on each merge to main . (IE: 1.0.0-dev.1 -> 1.0.0-dev.2)","title":"During Development"},{"location":"design/adr/devops/0007-Release-Automation/#release","text":"The release automation for Go Modules, Device and Application SDKs is used to set the final release version git tag. (IE: 1.0.0-dev.X -> 1.0.0) For each release, the Go Modules, Device and Application SDK repositories will be tagged with the release version.","title":"Release"},{"location":"design/adr/devops/0007-Release-Automation/#core-services-including-security-and-system-management-services-application-services-device-services-and-supporting-docker-images","text":"","title":"Core Services (Including Security and System Management services), Application Services, Device Services and Supporting Docker Images"},{"location":"design/adr/devops/0007-Release-Automation/#during-development_1","text":"For the Core Services, Application Services, Device Services and Supporting Docker Images we release Github tags and docker images. On every merge to the main branch we will do the following; increment a developmental version tag on GitHub, (IE: 1.0.0-dev.1 -> 1.0.0-dev.2), stage docker images in our Nexus repository (docker.staging).","title":"During Development"},{"location":"design/adr/devops/0007-Release-Automation/#release_1","text":"The release automation will need to do the following: Set version tag on GitHub. (IE: 1.0.0-dev.X -> 1.0.0) Promote docker images in our Nexus repository from docker.staging to docker.release and public Docker hub.","title":"Release"},{"location":"design/adr/devops/0007-Release-Automation/#supporting-assets-eg-edgex-cli","text":"","title":"Supporting Assets (e.g. edgex-cli)"},{"location":"design/adr/devops/0007-Release-Automation/#during-development_2","text":"For supporting release assets (e.g. edgex-cli) we release GitHub tags on every merge to the main branch. For every merge to main we will do the following; increment a developmental version tag on GitHub, (IE: 1.0.0-dev.1 -> 1.0.0-dev.2) and store the build artifacts in our Nexus repository.","title":"During Development"},{"location":"design/adr/devops/0007-Release-Automation/#release_2","text":"For EdgeX releases the release automation will set the final release version by creating a git tag (e.g. 1.0.0-dev.X -> 1.0.0) and produce a Github Release containing the binary assets targeted for release.","title":"Release"},{"location":"design/adr/devops/0010-Release-Artifacts/","text":"Release Artifacts Status Approved Context During the Geneva release of EdgeX Foundry the DevOps WG transformed the CI/CD process with new Jenkins pipeline functionality. After this new functionality was added we also started adding release automation. This new automation is outlined in ADR 0007 Release Automation. However, in ADR 0007 Release Automation only two release artifact types are outlined. This document is meant to be a living document to try to outlines all currently supported artifacts associated with an EdgeX Foundry release, and should be updated if/when this list changes. Release Artifact Types Docker Images Tied to Code Release? Yes Docker images are released for every named release of EdgeX Foundry. During development the community releases images to the docker.staging repository in Nexus . At the time of release we promote the last tested image from docker.staging to docker.release . In addition to that we will publish the docker image on DockerHub . Nexus Retention Policy docker.snapshots Retention Policy: 90 days since last download Contains: Docker images that are not expected to be released. This contains images to optimize the builds in the CI infrastructure. The definitions of these docker images can be found in the edgexfoundry/ci-build-images Github repository. Docker Tags Used: Version, Latest docker.staging Retention Policy: 180 days since last download Contains: Docker images built for potential release and testing purposes during development. Docker Tags Used: Version (ie: v1.x), Release Branch (master, fuji, etc), Latest docker.release Retention Policy: No automatic removal. Requires TSC approval to remove images from this repository. Contains: Officially released docker images for EdgeX. Docker Tags Used:\u2022Version (ie: v1.x), Latest Nexus Cleanup Policies Reference Docker Compose Files Tied to Code Release? Yes Docker compose files are released alongside the docker images for every release of EdgeX Foundry. During development the community maintains compose files a folder named nightly-build . These compose files are meant to be used by our testing frameworks. At the time of release the community makes compose files for that release in a folder matching it's name. (ie: geneva ) DockerHub Image Descriptions and Overviews Tied to Code Release? No After Docker images are published to DockerHub , automation should be run to update the image Overviews and Descriptions of the necessary images. This automation is located in the edgex-docker-hub-documentation branch of the cd-management repository. In preparation for the release the community makes changes to the Overview and Description metadata as appropriate. The Release Czar will coordinate the execution of the automation near the release time. Github Page: EdgeX Docs Tied to Code Release? No EdgeX Foundry releases a set of documentation for our project at http://docs.edgexfoundry.org . This page is a Github page that is managed by the edgex/foundry/edgex-docs Github repository. As a community we make our best effort to keep these docs up to date. On this page we are also versioning the docs with the semantic versions of the named releases. As a community we try to version our documentation site shortly after the official release date but documentation changes are addressed as we find them throughout the release cycle. GitHub Tags Tied to Code Release? Yes, for the final semantic version Github tags are used to track the releases of EdgeX Foundry. During development the tags are incremented automatically for each commit using a development suffix (ie: v1.1.1-dev.1 -> v1.1.1-dev.2 ). At the time of release we release a tag with the final semantic version (ie: v1.1.1 ). Snaps Tied to Code Release? Yes The building of snaps was removed from community scope in September 2020 but are still available on the snapcraft store . Canonical publishes daily arm64 and amd64 releases of the following snaps to latest/edge in the Snap Store. These builds take place on the Canonical Launchpad platform and use the latest code from the master branch of each EdgeX repository, versioned using the latest git tag. edgexfoundry edgex-app-service-configurable edgex-device-camera edgex-device-rest edgex-device-modbus edgex-device-mqtt edgex-device-grove edgex-cli (work-in-progress) Note - this list may expand over time. At code freeze the edgexfoundry snap revision in the edge channel is promoted to latest/beta and $TRACK/beta. Publishing to beta will trigger the Canonical checkbox automated tests, which include tests on a variety of hardware hosted by Canonical. When the project tags a release of any of the snaps listed above, the resulting snap revision is first promoted from the edge channel to latest/candidate and $TRACK/candidate. Canonical tests this revision, and if all looks good, releases to latest/stable and $TRACK/stable. Canonical may also publish updates to the EdgeX snaps after release to address high/critical bugs and CVEs (common vulnerabilities and exposures). Note - in the above descriptions, $TRACK corresponds to the named release tracks (e.g. fuji, geneva, hanoi, ...) which are created for every major/minor release of EdgeX Foundry. SwaggerHub API Docs Tied to Code Release? No In addition to our documentation site EdgeX foundry also releases our API specifications on Swaggerhub. Testing Framework Tied to Code Release? Yes The EdgeX Foundry community has a set of tests we maintain to do regression testing during development this framework is tracking the master branch of the components of EdgeX. At the time of release we will update the testing frameworks to point at the released Github tags and add a version tag to the testing frameworks themselves. This creates a snapshot of testing framework at the time of release for validation of the official release. GitHub Release Artifacts Tied to Code Release? Yes GitHub release functionality is utilized on some repositories to release binary artifacts/assets (e.g. zip/tar files). These are versioned with the semantic version and found on the repository's GitHub Release page under 'Assets'. Known Build Dependencies for EdgeX Foundry There are some internal build dependencies within the EdgeX Foundry organization. When building artifacts for validation or a release you will need to take into the account the build dependencies to make sure you build them in the correct order. Application services have a dependency on the Application Functions SDK. Go Device services have a dependency on the Go Device SDK. C Device services have a dependency on the C Device SDK. Decision Consequences This document is meant to be a living document of all the release artifacts of EdgeX Foundry. With this ADR we would have a good understanding on what needs to be released and when they are released. Without this document this information will remain tribal knowledge within the community.","title":"Release Artifacts"},{"location":"design/adr/devops/0010-Release-Artifacts/#release-artifacts","text":"","title":"Release Artifacts"},{"location":"design/adr/devops/0010-Release-Artifacts/#status","text":"Approved","title":"Status"},{"location":"design/adr/devops/0010-Release-Artifacts/#context","text":"During the Geneva release of EdgeX Foundry the DevOps WG transformed the CI/CD process with new Jenkins pipeline functionality. After this new functionality was added we also started adding release automation. This new automation is outlined in ADR 0007 Release Automation. However, in ADR 0007 Release Automation only two release artifact types are outlined. This document is meant to be a living document to try to outlines all currently supported artifacts associated with an EdgeX Foundry release, and should be updated if/when this list changes.","title":"Context"},{"location":"design/adr/devops/0010-Release-Artifacts/#release-artifact-types","text":"","title":"Release Artifact Types"},{"location":"design/adr/devops/0010-Release-Artifacts/#docker-images","text":"Tied to Code Release? Yes Docker images are released for every named release of EdgeX Foundry. During development the community releases images to the docker.staging repository in Nexus . At the time of release we promote the last tested image from docker.staging to docker.release . In addition to that we will publish the docker image on DockerHub .","title":"Docker Images"},{"location":"design/adr/devops/0010-Release-Artifacts/#nexus-retention-policy","text":"","title":"Nexus Retention Policy"},{"location":"design/adr/devops/0010-Release-Artifacts/#dockersnapshots","text":"Retention Policy: 90 days since last download Contains: Docker images that are not expected to be released. This contains images to optimize the builds in the CI infrastructure. The definitions of these docker images can be found in the edgexfoundry/ci-build-images Github repository. Docker Tags Used: Version, Latest","title":"docker.snapshots"},{"location":"design/adr/devops/0010-Release-Artifacts/#dockerstaging","text":"Retention Policy: 180 days since last download Contains: Docker images built for potential release and testing purposes during development. Docker Tags Used: Version (ie: v1.x), Release Branch (master, fuji, etc), Latest","title":"docker.staging"},{"location":"design/adr/devops/0010-Release-Artifacts/#dockerrelease","text":"Retention Policy: No automatic removal. Requires TSC approval to remove images from this repository. Contains: Officially released docker images for EdgeX. Docker Tags Used:\u2022Version (ie: v1.x), Latest Nexus Cleanup Policies Reference","title":"docker.release"},{"location":"design/adr/devops/0010-Release-Artifacts/#docker-compose-files","text":"Tied to Code Release? Yes Docker compose files are released alongside the docker images for every release of EdgeX Foundry. During development the community maintains compose files a folder named nightly-build . These compose files are meant to be used by our testing frameworks. At the time of release the community makes compose files for that release in a folder matching it's name. (ie: geneva )","title":"Docker Compose Files"},{"location":"design/adr/devops/0010-Release-Artifacts/#dockerhub-image-descriptions-and-overviews","text":"Tied to Code Release? No After Docker images are published to DockerHub , automation should be run to update the image Overviews and Descriptions of the necessary images. This automation is located in the edgex-docker-hub-documentation branch of the cd-management repository. In preparation for the release the community makes changes to the Overview and Description metadata as appropriate. The Release Czar will coordinate the execution of the automation near the release time.","title":"DockerHub Image Descriptions and Overviews"},{"location":"design/adr/devops/0010-Release-Artifacts/#github-page-edgex-docs","text":"Tied to Code Release? No EdgeX Foundry releases a set of documentation for our project at http://docs.edgexfoundry.org . This page is a Github page that is managed by the edgex/foundry/edgex-docs Github repository. As a community we make our best effort to keep these docs up to date. On this page we are also versioning the docs with the semantic versions of the named releases. As a community we try to version our documentation site shortly after the official release date but documentation changes are addressed as we find them throughout the release cycle.","title":"Github Page: EdgeX Docs"},{"location":"design/adr/devops/0010-Release-Artifacts/#github-tags","text":"Tied to Code Release? Yes, for the final semantic version Github tags are used to track the releases of EdgeX Foundry. During development the tags are incremented automatically for each commit using a development suffix (ie: v1.1.1-dev.1 -> v1.1.1-dev.2 ). At the time of release we release a tag with the final semantic version (ie: v1.1.1 ).","title":"GitHub Tags"},{"location":"design/adr/devops/0010-Release-Artifacts/#snaps","text":"Tied to Code Release? Yes The building of snaps was removed from community scope in September 2020 but are still available on the snapcraft store . Canonical publishes daily arm64 and amd64 releases of the following snaps to latest/edge in the Snap Store. These builds take place on the Canonical Launchpad platform and use the latest code from the master branch of each EdgeX repository, versioned using the latest git tag. edgexfoundry edgex-app-service-configurable edgex-device-camera edgex-device-rest edgex-device-modbus edgex-device-mqtt edgex-device-grove edgex-cli (work-in-progress) Note - this list may expand over time. At code freeze the edgexfoundry snap revision in the edge channel is promoted to latest/beta and $TRACK/beta. Publishing to beta will trigger the Canonical checkbox automated tests, which include tests on a variety of hardware hosted by Canonical. When the project tags a release of any of the snaps listed above, the resulting snap revision is first promoted from the edge channel to latest/candidate and $TRACK/candidate. Canonical tests this revision, and if all looks good, releases to latest/stable and $TRACK/stable. Canonical may also publish updates to the EdgeX snaps after release to address high/critical bugs and CVEs (common vulnerabilities and exposures). Note - in the above descriptions, $TRACK corresponds to the named release tracks (e.g. fuji, geneva, hanoi, ...) which are created for every major/minor release of EdgeX Foundry.","title":"Snaps"},{"location":"design/adr/devops/0010-Release-Artifacts/#swaggerhub-api-docs","text":"Tied to Code Release? No In addition to our documentation site EdgeX foundry also releases our API specifications on Swaggerhub.","title":"SwaggerHub API Docs"},{"location":"design/adr/devops/0010-Release-Artifacts/#testing-framework","text":"Tied to Code Release? Yes The EdgeX Foundry community has a set of tests we maintain to do regression testing during development this framework is tracking the master branch of the components of EdgeX. At the time of release we will update the testing frameworks to point at the released Github tags and add a version tag to the testing frameworks themselves. This creates a snapshot of testing framework at the time of release for validation of the official release.","title":"Testing Framework"},{"location":"design/adr/devops/0010-Release-Artifacts/#github-release-artifacts","text":"Tied to Code Release? Yes GitHub release functionality is utilized on some repositories to release binary artifacts/assets (e.g. zip/tar files). These are versioned with the semantic version and found on the repository's GitHub Release page under 'Assets'.","title":"GitHub Release Artifacts"},{"location":"design/adr/devops/0010-Release-Artifacts/#known-build-dependencies-for-edgex-foundry","text":"There are some internal build dependencies within the EdgeX Foundry organization. When building artifacts for validation or a release you will need to take into the account the build dependencies to make sure you build them in the correct order. Application services have a dependency on the Application Functions SDK. Go Device services have a dependency on the Go Device SDK. C Device services have a dependency on the C Device SDK.","title":"Known Build Dependencies for EdgeX Foundry"},{"location":"design/adr/devops/0010-Release-Artifacts/#decision","text":"","title":"Decision"},{"location":"design/adr/devops/0010-Release-Artifacts/#consequences","text":"This document is meant to be a living document of all the release artifacts of EdgeX Foundry. With this ADR we would have a good understanding on what needs to be released and when they are released. Without this document this information will remain tribal knowledge within the community.","title":"Consequences"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/","text":"Creation and Distribution of Secrets Status Approved Context This ADR seeks to clarify and prioritize the secret handling approach taken by EdgeX. EdgeX microservices need a number of secrets to be created and distributed in order to create a functional, secure system. Among these secrets are: Privileged administrator passwords (such as a database superuser password) Service account passwords (e.g. non-privileged database accounts) PKI private keys There is a lack of consistency on how secrets are created and distributed to EdgeX microservices, and when developers need to add new components to the system, it is unclear on what the preferred approach should be. This document assumes a threat model wherein the EdgeX services are sandboxed (such as in a snap or a container) and the host system is trusted, and all services running in a single snap share a trust boundary. Terms The following terms will be helpful for understading the subsequent discussion: SECRETSLOC is a protected file system path where bootstrapping secrets are stored. While EdgeX implements a sophisticated secret handling mechanism, that mechanism itself requires secrets. For example, every microservice that talks to Vault must have its own unique secret to authenticate: Vault itself cannot be used to distribute these secrets. SECRETSLOC fulfills the role that the non-routable instance data IP address, 169.254.169.254, fulfills in the public cloud: delivery of bootstrapping secrets. As EdgeX does not have a hypervisor nor virtual machines for this purpose, a protected file system path is used instead. SECRETSLOC is implementation-dependent. A desirable feature of SECRETSLOC would be that data written here is kept in RAM and is not persisted to storage media. This property is not achieveable in all circumstances. For Docker, a list of suggested paths--in preference order--is: /run/edgex/secrets (a tmpfs volume on a Linux host) /tmp/edgex/secrets (a temporary file area on Linux and MacOS hosts) A persistent docker volume (use when host bind mounts are not available) For snaps, a list of suggested paths-in preference order--is: * /run/snap. $SNAP_NAME / (a tmpfs volume on a Linux host) * $SNAP_DATA /secrets (a snap-specific persistent data area) * TBD (a content interface that allows for sharing of secrets from the core snap) Current practices survey A survey on the existing EdgeX secrets reveals the following appoaches. A designation of \"compliant\" means that the current implementation is aligned with the recommended practices documented in the next section. A designation of \"non-compliant\" means that the current implementation uses an implemention mechanism outside of the recommended practices documented in the next section. A \"non-compliant\" implementation is a candidate for refactoring to bring the implementation into conformance with the recommended practices. System-managed secrets PKI private keys Docker: PKI generated by standalone utility every cold start of the framework. Distribution via SECRETSLOC . (Compliant.) Snaps: PKI generated by standalone utility every cold start of the framework. Deployed to SECRETSLOC . (Compliant.) Secret store master password Docker: Distribution via persistent docker volume. (Non-compliant.) Snaps: Stored in $SNAP_DATA/config/security-secrets-setup/res . (Non-compliant.) Secret store per-service authentication tokens Docker: Distribution via SECRETSLOC generated every cold start of the framework. (Compliant.) Snaps: Distribution via SECRETSLOC , generated every cold start of the framework. (Compliant.) Postgres superuser password Docker: Hard-coded into docker-compose file, checked in to source control. (Non-compliant.) Snaps: Generated at snap install time via \"apg\" (\"automatic password generator\") tool, installed into Postgres, cached to $SNAP_DATA/config/postgres/kongpw (non-compliant), and passed to Kong via $KONG_PG_PASSWORD . MongoDB service account passwords Docker: Direct consumption from secret store. (Compliant.) Snaps: Direct consumption from secret store. (Compliant.) Redis authentication password Docker: Server--staged to secrets volume and injected via command line. (Non-compliant.). Clients--direct consumption from secret store. (Compliant.) Snaps: Server--staged to $SNAP_DATA/secrets/edgex-redis/redis5-password and injected via command line. (Non-compliant.). Clients--direct consumption from secret store. (Compliant.) Kong client authentication tokens Docker: System of reference is unencrypted Postgres database. (Non-compliant.) Snaps: System of reference is unencrypted Postgres database. (Non-compliant.) Note: in the current implementation, Consul is being operated as a public service. Consul will be a subject of a future \"bootstrapping ADR\" due to its role in serivce location. User-managed secrets User-managed secrets functionality is provided by app-functions-sdk-go . If security is enabled, secrets are retrieved from Vault. If security is disabled, secrets are retreived from the configuration provider. If the configuration provider is not available, secrets are read from the underlying .toml . It is taken as granted in this ADR that secrets originating in the configuration provider or from .toml configuration files are not secret. The fallback mechanism is provided as a convienience to the developer, who would otherwise have to litter their code with \"if (isSecurityEnabled())\" logic leading to implementation inconsistencies. The central database credential is supplied by GetDatabaseCredentials() and returns the database credential assigned to app-service-configurable . If security is enabled, database credentials are retreived using the standard flow. If security is disabled, secrets are retreived from the configuration provider from a special section called [Writable.InsecureSecrets] . If not found there, the configuration provider is searched for credentials stored in the legacy [Databases.Primary] section using the Username and Password keys. Each user application has its own exclusive-use area of the secret store that is accessed via GetSecrets() . If security is enabled, secret requests are passed along to go-mod-secrets using an application-specific access token. If security is disabled, secret requets are made to the configuration provider from the [Writable.InsecureSecrets] section. There is no fallback configuration location. As user-managed secrets have no framework support for initialization, a special StoreSecrets() method is made available to the application for the application to initialize its own secrets. This method is only available in security-enabled mode. No changes to user-managed secrets are being proposed in this ADR. Decision Creation of secrets Management of hardware-bound secrets is platform-specific and out-of-scope for the EdgeX framework. EdgeX open source will contain only the necessary hooks to integrate platform-specific functionality. For software-managed secrets, the system of reference of secrets in EdgeX is the EdgeX secret store. The EdgeX secret store provides for encryption of secrets at rest. This term means that if a secret is replicated, the EdgeX secret store is the authoritative source of truth of the secret. Whenever possible, the EdgeX secret store should also be the record of origin of a secret as well. This means creating secrets inside of the EdgeX secret store is preferable to importing an externally-created secret into the secret store. This can often be done for framework-managed secrets, but not possible for user-managed secrets. Choosing between alternative forms of secrets When given a choice between plain-text secrets and cryptographic keys, cryptographic keys should be preferred. An example situation would be the introduction of an MQTT message broker. A broker may support both TLS client authentication as well as username/password authentication. In such a situation, TLS client authentication would be preferred: The cryptographic key is typically longer in bits than a plain-text secret. A plain-text secret will require transport encryption in order to protect confidentiality of the secret, such as server-side TLS. Use of TLS client authentication typically eliminates the need for additional assets on the server side (such as a password database) to authenticate the client, by relying on digital signature instead. TLS client authentication should not be used unless there is a capability to revoke a compromised certificate, such as by replacing the certificate authority, or providing a certificate revokation list to the server. If certificate revokation is not supported, plain-text secrets (such as username/password) should be used instead, as they are typically easier to revoke. Distribution and consumption of secrets Prohibited practices Use of hard-coded secrets is an instance of CWE-798: Use of hard-coded credentials and is not allowed. A hard-coded secret is a secret that is the same across multiple EdgeX instances. Hard-coded secrets make devices susceptible to BORE (break-once-run-everywhere) attacks, where collections of machines can compromised by a single replicated secret. Specific cases where this is likely to come up are: Secrets embedded in source control EdgeX is an open-source project. Any secret that is present in an EdgeX repository is public to the world, and therefore not a secret, by definition. Configuration files, such as .toml files, .json files, .yaml files (including docker-compose.yml ) are specific instances of this practice. Secrets embedded in binaries Binaries are usually not protected against confidentiality threats, and binaries can be easily reverse-engineered to find any secrets therein. Binaries included compile executables as well as Docker images. Recommended practices Direct consumption from process-to-process interaction with secret store This approach is only possible for components that have native support for Hashicorp Vault . This includes any EdgeX service that links to go-mod-secrets. For example, if secretClient is an instance of the go-mod-secrets secret store client: secrets , err := secretClient . GetSecrets ( \"myservice\" , \"username\" , \"password\" ) The above code will retrieve the username and password properties of the myservice secret. Dynamic injection of secret into process environment space Environment variables are part of a process' environment block and are mapped into a process' memory. In this scenario, an intermediary makes a connection to the secret store to fetch a secret, store it into an environment variable, and then launches a target executable, thereby passing the secret in-memory to the target process. Existing examples of this functionality include vaultenv , envconsul , or env-aws-params . These tools authenticate to a remote network service, inject secrets into the process environment, and then exec's a replacment process that inherits the secret-enriched enviornment block. There are a few potential risks with this approach: Environment blocks are passed to child processes by default. Environment-variable-sniffing malware (introduced by compromised 3rd party libaries) is a proven attack method. Dynamic injection of secret into container-scoped tmpfs volume An example of this approach is consul-template . This approach is useful when a secret is required to be in a configuration file and cannot be passed via an environment variable or directly consumed from a secret store. Distribution via SECRETSLOC This option is the most widely supported secret distribution mechanism by container orchestrators. EdgeX supports runtime environments such as standard Docker and snaps that have no built-in secret management features. Generic Docker does not have a built-in secrets mechanism. Manual configuration of a SECRETSLOC should utilize either a host file file system path or a Docker volume. Snaps also do not have a built-in secrets mechanism. The options for SECRETSLOC are limited to designated snap-writable directories. For comparison: Docker Swarm: Swarm swarm mode is not officially supported by the EdgeX project. Docker Swarm secrets are shared via the /run/secrets volume, which is a Linux tmpfs volume created on the host and shared with the container. For an example of Docker Swarm secrets, see the docker-compose secrets stanza . Secrets distributed in this manner become part of the RaftDB, and thus it becomes necessary to enable swarm autolock mode, which prevents the Raft database encryption key from being stored plaintext on disk. Swarm secrets have an additional limitation in that they are not mutable at runtime. Kubernetes: Kubernetes is not officially supported by the EdgeX project. Kubernetes also supports the secrets volume approach, though the secrets volume can be mounted anywhere in the container namespace. For an example of Kubernetes secrets volumes, see the Kubernetes secrets documentation . Secrets distributed in this manner become part of the etcd database, and thus it becomes necessary to specify a KMS provider for data encryption to prevent etcd from storing plaintext versions of secrets. Consequences As the existing implementation is not fully-compliant with this ADR, significant scope will be added to current and future EdgeX releases in order to bring the project into compliance. List of needed improvements: PKI private keys All: Move to using Vault as system of origin for the PKI instead of the standalone security-secrets-setup utility. All: Cache the PKI for Consul and Vault on persistent disk; rotate occasionally. All: Investigate hardware protection of cached Consul and Vault PKI secret keys. (Vault cannot unseal its own TLS certificate.) Special case: Bring-your-own external Kong certificate and key The Kong external certificate and key is already stored in Vault, however, additional metadata is needed to signal whether these are auto-generated or manually-installed. A manually-installed certificate and key would not be overwritten by the framework bringup logic. Installing a custom certificate and key can then be implemented by overwriting the system-generated ones and setting a flag indicating that they were manually-installed. Secret store master password All: Enable hooks for hardware protection of secret store master password. Secret store per-service authentication tokens No changes required. Postgres superuser password Generate at install time or on cold start of the framework. Cache in Vault and inject into Kong using environment variable injection. MongoDB service account passwords No changes required. Redis(v5) authentication password All: Implement process-to-process injection: start Redis unauthenticated, with a post-start hook to read the secret out of Vault and set the Redis password. (Short race condition between Redis starting, password being set, and dependent services starting.) No changes on client side. Redis(v6) passwords (v6 adds multiple user support) Interim solution: handle like MongoDB service account passwords. Future ADR to propose use of a Vault database secrets engine. No changes on client side (each service accesses its own credential) Kong authentication tokens All: Implement in-transit authentication with TLS-protected Postgres interface. (Subject to change if it is decided not to enable a Postgres backend out of the box.) Additional research needed as PostgreSQL does not support transparent data encryption. References ADR for secret creation and distribution CWE-798: Use of hard-coded credentials Docker Swarm secrets EdgeX go-mod-secrets Hashicorp Vault","title":"Creation and Distribution of Secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#creation-and-distribution-of-secrets","text":"","title":"Creation and Distribution of Secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#status","text":"Approved","title":"Status"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#context","text":"This ADR seeks to clarify and prioritize the secret handling approach taken by EdgeX. EdgeX microservices need a number of secrets to be created and distributed in order to create a functional, secure system. Among these secrets are: Privileged administrator passwords (such as a database superuser password) Service account passwords (e.g. non-privileged database accounts) PKI private keys There is a lack of consistency on how secrets are created and distributed to EdgeX microservices, and when developers need to add new components to the system, it is unclear on what the preferred approach should be. This document assumes a threat model wherein the EdgeX services are sandboxed (such as in a snap or a container) and the host system is trusted, and all services running in a single snap share a trust boundary.","title":"Context"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#terms","text":"The following terms will be helpful for understading the subsequent discussion: SECRETSLOC is a protected file system path where bootstrapping secrets are stored. While EdgeX implements a sophisticated secret handling mechanism, that mechanism itself requires secrets. For example, every microservice that talks to Vault must have its own unique secret to authenticate: Vault itself cannot be used to distribute these secrets. SECRETSLOC fulfills the role that the non-routable instance data IP address, 169.254.169.254, fulfills in the public cloud: delivery of bootstrapping secrets. As EdgeX does not have a hypervisor nor virtual machines for this purpose, a protected file system path is used instead. SECRETSLOC is implementation-dependent. A desirable feature of SECRETSLOC would be that data written here is kept in RAM and is not persisted to storage media. This property is not achieveable in all circumstances. For Docker, a list of suggested paths--in preference order--is: /run/edgex/secrets (a tmpfs volume on a Linux host) /tmp/edgex/secrets (a temporary file area on Linux and MacOS hosts) A persistent docker volume (use when host bind mounts are not available) For snaps, a list of suggested paths-in preference order--is: * /run/snap. $SNAP_NAME / (a tmpfs volume on a Linux host) * $SNAP_DATA /secrets (a snap-specific persistent data area) * TBD (a content interface that allows for sharing of secrets from the core snap)","title":"Terms"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#current-practices-survey","text":"A survey on the existing EdgeX secrets reveals the following appoaches. A designation of \"compliant\" means that the current implementation is aligned with the recommended practices documented in the next section. A designation of \"non-compliant\" means that the current implementation uses an implemention mechanism outside of the recommended practices documented in the next section. A \"non-compliant\" implementation is a candidate for refactoring to bring the implementation into conformance with the recommended practices.","title":"Current practices survey"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#system-managed-secrets","text":"PKI private keys Docker: PKI generated by standalone utility every cold start of the framework. Distribution via SECRETSLOC . (Compliant.) Snaps: PKI generated by standalone utility every cold start of the framework. Deployed to SECRETSLOC . (Compliant.) Secret store master password Docker: Distribution via persistent docker volume. (Non-compliant.) Snaps: Stored in $SNAP_DATA/config/security-secrets-setup/res . (Non-compliant.) Secret store per-service authentication tokens Docker: Distribution via SECRETSLOC generated every cold start of the framework. (Compliant.) Snaps: Distribution via SECRETSLOC , generated every cold start of the framework. (Compliant.) Postgres superuser password Docker: Hard-coded into docker-compose file, checked in to source control. (Non-compliant.) Snaps: Generated at snap install time via \"apg\" (\"automatic password generator\") tool, installed into Postgres, cached to $SNAP_DATA/config/postgres/kongpw (non-compliant), and passed to Kong via $KONG_PG_PASSWORD . MongoDB service account passwords Docker: Direct consumption from secret store. (Compliant.) Snaps: Direct consumption from secret store. (Compliant.) Redis authentication password Docker: Server--staged to secrets volume and injected via command line. (Non-compliant.). Clients--direct consumption from secret store. (Compliant.) Snaps: Server--staged to $SNAP_DATA/secrets/edgex-redis/redis5-password and injected via command line. (Non-compliant.). Clients--direct consumption from secret store. (Compliant.) Kong client authentication tokens Docker: System of reference is unencrypted Postgres database. (Non-compliant.) Snaps: System of reference is unencrypted Postgres database. (Non-compliant.) Note: in the current implementation, Consul is being operated as a public service. Consul will be a subject of a future \"bootstrapping ADR\" due to its role in serivce location.","title":"System-managed secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#user-managed-secrets","text":"User-managed secrets functionality is provided by app-functions-sdk-go . If security is enabled, secrets are retrieved from Vault. If security is disabled, secrets are retreived from the configuration provider. If the configuration provider is not available, secrets are read from the underlying .toml . It is taken as granted in this ADR that secrets originating in the configuration provider or from .toml configuration files are not secret. The fallback mechanism is provided as a convienience to the developer, who would otherwise have to litter their code with \"if (isSecurityEnabled())\" logic leading to implementation inconsistencies. The central database credential is supplied by GetDatabaseCredentials() and returns the database credential assigned to app-service-configurable . If security is enabled, database credentials are retreived using the standard flow. If security is disabled, secrets are retreived from the configuration provider from a special section called [Writable.InsecureSecrets] . If not found there, the configuration provider is searched for credentials stored in the legacy [Databases.Primary] section using the Username and Password keys. Each user application has its own exclusive-use area of the secret store that is accessed via GetSecrets() . If security is enabled, secret requests are passed along to go-mod-secrets using an application-specific access token. If security is disabled, secret requets are made to the configuration provider from the [Writable.InsecureSecrets] section. There is no fallback configuration location. As user-managed secrets have no framework support for initialization, a special StoreSecrets() method is made available to the application for the application to initialize its own secrets. This method is only available in security-enabled mode. No changes to user-managed secrets are being proposed in this ADR.","title":"User-managed secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#decision","text":"","title":"Decision"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#creation-of-secrets","text":"Management of hardware-bound secrets is platform-specific and out-of-scope for the EdgeX framework. EdgeX open source will contain only the necessary hooks to integrate platform-specific functionality. For software-managed secrets, the system of reference of secrets in EdgeX is the EdgeX secret store. The EdgeX secret store provides for encryption of secrets at rest. This term means that if a secret is replicated, the EdgeX secret store is the authoritative source of truth of the secret. Whenever possible, the EdgeX secret store should also be the record of origin of a secret as well. This means creating secrets inside of the EdgeX secret store is preferable to importing an externally-created secret into the secret store. This can often be done for framework-managed secrets, but not possible for user-managed secrets.","title":"Creation of secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#choosing-between-alternative-forms-of-secrets","text":"When given a choice between plain-text secrets and cryptographic keys, cryptographic keys should be preferred. An example situation would be the introduction of an MQTT message broker. A broker may support both TLS client authentication as well as username/password authentication. In such a situation, TLS client authentication would be preferred: The cryptographic key is typically longer in bits than a plain-text secret. A plain-text secret will require transport encryption in order to protect confidentiality of the secret, such as server-side TLS. Use of TLS client authentication typically eliminates the need for additional assets on the server side (such as a password database) to authenticate the client, by relying on digital signature instead. TLS client authentication should not be used unless there is a capability to revoke a compromised certificate, such as by replacing the certificate authority, or providing a certificate revokation list to the server. If certificate revokation is not supported, plain-text secrets (such as username/password) should be used instead, as they are typically easier to revoke.","title":"Choosing between alternative forms of secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#distribution-and-consumption-of-secrets","text":"","title":"Distribution and consumption of secrets"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#prohibited-practices","text":"Use of hard-coded secrets is an instance of CWE-798: Use of hard-coded credentials and is not allowed. A hard-coded secret is a secret that is the same across multiple EdgeX instances. Hard-coded secrets make devices susceptible to BORE (break-once-run-everywhere) attacks, where collections of machines can compromised by a single replicated secret. Specific cases where this is likely to come up are: Secrets embedded in source control EdgeX is an open-source project. Any secret that is present in an EdgeX repository is public to the world, and therefore not a secret, by definition. Configuration files, such as .toml files, .json files, .yaml files (including docker-compose.yml ) are specific instances of this practice. Secrets embedded in binaries Binaries are usually not protected against confidentiality threats, and binaries can be easily reverse-engineered to find any secrets therein. Binaries included compile executables as well as Docker images.","title":"Prohibited practices"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#recommended-practices","text":"Direct consumption from process-to-process interaction with secret store This approach is only possible for components that have native support for Hashicorp Vault . This includes any EdgeX service that links to go-mod-secrets. For example, if secretClient is an instance of the go-mod-secrets secret store client: secrets , err := secretClient . GetSecrets ( \"myservice\" , \"username\" , \"password\" ) The above code will retrieve the username and password properties of the myservice secret. Dynamic injection of secret into process environment space Environment variables are part of a process' environment block and are mapped into a process' memory. In this scenario, an intermediary makes a connection to the secret store to fetch a secret, store it into an environment variable, and then launches a target executable, thereby passing the secret in-memory to the target process. Existing examples of this functionality include vaultenv , envconsul , or env-aws-params . These tools authenticate to a remote network service, inject secrets into the process environment, and then exec's a replacment process that inherits the secret-enriched enviornment block. There are a few potential risks with this approach: Environment blocks are passed to child processes by default. Environment-variable-sniffing malware (introduced by compromised 3rd party libaries) is a proven attack method. Dynamic injection of secret into container-scoped tmpfs volume An example of this approach is consul-template . This approach is useful when a secret is required to be in a configuration file and cannot be passed via an environment variable or directly consumed from a secret store. Distribution via SECRETSLOC This option is the most widely supported secret distribution mechanism by container orchestrators. EdgeX supports runtime environments such as standard Docker and snaps that have no built-in secret management features. Generic Docker does not have a built-in secrets mechanism. Manual configuration of a SECRETSLOC should utilize either a host file file system path or a Docker volume. Snaps also do not have a built-in secrets mechanism. The options for SECRETSLOC are limited to designated snap-writable directories. For comparison: Docker Swarm: Swarm swarm mode is not officially supported by the EdgeX project. Docker Swarm secrets are shared via the /run/secrets volume, which is a Linux tmpfs volume created on the host and shared with the container. For an example of Docker Swarm secrets, see the docker-compose secrets stanza . Secrets distributed in this manner become part of the RaftDB, and thus it becomes necessary to enable swarm autolock mode, which prevents the Raft database encryption key from being stored plaintext on disk. Swarm secrets have an additional limitation in that they are not mutable at runtime. Kubernetes: Kubernetes is not officially supported by the EdgeX project. Kubernetes also supports the secrets volume approach, though the secrets volume can be mounted anywhere in the container namespace. For an example of Kubernetes secrets volumes, see the Kubernetes secrets documentation . Secrets distributed in this manner become part of the etcd database, and thus it becomes necessary to specify a KMS provider for data encryption to prevent etcd from storing plaintext versions of secrets.","title":"Recommended practices"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#consequences","text":"As the existing implementation is not fully-compliant with this ADR, significant scope will be added to current and future EdgeX releases in order to bring the project into compliance. List of needed improvements: PKI private keys All: Move to using Vault as system of origin for the PKI instead of the standalone security-secrets-setup utility. All: Cache the PKI for Consul and Vault on persistent disk; rotate occasionally. All: Investigate hardware protection of cached Consul and Vault PKI secret keys. (Vault cannot unseal its own TLS certificate.) Special case: Bring-your-own external Kong certificate and key The Kong external certificate and key is already stored in Vault, however, additional metadata is needed to signal whether these are auto-generated or manually-installed. A manually-installed certificate and key would not be overwritten by the framework bringup logic. Installing a custom certificate and key can then be implemented by overwriting the system-generated ones and setting a flag indicating that they were manually-installed. Secret store master password All: Enable hooks for hardware protection of secret store master password. Secret store per-service authentication tokens No changes required. Postgres superuser password Generate at install time or on cold start of the framework. Cache in Vault and inject into Kong using environment variable injection. MongoDB service account passwords No changes required. Redis(v5) authentication password All: Implement process-to-process injection: start Redis unauthenticated, with a post-start hook to read the secret out of Vault and set the Redis password. (Short race condition between Redis starting, password being set, and dependent services starting.) No changes on client side. Redis(v6) passwords (v6 adds multiple user support) Interim solution: handle like MongoDB service account passwords. Future ADR to propose use of a Vault database secrets engine. No changes on client side (each service accesses its own credential) Kong authentication tokens All: Implement in-transit authentication with TLS-protected Postgres interface. (Subject to change if it is decided not to enable a Postgres backend out of the box.) Additional research needed as PostgreSQL does not support transparent data encryption.","title":"Consequences"},{"location":"design/adr/security/0008-Secret-Creation-and-Distribution/#references","text":"ADR for secret creation and distribution CWE-798: Use of hard-coded credentials Docker Swarm secrets EdgeX go-mod-secrets Hashicorp Vault","title":"References"},{"location":"design/adr/security/0009-Secure-Bootstrapping/","text":"Secure Bootstrapping of EdgeX Secure Bootstrapping of EdgeX Status Context History Decision Stage-gate mechanism Docker-specific service changes \"As-is\" startup flow \"To-be\" startup flow New Bootstrap/RTR container Consequences Benefits Drawbacks Alternatives Event-driven vs commanded staging System management agent (SMA) as the coordinator Create a mega-install container Manual secret provisioning References Status Approved Context Docker-compose, the tool used by EdgeX to manage its Docker-based stack, lags in its support for initialization logic. Docker-compose v2.x used to have a depends_on / condition directive that would test a service's HEALTHCHECK and block startup until the service was \"healthy\". Unfortunately, this feature was removed in 3.x docker-compose. (This feature is also unsuppported in swarm mode as well.) Snaps have an explicit install phase and Kubernetes PODs have optional init containers. In other frameworks, initialization is allowed to run to completion prior to application components being started in production mode. This functionality does not exist in Docker nor docker-compose. The current lack of an initialization phase is a blocking issue for implementing microservice communication security, as critical EdgeX core components that are involved with microservice communication (specifically Consul) are being brought up in an insecure configuration. (Consul's insecure configuration is will be addressed in a separate ADR .) Activities that are best done in the initialization phase include the following: Bootstrapping of crytographic secrets needed by the application. Bootstrapping of database users and passwords. Installation of database schema needed for application logic to function. Initialization of authorization frameworks such as configuring RBAC or ACLs. Other one-time initialization activities. Workarounds when an installation phase is not present include: Perform initialization tasks manually, and manually seed secrets into static configuration files. Ship with known hard-coded secrets in static configuration files. Start in an insecure configuration and remain that way. Provision some secrets at runtime. EdgeX does not have a manual installation flow, and uses a combination of the last three approaches. The objective of this ADR is to define a framework for Docker-based initialization logic in EdgeX. This will enable the removal of certain hard-coded secrets in EdgeX and enable certain components (such as Consul) to be started in a secure configuration. These improvement are necessary pre-requisites to implementing microservice communication security. History In previous releases, container startup sequencing has been primarily been driven by Consul service health checks backed healthcheck endpoints of particular services or by sentinel files placed in the file system when certain intialization milestones are reached. The implementation has been plagued by several issues: Sentinel files are not cleaned up if the framework fails or is shut down. Invalid state left over from previous instantiations of the framework causes difficult-to-resolve race conditions. (Implementation of this ADR will try to remove as many as possible, focusing on those that are used to gate startup. Some use of sentinel files may still be required to indicate completion of initialization steps so that they are not re-done if there is no API-based mechanism to determine if such initialization has been completed.) Consul healh checks are reported in a difficult-to-parse JSON structure, which has lead to the creation of specialized tools that are insensitive to libc implementations used by different container images. Consul is being used not only for service health, but for service location and configuration as well . The requirement to synchronize framework startup for the purpose of securely initializing Consul means that a non-Consul mechanism must be used to stage-gate EdgeX initialization. This last point is the primary motivator of this ADR. Decision Stage-gate mechanism The stage-gate mechanism must work in the following environments: docker-compose in Linux on a single node/system docker-compose in Microsoft Windows on a single node/system docker-compose in Apple MacOS on a single node/system Startup sequencing will be driven by two primary mechanisms: Use of entrypoint scripts to: Block on stage-gate and service dependencies Perform first-boot initialization phase activities as noted in Context The bootstrap container will inject entrypoint scripts into the other containers in the case where EdgeX is directly consuming an upstream container. Docker will automatically retry restarting containers if its entrypoint script is missing. Use of open TCP sockets as semaphores to gate startup sequencing Use of TCP sockets for startup sequencing is commonly used in Docker environments. Due to its popularlity, there are several existing tools for this, including wait-for-it , dockerize , and wait-for . The TCP mechanism is portable across platforms and will work in distributed multi-node scenarios. At least three new ports will be added to EdgeX for sequencing purposes: bootstrap port. This port will be opened once first-time initialization has been completed. tokens_ready port. This port signals that secret-store tokens have been provisioned and are valid. ready_to_run port. This port will be opened once stateful services have completed initialization and it is safe for the majority of EdgeX core services to start. The stateless EdgeX services should block on ready_to_run port. Docker-specific service changes \"As-is\" startup flow The following diagram shows the \"as-is\" startup flow. There are several components being removed via activity unrelated with this ADR. These proposed edits are shown to reduce clutter in the TO-BE diagram. * secrets-setup is being eliminated through a separate ADR to eliminate TLS for single-node usage. * kong-migrations is being combined with the kong service via an entrypoint script. * bootstrap-redis will be incorporated into the Redis entrypoint script to set the Redis password before Redis starts to fix the time delay before a Redis password is set. \"To-be\" startup flow The following diagram shows the \"to-be\" startup flow. Note that the bootstrap flows are always processed, but can be short-circuited. Another difference to note in the \"to-be\" diagram is that the Vault depdendency on Consul is reversed in order to provide better security . New Bootstrap/RTR container The purpose of this new container is to: Inject entrypoint scripts into third-party containers (such as Vault, Redis, Consul, PostgreSQL, Kong) in order to perform first-time initialization and wait on service dependencies Raise the bootstrap semaphore Wait on dependent semaphores required to raise the ready_to_run semaphore (these are the stateful components such as databases, and blocking waiting for sercret store tokens to be provisioned) Raise the ready_to_run semaphore Wait forever (in order to leave TCP sockets open) Consequences Benefits This ADR is expected to yield the following benefits after completion of the related engineering tasks: Standardization of the stage-gate mechanism. Standardized approach to component initialization in Docker. Reduced fragility in the framework startup flow. Vault no longer uses Consul as its data store (uses file system instead). Ability to use a stock Consul container instead of creating a custom one for EdgeX Elimination of several sentinel files used for Consul health checks /tmp/edgex/secrets/ca/.security-secrets-setup.complete /tmp/edgex/secrets/edgex-consul/.secretstore-setup-done Drawbacks Introduction of a new container into the startup flow (but other containers are eliminated or combined). Expanded scope and responsibility of entrypoint scripts, which must not only block component startup, but now must also configure a component for secure operation. Alternatives Event-driven vs commanded staging In this scenario, instead of a service waiting on a TCP-socket semaphore created by another service, services would open a socket and wait for a coordinator/controller to issue a \"go\" command. This solution was not chosen for several reasons: The code required to open a socket and wait for a command is much more complicated than the code required to check for an open socket. Many open source utilities exist to block on a socket opening; there are no such examples for the reverse. This solution would would duplicate the information regarding which services need to run: once in the docker-compose file, and once as a configuration file to the coordinator/controller. System management agent (SMA) as the coordinator In this scenario, the system management agent is responsbile bringing up the EdgeX framework. Since the system management agent has access to the Docker socket, it has the ability to start services in a prescribed order, and as a management agent, has knowledge about the desired state of the framework. This solution was not chosen for several reasons: SMA is an optional EdgeX component--use in this way would make SMA a required core component. SMA, in order to authenticate an authorize remote management requests, requires access to persistent state and secrets. To make the same component responsible for initializing that state and secrets upon which it depends would make the design convoluted. Create a mega-install container This alternative would create a mega-install container that has locally installed verions of critical components needed for bootstrapping such as Vault, Consul, PostgreSQL, and others. A sequential script would start each component in turn, intiailizing each to run in a secure configuration, and then shut them all down again. The same stage-gate mechanism would be used to block startup of these same components, but Docker would start them in production configuration. Manual secret provisioning A typical cloud-based microservice architecture typically has a manual provisioning step. This step would include activities such as configuring Vault, installing a database schema, setting up database service account passwords, and seeding initial secrets such as PKI private keys that have been generated offline (possibly requiring several days of lead time). A cloud team may have weeks or months to prepare for this event, and it might take the greater part of a day. In contrast, EdgeX up to this point has been a \"turnkey\" middleware framework: it can be deployed with the same ease as an application, such as via a docker-compose file, or via a snap install. This means that most of the secret provisioning must be automated and the provisioning logic must be built into the framework in some way. The proposals presented in this ADR are compatibile with continuance of this functionality. References ADR 0008 - Creation and Distribution of Secrets ADR 0015 - Encryption between microservices , Hashicorp Consul Hashicorp Vault Issue: ADR for securing access to Consul Issue: Service registry ADR","title":"Secure Bootstrapping of EdgeX"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#secure-bootstrapping-of-edgex","text":"Secure Bootstrapping of EdgeX Status Context History Decision Stage-gate mechanism Docker-specific service changes \"As-is\" startup flow \"To-be\" startup flow New Bootstrap/RTR container Consequences Benefits Drawbacks Alternatives Event-driven vs commanded staging System management agent (SMA) as the coordinator Create a mega-install container Manual secret provisioning References","title":"Secure Bootstrapping of EdgeX"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#status","text":"Approved","title":"Status"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#context","text":"Docker-compose, the tool used by EdgeX to manage its Docker-based stack, lags in its support for initialization logic. Docker-compose v2.x used to have a depends_on / condition directive that would test a service's HEALTHCHECK and block startup until the service was \"healthy\". Unfortunately, this feature was removed in 3.x docker-compose. (This feature is also unsuppported in swarm mode as well.) Snaps have an explicit install phase and Kubernetes PODs have optional init containers. In other frameworks, initialization is allowed to run to completion prior to application components being started in production mode. This functionality does not exist in Docker nor docker-compose. The current lack of an initialization phase is a blocking issue for implementing microservice communication security, as critical EdgeX core components that are involved with microservice communication (specifically Consul) are being brought up in an insecure configuration. (Consul's insecure configuration is will be addressed in a separate ADR .) Activities that are best done in the initialization phase include the following: Bootstrapping of crytographic secrets needed by the application. Bootstrapping of database users and passwords. Installation of database schema needed for application logic to function. Initialization of authorization frameworks such as configuring RBAC or ACLs. Other one-time initialization activities. Workarounds when an installation phase is not present include: Perform initialization tasks manually, and manually seed secrets into static configuration files. Ship with known hard-coded secrets in static configuration files. Start in an insecure configuration and remain that way. Provision some secrets at runtime. EdgeX does not have a manual installation flow, and uses a combination of the last three approaches. The objective of this ADR is to define a framework for Docker-based initialization logic in EdgeX. This will enable the removal of certain hard-coded secrets in EdgeX and enable certain components (such as Consul) to be started in a secure configuration. These improvement are necessary pre-requisites to implementing microservice communication security.","title":"Context"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#history","text":"In previous releases, container startup sequencing has been primarily been driven by Consul service health checks backed healthcheck endpoints of particular services or by sentinel files placed in the file system when certain intialization milestones are reached. The implementation has been plagued by several issues: Sentinel files are not cleaned up if the framework fails or is shut down. Invalid state left over from previous instantiations of the framework causes difficult-to-resolve race conditions. (Implementation of this ADR will try to remove as many as possible, focusing on those that are used to gate startup. Some use of sentinel files may still be required to indicate completion of initialization steps so that they are not re-done if there is no API-based mechanism to determine if such initialization has been completed.) Consul healh checks are reported in a difficult-to-parse JSON structure, which has lead to the creation of specialized tools that are insensitive to libc implementations used by different container images. Consul is being used not only for service health, but for service location and configuration as well . The requirement to synchronize framework startup for the purpose of securely initializing Consul means that a non-Consul mechanism must be used to stage-gate EdgeX initialization. This last point is the primary motivator of this ADR.","title":"History"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#decision","text":"","title":"Decision"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#stage-gate-mechanism","text":"The stage-gate mechanism must work in the following environments: docker-compose in Linux on a single node/system docker-compose in Microsoft Windows on a single node/system docker-compose in Apple MacOS on a single node/system Startup sequencing will be driven by two primary mechanisms: Use of entrypoint scripts to: Block on stage-gate and service dependencies Perform first-boot initialization phase activities as noted in Context The bootstrap container will inject entrypoint scripts into the other containers in the case where EdgeX is directly consuming an upstream container. Docker will automatically retry restarting containers if its entrypoint script is missing. Use of open TCP sockets as semaphores to gate startup sequencing Use of TCP sockets for startup sequencing is commonly used in Docker environments. Due to its popularlity, there are several existing tools for this, including wait-for-it , dockerize , and wait-for . The TCP mechanism is portable across platforms and will work in distributed multi-node scenarios. At least three new ports will be added to EdgeX for sequencing purposes: bootstrap port. This port will be opened once first-time initialization has been completed. tokens_ready port. This port signals that secret-store tokens have been provisioned and are valid. ready_to_run port. This port will be opened once stateful services have completed initialization and it is safe for the majority of EdgeX core services to start. The stateless EdgeX services should block on ready_to_run port.","title":"Stage-gate mechanism"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#docker-specific-service-changes","text":"","title":"Docker-specific service changes"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#as-is-startup-flow","text":"The following diagram shows the \"as-is\" startup flow. There are several components being removed via activity unrelated with this ADR. These proposed edits are shown to reduce clutter in the TO-BE diagram. * secrets-setup is being eliminated through a separate ADR to eliminate TLS for single-node usage. * kong-migrations is being combined with the kong service via an entrypoint script. * bootstrap-redis will be incorporated into the Redis entrypoint script to set the Redis password before Redis starts to fix the time delay before a Redis password is set.","title":"\"As-is\" startup flow"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#to-be-startup-flow","text":"The following diagram shows the \"to-be\" startup flow. Note that the bootstrap flows are always processed, but can be short-circuited. Another difference to note in the \"to-be\" diagram is that the Vault depdendency on Consul is reversed in order to provide better security .","title":"\"To-be\" startup flow"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#new-bootstraprtr-container","text":"The purpose of this new container is to: Inject entrypoint scripts into third-party containers (such as Vault, Redis, Consul, PostgreSQL, Kong) in order to perform first-time initialization and wait on service dependencies Raise the bootstrap semaphore Wait on dependent semaphores required to raise the ready_to_run semaphore (these are the stateful components such as databases, and blocking waiting for sercret store tokens to be provisioned) Raise the ready_to_run semaphore Wait forever (in order to leave TCP sockets open)","title":"New Bootstrap/RTR container"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#consequences","text":"","title":"Consequences"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#benefits","text":"This ADR is expected to yield the following benefits after completion of the related engineering tasks: Standardization of the stage-gate mechanism. Standardized approach to component initialization in Docker. Reduced fragility in the framework startup flow. Vault no longer uses Consul as its data store (uses file system instead). Ability to use a stock Consul container instead of creating a custom one for EdgeX Elimination of several sentinel files used for Consul health checks /tmp/edgex/secrets/ca/.security-secrets-setup.complete /tmp/edgex/secrets/edgex-consul/.secretstore-setup-done","title":"Benefits"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#drawbacks","text":"Introduction of a new container into the startup flow (but other containers are eliminated or combined). Expanded scope and responsibility of entrypoint scripts, which must not only block component startup, but now must also configure a component for secure operation.","title":"Drawbacks"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#alternatives","text":"","title":"Alternatives"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#event-driven-vs-commanded-staging","text":"In this scenario, instead of a service waiting on a TCP-socket semaphore created by another service, services would open a socket and wait for a coordinator/controller to issue a \"go\" command. This solution was not chosen for several reasons: The code required to open a socket and wait for a command is much more complicated than the code required to check for an open socket. Many open source utilities exist to block on a socket opening; there are no such examples for the reverse. This solution would would duplicate the information regarding which services need to run: once in the docker-compose file, and once as a configuration file to the coordinator/controller.","title":"Event-driven vs commanded staging"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#system-management-agent-sma-as-the-coordinator","text":"In this scenario, the system management agent is responsbile bringing up the EdgeX framework. Since the system management agent has access to the Docker socket, it has the ability to start services in a prescribed order, and as a management agent, has knowledge about the desired state of the framework. This solution was not chosen for several reasons: SMA is an optional EdgeX component--use in this way would make SMA a required core component. SMA, in order to authenticate an authorize remote management requests, requires access to persistent state and secrets. To make the same component responsible for initializing that state and secrets upon which it depends would make the design convoluted.","title":"System management agent (SMA) as the coordinator"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#create-a-mega-install-container","text":"This alternative would create a mega-install container that has locally installed verions of critical components needed for bootstrapping such as Vault, Consul, PostgreSQL, and others. A sequential script would start each component in turn, intiailizing each to run in a secure configuration, and then shut them all down again. The same stage-gate mechanism would be used to block startup of these same components, but Docker would start them in production configuration.","title":"Create a mega-install container"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#manual-secret-provisioning","text":"A typical cloud-based microservice architecture typically has a manual provisioning step. This step would include activities such as configuring Vault, installing a database schema, setting up database service account passwords, and seeding initial secrets such as PKI private keys that have been generated offline (possibly requiring several days of lead time). A cloud team may have weeks or months to prepare for this event, and it might take the greater part of a day. In contrast, EdgeX up to this point has been a \"turnkey\" middleware framework: it can be deployed with the same ease as an application, such as via a docker-compose file, or via a snap install. This means that most of the secret provisioning must be automated and the provisioning logic must be built into the framework in some way. The proposals presented in this ADR are compatibile with continuance of this functionality.","title":"Manual secret provisioning"},{"location":"design/adr/security/0009-Secure-Bootstrapping/#references","text":"ADR 0008 - Creation and Distribution of Secrets ADR 0015 - Encryption between microservices , Hashicorp Consul Hashicorp Vault Issue: ADR for securing access to Consul Issue: Service registry ADR","title":"References"},{"location":"design/adr/security/0015-in-cluster-tls/","text":"Use of encryption to secure in-cluster EdgeX communications Status Approved Context This ADR seeks to define the EdgeX direction on using encryption to secure \"in-cluster\" EdgeX communications, that is, internal microservice-to-microservice communication. This ADR will seek to clarify the EdgeX direction in several aspects with regard to: EdgeX services communicating within a single host EdgeX services communicating across multiple hosts Using encryption for confidentiality or integrity in communication Using encryption for authentication between microservices This ADR will be used to triage EdgeX feature requests in this space. Background Why encrypt? Why consider encryption in the first place? Simple. Encryption helps with the following problems: Client authentication of servers. The client knows that it is talking to the correct server. This is typically achieved using TLS server certificates that the client checks against a trusted root certificate authority. Since the client is not in charge of network routing, TLS server authentication provides a good assurance that the requests are being routed to the correct server. Server authentication of clients. The server knows the identity of the client that has connected to it. There are a variety of mechanims to achieve this, such as usernames and passwords, tokens, claims, et cetera, but the mechanism under consideration by this ADR is TLS client authentication using TLS client certificates. Confidentiality of messages exchanged between services. Confidentiality is needed to protect authentication data flowing between communicating microservices as well as to protect the message payloads if they contain nonpublic data. TLS provides communication channel confidentiality. Integrity of messages exchanged between services. Integrity is needed to ensure that messages between communicating microservices are not maliciously altered, such as inserting or deleting data in the middle of the exchange. TLS provides communication channel integrity. A microservice architecture normally strives for all of the above protections. Besides TLS, there are other mechanisms that can be used to provide some of the above properties. For example, IPSec tunnels provide confidentity, integrity, and authentication of the hosts (network-level protection). SSH tunnels provide confidentiality, integrity, and authentication of the tunnel endpoints (also network-level protection). TLS, however, is preferred, because it operates in-process at the application level and provides better point-to-point security. Why to not encrypt? In the case of TLS communications, microservices depend on an asymmetric private key to prove their identity. To be of value, this private key must be kept secret. Applications typically depend on process-level isolation and/or file system protections for the private key. Moreover, interprocess communication using sockets is mediated by the operating system kernel. An attacker running at the privilege of the operating system has the ability to compromise TLS protections, such as by substituting a private key or certificate authority of their choice, accessing the unencrypted data in process memory, or intercepting the network communications that flow through the kernel. Therefore, within a single host, TLS protections may slow down an attacker, but are not likely to stop them. Additionally, use of TLS requires management of additional security assets in the form of TLS private keys. Microservice communcation across hosts, however, is vulnerable to intereception, and must be protected via some mechanism such as, but not limited to: IPSec or SSH tunnels, encrypted overlay networks, service mesh middlewares, or application-level TLS. Another reason to not encrypt is that TLS adds overhead to microservice communication in the form of additional network around-trips when opening connections and performing cryptographic public key and symmetric key operations. Decision At this time, EdgeX is primarily a single-node IoT application framework. Should this position change, this ADR should be revisited. Based on the single-node assumption: TLS will not be used for confidentiality and integrity of internal on-host microservice communication. TLS will be avoided as an authentication mechanism of peer microservices. Integrity and confidentiality of microservice communcations crossing host boundaries is required to secure EdgeX, but are an EdgeX customer responsibility. EdgeX customers are welcome to add extra security to their own EdgeX deployments. Consequences This ADR if approved would close the following issues as will-not-fix. https://github.com/edgexfoundry/edgex-go/issues/1942 https://github.com/edgexfoundry/edgex-go/issues/1941 https://github.com/edgexfoundry/edgex-go/issues/2454 https://github.com/edgexfoundry/developer-scripts/issues/240 https://github.com/edgexfoundry/edgex-go/issues/2495 It would also close https://github.com/edgexfoundry/edgex-go/issues/1925 as there is no current need for TLS as a mutual authentication strategy. Alternatives Encrypted overlay networks Encrypted overlay networks provide varying protection based on the product used. Some can only encrypt data, such as an IPsec tunnel. Some can encrypt and provide for network microsegmentation, such as Docker Swarm networks with encryption enabled. Some can encrypt and enforce network policy such as restrictions on ingress traffic or restrictions on egress traffic. Service mesh middleware Service mesh middleware is an alternative that should be investigated if EdgeX decides to fully support a Kubernetes-based deployment using distributed Kubernetes pods. A service mesh typically achieves most of the security objectives of security microservice commuication by intercepting microservice communications and imposing a configuration-driven policy that typically includes confidentiality and integrity protection. These middlewares typically rely on the Kubernetes pod construct and are difficult to support for non-Kubernetes deployments. EdgeX public key infrastructure An EdgeX public key infrastructure that is natively supported by the architecture should be considered if EdgeX decides to support an out-of-box distributed deployment on non-Kubernetes platforms. Native support of TLS requires a significant amount of glue logic, and exceeds the availble resources in the security working group to implement this strategy. The following text outlines a proposed strategy for supporting native TLS in the EdgeX framework: EdgeX will use Hashicorp Vault to secure the EdgeX PKI, through the use of the Vault PKI secrets engine. Vault will be configured with a root CA at initialization time, and a Vault-based sub-CA for dynamic generation of TLS leaf certificates. The root CA will be restricted to be used only by the Vault root token. EdgeX microservices that are based on third-party containers require special support unless they can talk natively to Vault for their secrets. Certain tools, such as those mentioned in the \"Creation and Distribution of Secrets\" ADR ( envconsul , consul-template , and others) can be used to facilitiate third-party container integration. These services are: Consul : Requires TLS certificate set by configuration file or command line, with a TLS certificate injected into the container. Vault : As Vault's database is encrypted, Vault cannot natively bootstrap its own TLS certificate. Requires TLS certificate to be injected into container and its location set in a configuration file. PostgreSQL : Requires TLS certificate to be injected into '$PGDATA' (default: /var/lib/postgresql/data ) which is where the writable database files are kept. Kong (admin) : Requires environment variable to be set to secure admin port with TLS, with a TLS certificates injected into the container. Kong (external) : Requires a bring-your-own (BYO) external certificate, or as a fallback, a default one should be generated using a configurable external hostname. (The Kong ACME plugin could possibly be used to automate this process.) Redis (v6) : Requires TLS certificate set by configuration file or command line, with a TLS certificate injected into the container. Mosquitto : Requires TLS certificate set by configuration file, with a TLS certificate injected into the container. Additionally, every EdgeX microservice consumer will require access to the root CA for certificate verification purposes, and every EdgeX microservice server will need a TLS leaf certificate and private key. Note that Vault bootstrapping its own PKI is tricky and not natively supported by Vault. Expect that a non-trivial amount of effort will need to be put into starting Vault in non-secure mode to create the CA hierarchy and a TLS certificate for Vault itself, and then restarting Vault in a TLS-enabled configuration. Periodic certificate rotation is a non-trivial challenge as well. The Vault bootstrapping flow would look something like this: Bring up vault on localhost with TLS disabled (bootstrapping configuration) Initialize a blank Vault and immediately unseal it Encrypt the Vault keyshares and revoke the root token Generate a new root from the keyshares Generate an on-device root CA (see https://learn.hashicorp.com/vault/secrets-management/sm-pki-engine) Create an intermediate CA for TLS server authentication Sign the intermediate CA using the root CA Configure policy for intermediate CA Generate and store leaf certificates for Consul, Vault, PostgreSQL, Kong (admin), Kong (external), Redis (v6), Mosquitto Deploy the PKI to the respective services' secrets area Write the production Vault configuration (TLS-enabled) to a Docker volume There are no current plans for mutual auth TLS. Supporting mutual auth TLS would require creation of a separate PKI hierarchy for generation of TLS client certificates and glue logic to persist the certificates in the service's key-value secret store and provide them when connecting to other EdgeX services.","title":"Use of encryption to secure in-cluster EdgeX communications"},{"location":"design/adr/security/0015-in-cluster-tls/#use-of-encryption-to-secure-in-cluster-edgex-communications","text":"","title":"Use of encryption to secure in-cluster EdgeX communications"},{"location":"design/adr/security/0015-in-cluster-tls/#status","text":"Approved","title":"Status"},{"location":"design/adr/security/0015-in-cluster-tls/#context","text":"This ADR seeks to define the EdgeX direction on using encryption to secure \"in-cluster\" EdgeX communications, that is, internal microservice-to-microservice communication. This ADR will seek to clarify the EdgeX direction in several aspects with regard to: EdgeX services communicating within a single host EdgeX services communicating across multiple hosts Using encryption for confidentiality or integrity in communication Using encryption for authentication between microservices This ADR will be used to triage EdgeX feature requests in this space.","title":"Context"},{"location":"design/adr/security/0015-in-cluster-tls/#background","text":"","title":"Background"},{"location":"design/adr/security/0015-in-cluster-tls/#why-encrypt","text":"Why consider encryption in the first place? Simple. Encryption helps with the following problems: Client authentication of servers. The client knows that it is talking to the correct server. This is typically achieved using TLS server certificates that the client checks against a trusted root certificate authority. Since the client is not in charge of network routing, TLS server authentication provides a good assurance that the requests are being routed to the correct server. Server authentication of clients. The server knows the identity of the client that has connected to it. There are a variety of mechanims to achieve this, such as usernames and passwords, tokens, claims, et cetera, but the mechanism under consideration by this ADR is TLS client authentication using TLS client certificates. Confidentiality of messages exchanged between services. Confidentiality is needed to protect authentication data flowing between communicating microservices as well as to protect the message payloads if they contain nonpublic data. TLS provides communication channel confidentiality. Integrity of messages exchanged between services. Integrity is needed to ensure that messages between communicating microservices are not maliciously altered, such as inserting or deleting data in the middle of the exchange. TLS provides communication channel integrity. A microservice architecture normally strives for all of the above protections. Besides TLS, there are other mechanisms that can be used to provide some of the above properties. For example, IPSec tunnels provide confidentity, integrity, and authentication of the hosts (network-level protection). SSH tunnels provide confidentiality, integrity, and authentication of the tunnel endpoints (also network-level protection). TLS, however, is preferred, because it operates in-process at the application level and provides better point-to-point security.","title":"Why encrypt?"},{"location":"design/adr/security/0015-in-cluster-tls/#why-to-not-encrypt","text":"In the case of TLS communications, microservices depend on an asymmetric private key to prove their identity. To be of value, this private key must be kept secret. Applications typically depend on process-level isolation and/or file system protections for the private key. Moreover, interprocess communication using sockets is mediated by the operating system kernel. An attacker running at the privilege of the operating system has the ability to compromise TLS protections, such as by substituting a private key or certificate authority of their choice, accessing the unencrypted data in process memory, or intercepting the network communications that flow through the kernel. Therefore, within a single host, TLS protections may slow down an attacker, but are not likely to stop them. Additionally, use of TLS requires management of additional security assets in the form of TLS private keys. Microservice communcation across hosts, however, is vulnerable to intereception, and must be protected via some mechanism such as, but not limited to: IPSec or SSH tunnels, encrypted overlay networks, service mesh middlewares, or application-level TLS. Another reason to not encrypt is that TLS adds overhead to microservice communication in the form of additional network around-trips when opening connections and performing cryptographic public key and symmetric key operations.","title":"Why to not encrypt?"},{"location":"design/adr/security/0015-in-cluster-tls/#decision","text":"At this time, EdgeX is primarily a single-node IoT application framework. Should this position change, this ADR should be revisited. Based on the single-node assumption: TLS will not be used for confidentiality and integrity of internal on-host microservice communication. TLS will be avoided as an authentication mechanism of peer microservices. Integrity and confidentiality of microservice communcations crossing host boundaries is required to secure EdgeX, but are an EdgeX customer responsibility. EdgeX customers are welcome to add extra security to their own EdgeX deployments.","title":"Decision"},{"location":"design/adr/security/0015-in-cluster-tls/#consequences","text":"This ADR if approved would close the following issues as will-not-fix. https://github.com/edgexfoundry/edgex-go/issues/1942 https://github.com/edgexfoundry/edgex-go/issues/1941 https://github.com/edgexfoundry/edgex-go/issues/2454 https://github.com/edgexfoundry/developer-scripts/issues/240 https://github.com/edgexfoundry/edgex-go/issues/2495 It would also close https://github.com/edgexfoundry/edgex-go/issues/1925 as there is no current need for TLS as a mutual authentication strategy.","title":"Consequences"},{"location":"design/adr/security/0015-in-cluster-tls/#alternatives","text":"","title":"Alternatives"},{"location":"design/adr/security/0015-in-cluster-tls/#encrypted-overlay-networks","text":"Encrypted overlay networks provide varying protection based on the product used. Some can only encrypt data, such as an IPsec tunnel. Some can encrypt and provide for network microsegmentation, such as Docker Swarm networks with encryption enabled. Some can encrypt and enforce network policy such as restrictions on ingress traffic or restrictions on egress traffic.","title":"Encrypted overlay networks"},{"location":"design/adr/security/0015-in-cluster-tls/#service-mesh-middleware","text":"Service mesh middleware is an alternative that should be investigated if EdgeX decides to fully support a Kubernetes-based deployment using distributed Kubernetes pods. A service mesh typically achieves most of the security objectives of security microservice commuication by intercepting microservice communications and imposing a configuration-driven policy that typically includes confidentiality and integrity protection. These middlewares typically rely on the Kubernetes pod construct and are difficult to support for non-Kubernetes deployments.","title":"Service mesh middleware"},{"location":"design/adr/security/0015-in-cluster-tls/#edgex-public-key-infrastructure","text":"An EdgeX public key infrastructure that is natively supported by the architecture should be considered if EdgeX decides to support an out-of-box distributed deployment on non-Kubernetes platforms. Native support of TLS requires a significant amount of glue logic, and exceeds the availble resources in the security working group to implement this strategy. The following text outlines a proposed strategy for supporting native TLS in the EdgeX framework: EdgeX will use Hashicorp Vault to secure the EdgeX PKI, through the use of the Vault PKI secrets engine. Vault will be configured with a root CA at initialization time, and a Vault-based sub-CA for dynamic generation of TLS leaf certificates. The root CA will be restricted to be used only by the Vault root token. EdgeX microservices that are based on third-party containers require special support unless they can talk natively to Vault for their secrets. Certain tools, such as those mentioned in the \"Creation and Distribution of Secrets\" ADR ( envconsul , consul-template , and others) can be used to facilitiate third-party container integration. These services are: Consul : Requires TLS certificate set by configuration file or command line, with a TLS certificate injected into the container. Vault : As Vault's database is encrypted, Vault cannot natively bootstrap its own TLS certificate. Requires TLS certificate to be injected into container and its location set in a configuration file. PostgreSQL : Requires TLS certificate to be injected into '$PGDATA' (default: /var/lib/postgresql/data ) which is where the writable database files are kept. Kong (admin) : Requires environment variable to be set to secure admin port with TLS, with a TLS certificates injected into the container. Kong (external) : Requires a bring-your-own (BYO) external certificate, or as a fallback, a default one should be generated using a configurable external hostname. (The Kong ACME plugin could possibly be used to automate this process.) Redis (v6) : Requires TLS certificate set by configuration file or command line, with a TLS certificate injected into the container. Mosquitto : Requires TLS certificate set by configuration file, with a TLS certificate injected into the container. Additionally, every EdgeX microservice consumer will require access to the root CA for certificate verification purposes, and every EdgeX microservice server will need a TLS leaf certificate and private key. Note that Vault bootstrapping its own PKI is tricky and not natively supported by Vault. Expect that a non-trivial amount of effort will need to be put into starting Vault in non-secure mode to create the CA hierarchy and a TLS certificate for Vault itself, and then restarting Vault in a TLS-enabled configuration. Periodic certificate rotation is a non-trivial challenge as well. The Vault bootstrapping flow would look something like this: Bring up vault on localhost with TLS disabled (bootstrapping configuration) Initialize a blank Vault and immediately unseal it Encrypt the Vault keyshares and revoke the root token Generate a new root from the keyshares Generate an on-device root CA (see https://learn.hashicorp.com/vault/secrets-management/sm-pki-engine) Create an intermediate CA for TLS server authentication Sign the intermediate CA using the root CA Configure policy for intermediate CA Generate and store leaf certificates for Consul, Vault, PostgreSQL, Kong (admin), Kong (external), Redis (v6), Mosquitto Deploy the PKI to the respective services' secrets area Write the production Vault configuration (TLS-enabled) to a Docker volume There are no current plans for mutual auth TLS. Supporting mutual auth TLS would require creation of a separate PKI hierarchy for generation of TLS client certificates and glue logic to persist the certificates in the service's key-value secret store and provide them when connecting to other EdgeX services.","title":"EdgeX public key infrastructure"},{"location":"design/adr/security/0016-docker-image-guidelines/","text":"Docker image guidelines Status Approved Context When deploying the EdgeX Docker containers some security measures are recommended to ensure the integrity of the software stack. Decision When deploying Docker images, the following flags should be set for heightened security. To avoid escalation of privileges each docker container should use the no-new-privileges option in their Docker compose file (example below). More details about this flag can be found here . This follows Rule #4 for Docker security found here . security_opt: - \"no-new-privileges:true\" NOTE: Alternatively an AppArmor security profile can be used to isolate the docker container. More details about apparmor profiles can be found here security_opt: [ \"apparmor:unconfined\" ] To further prevent privilege escalation attacks the user should be set for the docker container using the --user= or -u= option in their Docker compose file (example below). More details about this flag can be found here . This follows Rule #2 for Docker security found here . services: device-virtual: image: ${ REPOSITORY } /docker-device-virtual-go ${ ARCH } : ${ DEVICE_VIRTUAL_VERSION } user: $CONTAINER -PORT: $CONTAINER -PORT # user option using an unprivileged user ports: - \"127.0.0.1:49990:49990\" container_name: edgex-device-virtual hostname: edgex-device-virtual networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-virtual depends_on: - consul - data - metadata NOTE: exception Sometimes containers will require root access to perform their fuctions. For example the System Management Agent requires root access to control other Docker containers. In this case you would allow it run as default root user. To avoid a faulty or compromised containers from consuming excess amounts of the host of its resources resource limits should be set for each container. More details about resource limits can be found here . This follows Rule #7 for Docker security found here . services: device-virtual: image: ${ REPOSITORY } /docker-device-virtual-go ${ ARCH } : ${ DEVICE_VIRTUAL_VERSION } user: 4000 :4000 # user option using an unprivileged user ports: - \"127.0.0.1:49990:49990\" container_name: edgex-device-virtual hostname: edgex-device-virtual networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-virtual depends_on: - consul - data - metadata deploy: # Deployment resource limits resources: limits: cpus: '0.001' memory: 50M reservations: cpus: '0.0001' memory: 20M To avoid attackers from writing data to the containers and modifying their files the --read_only flag should be set. More details about this flag can be found here . This follows Rule #8 for Docker security found here . device-rest: image: ${ REPOSITORY } /docker-device-rest-go ${ ARCH } : ${ DEVICE_REST_VERSION } ports: - \"127.0.0.1:49986:49986\" container_name: edgex-device-rest hostname: edgex-device-rest read_only: true # read_only option networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-rest depends_on: - data - command NOTE: exception If a container is required to have write permission to function, then this flag will not work. For example, the vault needs to run setcap in order to lock pages in memory. In this case the --read_only flag will not be used. NOTE: Volumes If writing persistent data is required then a volume can be used. A volume can be attached to the container in the following way device-rest: image: ${ REPOSITORY } /docker-device-rest-go ${ ARCH } : ${ DEVICE_REST_VERSION } ports: - \"127.0.0.1:49986:49986\" container_name: edgex-device-rest hostname: edgex-device-rest read_only: true # read_only option networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-rest depends_on: - data - command volumes: - consul-config:/consul/config:z NOTE: alternatives If writing non-persistent data is required (ex. a config file) then a temporary filesystem mount can be used to accomplish this goal while still enforcing --read_only . Mounting a tmpfs in Docker gives the container a temporary location in the host systems memory to modify files. This location will be removed once the container is stopped. More details about tmpfs can be found here for additional docker security rules and guidelines please check the Docker security cheatsheet Consequences Create a more secure Docker environment References Docker-compose reference https://docs.docker.com/compose/compose-file OWASP Docker Recommendations https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html CIS Docker Benchmark https://workbench.cisecurity.org/files/2433/download/2786 (registration required)","title":"Docker image guidelines"},{"location":"design/adr/security/0016-docker-image-guidelines/#docker-image-guidelines","text":"","title":"Docker image guidelines"},{"location":"design/adr/security/0016-docker-image-guidelines/#status","text":"Approved","title":"Status"},{"location":"design/adr/security/0016-docker-image-guidelines/#context","text":"When deploying the EdgeX Docker containers some security measures are recommended to ensure the integrity of the software stack.","title":"Context"},{"location":"design/adr/security/0016-docker-image-guidelines/#decision","text":"When deploying Docker images, the following flags should be set for heightened security. To avoid escalation of privileges each docker container should use the no-new-privileges option in their Docker compose file (example below). More details about this flag can be found here . This follows Rule #4 for Docker security found here . security_opt: - \"no-new-privileges:true\" NOTE: Alternatively an AppArmor security profile can be used to isolate the docker container. More details about apparmor profiles can be found here security_opt: [ \"apparmor:unconfined\" ] To further prevent privilege escalation attacks the user should be set for the docker container using the --user= or -u= option in their Docker compose file (example below). More details about this flag can be found here . This follows Rule #2 for Docker security found here . services: device-virtual: image: ${ REPOSITORY } /docker-device-virtual-go ${ ARCH } : ${ DEVICE_VIRTUAL_VERSION } user: $CONTAINER -PORT: $CONTAINER -PORT # user option using an unprivileged user ports: - \"127.0.0.1:49990:49990\" container_name: edgex-device-virtual hostname: edgex-device-virtual networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-virtual depends_on: - consul - data - metadata NOTE: exception Sometimes containers will require root access to perform their fuctions. For example the System Management Agent requires root access to control other Docker containers. In this case you would allow it run as default root user. To avoid a faulty or compromised containers from consuming excess amounts of the host of its resources resource limits should be set for each container. More details about resource limits can be found here . This follows Rule #7 for Docker security found here . services: device-virtual: image: ${ REPOSITORY } /docker-device-virtual-go ${ ARCH } : ${ DEVICE_VIRTUAL_VERSION } user: 4000 :4000 # user option using an unprivileged user ports: - \"127.0.0.1:49990:49990\" container_name: edgex-device-virtual hostname: edgex-device-virtual networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-virtual depends_on: - consul - data - metadata deploy: # Deployment resource limits resources: limits: cpus: '0.001' memory: 50M reservations: cpus: '0.0001' memory: 20M To avoid attackers from writing data to the containers and modifying their files the --read_only flag should be set. More details about this flag can be found here . This follows Rule #8 for Docker security found here . device-rest: image: ${ REPOSITORY } /docker-device-rest-go ${ ARCH } : ${ DEVICE_REST_VERSION } ports: - \"127.0.0.1:49986:49986\" container_name: edgex-device-rest hostname: edgex-device-rest read_only: true # read_only option networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-rest depends_on: - data - command NOTE: exception If a container is required to have write permission to function, then this flag will not work. For example, the vault needs to run setcap in order to lock pages in memory. In this case the --read_only flag will not be used. NOTE: Volumes If writing persistent data is required then a volume can be used. A volume can be attached to the container in the following way device-rest: image: ${ REPOSITORY } /docker-device-rest-go ${ ARCH } : ${ DEVICE_REST_VERSION } ports: - \"127.0.0.1:49986:49986\" container_name: edgex-device-rest hostname: edgex-device-rest read_only: true # read_only option networks: - edgex-network env_file: - common.env environment: SERVICE_HOST: edgex-device-rest depends_on: - data - command volumes: - consul-config:/consul/config:z NOTE: alternatives If writing non-persistent data is required (ex. a config file) then a temporary filesystem mount can be used to accomplish this goal while still enforcing --read_only . Mounting a tmpfs in Docker gives the container a temporary location in the host systems memory to modify files. This location will be removed once the container is stopped. More details about tmpfs can be found here for additional docker security rules and guidelines please check the Docker security cheatsheet","title":"Decision"},{"location":"design/adr/security/0016-docker-image-guidelines/#consequences","text":"Create a more secure Docker environment","title":"Consequences"},{"location":"design/adr/security/0016-docker-image-guidelines/#references","text":"Docker-compose reference https://docs.docker.com/compose/compose-file OWASP Docker Recommendations https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html CIS Docker Benchmark https://workbench.cisecurity.org/files/2433/download/2786 (registration required)","title":"References"},{"location":"design/adr/security/0017-consul-security/","text":"Securing access to Consul Status Approved Context This ADR defines the motiviation and approach used to secure access to the Consul component in the EdgeX architecture for security-enabled configurations only . Non-secure configuations continue to use Consul in anonymous read-write mode. As this Consul security feature requires Vault to function, if EDGEX_SECURITY_SECRET_STORE=false and Vault is not present, the legacy behavior (unauthenticated Consul access) will be preserved. Consul provides several services for the EdgeX architecture: Service registry (see ADR in references below) Service health monitoring Mutable configuration data Use of the services provided by Consul is optional on a service-by-service basis. Use of the registry is controlled by the -r or --registry flag provided to an EdgeX service. Use of mutable configuration data is controlled by the -cp or --configProvider flag provided to an EdgeX service. When Consul is enabled as a configuration provider, the configuration.toml is parsed into individual settings and seeded into the Consul key-value store on the first start of a service. Configuration reads and writes are then done to Consul if it is specified as the configuration provider, otherwise the static configuration.toml is used. Writes to the [Writable] section in Consul trigger per-service callbacks notifying the application of the changed data. Updates to non- [Writable] sections are parsed only once at startup and require a service restart to take effect. Since configuration data can affect the runtime behavior of services, compensating controls must be introduced in order to mitigate the risks introduced by moving configuration from a static file into to an HTTP-accessible service with mutable state. The current practice is that Consul is exposed via unencrypted HTTP in anonymous read/write mode to all processes and EdgeX services running on the host machine. Decision Consul will be configured with access control list (ACL) functionality enabled, and each EdgeX service will utilize a Consul access token to authenticate to Consul. Consul access tokens will be requested from the Vault Consul secrets engine (to avoid introducing additional bootstrapping secrets). DNS will be disabled via configuration as it is not used in EdgeX. Consul Access Via API Gateway In security enabled EdgeX, the API gateway will be configured to proxy the Consul service over the /consul path, using the request-transformer plugin to add the global management token to incoming requests via the X-Consul-Token HTTP header. Thus, ability to access remote APIs also grants the ability to modify Consul's key-value store. At this time, service access via API gateway is all-or-nothing, but this does not preclude future fine-grained authorization at the API gateway layer to specific microservices, including Consul. Proxying of the Consul UI is problematic and there is no current solution, which would involve proper balacing of the externally-visible URL, the path-stripping effect (or not) of the proxy, Consul's ui_content_path , and UI authentication (the request-transfomer does not work on the UI). Consequences Full implementation of this ADR will deny Consul access to all existing Consul clients. To limit the impacts of the change, deployment will take place in phases. Phase 1 is basic plumbing work and leaves Consul configured in a permissive mode and thus is not a breaking change. Phase 2 will affect the APIs of Go modules and will change the default policy to \"deny\", both of which are breaking changes. Phase 3 is a refinement of access control; presuming the existing services are \"well-behaved\", that is, they do not access configuration of other services, Phase 3 will not introduce any breaking changes on top of the Phase 2 breaking changes. Phase 1 (completed in Ireland release) Vault bootstrapper will install Vault Consul secrets engine. Secretstore-setup will create a Vault token for consul secrets engine configuration. Consul will be started with Consul ACLs enabled with persistent agent tokens and a default \"allow\" policy. Consul bootstrapper will create a bootstrap management token and use the provided Vault token to (re)configure the Consul secrets engine in Vault. Do to a quirk in Consul's ACL behavior that inverts the meaning of an ACL in default-allow mode, in phase 1 the Consul bootstrapper will create an agent token with the global-management policy and install it into the agent. During phase 2, it will be changed to a specific, limited, policy. (This change should not be visible to Consul API clients.) The bootstrap management token will also be stored persistently to be used by the API gateway for proxy authentication, and will also be needed for local access to Consul's web user interface. (Docker-only) Open a port to signal that Consul bootstrapping is completed. (Integrate with ready_to_run signal.) Phase 2 (completed in Ireland release) Consul bootstrapper will install a role in Vault that creates global-management tokens in Consul with no TTL. Registry and configuration client libraries will be modified to accept a Consul access token. go-mod-bootstrap will have contain the necessary glue logic to request a service-specifc Consul access token from Vault every time the service is started. Consul configuration will be changed to a default \"deny\" policy once all services have been changed to authenticated access mode. The agent tokens' policy will be changed to a specific agent policy instead of the global-management policy. Phase 3 (for Jakarta release) Introduce per-service roles and ACL policies that give each service access to its own subset of the Consul key-value store and to register in the service registry. Consul access tokens will be scoped to the needs of the particular service (ability to update that service's registry data, an access that services's KV store). Create a separate management token (non-bootstrap) for API gateway proxy authentication and Consul UI access that is different from boostrap management token stored in Vault. This token will need to be requested outside of Vault in order for it to be non-expiring. Glue logic will ensure that expired Consul tokens are replaced with fresh ones (token freshness can be pre-checked by a request made to /acl/token/self ). Unintended consequences and mitigation (for Jakarta stabilization release) Consul token lifetime will be tied to the Vault token lifetime. Vault deliberately revokes any Consul tokens that it issues in order to ensure that they don't outlive the parent token's lifetime. If Consul is not fully initialized when token revokation is attempted, Vault will be unable to revoke these tokens. Migtigations: Consul will be started concurrently with Vault to give time for Consul to fully initialize. secretstore-setup will delay starting until Consul has completed leader election. secretstore-setup will be modified to less aggressively revoke tokens. Alternatives include revoke-and-orphan which should leave the Consul tokens intact if the secret store is restarted but may leave garbage tokens in the Consul database, or tidy-tokens which cleans up invalid entries in the token database, or simply leave Vault to its own devices and let Vault clean itself up. Testing will be performed and an appropriate mechanism selected. References ADR for secret creation and distribution ADR for secure bootstrapping ADR for service registry Hashicorp Vault","title":"Securing access to Consul"},{"location":"design/adr/security/0017-consul-security/#securing-access-to-consul","text":"","title":"Securing access to Consul"},{"location":"design/adr/security/0017-consul-security/#status","text":"Approved","title":"Status"},{"location":"design/adr/security/0017-consul-security/#context","text":"This ADR defines the motiviation and approach used to secure access to the Consul component in the EdgeX architecture for security-enabled configurations only . Non-secure configuations continue to use Consul in anonymous read-write mode. As this Consul security feature requires Vault to function, if EDGEX_SECURITY_SECRET_STORE=false and Vault is not present, the legacy behavior (unauthenticated Consul access) will be preserved. Consul provides several services for the EdgeX architecture: Service registry (see ADR in references below) Service health monitoring Mutable configuration data Use of the services provided by Consul is optional on a service-by-service basis. Use of the registry is controlled by the -r or --registry flag provided to an EdgeX service. Use of mutable configuration data is controlled by the -cp or --configProvider flag provided to an EdgeX service. When Consul is enabled as a configuration provider, the configuration.toml is parsed into individual settings and seeded into the Consul key-value store on the first start of a service. Configuration reads and writes are then done to Consul if it is specified as the configuration provider, otherwise the static configuration.toml is used. Writes to the [Writable] section in Consul trigger per-service callbacks notifying the application of the changed data. Updates to non- [Writable] sections are parsed only once at startup and require a service restart to take effect. Since configuration data can affect the runtime behavior of services, compensating controls must be introduced in order to mitigate the risks introduced by moving configuration from a static file into to an HTTP-accessible service with mutable state. The current practice is that Consul is exposed via unencrypted HTTP in anonymous read/write mode to all processes and EdgeX services running on the host machine.","title":"Context"},{"location":"design/adr/security/0017-consul-security/#decision","text":"Consul will be configured with access control list (ACL) functionality enabled, and each EdgeX service will utilize a Consul access token to authenticate to Consul. Consul access tokens will be requested from the Vault Consul secrets engine (to avoid introducing additional bootstrapping secrets). DNS will be disabled via configuration as it is not used in EdgeX. Consul Access Via API Gateway In security enabled EdgeX, the API gateway will be configured to proxy the Consul service over the /consul path, using the request-transformer plugin to add the global management token to incoming requests via the X-Consul-Token HTTP header. Thus, ability to access remote APIs also grants the ability to modify Consul's key-value store. At this time, service access via API gateway is all-or-nothing, but this does not preclude future fine-grained authorization at the API gateway layer to specific microservices, including Consul. Proxying of the Consul UI is problematic and there is no current solution, which would involve proper balacing of the externally-visible URL, the path-stripping effect (or not) of the proxy, Consul's ui_content_path , and UI authentication (the request-transfomer does not work on the UI).","title":"Decision"},{"location":"design/adr/security/0017-consul-security/#consequences","text":"Full implementation of this ADR will deny Consul access to all existing Consul clients. To limit the impacts of the change, deployment will take place in phases. Phase 1 is basic plumbing work and leaves Consul configured in a permissive mode and thus is not a breaking change. Phase 2 will affect the APIs of Go modules and will change the default policy to \"deny\", both of which are breaking changes. Phase 3 is a refinement of access control; presuming the existing services are \"well-behaved\", that is, they do not access configuration of other services, Phase 3 will not introduce any breaking changes on top of the Phase 2 breaking changes.","title":"Consequences"},{"location":"design/adr/security/0017-consul-security/#phase-1-completed-in-ireland-release","text":"Vault bootstrapper will install Vault Consul secrets engine. Secretstore-setup will create a Vault token for consul secrets engine configuration. Consul will be started with Consul ACLs enabled with persistent agent tokens and a default \"allow\" policy. Consul bootstrapper will create a bootstrap management token and use the provided Vault token to (re)configure the Consul secrets engine in Vault. Do to a quirk in Consul's ACL behavior that inverts the meaning of an ACL in default-allow mode, in phase 1 the Consul bootstrapper will create an agent token with the global-management policy and install it into the agent. During phase 2, it will be changed to a specific, limited, policy. (This change should not be visible to Consul API clients.) The bootstrap management token will also be stored persistently to be used by the API gateway for proxy authentication, and will also be needed for local access to Consul's web user interface. (Docker-only) Open a port to signal that Consul bootstrapping is completed. (Integrate with ready_to_run signal.)","title":"Phase 1 (completed in Ireland release)"},{"location":"design/adr/security/0017-consul-security/#phase-2-completed-in-ireland-release","text":"Consul bootstrapper will install a role in Vault that creates global-management tokens in Consul with no TTL. Registry and configuration client libraries will be modified to accept a Consul access token. go-mod-bootstrap will have contain the necessary glue logic to request a service-specifc Consul access token from Vault every time the service is started. Consul configuration will be changed to a default \"deny\" policy once all services have been changed to authenticated access mode. The agent tokens' policy will be changed to a specific agent policy instead of the global-management policy.","title":"Phase 2 (completed in Ireland release)"},{"location":"design/adr/security/0017-consul-security/#phase-3-for-jakarta-release","text":"Introduce per-service roles and ACL policies that give each service access to its own subset of the Consul key-value store and to register in the service registry. Consul access tokens will be scoped to the needs of the particular service (ability to update that service's registry data, an access that services's KV store). Create a separate management token (non-bootstrap) for API gateway proxy authentication and Consul UI access that is different from boostrap management token stored in Vault. This token will need to be requested outside of Vault in order for it to be non-expiring. Glue logic will ensure that expired Consul tokens are replaced with fresh ones (token freshness can be pre-checked by a request made to /acl/token/self ).","title":"Phase 3 (for Jakarta release)"},{"location":"design/adr/security/0017-consul-security/#unintended-consequences-and-mitigation-for-jakarta-stabilization-release","text":"Consul token lifetime will be tied to the Vault token lifetime. Vault deliberately revokes any Consul tokens that it issues in order to ensure that they don't outlive the parent token's lifetime. If Consul is not fully initialized when token revokation is attempted, Vault will be unable to revoke these tokens. Migtigations: Consul will be started concurrently with Vault to give time for Consul to fully initialize. secretstore-setup will delay starting until Consul has completed leader election. secretstore-setup will be modified to less aggressively revoke tokens. Alternatives include revoke-and-orphan which should leave the Consul tokens intact if the secret store is restarted but may leave garbage tokens in the Consul database, or tidy-tokens which cleans up invalid entries in the token database, or simply leave Vault to its own devices and let Vault clean itself up. Testing will be performed and an appropriate mechanism selected.","title":"Unintended consequences and mitigation (for Jakarta stabilization release)"},{"location":"design/adr/security/0017-consul-security/#references","text":"ADR for secret creation and distribution ADR for secure bootstrapping ADR for service registry Hashicorp Vault","title":"References"},{"location":"design/adr/security/0020-spiffe/","text":"Use SPIFFE/SPIRE for On-demand Secret Store Token Generation Status Approved via TSC vote on 2021-12-14 Context In security-enabled EdgeX, there is a component called security-secretstore-setup that seeds authentication tokens for Hashicorp Vault--EdgeX's secret store--into directories reserved for each EdgeX microservice. The implementation is provided by a sub-component, security-file-token-provider , that works off of a static configuration file ( token-config.json ) that configures known EdgeX services, and an environment variable that lists additional services that require tokens. The token provider creates a unique token for each service and attaches a custom policy to each token that limits token access in a manner that paritions the secret store's namespace. The current solution has some problematic aspects: These tokens have an initial TTL of one hour (1h) and become invalid if not used and renewed within that time period. It is not possible to delay the start of EdgeX services until a later time (that is, greater than the default token TTL), as they will not be able to connect to the EdgeX secret store to obtain required secrets. Transmission of the authentication token requires one or more shared file systems between the service and security-secretstore-setup . In the Docker implementation, this shared file system is constructed by bind-mounting a host-based directory to multiple containers. The snap implementation is similar, utilizing a content-interface between snaps. In a Kubernetes implementation limited to a single worker node, a CSI storage driver that provided RWO volumes would suffice. The current approach cannot support distributed services without an underlying distributed file system to distribute tokens, such as GlusterFS, running across the participating nodes. For Kubernetes, the requirement would be a remote shared file system persistent volume (RWX volume). Decision EdgeX will create a new service, security-spiffe-token-provider . This service will be a mutual-auth TLS service that exchanges a SPIFFE X.509 SVID for a secret store token. An SPIFFE identifier is a URI of the format spiffe://trust domain/workload identifier . For example: spiffe://edgexfoundry.org/service/core-data . A SPIFFE Verifiable Identity Document (SVID) is a cryptographically-signed version of a SPIFFE ID, typically a X.509 certificate with the SPIFFE ID encoded into the subjectAltName certificate extension, or a JSON web token (encoded into the sub claim). The EdgeX implementation will use a naming convention on the path component, such as the above, in order to be able to extract the requesting service from the SPIFFE ID. The SPIFFE token provider will take three parameters: An X.509 SVID used in mutual-auth TLS for the token provider and the service to cross-authenticate. The reqested service key. If blank, the service key will default to the service name encoded in the SVID. If the service name follows the pattern device-(name) , then the service key must follow the format device-(name) or device-name-* . If the service name is app-service-configurable , then the service key must follow the format app-* . (This is an accomodation for the Unix workload attester not being able to distingish workloads that are launched using the same executable binary. Custom app services that support multiple instances won't be supported unless they name the executable the same as the standard app service binary or modify this logic.) A list of \"known secret\" identifiers that will allow new services to request database passwords or other \"known secrets\" to be seeded into their service's partition in the secret store. The go-mod-secrets module will be modified to enable a new mode whereby a secret store token is obtained by: Obtaining an X.509 SVID by contacting a local SPIFFE agent's workload API on a local Unix domain socket. Connecting to the security-spiffe-token-provider service using the X.509 SVID to request a secret store token. The SPIFFE authentication mode will be an opt-in feature. The SPIFFE implementation will be user-replaceable; specifically, the workload API socket will be configurable, as well as the parsing of the SPIFFE ID. Reasons for doing so might include: changing the name of the trust domain in the SPIFFE ID, or moving the SPIFFE server out of the edge. This feature is estimated to be a \"large\" or \"extra large\" effort that could be implemented in a single release cycle. Technical Architecture The work flow is as follows: Create a root CA for the SPIFFE user to use for creation of sub-CA's. The SPIFFE server is started. The server creates a sub-CA for issuing new identities. The trust bundle (certificate authority) data is exported from the SPIFFE server and stored on a shared volume readable by other EdgeX microservices (i.e. the existing secrets volume used for sharing secret store tokens). A join token for the SPIFFE agent is created using token generate and shared to the EdgeX secrets volume. Workload entries are loaded into the SPIFFE server database, using the join-identity of the agent created in the previous step as the parent ID of the workload. The SPIFFE agent is started with the join token created in a previous step to add it to the cluster. Vault is started and security-secret-store-setup initializes it and creates an admin token for security-spiffe-token-provider to use. The security-spiffe-token-provider service is started. It obtains an SVID from the SIFFE agent and uses it as a TLS server certificate. An EdgeX microservice starts and obtains another SVID from the SPIFFE agent and uses it as a TLS client certificate to contact the security-spiffe-token-provider service. The EdgeX microservice uses the trust bundle as a server CA to verify the TLS certificate of the remote service. security-spiffe-token-provider verifies the SVID using the trust bundle as client CA to verify the client, extracts the service key, and issues an appropriate Vault service token. The EdgeX microservice accesses Vault as usual. Workload Registration and Agent Sockets The server uses a workload registration Unix domain socket that allows authorization entries to be added to the authorization database. This socket is protected by Unix file system permissions to control who is allowed to add entries to the database. In this proposal, a subcommand will be added to the EdgeX secrets-config utility to simplify the process of registering new services that uses the registration socket above. The agent uses a workload attesation Unix domain socket that is open to the world. This socket is shared via a snap content-interface of via a shared host bind mount for Docker. There is one agent per node. Trust Bundle SVID's must be traceable back to a known issuing authority (certificate authority) to determine their validity. In the proposed implementation, we will generate a CA on first boot and store it persistently. This root CA will be distributed as the trust bundle. The SPIFFE server will then generate a rotating sub-CA for issuing SVIDs, and the issued SVID will include both the leaf certificate and the intermediate certificate. This implementation differs from the default implementation, which uses a transient CA that is rotated periodically and that keeps a log of past CA's. The default implementation is not suitable because only the Kubernetes reference implementation of the SPIRE server has a notification hook that is invoked when the CA is rotated. CA rotation would just result in issuing of SVIDs that are not trusted by microservices that received only the initial CA. The SPIFFE implementation is replaceable. The user is free to replace this default implementation with potentally a cloud-based SPIFFE server and a cloud-based CA. Workload Authorization Workloads are authenticated by connecting to the spiffe-agent via a Unix domain socket, which is capable of identifying the process ID of the remote client. The process ID is fed into one of following workload attesters, which gather additional metadata about the caller: The Unix workload attester gathers UID, GID, path, and SHA-256 hash of the executable. The Unix workload attester would be used native services and snaps. The Docker workload attester gathers container labels that are added by docker-compose when the container is launched. The Docker workload attester would be used for Docker-based EdgeX deployments. An example label is docker:label:com.docker.compose.service:edgex-core-data where the service label is the key value in the services section of the docker-compose.yml . It is also possible to refer to labels built-in to the container image. The Kubernetes workload attester gathers a wealth of pod and container metadata. Once authenticated, the metadata is sent to the SPIFFE server to authorize the workload. Workloads are authorized via an authorization database connected to the SPIFFE server. Supported databases are SQLite (default), PostgreSQL, and MySQL. Due to startup ordering issues, SQLite will be used. (Disclaimer: SQlite, according for the Turtle book is intended for development and test only. We will use SQlite anyway because because Redis is not supported.) The only service that needs to be seeded to the database as this time is security-spiffe-token-provier . For example: spire-server entry create -parentID \" ${ local_agent_svid } \" -dns edgex-spiffe-token-provider -spiffeID \" ${ svid_service_base } /edgex-spiffe-token-provider\" -selector \"docker:label:com.docker.compose.service:edgex-spiffe-token-provider\" The above command associates a SPIFFE ID with a selector , in this case, a container label, and configures a DNS subjectAltName in the X.509 certificate for server-side TLS. A snap-based installation of EdgeX would use a unix:path or unix:sha256 selector instead. There are two extension mechanims for authorization additional workloads: Inject a config file or environment variable to authorize additional workloads. The container will parse and issue spire-server entry create commands for each additional service. Run the edgex-secrets-config utility (that will wrap the spire-server entry create command) for ad-hoc authorization of new services. The authorization database is persistent across reboots. Consequences This proposal will require addition of several new, optional, EdgeX microservices: security-spiffe-token-provider , running on the main node spiffe-agent , running on the main node and each remote node spiffe-server , running on the main node spiffe-config , a one-shot service running on the main node Note that like Vault, the recommended SPIFFE configuration is to run the SPIFFE server on a dedicated node. If this is a concern, bring your own SPIFFE implementation. Minor changes will be needed to security-secretstore-setup to preserve the token-creating-token used by security-file-token-provider so that it can be used by security-spiffe-token-provider . The startup flow of the framework will be adjusted as follows: Bootstrap service (original) spiffe-server spiffe-config (can be combined with spifee-server ) spiffe-agent Vault service (original) Secret store setup service (original) security-spiffe-token-provider Consul (original) Postgres (original) There is no direct dependency between spiffe-server and any other microservice. security-spiffe-token-provider requires an SVID from spiffe-agent and a Vault admin token. None of these new services will be proxied via the API gateway. In the future, this mechanism may become the default secret store distribution mechanism, as it eliminates several secrets volumes used to share secrets between security-secretstore-setup and various EdgeX microservices. The EdgeX automation will only configure the SPIFEE agent on the main node. Additional nodes can be manually added by the operator by obtaining a join token from the main node and using it to bootstrap a remote node. SPIFFE/SPIRE has native support for Kubernetes and can distribute the trust bundle via a Kubernetes ConfigMap to more easily enable distributed scenarios, removing a major roadblock to usage of EdgeX in a Kubernetes environment. Footprint NOTE: This data is limited by the fact that the pre-built SPIRE reference binaries are compiled with CGO enabled. SPIRE Server 69 MB executable, dynamically linked 151 MB inside of a Debian-slim container 30 MB memory usage, as container SPIRE Agent 33 MB executable, dynamically linked 114 MB inside of a Debian-slim container 64 MB memory usage, as container SPIFFE-base Secret Store Token Provider The following is the minimum size: > 6 MB executable (likely much larger) > 29 MB memory usage, as container Limitations The following are known limitations with this proposal: The capabilities enabled by this solution would only be enabled on Linux platforms. SIFFE/SPIRE Agent is not available for native Windows and pre-built binaries are only avaiable for Linux. (It is unclear as to whether other *nix'es are supported.) The capabilities enabled by this solution would only be supported for Go-based services. The SPIFFE API's are implemented in gRPC, which is only ported to C#, C++, Dart, Go, Java, Kotlin, Node, Objective-C, PHP, Python, and Ruby. Notably, the C language is not supported, and the only other EdgeX supported language is Go. That default TTL of an x.509 SVID is one hour. As such, all SVID consumers must be capable of auto-renewal of SVIDs on both the client and server side. Alternatives Overcoming lack of a supported GRPC-C library Leave C-SDK device services behind. In this option, C device services would be unable to participate in the delayed-start services architecture. Fork a grpc-c library. Forking a grpc-c library and rehabilitating it is one option. There is at least one grpc-c library that has been proven to work, but it requires additional features to make it compatible with the SPIRE workload agent. However, the project is extremely large and it is unlikely that EdgeX is big enough to carry the project. Available libraries include: https://github.com/lixiangyun/grpc-c This library is several years out-of-date, does not compile on current Linux distributions without some rework, and does not pass per-request metadata tags. Proved to work via manual patching. Not supportable. https://github.com/Juniper/grpc-c This library is serveral years out-of-date, also does not compile on current Linux distributiosn without some rework. Uses hard-coded Unix domain socket paths. May support per-request metadata tags, but did not test. Not supportable. https://github.com/HewlettPackard/c-spiffe This library is yet untested. Rather than a gRPC library, this library implements the workload API client directly. Ultimately, this library also wraps the gRPC C++ library, and statically links to it. There is no benefit to the EdgeX project to use this library as we can call the underlying library directly. Hybrid device services. In this model, device services would always be written in Go, but in the case where linking to a C language library is required, CGO features would be used to invoke native C functions from golang. This option would commit the EdgeX project to a one-time investment to port the existing C device services to the new hybrid model. This option is the best choice if the long-term strategy is to end-of-life the C Device SDK. Bridge. In this model, the C++ implementation to invoke the SPIFFE/SPIRE workload API would be hidden behind a dynamic shared library with C linkage. This would require minimal change to the existing C SDK. However, the resulting binaries would have be based on GLIBC vs MUSL in order to get dlopen() support. This will also limit the choice of container base images for containerized services. Modernize. In this model, the Device SDK would be rewritten either partially or in-full in C++. Under this model, the SPIFFE/SPIRE workload API could be accessed via a community-supported C++ GRPC SDK. There are many implementation options: A \"C++ compilation-switch\" where the C SDK could be compiled in C-mode or C++-mode with enhanced functionality. A C++ extension API. The original C SDK would remain as-is, but if compiling with __cplusplus defined, additional API methods would be exposed. The SDK could thus be composed of a mixture of .c files with C linkage and .cc files with C++ linkage. The linker would ultimately determine whether or not the C++ runtime library needed to be linked in. Native C++ device SDK with legacy C wrapper facade. Compile existing code in C++ mode, with optional C++ facade. Opt-in or Standard Feature If one of the following things were to happen, it would push this proposal \"over the edge\" from being an optional opt-in feature to a required standard feature for security: The \"on-demand\" method of obtaining a secret store token is the default method of obtaining a token for non-core EdgeX services. The \"on-demand\" method of obtaining a secret store token is the default method for all EdgeX services. SPIFFE SVID's become the implementation mechanism for microservice-level authentication. (Not in scope for this ADR.) Merge security-file-token-provider and security-spiffe-token-provider Keeping these as separate executables clearly separates the on-demand secret store tokens feature as an optional service. It is possible to combine the services, but there would need to be a configuration switch in order to enable the SPIFFE feature. It would also increase the base executable size to include the extra logic. Alternatives regarding SPIFFE CA Transient CA option The SPIFFE server can be configured with no \"upstream authority\" (certificate authority), and the server will periodically generate a new, transient CA, and keep a bounded history of previous CA's. A rotating trust bundle only practically works in a Kubernetes environment, since a configmap can be updated real-time. For everyone else, we need a static CA that can be pre-distributed to remote nodes. Thus, this solution was not chosen. Vault-based CA option The SPIFFE server can be configured to make requests to a Hashicorp Vault PKI secrets engine to generate intermediate CA certificates for signing SVID's. This is an option for future integrations, but is omitted from this proposal due to the jump in implementation complexity and the desire that the current proposal be on add-on feature. The current implementation allows the SPIFFE server and Vault to be started simultaneously. Using a Vault-based CA would require a complex interlocking sequence of steps. References Issue to create ADR for handling delayed-start services 0018 Service Registry ADR Service List ADR SPIFFE SPIFFE ID X.500 SVID JWT SVID Turtle book","title":"Use SPIFFE/SPIRE for On-demand Secret Store Token Generation"},{"location":"design/adr/security/0020-spiffe/#use-spiffespire-for-on-demand-secret-store-token-generation","text":"","title":"Use SPIFFE/SPIRE for On-demand Secret Store Token Generation"},{"location":"design/adr/security/0020-spiffe/#status","text":"Approved via TSC vote on 2021-12-14","title":"Status"},{"location":"design/adr/security/0020-spiffe/#context","text":"In security-enabled EdgeX, there is a component called security-secretstore-setup that seeds authentication tokens for Hashicorp Vault--EdgeX's secret store--into directories reserved for each EdgeX microservice. The implementation is provided by a sub-component, security-file-token-provider , that works off of a static configuration file ( token-config.json ) that configures known EdgeX services, and an environment variable that lists additional services that require tokens. The token provider creates a unique token for each service and attaches a custom policy to each token that limits token access in a manner that paritions the secret store's namespace. The current solution has some problematic aspects: These tokens have an initial TTL of one hour (1h) and become invalid if not used and renewed within that time period. It is not possible to delay the start of EdgeX services until a later time (that is, greater than the default token TTL), as they will not be able to connect to the EdgeX secret store to obtain required secrets. Transmission of the authentication token requires one or more shared file systems between the service and security-secretstore-setup . In the Docker implementation, this shared file system is constructed by bind-mounting a host-based directory to multiple containers. The snap implementation is similar, utilizing a content-interface between snaps. In a Kubernetes implementation limited to a single worker node, a CSI storage driver that provided RWO volumes would suffice. The current approach cannot support distributed services without an underlying distributed file system to distribute tokens, such as GlusterFS, running across the participating nodes. For Kubernetes, the requirement would be a remote shared file system persistent volume (RWX volume).","title":"Context"},{"location":"design/adr/security/0020-spiffe/#decision","text":"EdgeX will create a new service, security-spiffe-token-provider . This service will be a mutual-auth TLS service that exchanges a SPIFFE X.509 SVID for a secret store token. An SPIFFE identifier is a URI of the format spiffe://trust domain/workload identifier . For example: spiffe://edgexfoundry.org/service/core-data . A SPIFFE Verifiable Identity Document (SVID) is a cryptographically-signed version of a SPIFFE ID, typically a X.509 certificate with the SPIFFE ID encoded into the subjectAltName certificate extension, or a JSON web token (encoded into the sub claim). The EdgeX implementation will use a naming convention on the path component, such as the above, in order to be able to extract the requesting service from the SPIFFE ID. The SPIFFE token provider will take three parameters: An X.509 SVID used in mutual-auth TLS for the token provider and the service to cross-authenticate. The reqested service key. If blank, the service key will default to the service name encoded in the SVID. If the service name follows the pattern device-(name) , then the service key must follow the format device-(name) or device-name-* . If the service name is app-service-configurable , then the service key must follow the format app-* . (This is an accomodation for the Unix workload attester not being able to distingish workloads that are launched using the same executable binary. Custom app services that support multiple instances won't be supported unless they name the executable the same as the standard app service binary or modify this logic.) A list of \"known secret\" identifiers that will allow new services to request database passwords or other \"known secrets\" to be seeded into their service's partition in the secret store. The go-mod-secrets module will be modified to enable a new mode whereby a secret store token is obtained by: Obtaining an X.509 SVID by contacting a local SPIFFE agent's workload API on a local Unix domain socket. Connecting to the security-spiffe-token-provider service using the X.509 SVID to request a secret store token. The SPIFFE authentication mode will be an opt-in feature. The SPIFFE implementation will be user-replaceable; specifically, the workload API socket will be configurable, as well as the parsing of the SPIFFE ID. Reasons for doing so might include: changing the name of the trust domain in the SPIFFE ID, or moving the SPIFFE server out of the edge. This feature is estimated to be a \"large\" or \"extra large\" effort that could be implemented in a single release cycle.","title":"Decision"},{"location":"design/adr/security/0020-spiffe/#technical-architecture","text":"The work flow is as follows: Create a root CA for the SPIFFE user to use for creation of sub-CA's. The SPIFFE server is started. The server creates a sub-CA for issuing new identities. The trust bundle (certificate authority) data is exported from the SPIFFE server and stored on a shared volume readable by other EdgeX microservices (i.e. the existing secrets volume used for sharing secret store tokens). A join token for the SPIFFE agent is created using token generate and shared to the EdgeX secrets volume. Workload entries are loaded into the SPIFFE server database, using the join-identity of the agent created in the previous step as the parent ID of the workload. The SPIFFE agent is started with the join token created in a previous step to add it to the cluster. Vault is started and security-secret-store-setup initializes it and creates an admin token for security-spiffe-token-provider to use. The security-spiffe-token-provider service is started. It obtains an SVID from the SIFFE agent and uses it as a TLS server certificate. An EdgeX microservice starts and obtains another SVID from the SPIFFE agent and uses it as a TLS client certificate to contact the security-spiffe-token-provider service. The EdgeX microservice uses the trust bundle as a server CA to verify the TLS certificate of the remote service. security-spiffe-token-provider verifies the SVID using the trust bundle as client CA to verify the client, extracts the service key, and issues an appropriate Vault service token. The EdgeX microservice accesses Vault as usual.","title":"Technical Architecture"},{"location":"design/adr/security/0020-spiffe/#workload-registration-and-agent-sockets","text":"The server uses a workload registration Unix domain socket that allows authorization entries to be added to the authorization database. This socket is protected by Unix file system permissions to control who is allowed to add entries to the database. In this proposal, a subcommand will be added to the EdgeX secrets-config utility to simplify the process of registering new services that uses the registration socket above. The agent uses a workload attesation Unix domain socket that is open to the world. This socket is shared via a snap content-interface of via a shared host bind mount for Docker. There is one agent per node.","title":"Workload Registration and Agent Sockets"},{"location":"design/adr/security/0020-spiffe/#trust-bundle","text":"SVID's must be traceable back to a known issuing authority (certificate authority) to determine their validity. In the proposed implementation, we will generate a CA on first boot and store it persistently. This root CA will be distributed as the trust bundle. The SPIFFE server will then generate a rotating sub-CA for issuing SVIDs, and the issued SVID will include both the leaf certificate and the intermediate certificate. This implementation differs from the default implementation, which uses a transient CA that is rotated periodically and that keeps a log of past CA's. The default implementation is not suitable because only the Kubernetes reference implementation of the SPIRE server has a notification hook that is invoked when the CA is rotated. CA rotation would just result in issuing of SVIDs that are not trusted by microservices that received only the initial CA. The SPIFFE implementation is replaceable. The user is free to replace this default implementation with potentally a cloud-based SPIFFE server and a cloud-based CA.","title":"Trust Bundle"},{"location":"design/adr/security/0020-spiffe/#workload-authorization","text":"Workloads are authenticated by connecting to the spiffe-agent via a Unix domain socket, which is capable of identifying the process ID of the remote client. The process ID is fed into one of following workload attesters, which gather additional metadata about the caller: The Unix workload attester gathers UID, GID, path, and SHA-256 hash of the executable. The Unix workload attester would be used native services and snaps. The Docker workload attester gathers container labels that are added by docker-compose when the container is launched. The Docker workload attester would be used for Docker-based EdgeX deployments. An example label is docker:label:com.docker.compose.service:edgex-core-data where the service label is the key value in the services section of the docker-compose.yml . It is also possible to refer to labels built-in to the container image. The Kubernetes workload attester gathers a wealth of pod and container metadata. Once authenticated, the metadata is sent to the SPIFFE server to authorize the workload. Workloads are authorized via an authorization database connected to the SPIFFE server. Supported databases are SQLite (default), PostgreSQL, and MySQL. Due to startup ordering issues, SQLite will be used. (Disclaimer: SQlite, according for the Turtle book is intended for development and test only. We will use SQlite anyway because because Redis is not supported.) The only service that needs to be seeded to the database as this time is security-spiffe-token-provier . For example: spire-server entry create -parentID \" ${ local_agent_svid } \" -dns edgex-spiffe-token-provider -spiffeID \" ${ svid_service_base } /edgex-spiffe-token-provider\" -selector \"docker:label:com.docker.compose.service:edgex-spiffe-token-provider\" The above command associates a SPIFFE ID with a selector , in this case, a container label, and configures a DNS subjectAltName in the X.509 certificate for server-side TLS. A snap-based installation of EdgeX would use a unix:path or unix:sha256 selector instead. There are two extension mechanims for authorization additional workloads: Inject a config file or environment variable to authorize additional workloads. The container will parse and issue spire-server entry create commands for each additional service. Run the edgex-secrets-config utility (that will wrap the spire-server entry create command) for ad-hoc authorization of new services. The authorization database is persistent across reboots.","title":"Workload Authorization"},{"location":"design/adr/security/0020-spiffe/#consequences","text":"This proposal will require addition of several new, optional, EdgeX microservices: security-spiffe-token-provider , running on the main node spiffe-agent , running on the main node and each remote node spiffe-server , running on the main node spiffe-config , a one-shot service running on the main node Note that like Vault, the recommended SPIFFE configuration is to run the SPIFFE server on a dedicated node. If this is a concern, bring your own SPIFFE implementation. Minor changes will be needed to security-secretstore-setup to preserve the token-creating-token used by security-file-token-provider so that it can be used by security-spiffe-token-provider . The startup flow of the framework will be adjusted as follows: Bootstrap service (original) spiffe-server spiffe-config (can be combined with spifee-server ) spiffe-agent Vault service (original) Secret store setup service (original) security-spiffe-token-provider Consul (original) Postgres (original) There is no direct dependency between spiffe-server and any other microservice. security-spiffe-token-provider requires an SVID from spiffe-agent and a Vault admin token. None of these new services will be proxied via the API gateway. In the future, this mechanism may become the default secret store distribution mechanism, as it eliminates several secrets volumes used to share secrets between security-secretstore-setup and various EdgeX microservices. The EdgeX automation will only configure the SPIFEE agent on the main node. Additional nodes can be manually added by the operator by obtaining a join token from the main node and using it to bootstrap a remote node. SPIFFE/SPIRE has native support for Kubernetes and can distribute the trust bundle via a Kubernetes ConfigMap to more easily enable distributed scenarios, removing a major roadblock to usage of EdgeX in a Kubernetes environment.","title":"Consequences"},{"location":"design/adr/security/0020-spiffe/#footprint","text":"NOTE: This data is limited by the fact that the pre-built SPIRE reference binaries are compiled with CGO enabled.","title":"Footprint"},{"location":"design/adr/security/0020-spiffe/#spire-server","text":"69 MB executable, dynamically linked 151 MB inside of a Debian-slim container 30 MB memory usage, as container","title":"SPIRE Server"},{"location":"design/adr/security/0020-spiffe/#spire-agent","text":"33 MB executable, dynamically linked 114 MB inside of a Debian-slim container 64 MB memory usage, as container","title":"SPIRE Agent"},{"location":"design/adr/security/0020-spiffe/#spiffe-base-secret-store-token-provider","text":"The following is the minimum size: > 6 MB executable (likely much larger) > 29 MB memory usage, as container","title":"SPIFFE-base Secret Store Token Provider"},{"location":"design/adr/security/0020-spiffe/#limitations","text":"The following are known limitations with this proposal: The capabilities enabled by this solution would only be enabled on Linux platforms. SIFFE/SPIRE Agent is not available for native Windows and pre-built binaries are only avaiable for Linux. (It is unclear as to whether other *nix'es are supported.) The capabilities enabled by this solution would only be supported for Go-based services. The SPIFFE API's are implemented in gRPC, which is only ported to C#, C++, Dart, Go, Java, Kotlin, Node, Objective-C, PHP, Python, and Ruby. Notably, the C language is not supported, and the only other EdgeX supported language is Go. That default TTL of an x.509 SVID is one hour. As such, all SVID consumers must be capable of auto-renewal of SVIDs on both the client and server side.","title":"Limitations"},{"location":"design/adr/security/0020-spiffe/#alternatives","text":"","title":"Alternatives"},{"location":"design/adr/security/0020-spiffe/#overcoming-lack-of-a-supported-grpc-c-library","text":"Leave C-SDK device services behind. In this option, C device services would be unable to participate in the delayed-start services architecture. Fork a grpc-c library. Forking a grpc-c library and rehabilitating it is one option. There is at least one grpc-c library that has been proven to work, but it requires additional features to make it compatible with the SPIRE workload agent. However, the project is extremely large and it is unlikely that EdgeX is big enough to carry the project. Available libraries include: https://github.com/lixiangyun/grpc-c This library is several years out-of-date, does not compile on current Linux distributions without some rework, and does not pass per-request metadata tags. Proved to work via manual patching. Not supportable. https://github.com/Juniper/grpc-c This library is serveral years out-of-date, also does not compile on current Linux distributiosn without some rework. Uses hard-coded Unix domain socket paths. May support per-request metadata tags, but did not test. Not supportable. https://github.com/HewlettPackard/c-spiffe This library is yet untested. Rather than a gRPC library, this library implements the workload API client directly. Ultimately, this library also wraps the gRPC C++ library, and statically links to it. There is no benefit to the EdgeX project to use this library as we can call the underlying library directly. Hybrid device services. In this model, device services would always be written in Go, but in the case where linking to a C language library is required, CGO features would be used to invoke native C functions from golang. This option would commit the EdgeX project to a one-time investment to port the existing C device services to the new hybrid model. This option is the best choice if the long-term strategy is to end-of-life the C Device SDK. Bridge. In this model, the C++ implementation to invoke the SPIFFE/SPIRE workload API would be hidden behind a dynamic shared library with C linkage. This would require minimal change to the existing C SDK. However, the resulting binaries would have be based on GLIBC vs MUSL in order to get dlopen() support. This will also limit the choice of container base images for containerized services. Modernize. In this model, the Device SDK would be rewritten either partially or in-full in C++. Under this model, the SPIFFE/SPIRE workload API could be accessed via a community-supported C++ GRPC SDK. There are many implementation options: A \"C++ compilation-switch\" where the C SDK could be compiled in C-mode or C++-mode with enhanced functionality. A C++ extension API. The original C SDK would remain as-is, but if compiling with __cplusplus defined, additional API methods would be exposed. The SDK could thus be composed of a mixture of .c files with C linkage and .cc files with C++ linkage. The linker would ultimately determine whether or not the C++ runtime library needed to be linked in. Native C++ device SDK with legacy C wrapper facade. Compile existing code in C++ mode, with optional C++ facade.","title":"Overcoming lack of a supported GRPC-C library"},{"location":"design/adr/security/0020-spiffe/#opt-in-or-standard-feature","text":"If one of the following things were to happen, it would push this proposal \"over the edge\" from being an optional opt-in feature to a required standard feature for security: The \"on-demand\" method of obtaining a secret store token is the default method of obtaining a token for non-core EdgeX services. The \"on-demand\" method of obtaining a secret store token is the default method for all EdgeX services. SPIFFE SVID's become the implementation mechanism for microservice-level authentication. (Not in scope for this ADR.)","title":"Opt-in or Standard Feature"},{"location":"design/adr/security/0020-spiffe/#merge-security-file-token-provider-and-security-spiffe-token-provider","text":"Keeping these as separate executables clearly separates the on-demand secret store tokens feature as an optional service. It is possible to combine the services, but there would need to be a configuration switch in order to enable the SPIFFE feature. It would also increase the base executable size to include the extra logic.","title":"Merge security-file-token-provider and security-spiffe-token-provider"},{"location":"design/adr/security/0020-spiffe/#alternatives-regarding-spiffe-ca","text":"","title":"Alternatives regarding SPIFFE CA"},{"location":"design/adr/security/0020-spiffe/#transient-ca-option","text":"The SPIFFE server can be configured with no \"upstream authority\" (certificate authority), and the server will periodically generate a new, transient CA, and keep a bounded history of previous CA's. A rotating trust bundle only practically works in a Kubernetes environment, since a configmap can be updated real-time. For everyone else, we need a static CA that can be pre-distributed to remote nodes. Thus, this solution was not chosen.","title":"Transient CA option"},{"location":"design/adr/security/0020-spiffe/#vault-based-ca-option","text":"The SPIFFE server can be configured to make requests to a Hashicorp Vault PKI secrets engine to generate intermediate CA certificates for signing SVID's. This is an option for future integrations, but is omitted from this proposal due to the jump in implementation complexity and the desire that the current proposal be on add-on feature. The current implementation allows the SPIFFE server and Vault to be started simultaneously. Using a Vault-based CA would require a complex interlocking sequence of steps.","title":"Vault-based CA option"},{"location":"design/adr/security/0020-spiffe/#references","text":"Issue to create ADR for handling delayed-start services 0018 Service Registry ADR Service List ADR SPIFFE SPIFFE ID X.500 SVID JWT SVID Turtle book","title":"References"},{"location":"design/legacy-design/","text":"Legacy Design Documents Name/Link Short Description Registry Abstraction Decouple EdgeX services from Consul device-service/Discovery Dynamically discover new devices","title":"Legacy Design Documents"},{"location":"design/legacy-design/#legacy-design-documents","text":"Name/Link Short Description Registry Abstraction Decouple EdgeX services from Consul device-service/Discovery Dynamically discover new devices","title":"Legacy Design Documents"},{"location":"design/legacy-design/device-service/discovery/","text":"Dynamic Device Discovery Overview Some device protocols allow for devices to be discovered automatically. A Device Service may include a capability for discovering devices and creating the corresponding Device objects within EdgeX. A framework for doing so will be implemented in the Device Service SDKs. The discovery process will operate as follows: Discovery is triggered either on an internal timer or by a call to a REST endpoint The SDK will call a function provided by the DS implementation to request a device scan The implementation calls back to the SDK with details of devices which it has found The SDK filters these devices against a set of acceptance criteria The SDK adds accepted devices in core-metadata. These are now available in the EdgeX system Triggering Discovery A boolean configuration value Device/Discovery/Enabled defaults to false. If this value is set true, and the DS implementation supports discovery, discovery is enabled. The SDK will respond to POST requests on the the /discovery endpoint. No content is required in the request. This call will return one of the following codes: 202: discovery has been triggered or is already running. The response should indicate which, and contain the correlation id that will be used by any resulting requests for device addition 423: the service is locked (admin state) or disabled (operating state) 500: unknown or unanticipated issues exist 501: discovery is not supported by this protocol implementation 503: discovery is disabled by configuration In each of the failure cases a meaningful error message should be returned. In the case where discovery is triggered, the discovery process will run in a new thread or goroutine, so that the REST call may return immediately. An integer configuration value Device/Discovery/Interval defaults to zero. If this value is set to a positive value, and discovery is enabled, the discovery process will be triggered at the specified interval (in seconds). Finding Devices When discovery is triggered, the SDK calls the implementation function provided by the Device Service. This should perform whatever protocol-specific procedure is necessary to find devices, and pass these devices into the SDK by calling the SDK's filtered device addition function. Note: The implementation should call back for every device found. The SDK is to take responsibility for filtering out devices which have already been added. The information required for a found device is as follows: An autogenerated device name The Protocol Properties of the device Optionally, a description string Optionally, a list of label strings The filtered device addition function will take as an argument a collection of structs containing the above data. An implementation may choose to make one call per discovered device, but implementors are encouraged to batch the devices if practical, as in future EdgeX versions it will be possible for the SDK to create all required new devices in a single call to core-metadata. Rationale: An alternative design would have the implementation function return the collection of discovered devices to the SDK. Using a callback mechanism instead has the following advantages: Allows for asynchronous operation. In this mode the DS implementation will intiate discovery and return immediately. For example discovery may be initiated by sending a broadcast packet. Devices will then send return packets indicating their existence. The thread handling inbound network traffic can on receipt of such packets call the filtered device addition function directly. Allows DS implementations where devices self-announce to call the filtered device addition function independent of the discovery process Filtered Device Addition The filter criteria for discovered devices are represented by Provision Watchers. A Provision Watcher contains the following fields: Identifiers : A set of name-value pairs against which a new device's ProtocolProperties are matched BlockingIdentifiers : A further set of name-value pairs which are also matched against a new device's ProtocolProperties Profile : The name of a DeviceProfile which should be assigned to new devices which pass this ProvisionWatcher AdminState : The initial Administrative State for new devices which pass this ProvisionWatcher A candidate new device passes a ProvisionWatcher if all of the Identifiers match, and none of the BlockingIdentifiers . For devices with multiple Device.Protocols , each Device.Protocol is considered separately. A pass (as described above) on any of the protocols results in the device being added. The values specified in Identifiers are regular expressions. Note: If a discovered Device is manually removed from EdgeX, it will be necessary to adjust the ProvisionWatcher via which it was added, either by making the Identifiers more specific or by adding BlockingIdentifiers , otherwise the Device will be re-added the next time Discovery is initiated. Note: ProvisionWatchers are stored in core-metadata. A facility for managing ProvisionWatchers is needed, eg edgex-cli could be extended","title":"Discovery"},{"location":"design/legacy-design/device-service/discovery/#dynamic-device-discovery","text":"","title":"Dynamic Device Discovery"},{"location":"design/legacy-design/device-service/discovery/#overview","text":"Some device protocols allow for devices to be discovered automatically. A Device Service may include a capability for discovering devices and creating the corresponding Device objects within EdgeX. A framework for doing so will be implemented in the Device Service SDKs. The discovery process will operate as follows: Discovery is triggered either on an internal timer or by a call to a REST endpoint The SDK will call a function provided by the DS implementation to request a device scan The implementation calls back to the SDK with details of devices which it has found The SDK filters these devices against a set of acceptance criteria The SDK adds accepted devices in core-metadata. These are now available in the EdgeX system","title":"Overview"},{"location":"design/legacy-design/device-service/discovery/#triggering-discovery","text":"A boolean configuration value Device/Discovery/Enabled defaults to false. If this value is set true, and the DS implementation supports discovery, discovery is enabled. The SDK will respond to POST requests on the the /discovery endpoint. No content is required in the request. This call will return one of the following codes: 202: discovery has been triggered or is already running. The response should indicate which, and contain the correlation id that will be used by any resulting requests for device addition 423: the service is locked (admin state) or disabled (operating state) 500: unknown or unanticipated issues exist 501: discovery is not supported by this protocol implementation 503: discovery is disabled by configuration In each of the failure cases a meaningful error message should be returned. In the case where discovery is triggered, the discovery process will run in a new thread or goroutine, so that the REST call may return immediately. An integer configuration value Device/Discovery/Interval defaults to zero. If this value is set to a positive value, and discovery is enabled, the discovery process will be triggered at the specified interval (in seconds).","title":"Triggering Discovery"},{"location":"design/legacy-design/device-service/discovery/#finding-devices","text":"When discovery is triggered, the SDK calls the implementation function provided by the Device Service. This should perform whatever protocol-specific procedure is necessary to find devices, and pass these devices into the SDK by calling the SDK's filtered device addition function. Note: The implementation should call back for every device found. The SDK is to take responsibility for filtering out devices which have already been added. The information required for a found device is as follows: An autogenerated device name The Protocol Properties of the device Optionally, a description string Optionally, a list of label strings The filtered device addition function will take as an argument a collection of structs containing the above data. An implementation may choose to make one call per discovered device, but implementors are encouraged to batch the devices if practical, as in future EdgeX versions it will be possible for the SDK to create all required new devices in a single call to core-metadata. Rationale: An alternative design would have the implementation function return the collection of discovered devices to the SDK. Using a callback mechanism instead has the following advantages: Allows for asynchronous operation. In this mode the DS implementation will intiate discovery and return immediately. For example discovery may be initiated by sending a broadcast packet. Devices will then send return packets indicating their existence. The thread handling inbound network traffic can on receipt of such packets call the filtered device addition function directly. Allows DS implementations where devices self-announce to call the filtered device addition function independent of the discovery process","title":"Finding Devices"},{"location":"design/legacy-design/device-service/discovery/#filtered-device-addition","text":"The filter criteria for discovered devices are represented by Provision Watchers. A Provision Watcher contains the following fields: Identifiers : A set of name-value pairs against which a new device's ProtocolProperties are matched BlockingIdentifiers : A further set of name-value pairs which are also matched against a new device's ProtocolProperties Profile : The name of a DeviceProfile which should be assigned to new devices which pass this ProvisionWatcher AdminState : The initial Administrative State for new devices which pass this ProvisionWatcher A candidate new device passes a ProvisionWatcher if all of the Identifiers match, and none of the BlockingIdentifiers . For devices with multiple Device.Protocols , each Device.Protocol is considered separately. A pass (as described above) on any of the protocols results in the device being added. The values specified in Identifiers are regular expressions. Note: If a discovered Device is manually removed from EdgeX, it will be necessary to adjust the ProvisionWatcher via which it was added, either by making the Identifiers more specific or by adding BlockingIdentifiers , otherwise the Device will be re-added the next time Discovery is initiated. Note: ProvisionWatchers are stored in core-metadata. A facility for managing ProvisionWatchers is needed, eg edgex-cli could be extended","title":"Filtered Device Addition"},{"location":"design/legacy-requirements/","text":"Legacy Requirements Name/Link Short Description Device Service Device Service SDK required functionality","title":"Legacy Requirements"},{"location":"design/legacy-requirements/#legacy-requirements","text":"Name/Link Short Description Device Service Device Service SDK required functionality","title":"Legacy Requirements"},{"location":"design/legacy-requirements/device-service/","text":"Device SDK Required Functionality Overview This document sets out the required functionality of a Device SDK other than the implementation of its REST API (see ADR 0011 ) and the Dynamic Discovery mechanism (see Discovery ). This functionality is categorised into three areas - actions required at startup, configuration options to be supported, and support for push-style event generation. Startup When the device service is started, in addition to any actions required to support functionality defined elsewhere, the SDK must: Manage the device service's registration in metadata Provide initialization information to the protocol-specific implementation Registration The core-metadata service maintains an extent of device service registrations so that it may route requests relating to particular devices to the correct device service. The SDK should create (on first run) or update its record appropriately. Device service registrations contain the following fields: Name - the name of the device service Description - an optional brief description of the service Labels - optional string labels BaseAddress - URL of the base of the service's REST API The default device service Name is to be hardcoded into every device service implementation. A suffix may be added to this name at runtime by means of commandline option or environment variable. Service names must be unique in a particular EdgeX instance; the suffix mechanism allows for running multiple instances of a given device service. The Description and Labels are configured in the [Service] section of the device service configuration. BaseAddress may be constructed using the [Service]/Host and [Service]/Port entries in the device service configuration. Initialization During startup the SDK must supply to the implementation that part of the service configuration which is specific to the implementation. This configuration is held in the Driver section of the configuration file or registry. The SDK must also supply a logging facility at this stage. This facility should by default emit logs locally (configurable to file or to stdout) but instead should use the optional logging service if the configuration element Logging/EnableRemote is set true . Note: the logging service is deprecated and support for it will be removed in EdgeX v2.0 The implementation on receipt of its configuration should perform any necessary initialization of its own. It may return an error in the event of unrecoverable problems, this should cause the service startup itself to fail. Configuration Configuration should be supported by the SDK, in accordance with ADR 0005 Commandline processing The SDK should handle commandline processing on behalf of the device service. In addition to the common EdgeX service options, the --instance / -i flag should be supported. This specifies a suffix to append to the device service name. Environment variables The SDK should also handle environment variables. In addition to the common EdgeX variables, EDGEX_INSTANCE_NAME should if set override the --instance setting. Configuration file and Registry The SDK should use (or for non-Go implementations, re-implement) the standard mechanisms for obtaining configuration from a file or registry. The configuration parameters to be supported are: Service section Option Type Notes Host String This is the hostname to use when registering the service in core-metadata. As such it is used by other services to connect to the device service, and therefore must be resolvable by other services in the EdgeX deployment. Port Int Port on which to accept the device service's REST API. The assigned port for experimental / in-development device services is 49999. Timeout Int Time (in milliseconds) to wait between attempts to contact core-data and core-metadata when starting up. ConnectRetries Int Number of times to attempt to contact core-data and core-metadata when starting up. StartupMsg String Message to log on successful startup. CheckInterval String The checking interval to request if registering with Consul. Consul will ping the service at this interval to monitor its liveliness. ServerBindAddr String The interface on which the service's REST server should listen. By default the server is to listen on the interface to which the Host option resolves. A value of 0.0.0.0 means listen on all available interfaces. Clients section Defines the endpoints for other microservices in an EdgeX system. Not required when using Registry. Data Option Type Notes Host String Hostname on which to contact the core-data service. Port Int Port on which to contact the core-data service. Metadata Option Type Notes Host String Hostname on which to contact the core-metadata service. Port Int Port on which to contact the core-metadata service. Device section Option Type Notes DataTransform Bool For enabling/disabling transformations on data between the device and EdgeX. Defaults to true (enabled). Discovery/Enabled Bool For enabling/disabling device discovery. Defaults to true (enabled). Discovery/Interval Int Time between automatic discovery runs, in seconds. Defaults to zero (do not run discovery automatically). MaxCmdOps Int Defines the maximum number of resource operations that can be sent to the driver in a single command. MaxCmdResultLen Int Maximum string length for command results returned from the driver. UpdateLastConnected Bool If true, update the LastConnected attribute of a device whenever it is successfully accessed (read or write). Defaults to false. Logging section Option Type Notes LogLevel String Sets the logging level. Available settings in order of increasing severity are: TRACE , DEBUG , INFO , WARNING , ERROR . Driver section This section is for options specific to the protocol driver. Any configuration specified here will be passed to the driver implementation during initialization. Push Events The SDK should implement methods for generating Events other than on receipt of device GET requests. The AutoEvent mechanism provides for generating Events at fixed intervals. The asynchronous event queue enables the device service to generate events at arbitrary times, according to implementation-specific logic. AutoEvents Each device may have as part of its definition in Metadata a number of AutoEvents associated with it. An AutoEvent has the following fields: resource : the name of a deviceResource or deviceCommand indicating what to read. frequency : a string indicating the time to wait between reading events, expressed as an integer followed by units of ms, s, m or h. onchange : a boolean: if set to true, only generate new events if one or more of the contained readings has changed since the last event. The device SDK should schedule device readings from the implementation according to these AutoEvent defininitions. It should use the same logic as it would if the readings were being requested via REST. Asynchronous Event Queue The SDK should provide a mechanism whereby the implementation may submit device readings at any time without blocking. This may be done in a manner appropriate to the implementation language, eg the Go SDK provides a channel on which readings may be pushed, the C SDK provides a function which submits readings to a workqueue.","title":"Device SDK Required Functionality"},{"location":"design/legacy-requirements/device-service/#device-sdk-required-functionality","text":"","title":"Device SDK Required Functionality"},{"location":"design/legacy-requirements/device-service/#overview","text":"This document sets out the required functionality of a Device SDK other than the implementation of its REST API (see ADR 0011 ) and the Dynamic Discovery mechanism (see Discovery ). This functionality is categorised into three areas - actions required at startup, configuration options to be supported, and support for push-style event generation.","title":"Overview"},{"location":"design/legacy-requirements/device-service/#startup","text":"When the device service is started, in addition to any actions required to support functionality defined elsewhere, the SDK must: Manage the device service's registration in metadata Provide initialization information to the protocol-specific implementation","title":"Startup"},{"location":"design/legacy-requirements/device-service/#registration","text":"The core-metadata service maintains an extent of device service registrations so that it may route requests relating to particular devices to the correct device service. The SDK should create (on first run) or update its record appropriately. Device service registrations contain the following fields: Name - the name of the device service Description - an optional brief description of the service Labels - optional string labels BaseAddress - URL of the base of the service's REST API The default device service Name is to be hardcoded into every device service implementation. A suffix may be added to this name at runtime by means of commandline option or environment variable. Service names must be unique in a particular EdgeX instance; the suffix mechanism allows for running multiple instances of a given device service. The Description and Labels are configured in the [Service] section of the device service configuration. BaseAddress may be constructed using the [Service]/Host and [Service]/Port entries in the device service configuration.","title":"Registration"},{"location":"design/legacy-requirements/device-service/#initialization","text":"During startup the SDK must supply to the implementation that part of the service configuration which is specific to the implementation. This configuration is held in the Driver section of the configuration file or registry. The SDK must also supply a logging facility at this stage. This facility should by default emit logs locally (configurable to file or to stdout) but instead should use the optional logging service if the configuration element Logging/EnableRemote is set true . Note: the logging service is deprecated and support for it will be removed in EdgeX v2.0 The implementation on receipt of its configuration should perform any necessary initialization of its own. It may return an error in the event of unrecoverable problems, this should cause the service startup itself to fail.","title":"Initialization"},{"location":"design/legacy-requirements/device-service/#configuration","text":"Configuration should be supported by the SDK, in accordance with ADR 0005","title":"Configuration"},{"location":"design/legacy-requirements/device-service/#commandline-processing","text":"The SDK should handle commandline processing on behalf of the device service. In addition to the common EdgeX service options, the --instance / -i flag should be supported. This specifies a suffix to append to the device service name.","title":"Commandline processing"},{"location":"design/legacy-requirements/device-service/#environment-variables","text":"The SDK should also handle environment variables. In addition to the common EdgeX variables, EDGEX_INSTANCE_NAME should if set override the --instance setting.","title":"Environment variables"},{"location":"design/legacy-requirements/device-service/#configuration-file-and-registry","text":"The SDK should use (or for non-Go implementations, re-implement) the standard mechanisms for obtaining configuration from a file or registry. The configuration parameters to be supported are:","title":"Configuration file and Registry"},{"location":"design/legacy-requirements/device-service/#service-section","text":"Option Type Notes Host String This is the hostname to use when registering the service in core-metadata. As such it is used by other services to connect to the device service, and therefore must be resolvable by other services in the EdgeX deployment. Port Int Port on which to accept the device service's REST API. The assigned port for experimental / in-development device services is 49999. Timeout Int Time (in milliseconds) to wait between attempts to contact core-data and core-metadata when starting up. ConnectRetries Int Number of times to attempt to contact core-data and core-metadata when starting up. StartupMsg String Message to log on successful startup. CheckInterval String The checking interval to request if registering with Consul. Consul will ping the service at this interval to monitor its liveliness. ServerBindAddr String The interface on which the service's REST server should listen. By default the server is to listen on the interface to which the Host option resolves. A value of 0.0.0.0 means listen on all available interfaces.","title":"Service section"},{"location":"design/legacy-requirements/device-service/#clients-section","text":"Defines the endpoints for other microservices in an EdgeX system. Not required when using Registry.","title":"Clients section"},{"location":"design/legacy-requirements/device-service/#data","text":"Option Type Notes Host String Hostname on which to contact the core-data service. Port Int Port on which to contact the core-data service.","title":"Data"},{"location":"design/legacy-requirements/device-service/#metadata","text":"Option Type Notes Host String Hostname on which to contact the core-metadata service. Port Int Port on which to contact the core-metadata service.","title":"Metadata"},{"location":"design/legacy-requirements/device-service/#device-section","text":"Option Type Notes DataTransform Bool For enabling/disabling transformations on data between the device and EdgeX. Defaults to true (enabled). Discovery/Enabled Bool For enabling/disabling device discovery. Defaults to true (enabled). Discovery/Interval Int Time between automatic discovery runs, in seconds. Defaults to zero (do not run discovery automatically). MaxCmdOps Int Defines the maximum number of resource operations that can be sent to the driver in a single command. MaxCmdResultLen Int Maximum string length for command results returned from the driver. UpdateLastConnected Bool If true, update the LastConnected attribute of a device whenever it is successfully accessed (read or write). Defaults to false.","title":"Device section"},{"location":"design/legacy-requirements/device-service/#logging-section","text":"Option Type Notes LogLevel String Sets the logging level. Available settings in order of increasing severity are: TRACE , DEBUG , INFO , WARNING , ERROR .","title":"Logging section"},{"location":"design/legacy-requirements/device-service/#driver-section","text":"This section is for options specific to the protocol driver. Any configuration specified here will be passed to the driver implementation during initialization.","title":"Driver section"},{"location":"design/legacy-requirements/device-service/#push-events","text":"The SDK should implement methods for generating Events other than on receipt of device GET requests. The AutoEvent mechanism provides for generating Events at fixed intervals. The asynchronous event queue enables the device service to generate events at arbitrary times, according to implementation-specific logic.","title":"Push Events"},{"location":"design/legacy-requirements/device-service/#autoevents","text":"Each device may have as part of its definition in Metadata a number of AutoEvents associated with it. An AutoEvent has the following fields: resource : the name of a deviceResource or deviceCommand indicating what to read. frequency : a string indicating the time to wait between reading events, expressed as an integer followed by units of ms, s, m or h. onchange : a boolean: if set to true, only generate new events if one or more of the contained readings has changed since the last event. The device SDK should schedule device readings from the implementation according to these AutoEvent defininitions. It should use the same logic as it would if the readings were being requested via REST.","title":"AutoEvents"},{"location":"design/legacy-requirements/device-service/#asynchronous-event-queue","text":"The SDK should provide a mechanism whereby the implementation may submit device readings at any time without blocking. This may be done in a manner appropriate to the implementation language, eg the Go SDK provides a channel on which readings may be pushed, the C SDK provides a function which submits readings to a workqueue.","title":"Asynchronous Event Queue"},{"location":"examples/","text":"EdgeX Examples In addition to the examples listed in this section of the documentation, you will find other examples in the EdgeX Examples Repository . The tabs below provide a listing (may be partial based on latest updates) for reference. Application Services Deployment Device Services Security See App Service Examples for a listing of custom and configurable application service examples. Example Location Helm (Kubernetes) Github - examples, deployment Raspberry Pi 4 Github - examples, raspberry-pi-4 Cloud deployments Github - examples, cloud deployment templates Example Location Random Number Device Service (simulation) Github - examples, device-random Grove Device Service in C Github - examples, device-grove-c Example Location Docker Swarm, remote device service via overlay network Github - Docker Swarm SSH Tunneling, remote device service via SSH tunneling Github - SSH Tunneling Warning Not all the examples in the EdgeX Examples repository are available for all EdgeX releases. Check the documentation for details.","title":"EdgeX Examples"},{"location":"examples/#edgex-examples","text":"In addition to the examples listed in this section of the documentation, you will find other examples in the EdgeX Examples Repository . The tabs below provide a listing (may be partial based on latest updates) for reference. Application Services Deployment Device Services Security See App Service Examples for a listing of custom and configurable application service examples. Example Location Helm (Kubernetes) Github - examples, deployment Raspberry Pi 4 Github - examples, raspberry-pi-4 Cloud deployments Github - examples, cloud deployment templates Example Location Random Number Device Service (simulation) Github - examples, device-random Grove Device Service in C Github - examples, device-grove-c Example Location Docker Swarm, remote device service via overlay network Github - Docker Swarm SSH Tunneling, remote device service via SSH tunneling Github - SSH Tunneling Warning Not all the examples in the EdgeX Examples repository are available for all EdgeX releases. Check the documentation for details.","title":"EdgeX Examples"},{"location":"examples/AppServiceExamples/","text":"App Service Examples The following is a list of examples we currently have available that demonstrate various ways that the Application Functions SDK or App Service Configurable can be used. All of the examples can be found here in the edgex-examples repo. They focus on how to leverage various built in provided functions as mentioned above as well as how to write your own in the case that the SDK does not provide what is needed. Example Name Description Simple Filter XML Demonstrates Filtering of Events by Device names and transforming data to XML Simple Filter XML HTTP Same example as #1, but result published to HTTP Endpoint Simple Filter XML MQTT Same example as #1, but result published to MQTT Broker Simple CBOR Filter Demonstrates Filtering of Events by Resource names for Event that is CBOR encoded containing a binary reading Advanced Filter Convert Publish Demonstrates Filtering of Events by Resource names, custom function to convert the reading and them publish the modified Event back to the MessageBus under a different topic. Advanced Target Type Demonstrates use of custom Target Type and use of HTTP Trigger Cloud Export MQTT Demonstrates simple custom Cloud transform and exporting to Cloud MQTT Broker. Cloud Event Transform Demonstrates custom transforms that convert Event/Readings to and from Cloud Events Send Command Demonstrates sending commands to a Device via the Command Client. Secrets Demonstrates how to retrieve secrets from the service SecretStore Custom Trigger Demonstrates how to create and use a custom trigger Fledge Export Demonstrates custom conversion of Event/Reading to Fledge format and then exporting to Fledge service REST endpoint Influxdb Export Demonstrates custom conversion of Event/Reading to InfluxDB timeseries format and then exporting to InFluxDB via MQTT Json Logic Demonstrates using the built in JSONLogic Evaluate pipeline function IBM Export Profile Demonstrates a custom App Service Configurable profile for exporting to IBM Cloud","title":"App Service Examples"},{"location":"examples/AppServiceExamples/#app-service-examples","text":"The following is a list of examples we currently have available that demonstrate various ways that the Application Functions SDK or App Service Configurable can be used. All of the examples can be found here in the edgex-examples repo. They focus on how to leverage various built in provided functions as mentioned above as well as how to write your own in the case that the SDK does not provide what is needed. Example Name Description Simple Filter XML Demonstrates Filtering of Events by Device names and transforming data to XML Simple Filter XML HTTP Same example as #1, but result published to HTTP Endpoint Simple Filter XML MQTT Same example as #1, but result published to MQTT Broker Simple CBOR Filter Demonstrates Filtering of Events by Resource names for Event that is CBOR encoded containing a binary reading Advanced Filter Convert Publish Demonstrates Filtering of Events by Resource names, custom function to convert the reading and them publish the modified Event back to the MessageBus under a different topic. Advanced Target Type Demonstrates use of custom Target Type and use of HTTP Trigger Cloud Export MQTT Demonstrates simple custom Cloud transform and exporting to Cloud MQTT Broker. Cloud Event Transform Demonstrates custom transforms that convert Event/Readings to and from Cloud Events Send Command Demonstrates sending commands to a Device via the Command Client. Secrets Demonstrates how to retrieve secrets from the service SecretStore Custom Trigger Demonstrates how to create and use a custom trigger Fledge Export Demonstrates custom conversion of Event/Reading to Fledge format and then exporting to Fledge service REST endpoint Influxdb Export Demonstrates custom conversion of Event/Reading to InfluxDB timeseries format and then exporting to InFluxDB via MQTT Json Logic Demonstrates using the built in JSONLogic Evaluate pipeline function IBM Export Profile Demonstrates a custom App Service Configurable profile for exporting to IBM Cloud","title":"App Service Examples"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/","text":"Command Devices with eKuiper Rules Engine Overview This document describes how to actuate a device with rules trigger by the eKuiper rules engine. To make the example simple, the virtual device device-virtual is used as the actuated device. The eKuiper rules engine analyzes the data sent from device-virtual services, and then sends a command to virtual device based a rule firing in eKuiper based on that analysis. It should be noted that an application service is used to route core data through the rules engine. Use Case Scenarios Rules will be created in eKuiper to watch for two circumstances: monitor for events coming from the Random-UnsignedInteger-Device device (one of the default virtual device managed devices), and if a uint8 reading value is found larger than 20 in the event, then send a command to Random-Boolean-Device device to start generating random numbers (specifically - set random generation bool to true). monitor for events coming from the Random-Integer-Device device (another of the default virtual device managed devices), and if the average for int8 reading values (within 20 seconds) is larger than 0, then send a command to Random-Boolean-Device device to stop generating random numbers (specifically - set random generation bool to false). These use case scenarios do not have any real business meaning, but easily demonstrate the features of EdgeX automatic actuation accomplished via the eKuiper rule engine. Prerequisite Knowledge This document will not cover basic operations of EdgeX or LF Edge eKuiper. Readers should have basic knowledge of: Get and start EdgeX. Refer to Quick Start for how to get and start EdgeX with the virtual device service. Run the eKuiper Rules Engine. Refer to EdgeX eKuiper Rule Engine Tutorial to understand the basics of eKuiper and EdgeX. Start eKuiper and Create an EdgeX Stream Make sure you read the EdgeX eKuiper Rule Engine Tutorial and successfully run eKuiper with EdgeX. First create a stream that can consume streaming data from the EdgeX application service (rules engine profile). This step is not required if you already finished the EdgeX eKuiper Rule Engine Tutorial . curl -X POST \\ http:// $ekuiper_docker :59720/streams \\ -H 'Content-Type: application/json' \\ -d '{\"sql\": \"create stream demo() WITH (FORMAT=\\\"JSON\\\", TYPE=\\\"edgex\\\")\"}' Get and Test the Command URL Since both use case scenario rules will send commands to the Random-Boolean-Device virtual device, use the curl request below to get a list of available commands for this device. curl http://127.0.0.1:59882/api/v2/device/name/Random-Boolean-Device | jq It should print results like those below. { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"deviceCoreCommand\" : { \"deviceName\" : \"Random-Boolean-Device\" , \"profileName\" : \"Random-Boolean-Device\" , \"coreCommands\" : [ { \"name\" : \"WriteBoolValue\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/WriteBoolValue\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Bool\" , \"valueType\" : \"Bool\" }, { \"resourceName\" : \"EnableRandomization_Bool\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"WriteBoolArrayValue\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/WriteBoolArrayValue\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"BoolArray\" , \"valueType\" : \"BoolArray\" }, { \"resourceName\" : \"EnableRandomization_BoolArray\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"Bool\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/Bool\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Bool\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"BoolArray\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/BoolArray\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"BoolArray\" , \"valueType\" : \"BoolArray\" } ] } ] } } From this output, look for the URL associated to the PUT command (the first URL listed). This is the command eKuiper will use to call on the device. There are two parameters for this command: Bool : Set the returned value when other services want to get device data. The parameter will be used only when EnableRandomization_Bool is set to false. EnableRandomization_Bool : Enable/disable the randomization generation of bool values. If this value is set to true, then the 1st parameter will be ignored. You can test calling this command with its parameters using curl as shown below. curl -X PUT \\ http://edgex-core-command:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue \\ -H 'Content-Type: application/json' \\ -d '{\"Bool\":\"true\", \"EnableRandomization_Bool\": \"true\"}' Create rules Now that you have EdgeX and eKuiper running, the EdgeX stream defined, and you know the command to actuate Random-Boolean-Device , it is time to build the eKuiper rules. The first rule Again, the 1st rule is to monitor for events coming from the Random-UnsignedInteger-Device device (one of the default virtual device managed devices), and if a uint8 reading value is found larger than 20 in the event, then send the command to Random-Boolean-Device device to start generating random numbers (specifically - set random generation bool to true). Given the URL and parameters to the command, below is the curl command to declare the first rule in eKuiper. curl -X POST \\ http:// $ekuiper_server :59720/rules \\ -H 'Content-Type: application/json' \\ -d '{ \"id\": \"rule1\", \"sql\": \"SELECT uint8 FROM demo WHERE uint8 > 20\", \"actions\": [ { \"rest\": { \"url\": \"http://edgex-core-command:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue\", \"method\": \"put\", \"dataTemplate\": \"{\\\"Bool\\\":\\\"true\\\", \\\"EnableRandomization_Bool\\\": \\\"true\\\"}\", \"sendSingle\": true } }, { \"log\":{} } ] }' The second rule The 2nd rule is to monitor for events coming from the Random-Integer-Device device (another of the default virtual device managed devices), and if the average for int8 reading values (within 20 seconds) is larger than 0, then send a command to Random-Boolean-Device device to stop generating random numbers (specifically - set random generation bool to false). Here is the curl request to setup the second rule in eKuiper. The same command URL is used as the same device action ( Random-Boolean-Device's PUT bool command ) is being actuated, but with different parameters. curl -X POST \\ http:// $ekuiper_server :59720/rules \\ -H 'Content-Type: application/json' \\ -d '{ \"id\": \"rule2\", \"sql\": \"SELECT avg(int8) AS avg_int8 FROM demo WHERE int8 != nil GROUP BY TUMBLINGWINDOW(ss, 20) HAVING avg(int8) > 0\", \"actions\": [ { \"rest\": { \"url\": \"http://edgex-core-command:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue\", \"method\": \"put\", \"dataTemplate\": \"{\\\"Bool\\\":\\\"false\\\", \\\"EnableRandomization_Bool\\\": \\\"false\\\"}\", \"sendSingle\": true } }, { \"log\":{} } ] }' Watch the eKuiper Logs Both rules are now created in eKuiper. eKuiper is busy analyzing the event data coming for the virtual devices looking for readings that match the rules you created. You can watch the edgex-kuiper container logs for the rule triggering and command execution. docker logs edgex-kuiper Explore the Results You can also explore the eKuiper analysis that caused the commands to be sent to the service. To see the the data from the analysis, use the SQL below to query eKuiper filtering data. SELECT int8 , \"true\" AS randomization FROM demo WHERE uint8 > 20 The output of the SQL should look similar to the results below. [{ \"int8\" : -75 , \"randomization\" : \"true\" }] Extended Reading Use these resources to learn more about the features of LF Edge eKuiper. eKuiper Github code repository eKuiper reference guide","title":"Command Devices with eKuiper Rules Engine"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#command-devices-with-ekuiper-rules-engine","text":"","title":"Command Devices with eKuiper Rules Engine"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#overview","text":"This document describes how to actuate a device with rules trigger by the eKuiper rules engine. To make the example simple, the virtual device device-virtual is used as the actuated device. The eKuiper rules engine analyzes the data sent from device-virtual services, and then sends a command to virtual device based a rule firing in eKuiper based on that analysis. It should be noted that an application service is used to route core data through the rules engine.","title":"Overview"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#use-case-scenarios","text":"Rules will be created in eKuiper to watch for two circumstances: monitor for events coming from the Random-UnsignedInteger-Device device (one of the default virtual device managed devices), and if a uint8 reading value is found larger than 20 in the event, then send a command to Random-Boolean-Device device to start generating random numbers (specifically - set random generation bool to true). monitor for events coming from the Random-Integer-Device device (another of the default virtual device managed devices), and if the average for int8 reading values (within 20 seconds) is larger than 0, then send a command to Random-Boolean-Device device to stop generating random numbers (specifically - set random generation bool to false). These use case scenarios do not have any real business meaning, but easily demonstrate the features of EdgeX automatic actuation accomplished via the eKuiper rule engine.","title":"Use Case Scenarios"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#prerequisite-knowledge","text":"This document will not cover basic operations of EdgeX or LF Edge eKuiper. Readers should have basic knowledge of: Get and start EdgeX. Refer to Quick Start for how to get and start EdgeX with the virtual device service. Run the eKuiper Rules Engine. Refer to EdgeX eKuiper Rule Engine Tutorial to understand the basics of eKuiper and EdgeX.","title":"Prerequisite Knowledge"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#start-ekuiper-and-create-an-edgex-stream","text":"Make sure you read the EdgeX eKuiper Rule Engine Tutorial and successfully run eKuiper with EdgeX. First create a stream that can consume streaming data from the EdgeX application service (rules engine profile). This step is not required if you already finished the EdgeX eKuiper Rule Engine Tutorial . curl -X POST \\ http:// $ekuiper_docker :59720/streams \\ -H 'Content-Type: application/json' \\ -d '{\"sql\": \"create stream demo() WITH (FORMAT=\\\"JSON\\\", TYPE=\\\"edgex\\\")\"}'","title":"Start eKuiper and Create an EdgeX Stream"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#get-and-test-the-command-url","text":"Since both use case scenario rules will send commands to the Random-Boolean-Device virtual device, use the curl request below to get a list of available commands for this device. curl http://127.0.0.1:59882/api/v2/device/name/Random-Boolean-Device | jq It should print results like those below. { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"deviceCoreCommand\" : { \"deviceName\" : \"Random-Boolean-Device\" , \"profileName\" : \"Random-Boolean-Device\" , \"coreCommands\" : [ { \"name\" : \"WriteBoolValue\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/WriteBoolValue\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Bool\" , \"valueType\" : \"Bool\" }, { \"resourceName\" : \"EnableRandomization_Bool\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"WriteBoolArrayValue\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/WriteBoolArrayValue\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"BoolArray\" , \"valueType\" : \"BoolArray\" }, { \"resourceName\" : \"EnableRandomization_BoolArray\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"Bool\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/Bool\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Bool\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"BoolArray\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Boolean-Device/BoolArray\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"BoolArray\" , \"valueType\" : \"BoolArray\" } ] } ] } } From this output, look for the URL associated to the PUT command (the first URL listed). This is the command eKuiper will use to call on the device. There are two parameters for this command: Bool : Set the returned value when other services want to get device data. The parameter will be used only when EnableRandomization_Bool is set to false. EnableRandomization_Bool : Enable/disable the randomization generation of bool values. If this value is set to true, then the 1st parameter will be ignored. You can test calling this command with its parameters using curl as shown below. curl -X PUT \\ http://edgex-core-command:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue \\ -H 'Content-Type: application/json' \\ -d '{\"Bool\":\"true\", \"EnableRandomization_Bool\": \"true\"}'","title":"Get and Test the Command URL"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#create-rules","text":"Now that you have EdgeX and eKuiper running, the EdgeX stream defined, and you know the command to actuate Random-Boolean-Device , it is time to build the eKuiper rules.","title":"Create rules"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#the-first-rule","text":"Again, the 1st rule is to monitor for events coming from the Random-UnsignedInteger-Device device (one of the default virtual device managed devices), and if a uint8 reading value is found larger than 20 in the event, then send the command to Random-Boolean-Device device to start generating random numbers (specifically - set random generation bool to true). Given the URL and parameters to the command, below is the curl command to declare the first rule in eKuiper. curl -X POST \\ http:// $ekuiper_server :59720/rules \\ -H 'Content-Type: application/json' \\ -d '{ \"id\": \"rule1\", \"sql\": \"SELECT uint8 FROM demo WHERE uint8 > 20\", \"actions\": [ { \"rest\": { \"url\": \"http://edgex-core-command:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue\", \"method\": \"put\", \"dataTemplate\": \"{\\\"Bool\\\":\\\"true\\\", \\\"EnableRandomization_Bool\\\": \\\"true\\\"}\", \"sendSingle\": true } }, { \"log\":{} } ] }'","title":"The first rule"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#the-second-rule","text":"The 2nd rule is to monitor for events coming from the Random-Integer-Device device (another of the default virtual device managed devices), and if the average for int8 reading values (within 20 seconds) is larger than 0, then send a command to Random-Boolean-Device device to stop generating random numbers (specifically - set random generation bool to false). Here is the curl request to setup the second rule in eKuiper. The same command URL is used as the same device action ( Random-Boolean-Device's PUT bool command ) is being actuated, but with different parameters. curl -X POST \\ http:// $ekuiper_server :59720/rules \\ -H 'Content-Type: application/json' \\ -d '{ \"id\": \"rule2\", \"sql\": \"SELECT avg(int8) AS avg_int8 FROM demo WHERE int8 != nil GROUP BY TUMBLINGWINDOW(ss, 20) HAVING avg(int8) > 0\", \"actions\": [ { \"rest\": { \"url\": \"http://edgex-core-command:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue\", \"method\": \"put\", \"dataTemplate\": \"{\\\"Bool\\\":\\\"false\\\", \\\"EnableRandomization_Bool\\\": \\\"false\\\"}\", \"sendSingle\": true } }, { \"log\":{} } ] }'","title":"The second rule"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#watch-the-ekuiper-logs","text":"Both rules are now created in eKuiper. eKuiper is busy analyzing the event data coming for the virtual devices looking for readings that match the rules you created. You can watch the edgex-kuiper container logs for the rule triggering and command execution. docker logs edgex-kuiper","title":"Watch the eKuiper Logs"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#explore-the-results","text":"You can also explore the eKuiper analysis that caused the commands to be sent to the service. To see the the data from the analysis, use the SQL below to query eKuiper filtering data. SELECT int8 , \"true\" AS randomization FROM demo WHERE uint8 > 20 The output of the SQL should look similar to the results below. [{ \"int8\" : -75 , \"randomization\" : \"true\" }]","title":"Explore the Results"},{"location":"examples/Ch-CommandingDeviceThroughRulesEngine/#extended-reading","text":"Use these resources to learn more about the features of LF Edge eKuiper. eKuiper Github code repository eKuiper reference guide","title":"Extended Reading"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/","text":"MQTT EdgeX - Jakarta Release Overview In this example, we use a script to simulate a custom-defined MQTT device, instead of a real device. This provides a straight-forward way to test the device-mqtt features using an MQTT-broker. Note Multi-Level Topics move metadata (i.e. device name, command name,... etc) from the payload into the MQTT topics. Notice the sections marked with Using Multi-level Topic: for relevant input/output throughout this example. Prepare the Custom Device Configuration In this section, we create folders that contain files required for deployment of a customized device configuration to work with the existing device service: - custom-config |- devices |- my.custom.device.config.toml |- profiles |- my.custom.device.profile.yml Device Configuration Use this configuration file to define devices and schedule jobs. device-mqtt generates a relative instance on start-up. Create the device configuration file, named my.custom.device.config.toml , as shown below: # Pre-define Devices [[DeviceList]] Name = \"my-custom-device\" ProfileName = \"my-custom-device-profile\" Description = \"MQTT device is created for test purpose\" Labels = [ \"MQTT\" , \"test\" ] [DeviceList.Protocols] [DeviceList.Protocols.mqtt] # Comment out/remove below to use multi-level topics CommandTopic = \"CommandTopic\" # Uncomment below to use multi-level topics # CommandTopic = \"command/my-custom-device\" [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"message\" Note CommandTopic is used to publish the GET or SET command request Device Profile The DeviceProfile defines the device's values and operation method, which can be Read or Write. Create a device profile, named my.custom.device.profile.yml , with the following content: name : \"my-custom-device-profile\" manufacturer : \"iot\" model : \"MQTT-DEVICE\" description : \"Test device profile\" labels : - \"mqtt\" - \"test\" deviceResources : - name : randnum isHidden : true description : \"device random number\" properties : valueType : \"Float32\" readWrite : \"R\" - name : ping isHidden : true description : \"device awake\" properties : valueType : \"String\" readWrite : \"R\" - name : message isHidden : false description : \"device message\" properties : valueType : \"String\" readWrite : \"RW\" - name : json isHidden : false description : \"JSON message\" properties : valueType : \"Object\" readWrite : \"RW\" mediaType : \"application/json\" deviceCommands : - name : values readWrite : \"R\" isHidden : false resourceOperations : - { deviceResource : \"randnum\" } - { deviceResource : \"ping\" } - { deviceResource : \"message\" } Prepare docker-compose file Clone edgex-compose $ git clone git@github.com:edgexfoundry/edgex-compose.git $ git checkout main !!! note Use main branch until jakarta is released. Generate the docker-compose.yml file (notice this includes mqtt-broker) $ cd edgex-compose/compose-builder $ make gen ds-mqtt mqtt-broker no-secty ui Check the generated file $ ls | grep 'docker-compose.yml' docker-compose.yml Mount the custom-config Open the edgex-compose/compose-builder/docker-compose.yml file and then add volumes path and environment as shown below: # docker-compose.yml device-mqtt : ... environment : DEVICE_DEVICESDIR : /custom-config/devices DEVICE_PROFILESDIR : /custom-config/profiles ... volumes : - /path/to/custom-config:/custom-config ... Note Replace the /path/to/custom-config in the example with the correct path Enabling Multi-Level Topics To use the optional setting for MQTT device services with multi-level topics, make the following changes in the device service configuration files: There are two ways to set the environment variables for multi-level topics. If the code is built with compose builder, modify the docker-compose.yml file in edgex-compose/compose-builder: # docker-compose.yml device-mqtt : ... environment : MQTTBROKERINFO_INCOMINGTOPIC : \"incoming/data/#\" MQTTBROKERINFO_RESPONSETOPIC : \"command/response/#\" MQTTBROKERINFO_USETOPICLEVELS : \"true\" ... Otherwise if the device service is built locally, modify these lines in configuration.toml : # Comment out/remove when using multi-level topics #IncomingTopic = \"DataTopic\" #ResponseTopic = \"ResponseTopic\" #UseTopicLevels = false # Uncomment to use multi-level topics IncomingTopic = \"incoming/data/#\" ResponseTopic = \"command/response/#\" UseTopicLevels = true Note If you have previously run Device MQTT locally, you will need to remove the services configuration from Consul. This can be done with: curl --request DELETE http://localhost:8500/v1/kv/edgex/devices/2.0/device-mqtt?recurse=true In my.custom.device.config.toml : [DeviceList.Protocols] [DeviceList.Protocols.mqtt] # Comment out/remove below to use multi-level topics # CommandTopic = \"CommandTopic\" # Uncomment below to use multi-level topics CommandTopic = \"command/my-custom-device\" Note If you have run Device-MQTT before, you will need to delete the previously registered device(s) by replacing in the command below: curl --request DELETE http://localhost:59881/api/v2/device/name/ where can be found by running: curl --request GET http://localhost:59881/api/v2/device/all | json_pp Start EdgeX Foundry on Docker Deploy EdgeX using the following commands: $ cd edgex-compose/compose-builder $ docker-compose pull $ docker-compose up -d Using a MQTT Device Simulator Overview Expected Behaviors Using the detailed script below as a simulator, there are three behaviors: Publish random number data every 15 seconds. Default (single-level) Topic: The simulator publishes the data to the MQTT broker with topic DataTopic and the message is similar to the following: {\"name\":\"my-custom-device\", \"cmd\":\"randnum\", \"method\":\"get\", \"randnum\":4161.3549} Using Multi-level Topic: The simulator publishes the data to the MQTT broker with topic incoming/data/my-custom-device/randnum and the message is similar to the following: {\"randnum\":4161.3549} Receive the reading request, then return the response. Default (single-level) Topic: The simulator receives the request from the MQTT broker, the topic is CommandTopic and the message is similar to the following: {\"cmd\":\"randnum\", \"method\":\"get\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\"} The simulator returns the response to the MQTT broker, the topic is ResponseTopic and the message is similar to the following: {\"cmd\":\"randnum\", \"method\":\"get\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\", \"randnum\":42.0} Using Multi-level Topic: The simulator receives the request from the MQTT broker, the topic is command/my-custom-device/randnum/get/293d7a00-66e1-4374-ace0-07520103c95f and message returned is similar to the following: {\"randnum\":\"42.0\"} The simulator returns the response to the MQTT broker, the topic is command/response/# and the message is similar to the following: {\"randnum\":\"4.20e+01\"} Receive the set request, then change the device value. Default (single-level) Topic: The simulator receives the request from the MQTT broker, the topic is CommandTopic and the message is similar to the following: {\"cmd\":\"message\", \"method\":\"set\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\", \"message\":\"test message...\"} The simulator changes the device value and returns the response to the MQTT broker, the topic is ResponseTopic and the message is similar to the following: {\"cmd\":\"message\", \"method\":\"set\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\"} Using Multi-level Topic: The simulator receives the request from the MQTT broker, the topic is command/my-custom-device/testmessage/set/293d7a00-66e1-4374-ace0-07520103c95f and the message is similar to the following: {\"message\":\"test message...\"} The simulator changes the device value and returns the response to the MQTT broker, the topic is command/response/# and the message is similar to the following: {\"message\":\"test message...\"} Creating and Running a MQTT Device Simulator To implement the simulated custom-defined MQTT device, create a javascript, named mock-device.js , with the following content: Default (single-level) Topic: function getRandomFloat ( min , max ) { return Math . random () * ( max - min ) + min ; } const deviceName = \"my-custom-device\" ; let message = \"test-message\" ; let json = { \"name\" : \"My JSON\" }; // DataSender sends async value to MQTT broker every 15 seconds schedule ( '*/15 * * * * *' , ()=>{ let body = { \"name\" : deviceName , \"cmd\" : \"randnum\" , \"randnum\" : getRandomFloat ( 25 , 29 ). toFixed ( 1 ) }; publish ( 'DataTopic' , JSON . stringify ( body )); }); // CommandHandler receives commands and sends response to MQTT broker // 1. Receive the reading request, then return the response // 2. Receive the set request, then change the device value subscribe ( \"CommandTopic\" , ( topic , val ) => { var data = val ; if ( data . method == \"set\" ) { switch ( data . cmd ) { case \"message\" : message = data [ data . cmd ]; break ; case \"json\" : json = data [ data . cmd ]; break ; } } else { switch ( data . cmd ) { case \"ping\" : data . ping = \"pong\" ; break ; case \"message\" : data . message = message ; break ; case \"randnum\" : data . randnum = 12.123 ; break ; case \"json\" : data . json = json ; break ; } } publish ( \"ResponseTopic\" , JSON . stringify ( data )); }); Using Multi-level Topic: function getRandomFloat ( min , max ) { return Math . random () * ( max - min ) + min ; } const deviceName = \"my-custom-device\" ; let message = \"test-message\" ; let json = { \"name\" : \"My JSON\" }; // DataSender sends async value to MQTT broker every 15 seconds schedule ( '*/15 * * * * *' , ()=>{ let body = getRandomFloat ( 25 , 29 ). toFixed ( 1 ); publish ( 'incoming/data/my-custom-device/randnum' , body ); }); // CommandHandler receives commands and sends response to MQTT broker // 1. Receive the reading request, then return the response // 2. Receive the set request, then change the device value subscribe ( \"command/my-custom-device/#\" , ( topic , val ) => { const words = topic . split ( '/' ); var cmd = words [ 2 ]; var method = words [ 3 ]; var uuid = words [ 4 ]; var response = {}; var data = val ; if ( method == \"set\" ) { switch ( cmd ) { case \"message\" : message = data [ cmd ]; break ; case \"json\" : json = data [ cmd ]; break ; } } else { switch ( cmd ) { case \"ping\" : response . ping = \"pong\" ; break ; case \"message\" : response . message = message ; break ; case \"randnum\" : response . randnum = 12.123 ; break ; case \"json\" : response . json = json ; break ; } } var sendTopic = \"command/response/\" + uuid ; publish ( sendTopic , JSON . stringify ( response )); }); To run the device simulator, enter the commands shown below with the following changes: $ mv mock-device.js /path/to/mqtt-scripts $ docker run -d --restart=always --name=mqtt-scripts \\ -v /path/to/mqtt-scripts:/scripts \\ dersimn/mqtt-scripts --url mqtt://172.17.0.1 --dir /scripts Note Replace the /path/to/mqtt-scripts in the example mv command with the correct path Execute Commands Now we're ready to run some commands. Find Executable Commands Use the following query to find executable commands: $ curl h tt p : //localhost:59882/api/v2/device/all | json_pp { \"deviceCoreCommands\" : [ { \"profileName\" : \"my-custom-device-profile\" , \"coreCommands\" : [ { \"name\" : \"values\" , \"get\" : true , \"path\" : \"/api/v2/device/name/my-custom-device/values\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"randnum\" , \"valueType\" : \"Float32\" }, { \"resourceName\" : \"ping\" , \"valueType\" : \"String\" }, { \"valueType\" : \"String\" , \"resourceName\" : \"message\" } ] }, { \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"message\" , \"valueType\" : \"String\" } ], \"name\" : \"message\" , \"get\" : true , \"path\" : \"/api/v2/device/name/my-custom-device/message\" , \"set\" : true }, { \"name\" : \"json\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/MQTT-test-device/json\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"json\" , \"valueType\" : \"Object\" } ] } ], \"deviceName\" : \"my-custom-device\" } ], \"apiVersion\" : \"v2\" , \"statusCode\" : 200 } Execute SET Command Execute a SET command according to the url and parameterNames, replacing [host] with the server IP when running the SET command. $ curl http://localhost:59882/api/v2/device/name/my-custom-device/message \\ -H \"Content-Type:application/json\" -X PUT \\ -d '{\"message\":\"Hello!\"}' Execute GET Command Execute a GET command as follows: $ curl h tt p : //localhost:59882/api/v2/device/name/my-custom-device/message | json_pp { \"event\" : { \"origin\" : 1624417689920618131 , \"readings\" : [ { \"resourceName\" : \"message\" , \"binaryValue\" : null , \"profileName\" : \"my-custom-device-profile\" , \"deviceName\" : \"my-custom-device\" , \"id\" : \"a3bb78c5-e76f-49a2-ad9d-b220a86c3e36\" , \"value\" : \"Hello!\" , \"valueType\" : \"String\" , \"origin\" : 1624417689920615828 , \"mediaType\" : \"\" } ], \"sourceName\" : \"message\" , \"deviceName\" : \"my-custom-device\" , \"apiVersion\" : \"v2\" , \"profileName\" : \"my-custom-device-profile\" , \"id\" : \"e0b29735-8b39-44d1-8f68-4d7252e14cc7\" }, \"apiVersion\" : \"v2\" , \"statusCode\" : 200 } Schedule Job The schedule job is defined in the [[DeviceList.AutoEvents]] section of the device configuration file: [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"message\" After the service starts, query core-data's reading API. The results show that the service auto-executes the command every 30 secs, as shown below: $ curl h tt p : //localhost:59880/api/v2/reading/resourceName/message | json_pp { \"statusCode\" : 200 , \"readings\" : [ { \"value\" : \"test-message\" , \"id\" : \"e91b8ca6-c5c4-4509-bb61-bd4b09fe835c\" , \"mediaType\" : \"\" , \"binaryValue\" : null , \"resourceName\" : \"message\" , \"origin\" : 1624418361324331392 , \"profileName\" : \"my-custom-device-profile\" , \"deviceName\" : \"my-custom-device\" , \"valueType\" : \"String\" }, { \"mediaType\" : \"\" , \"binaryValue\" : null , \"resourceName\" : \"message\" , \"value\" : \"test-message\" , \"id\" : \"1da58cb7-2bf4-47f0-bbb8-9519797149a2\" , \"deviceName\" : \"my-custom-device\" , \"valueType\" : \"String\" , \"profileName\" : \"my-custom-device-profile\" , \"origin\" : 1624418330822988843 }, ... ], \"apiVersion\" : \"v2\" } Async Device Reading The device-mqtt subscribes to a DataTopic , which waits for the real device to send value to MQTT broker , then device-mqtt parses the value and forward to the northbound. The data format contains the following values: name = device name cmd = deviceResource name method = get or set cmd = device reading The following results show that the mock device sent the reading every 15 secs: $ curl h tt p : //localhost:59880/api/v2/reading/resourceName/randnum | json_pp { \"readings\" : [ { \"origin\" : 1624418475007110946 , \"valueType\" : \"Float32\" , \"deviceName\" : \"my-custom-device\" , \"id\" : \"9b3d337e-8a8a-4a6c-8018-b4908b57abb8\" , \"binaryValue\" : null , \"resourceName\" : \"randnum\" , \"profileName\" : \"my-custom-device-profile\" , \"mediaType\" : \"\" , \"value\" : \"2.630000e+01\" }, { \"deviceName\" : \"my-custom-device\" , \"valueType\" : \"Float32\" , \"id\" : \"06918cbb-ada0-4752-8877-0ef8488620f6\" , \"origin\" : 1624418460007833720 , \"mediaType\" : \"\" , \"profileName\" : \"my-custom-device-profile\" , \"value\" : \"2.570000e+01\" , \"resourceName\" : \"randnum\" , \"binaryValue\" : null }, ... ], \"statusCode\" : 200 , \"apiVersion\" : \"v2\" } MQTT Device Service Configuration MQTT Device Service has the following configurations to implement the MQTT protocol. Configuration Default Value Description MQTTBrokerInfo.Schema tcp The URL schema MQTTBrokerInfo.Host 0.0.0.0 The URL host MQTTBrokerInfo.Port 1883 The URL port MQTTBrokerInfo.Qos 0 Quality of Service 0 (At most once), 1 (At least once) or 2 (Exactly once) MQTTBrokerInfo.KeepAlive 3600 Seconds between client ping when no active data flowing to avoid client being disconnected. Must be greater then 2 MQTTBrokerInfo.ClientId device-mqtt ClientId to connect to the broker with MQTTBrokerInfo.CredentialsRetryTime 120 The retry times to get the credential MQTTBrokerInfo.CredentialsRetryWait 1 The wait time(seconds) when retry to get the credential MQTTBrokerInfo.ConnEstablishingRetry 10 The retry times to establish the MQTT connection MQTTBrokerInfo.ConnRetryWaitTime 5 The wait time(seconds) when retry to establish the MQTT connection MQTTBrokerInfo.AuthMode none Indicates what to use when connecting to the broker. Must be one of \"none\" , \"usernamepassword\" MQTTBrokerInfo.CredentialsPath credentials Name of the path in secret provider to retrieve your secrets. Must be non-blank. MQTTBrokerInfo.IncomingTopic DataTopic (incoming/data/#) IncomingTopic is used to receive the async value MQTTBrokerInfo.ResponseTopic ResponseTopic (command/response/#) ResponseTopic is used to receive the command response from the device MQTTBrokerInfo.UseTopicLevels false (true) Boolean setting to use multi-level topics MQTTBrokerInfo.Writable.ResponseFetchInterval 500 ResponseFetchInterval specifies the retry interval(milliseconds) to fetch the command response from the MQTT broker Note Using Multi-level Topic: Remember to change the defaults in parentheses in the table above. Overriding with Environment Variables The user can override any of the above configurations using environment: variables to meet their requirement, for example: # docker-compose.yml device-mqtt : . . . environment : MQTTBROKERINFO_CLIENTID : \"my-device-mqtt\" MQTTBROKERINFO_CONNRETRYWAITTIME : \"10\" MQTTBROKERINFO_USETOPICLEVELS : \"false\" ...","title":"MQTT"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#mqtt","text":"EdgeX - Jakarta Release","title":"MQTT"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#overview","text":"In this example, we use a script to simulate a custom-defined MQTT device, instead of a real device. This provides a straight-forward way to test the device-mqtt features using an MQTT-broker. Note Multi-Level Topics move metadata (i.e. device name, command name,... etc) from the payload into the MQTT topics. Notice the sections marked with Using Multi-level Topic: for relevant input/output throughout this example.","title":"Overview"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#prepare-the-custom-device-configuration","text":"In this section, we create folders that contain files required for deployment of a customized device configuration to work with the existing device service: - custom-config |- devices |- my.custom.device.config.toml |- profiles |- my.custom.device.profile.yml","title":"Prepare the Custom Device Configuration"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#device-configuration","text":"Use this configuration file to define devices and schedule jobs. device-mqtt generates a relative instance on start-up. Create the device configuration file, named my.custom.device.config.toml , as shown below: # Pre-define Devices [[DeviceList]] Name = \"my-custom-device\" ProfileName = \"my-custom-device-profile\" Description = \"MQTT device is created for test purpose\" Labels = [ \"MQTT\" , \"test\" ] [DeviceList.Protocols] [DeviceList.Protocols.mqtt] # Comment out/remove below to use multi-level topics CommandTopic = \"CommandTopic\" # Uncomment below to use multi-level topics # CommandTopic = \"command/my-custom-device\" [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"message\" Note CommandTopic is used to publish the GET or SET command request","title":"Device Configuration"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#device-profile","text":"The DeviceProfile defines the device's values and operation method, which can be Read or Write. Create a device profile, named my.custom.device.profile.yml , with the following content: name : \"my-custom-device-profile\" manufacturer : \"iot\" model : \"MQTT-DEVICE\" description : \"Test device profile\" labels : - \"mqtt\" - \"test\" deviceResources : - name : randnum isHidden : true description : \"device random number\" properties : valueType : \"Float32\" readWrite : \"R\" - name : ping isHidden : true description : \"device awake\" properties : valueType : \"String\" readWrite : \"R\" - name : message isHidden : false description : \"device message\" properties : valueType : \"String\" readWrite : \"RW\" - name : json isHidden : false description : \"JSON message\" properties : valueType : \"Object\" readWrite : \"RW\" mediaType : \"application/json\" deviceCommands : - name : values readWrite : \"R\" isHidden : false resourceOperations : - { deviceResource : \"randnum\" } - { deviceResource : \"ping\" } - { deviceResource : \"message\" }","title":"Device Profile"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#prepare-docker-compose-file","text":"Clone edgex-compose $ git clone git@github.com:edgexfoundry/edgex-compose.git $ git checkout main !!! note Use main branch until jakarta is released. Generate the docker-compose.yml file (notice this includes mqtt-broker) $ cd edgex-compose/compose-builder $ make gen ds-mqtt mqtt-broker no-secty ui Check the generated file $ ls | grep 'docker-compose.yml' docker-compose.yml","title":"Prepare docker-compose file"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#mount-the-custom-config","text":"Open the edgex-compose/compose-builder/docker-compose.yml file and then add volumes path and environment as shown below: # docker-compose.yml device-mqtt : ... environment : DEVICE_DEVICESDIR : /custom-config/devices DEVICE_PROFILESDIR : /custom-config/profiles ... volumes : - /path/to/custom-config:/custom-config ... Note Replace the /path/to/custom-config in the example with the correct path","title":"Mount the custom-config"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#enabling-multi-level-topics","text":"To use the optional setting for MQTT device services with multi-level topics, make the following changes in the device service configuration files: There are two ways to set the environment variables for multi-level topics. If the code is built with compose builder, modify the docker-compose.yml file in edgex-compose/compose-builder: # docker-compose.yml device-mqtt : ... environment : MQTTBROKERINFO_INCOMINGTOPIC : \"incoming/data/#\" MQTTBROKERINFO_RESPONSETOPIC : \"command/response/#\" MQTTBROKERINFO_USETOPICLEVELS : \"true\" ... Otherwise if the device service is built locally, modify these lines in configuration.toml : # Comment out/remove when using multi-level topics #IncomingTopic = \"DataTopic\" #ResponseTopic = \"ResponseTopic\" #UseTopicLevels = false # Uncomment to use multi-level topics IncomingTopic = \"incoming/data/#\" ResponseTopic = \"command/response/#\" UseTopicLevels = true Note If you have previously run Device MQTT locally, you will need to remove the services configuration from Consul. This can be done with: curl --request DELETE http://localhost:8500/v1/kv/edgex/devices/2.0/device-mqtt?recurse=true In my.custom.device.config.toml : [DeviceList.Protocols] [DeviceList.Protocols.mqtt] # Comment out/remove below to use multi-level topics # CommandTopic = \"CommandTopic\" # Uncomment below to use multi-level topics CommandTopic = \"command/my-custom-device\" Note If you have run Device-MQTT before, you will need to delete the previously registered device(s) by replacing in the command below: curl --request DELETE http://localhost:59881/api/v2/device/name/ where can be found by running: curl --request GET http://localhost:59881/api/v2/device/all | json_pp","title":"Enabling Multi-Level Topics"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#start-edgex-foundry-on-docker","text":"Deploy EdgeX using the following commands: $ cd edgex-compose/compose-builder $ docker-compose pull $ docker-compose up -d","title":"Start EdgeX Foundry on Docker"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#using-a-mqtt-device-simulator","text":"","title":"Using a MQTT Device Simulator"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#overview_1","text":"","title":"Overview"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#expected-behaviors","text":"Using the detailed script below as a simulator, there are three behaviors: Publish random number data every 15 seconds. Default (single-level) Topic: The simulator publishes the data to the MQTT broker with topic DataTopic and the message is similar to the following: {\"name\":\"my-custom-device\", \"cmd\":\"randnum\", \"method\":\"get\", \"randnum\":4161.3549} Using Multi-level Topic: The simulator publishes the data to the MQTT broker with topic incoming/data/my-custom-device/randnum and the message is similar to the following: {\"randnum\":4161.3549} Receive the reading request, then return the response. Default (single-level) Topic: The simulator receives the request from the MQTT broker, the topic is CommandTopic and the message is similar to the following: {\"cmd\":\"randnum\", \"method\":\"get\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\"} The simulator returns the response to the MQTT broker, the topic is ResponseTopic and the message is similar to the following: {\"cmd\":\"randnum\", \"method\":\"get\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\", \"randnum\":42.0} Using Multi-level Topic: The simulator receives the request from the MQTT broker, the topic is command/my-custom-device/randnum/get/293d7a00-66e1-4374-ace0-07520103c95f and message returned is similar to the following: {\"randnum\":\"42.0\"} The simulator returns the response to the MQTT broker, the topic is command/response/# and the message is similar to the following: {\"randnum\":\"4.20e+01\"} Receive the set request, then change the device value. Default (single-level) Topic: The simulator receives the request from the MQTT broker, the topic is CommandTopic and the message is similar to the following: {\"cmd\":\"message\", \"method\":\"set\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\", \"message\":\"test message...\"} The simulator changes the device value and returns the response to the MQTT broker, the topic is ResponseTopic and the message is similar to the following: {\"cmd\":\"message\", \"method\":\"set\", \"uuid\":\"293d7a00-66e1-4374-ace0-07520103c95f\"} Using Multi-level Topic: The simulator receives the request from the MQTT broker, the topic is command/my-custom-device/testmessage/set/293d7a00-66e1-4374-ace0-07520103c95f and the message is similar to the following: {\"message\":\"test message...\"} The simulator changes the device value and returns the response to the MQTT broker, the topic is command/response/# and the message is similar to the following: {\"message\":\"test message...\"}","title":"Expected Behaviors"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#creating-and-running-a-mqtt-device-simulator","text":"To implement the simulated custom-defined MQTT device, create a javascript, named mock-device.js , with the following content: Default (single-level) Topic: function getRandomFloat ( min , max ) { return Math . random () * ( max - min ) + min ; } const deviceName = \"my-custom-device\" ; let message = \"test-message\" ; let json = { \"name\" : \"My JSON\" }; // DataSender sends async value to MQTT broker every 15 seconds schedule ( '*/15 * * * * *' , ()=>{ let body = { \"name\" : deviceName , \"cmd\" : \"randnum\" , \"randnum\" : getRandomFloat ( 25 , 29 ). toFixed ( 1 ) }; publish ( 'DataTopic' , JSON . stringify ( body )); }); // CommandHandler receives commands and sends response to MQTT broker // 1. Receive the reading request, then return the response // 2. Receive the set request, then change the device value subscribe ( \"CommandTopic\" , ( topic , val ) => { var data = val ; if ( data . method == \"set\" ) { switch ( data . cmd ) { case \"message\" : message = data [ data . cmd ]; break ; case \"json\" : json = data [ data . cmd ]; break ; } } else { switch ( data . cmd ) { case \"ping\" : data . ping = \"pong\" ; break ; case \"message\" : data . message = message ; break ; case \"randnum\" : data . randnum = 12.123 ; break ; case \"json\" : data . json = json ; break ; } } publish ( \"ResponseTopic\" , JSON . stringify ( data )); }); Using Multi-level Topic: function getRandomFloat ( min , max ) { return Math . random () * ( max - min ) + min ; } const deviceName = \"my-custom-device\" ; let message = \"test-message\" ; let json = { \"name\" : \"My JSON\" }; // DataSender sends async value to MQTT broker every 15 seconds schedule ( '*/15 * * * * *' , ()=>{ let body = getRandomFloat ( 25 , 29 ). toFixed ( 1 ); publish ( 'incoming/data/my-custom-device/randnum' , body ); }); // CommandHandler receives commands and sends response to MQTT broker // 1. Receive the reading request, then return the response // 2. Receive the set request, then change the device value subscribe ( \"command/my-custom-device/#\" , ( topic , val ) => { const words = topic . split ( '/' ); var cmd = words [ 2 ]; var method = words [ 3 ]; var uuid = words [ 4 ]; var response = {}; var data = val ; if ( method == \"set\" ) { switch ( cmd ) { case \"message\" : message = data [ cmd ]; break ; case \"json\" : json = data [ cmd ]; break ; } } else { switch ( cmd ) { case \"ping\" : response . ping = \"pong\" ; break ; case \"message\" : response . message = message ; break ; case \"randnum\" : response . randnum = 12.123 ; break ; case \"json\" : response . json = json ; break ; } } var sendTopic = \"command/response/\" + uuid ; publish ( sendTopic , JSON . stringify ( response )); }); To run the device simulator, enter the commands shown below with the following changes: $ mv mock-device.js /path/to/mqtt-scripts $ docker run -d --restart=always --name=mqtt-scripts \\ -v /path/to/mqtt-scripts:/scripts \\ dersimn/mqtt-scripts --url mqtt://172.17.0.1 --dir /scripts Note Replace the /path/to/mqtt-scripts in the example mv command with the correct path","title":"Creating and Running a MQTT Device Simulator"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#execute-commands","text":"Now we're ready to run some commands.","title":"Execute Commands"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#find-executable-commands","text":"Use the following query to find executable commands: $ curl h tt p : //localhost:59882/api/v2/device/all | json_pp { \"deviceCoreCommands\" : [ { \"profileName\" : \"my-custom-device-profile\" , \"coreCommands\" : [ { \"name\" : \"values\" , \"get\" : true , \"path\" : \"/api/v2/device/name/my-custom-device/values\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"randnum\" , \"valueType\" : \"Float32\" }, { \"resourceName\" : \"ping\" , \"valueType\" : \"String\" }, { \"valueType\" : \"String\" , \"resourceName\" : \"message\" } ] }, { \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"message\" , \"valueType\" : \"String\" } ], \"name\" : \"message\" , \"get\" : true , \"path\" : \"/api/v2/device/name/my-custom-device/message\" , \"set\" : true }, { \"name\" : \"json\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/MQTT-test-device/json\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"json\" , \"valueType\" : \"Object\" } ] } ], \"deviceName\" : \"my-custom-device\" } ], \"apiVersion\" : \"v2\" , \"statusCode\" : 200 }","title":"Find Executable Commands"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#execute-set-command","text":"Execute a SET command according to the url and parameterNames, replacing [host] with the server IP when running the SET command. $ curl http://localhost:59882/api/v2/device/name/my-custom-device/message \\ -H \"Content-Type:application/json\" -X PUT \\ -d '{\"message\":\"Hello!\"}'","title":"Execute SET Command"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#execute-get-command","text":"Execute a GET command as follows: $ curl h tt p : //localhost:59882/api/v2/device/name/my-custom-device/message | json_pp { \"event\" : { \"origin\" : 1624417689920618131 , \"readings\" : [ { \"resourceName\" : \"message\" , \"binaryValue\" : null , \"profileName\" : \"my-custom-device-profile\" , \"deviceName\" : \"my-custom-device\" , \"id\" : \"a3bb78c5-e76f-49a2-ad9d-b220a86c3e36\" , \"value\" : \"Hello!\" , \"valueType\" : \"String\" , \"origin\" : 1624417689920615828 , \"mediaType\" : \"\" } ], \"sourceName\" : \"message\" , \"deviceName\" : \"my-custom-device\" , \"apiVersion\" : \"v2\" , \"profileName\" : \"my-custom-device-profile\" , \"id\" : \"e0b29735-8b39-44d1-8f68-4d7252e14cc7\" }, \"apiVersion\" : \"v2\" , \"statusCode\" : 200 }","title":"Execute GET Command"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#schedule-job","text":"The schedule job is defined in the [[DeviceList.AutoEvents]] section of the device configuration file: [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"message\" After the service starts, query core-data's reading API. The results show that the service auto-executes the command every 30 secs, as shown below: $ curl h tt p : //localhost:59880/api/v2/reading/resourceName/message | json_pp { \"statusCode\" : 200 , \"readings\" : [ { \"value\" : \"test-message\" , \"id\" : \"e91b8ca6-c5c4-4509-bb61-bd4b09fe835c\" , \"mediaType\" : \"\" , \"binaryValue\" : null , \"resourceName\" : \"message\" , \"origin\" : 1624418361324331392 , \"profileName\" : \"my-custom-device-profile\" , \"deviceName\" : \"my-custom-device\" , \"valueType\" : \"String\" }, { \"mediaType\" : \"\" , \"binaryValue\" : null , \"resourceName\" : \"message\" , \"value\" : \"test-message\" , \"id\" : \"1da58cb7-2bf4-47f0-bbb8-9519797149a2\" , \"deviceName\" : \"my-custom-device\" , \"valueType\" : \"String\" , \"profileName\" : \"my-custom-device-profile\" , \"origin\" : 1624418330822988843 }, ... ], \"apiVersion\" : \"v2\" }","title":"Schedule Job"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#async-device-reading","text":"The device-mqtt subscribes to a DataTopic , which waits for the real device to send value to MQTT broker , then device-mqtt parses the value and forward to the northbound. The data format contains the following values: name = device name cmd = deviceResource name method = get or set cmd = device reading The following results show that the mock device sent the reading every 15 secs: $ curl h tt p : //localhost:59880/api/v2/reading/resourceName/randnum | json_pp { \"readings\" : [ { \"origin\" : 1624418475007110946 , \"valueType\" : \"Float32\" , \"deviceName\" : \"my-custom-device\" , \"id\" : \"9b3d337e-8a8a-4a6c-8018-b4908b57abb8\" , \"binaryValue\" : null , \"resourceName\" : \"randnum\" , \"profileName\" : \"my-custom-device-profile\" , \"mediaType\" : \"\" , \"value\" : \"2.630000e+01\" }, { \"deviceName\" : \"my-custom-device\" , \"valueType\" : \"Float32\" , \"id\" : \"06918cbb-ada0-4752-8877-0ef8488620f6\" , \"origin\" : 1624418460007833720 , \"mediaType\" : \"\" , \"profileName\" : \"my-custom-device-profile\" , \"value\" : \"2.570000e+01\" , \"resourceName\" : \"randnum\" , \"binaryValue\" : null }, ... ], \"statusCode\" : 200 , \"apiVersion\" : \"v2\" }","title":"Async Device Reading"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#mqtt-device-service-configuration","text":"MQTT Device Service has the following configurations to implement the MQTT protocol. Configuration Default Value Description MQTTBrokerInfo.Schema tcp The URL schema MQTTBrokerInfo.Host 0.0.0.0 The URL host MQTTBrokerInfo.Port 1883 The URL port MQTTBrokerInfo.Qos 0 Quality of Service 0 (At most once), 1 (At least once) or 2 (Exactly once) MQTTBrokerInfo.KeepAlive 3600 Seconds between client ping when no active data flowing to avoid client being disconnected. Must be greater then 2 MQTTBrokerInfo.ClientId device-mqtt ClientId to connect to the broker with MQTTBrokerInfo.CredentialsRetryTime 120 The retry times to get the credential MQTTBrokerInfo.CredentialsRetryWait 1 The wait time(seconds) when retry to get the credential MQTTBrokerInfo.ConnEstablishingRetry 10 The retry times to establish the MQTT connection MQTTBrokerInfo.ConnRetryWaitTime 5 The wait time(seconds) when retry to establish the MQTT connection MQTTBrokerInfo.AuthMode none Indicates what to use when connecting to the broker. Must be one of \"none\" , \"usernamepassword\" MQTTBrokerInfo.CredentialsPath credentials Name of the path in secret provider to retrieve your secrets. Must be non-blank. MQTTBrokerInfo.IncomingTopic DataTopic (incoming/data/#) IncomingTopic is used to receive the async value MQTTBrokerInfo.ResponseTopic ResponseTopic (command/response/#) ResponseTopic is used to receive the command response from the device MQTTBrokerInfo.UseTopicLevels false (true) Boolean setting to use multi-level topics MQTTBrokerInfo.Writable.ResponseFetchInterval 500 ResponseFetchInterval specifies the retry interval(milliseconds) to fetch the command response from the MQTT broker Note Using Multi-level Topic: Remember to change the defaults in parentheses in the table above.","title":"MQTT Device Service Configuration"},{"location":"examples/Ch-ExamplesAddingMQTTDevice/#overriding-with-environment-variables","text":"The user can override any of the above configurations using environment: variables to meet their requirement, for example: # docker-compose.yml device-mqtt : . . . environment : MQTTBROKERINFO_CLIENTID : \"my-device-mqtt\" MQTTBROKERINFO_CONNRETRYWAITTIME : \"10\" MQTTBROKERINFO_USETOPICLEVELS : \"false\" ...","title":"Overriding with Environment Variables"},{"location":"examples/Ch-ExamplesAddingModbusDevice/","text":"Modbus EdgeX - Ireland Release This page describes how to connect Modbus devices to EdgeX. In this example, we simulate the temperature sensor instead of using a real device. This provides a straightforward way to test the device service features. Temperature sensor: https://www.audon.co.uk/ethernet_sensors/NANO_TEMP.html User manual: http://download.inveo.com.pl/manual/nano_t/user_manual_en.pdf Important Notice To fulfill the issue #61 , there is an important incompatible change after v2 (Ireland release). In the Device Profile attributes section, the startingAddress becomes an integer data type and zero-based value. In v1, startingAddress was a string data type and one-based value. Environment You can use any operating system that can install docker and docker-compose. In this example, we use Ubuntu to deploy EdgeX using docker. Modbus Device Simulator 1.Download ModbusPal Download the fixed version of ModbusPal from the https://sourceforge.net/p/modbuspal/discussion/899955/thread/72cf35ee/cd1f/attachment/ModbusPal.jar . 2.Install required lib: sudo apt install librxtx-java 3.Startup the ModbusPal: sudo java -jar ModbusPal.jar Modbus Register Table You can find the available registers in the user manual. Modbus TCP \u2013 Holding Registers Address Name R/W Description 4000 ThermostatL R/W Lower alarm threshold 4001 ThermostatH R/W Upper alarm threshold 4002 Alarm mode R/W 1 - OFF (disabled), 2 - Lower, 3 - Higher, 4 - Lower or Higher 4004 Temperature x10 R Temperature x 10 (np. 10,5 st.C to 105) Setup ModbusPal To simulate the sensor, do the following: Add mock device: Add registers according to the register table: Add the ModbusPal support value auto-generator, which can bind to the registers: Run the Simulator Enable the value generator and click the Run button. Set Up Before Starting Services The following sections describe how to complete the set up before starting the services. If you prefer to start the services and then add the device, see Set Up After Starting Services Create a Custom configuration folder Run the following command: mkdir -p custom-config Set Up Device Profile Run the following command to create your device profile: cd custom-config nano temperature.profile.yml Fill in the device profile according to the Modbus Register Table , as shown below: name : \"Ethernet-Temperature-Sensor\" manufacturer : \"Audon Electronics\" model : \"Temperature\" labels : - \"Web\" - \"Modbus TCP\" - \"SNMP\" description : \"The NANO_TEMP is a Ethernet Thermometer measuring from -55\u00b0C to 125\u00b0C with a web interface and Modbus TCP communications.\" deviceResources : - name : \"ThermostatL\" isHidden : true description : \"Lower alarm threshold of the temperature\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 3999 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"RW\" scale : \"0.1\" - name : \"ThermostatH\" isHidden : true description : \"Upper alarm threshold of the temperature\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4000 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"RW\" scale : \"0.1\" - name : \"AlarmMode\" isHidden : true description : \"1 - OFF (disabled), 2 - Lower, 3 - Higher, 4 - Lower or Higher\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4001 } properties : valueType : \"Int16\" readWrite : \"RW\" - name : \"Temperature\" isHidden : false description : \"Temperature x 10 (np. 10,5 st.C to 105)\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4003 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.1\" deviceCommands : - name : \"AlarmThreshold\" readWrite : \"RW\" isHidden : false resourceOperations : - { deviceResource : \"ThermostatL\" } - { deviceResource : \"ThermostatH\" } - name : \"AlarmMode\" readWrite : \"RW\" isHidden : false resourceOperations : - { deviceResource : \"AlarmMode\" , mappings : { \"1\" : \"OFF\" , \"2\" : \"Lower\" , \"3\" : \"Higher\" , \"4\" : \"Lower or Higher\" } } In the Modbus protocol, we provide the following attributes: 1. primaryTable : HOLDING_REGISTERS, INPUT_REGISTERS, COILS, DISCRETES_INPUT 2. startingAddress This attribute defines the zero-based startingAddress in Modbus device. For example, the GET command requests data from the Modbus address 4004 to get the temperature data, so the starting register address should be 4003. Address Starting Address Name R/W Description 4004 4003 Temperature x10 R Temperature x 10 (np. 10,5 st.C to 105) 3. IS_BYTE_SWAP , IS_WORD_SWAP : To handle the different Modbus binary data order, we support Int32, Uint32, Float32 to do the swap operation before decoding the binary data. For example: { primaryTable: \"INPUT_REGISTERS\", startingAddress: \"4\", isByteSwap: \"false\", isWordSwap: \"true\" } 4. RAW_TYPE : This attribute defines the binary data read from the Modbus device, then we can use the value type to indicate the data type that the user wants to receive. We only support Int16 and Uint16 for rawType. The corresponding value type must be Float32 and Float64 . For example: deviceResources : - name : \"Temperature\" isHidden : false description : \"Temperature x 10 (np. 10,5 st.C to 105)\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4003 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.1\" In the device-modbus, the Property valueType decides how many registers will be read. Like Holding registers, a register has 16 bits. If the Modbus device's user manual specifies that a value has two registers, define it as Float32 or Int32 or Uint32 in the deviceProfile. Once we execute a command, device-modbus knows its value type and register type, startingAddress, and register length. So it can read or write value using the modbus protocol. Set Up Device Service Configuration Run the following command to create your device configuration: cd custom-config nano device.config.toml Fill in the device.config.toml file, as shown below: [[DeviceList]] Name = \"Modbus-TCP-Temperature-Sensor\" ProfileName = \"Ethernet-Temperature-Sensor\" Description = \"This device is a product for monitoring the temperature via the ethernet\" labels = [ \"temperature\" , \"modbus TCP\" ] [DeviceList.Protocols] [DeviceList.Protocols.modbus-tcp] Address = \"172.17.0.1\" Port = \"502\" UnitID = \"1\" Timeout = \"5\" IdleTimeout = \"5\" [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"Temperature\" The address 172.17.0.1 is point to the docker bridge network which means it can forward the request from docker network to the host. Use this configuration file to define devices and AutoEvent. Then the device-modbus will generate the relative instance on startup. The device-modbus offers two types of protocol, Modbus TCP and Modbus RTU, which can be defined as shown below: protocol Name Protocol Address Port UnitID BaudRate DataBits StopBits Parity Timeout IdleTimeout Modbus TCP Gateway address TCP 10.211.55.6 502 1 5 5 Modbus RTU Gateway address RTU /tmp/slave 502 2 19200 8 1 N 5 5 In the RTU protocol, Parity can be: N - None is 0 O - Odd is 1 E - Even is 2, default is E Prepare docker-compose file Clone edgex-compose $ git clone git@github.com:edgexfoundry/edgex-compose.git Generate the docker-compose.yml file $ cd edgex-compose/compose-builder $ make gen ds-modbus Add Custom Configuration to docker-compose File Add prepared configuration files to docker-compose file, you can mount them using volumes and change the environment for device-modbus internal use. Open the docker-compose.yml file and then add volumes path and environment as shown below: device-modbus : ... environment : ... DEVICE_DEVICESDIR : /custom-config DEVICE_PROFILESDIR : /custom-config volumes : ... - /path/to/custom-config:/custom-config Start EdgeX Foundry on Docker Since we generate the docker-compose.yml file at the previous step, we can deploy EdgeX as shown below: $ cd edgex-compose/compose-builder $ docker-compose up -d Creating network \"compose-builder_edgex-network\" with driver \"bridge\" Creating volume \"compose-builder_consul-acl-token\" with default driver ... Creating edgex-core-metadata ... done Creating edgex-core-command ... done Creating edgex-core-data ... done Creating edgex-device-modbus ... done Creating edgex-app-rules-engine ... done Creating edgex-sys-mgmt-agent ... done Set Up After Starting Services If the services are already running and you want to add a device, you can use the Core Metadata API as outlined in this section. If you set up the device profile and Service as described in Set Up Before Starting Services , you can skip this section. To add a device after starting the services, complete the following steps: Upload the device profile above to metadata with a POST to http://localhost:59881/api/v2/deviceprofile/uploadfile and add the file as key \"file\" to the body in form-data format, and the created ID will be returned. The following example command uses curl to send the request: $ curl http://localhost:59881/api/v2/deviceprofile/uploadfile \\ -F \"file=@temperature.profile.yml\" Ensure the Modbus device service is running, adjust the service name below to match if necessary or if using other device services. Add the device with a POST to http://localhost:59881/api/v2/device , the body will look something like: $ curl http://localhost:59881/api/v2/device -H \"Content-Type:application/json\" -X POST \\ -d '[ { \"apiVersion\": \"v2\", \"device\": { \"name\" :\"Modbus-TCP-Temperature-Sensor\", \"description\":\"This device is a product for monitoring the temperature via the ethernet\", \"labels\":[ \"Temperature\", \"Modbus TCP\" ], \"serviceName\": \"device-modbus\", \"profileName\": \"Ethernet-Temperature-Sensor\", \"protocols\":{ \"modbus-tcp\":{ \"Address\" : \"172.17.0.1\", \"Port\" : \"502\", \"UnitID\" : \"1\", \"Timeout\" : \"5\", \"IdleTimeout\" : \"5\" } }, \"autoEvents\":[ { \"Interval\":\"30s\", \"onChange\":false, \"SourceName\":\"Temperature\" } ], \"adminState\":\"UNLOCKED\", \"operatingState\":\"UP\" } } ]' The service name must match/refer to the target device service, and the profile name must match the device profile name from the previous steps. Execute Commands Now we're ready to run some commands. Find Executable Commands Use the following query to find executable commands: $ curl h tt p : //localhost:59882/api/v2/device/all | json_pp { \"apiVersion\" : \"v2\" , \"deviceCoreCommands\" : [ { \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"coreCommands\" : [ { \"url\" : \"http://edgex-core-command:59882\" , \"name\" : \"AlarmThreshold\" , \"get\" : true , \"set\" : true , \"parameters\" : [ { \"valueType\" : \"Float32\" , \"resourceName\" : \"ThermostatL\" }, { \"valueType\" : \"Float32\" , \"resourceName\" : \"ThermostatH\" } ], \"path\" : \"/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmThreshold\" }, { \"get\" : true , \"url\" : \"http://edgex-core-command:59882\" , \"name\" : \"AlarmMode\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmMode\" , \"parameters\" : [ { \"resourceName\" : \"AlarmMode\" , \"valueType\" : \"Int16\" } ] }, { \"get\" : true , \"url\" : \"http://edgex-core-command:59882\" , \"name\" : \"Temperature\" , \"path\" : \"/api/v2/device/name/Modbus-TCP-Temperature-Sensor/Temperature\" , \"parameters\" : [ { \"valueType\" : \"Float32\" , \"resourceName\" : \"Temperature\" } ] } ] } ], \"statusCode\" : 200 } Execute SET command Execute SET command according to url and parameterNames , replacing [host] with the server IP when running the SET command. $ curl http://localhost:59882/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmThreshold \\ -H \"Content-Type:application/json\" -X PUT \\ -d '{\"ThermostatL\":\"15\",\"ThermostatH\":\"100\"}' Execute GET command Replace \\ with the server IP when running the GET command. $ curl h tt p : //localhost:59882/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmThreshold | json_pp { \"statusCode\" : 200 , \"apiVersion\" : \"v2\" , \"event\" : { \"origin\" : 1624324686964377495 , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"f3d44a0f-d2c3-4ef6-9441-ad6b1bfb8a9e\" , \"sourceName\" : \"AlarmThreshold\" , \"readings\" : [ { \"resourceName\" : \"ThermostatL\" , \"value\" : \"1.500000e+01\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"9aa879a0-c184-476b-8124-34d35a2a51f3\" , \"valueType\" : \"Float32\" , \"mediaType\" : \"\" , \"binaryValue\" : null , \"origin\" : 1624324686963970614 , \"profileName\" : \"Ethernet-Temperature-Sensor\" }, { \"value\" : \"1.000000e+02\" , \"resourceName\" : \"ThermostatH\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"bf7df23b-4338-4b93-a8bd-7abd5e848379\" , \"valueType\" : \"Float32\" , \"mediaType\" : \"\" , \"binaryValue\" : null , \"origin\" : 1624324686964343768 , \"profileName\" : \"Ethernet-Temperature-Sensor\" } ], \"apiVersion\" : \"v2\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" } } AutoEvent The AutoEvent is defined in the [[DeviceList.AutoEvents]] section of the device configuration file: [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"Temperature\" After service startup, query core-data's API. The results show that the service auto-executes the command every 30 seconds. $ curl h tt p : //localhost:59880/api/v2/event/device/name/Modbus-TCP-Temperature-Sensor | json_pp { \"events\" : [ { \"readings\" : [ { \"value\" : \"5.300000e+01\" , \"binaryValue\" : null , \"origin\" : 1624325219186870396 , \"id\" : \"68a66a35-d3cf-48a2-9bf0-09578267a3f7\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"mediaType\" : \"\" , \"valueType\" : \"Float32\" , \"resourceName\" : \"Temperature\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" } ], \"apiVersion\" : \"v2\" , \"origin\" : 1624325219186977564 , \"id\" : \"4b235616-7304-419e-97ae-17a244911b1c\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"sourceName\" : \"Temperature\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" }, { \"readings\" : [ { \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"resourceName\" : \"Temperature\" , \"valueType\" : \"Float32\" , \"id\" : \"56b7e8be-7ce8-4fa9-89e2-3a1a7ef09050\" , \"origin\" : 1624325189184675483 , \"value\" : \"5.300000e+01\" , \"binaryValue\" : null , \"mediaType\" : \"\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" } ], \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"sourceName\" : \"Temperature\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"fbab44f5-9775-4c09-84bd-cbfb00001115\" , \"origin\" : 1624325189184721223 , \"apiVersion\" : \"v2\" }, ... ], \"apiVersion\" : \"v2\" , \"statusCode\" : 200 } Set up the Modbus RTU Device This section describes how to connect the Modbus RTU device. We use Ubuntu OS and a Modbus RTU device for this example. Modbus RTU device: http://www.icpdas.com/root/product/solutions/remote_io/rs-485/i-7000_m-7000/i-7055.html User manual: http://ftp.icpdas.com/pub/cd/8000cd/napdos/7000/manual/7000dio.pdf Connect the device Connect the device to your machine(laptop or gateway,etc.) via RS485/USB adaptor and power on. Execute a command on the machine, and you can find a message like the following: $ dmesg | grep tty ... ... [18006.167625] usb 1-1: FTDI USB Serial Device converter now attached to ttyUSB0 It shows the USB attach to ttyUSB0, then you can check whether the device path exists: $ ls /dev/ttyUSB0 /dev/ttyUSB0 Change the Owner of the Device For security reason, the EdgeX set up the user permission as below: device-modbus : ... user : 2002:2001 # UID:GID So we need to change the owner for the specified group by the following command: sudo chown :2001 /dev/ttyUSB0 # Or change the permissions for multiple files sudo chown :2001 /dev/tty* Note Since the owner will reset after the system reboot, we can add this script to the startup script. For Raspberry Pi as example, add script to /etc/rc.local , then the Pi will run this script at bootup. Mont the Device Path to the Docker Container Modify the docker-compose.yml file to mount the device path to the device-modbus, and here are two ways to mount the device path: Using devices : device-modbus : ... devices : - /dev/ttyUSB0 Or using volumes and device_cgroup_rules : device-modbus : ... volumes : ... - /dev:/dev device_cgroup_rules : - 'c 188:* rw' c: character device 188: device major number(188=USB) *: device minor number rw: read/write Deploy the EdgeX $ docker-compose up -d Add device to EdgeX Create the device profile according to the register table $ nano modbus.rtu.demo.profile.yml name : \"Modbus-RTU-IO-Module\" manufacturer : \"icpdas\" model : \"M-7055\" labels : - \"Modbus RTU\" - \"IO Module\" description : \"This IO module offers 8 isolated channels for digital input and 8 isolated channels for digital output.\" deviceResources : - name : \"DO0\" isHidden : true description : \"On/Off , 0-OFF 1-ON\" attributes : { primaryTable : \"COILS\" , startingAddress : 0 } properties : valueType : \"Bool\" readWrite : \"RW\" - name : \"DO1\" isHidden : true description : \"On/Off , 0-OFF 1-ON\" attributes : { primaryTable : \"COILS\" , startingAddress : 1 } properties : valueType : \"Bool\" readWrite : \"RW\" - name : \"DO2\" isHidden : true description : \"On/Off , 0-OFF 1-ON\" attributes : { primaryTable : \"COILS\" , startingAddress : 2 } properties : valueType : \"Bool\" readWrite : \"RW\" deviceCommands : - name : \"DO\" readWrite : \"RW\" isHidden : false resourceOperations : - { deviceResource : \"DO0\" } - { deviceResource : \"DO1\" } - { deviceResource : \"DO2\" } Upload the device profile $ curl http://localhost:59881/api/v2/deviceprofile/uploadfile \\ -F \"file=@modbus.rtu.demo.profile.yml\" Create the device entity to the EdgeX. You can find the Modbus RTU setting on the device or the user manual. $ curl h tt p : //localhost:59881/api/v2/device -H \"Content-Type:application/json\" -X POST \\ - d ' [ { \"apiVersion\" : \"v2\" , \"device\" : { \"name\" : \"Modbus-RTU-IO-Module\" , \"description\" : \"The device can be used to monitor the status of the digital input and digital output channels.\" , \"labels\" :[ \"IO Module\" , \"Modbus RTU\" ], \"serviceName\" : \"device-modbus\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"protocols\" :{ \"modbus-tcp\" :{ \"Address\" : \"/dev/ttyUSB0\" , \"BaudRate\" : \"19200\" , \"DataBits\" : \"8\" , \"StopBits\" : \"1\" , \"Parity\" : \"N\" , \"UnitID\" : \"1\" , \"Timeout\" : \"5\" , \"IdleTimeout\" : \"5\" } }, \"adminState\" : \"UNLOCKED\" , \"operatingState\" : \"UP\" } } ] ' Test the GET or SET command","title":"Modbus"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#modbus","text":"EdgeX - Ireland Release This page describes how to connect Modbus devices to EdgeX. In this example, we simulate the temperature sensor instead of using a real device. This provides a straightforward way to test the device service features. Temperature sensor: https://www.audon.co.uk/ethernet_sensors/NANO_TEMP.html User manual: http://download.inveo.com.pl/manual/nano_t/user_manual_en.pdf","title":"Modbus"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#important-notice","text":"To fulfill the issue #61 , there is an important incompatible change after v2 (Ireland release). In the Device Profile attributes section, the startingAddress becomes an integer data type and zero-based value. In v1, startingAddress was a string data type and one-based value.","title":"Important Notice"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#environment","text":"You can use any operating system that can install docker and docker-compose. In this example, we use Ubuntu to deploy EdgeX using docker.","title":"Environment"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#modbus-device-simulator","text":"1.Download ModbusPal Download the fixed version of ModbusPal from the https://sourceforge.net/p/modbuspal/discussion/899955/thread/72cf35ee/cd1f/attachment/ModbusPal.jar . 2.Install required lib: sudo apt install librxtx-java 3.Startup the ModbusPal: sudo java -jar ModbusPal.jar","title":"Modbus Device Simulator"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#modbus-register-table","text":"You can find the available registers in the user manual. Modbus TCP \u2013 Holding Registers Address Name R/W Description 4000 ThermostatL R/W Lower alarm threshold 4001 ThermostatH R/W Upper alarm threshold 4002 Alarm mode R/W 1 - OFF (disabled), 2 - Lower, 3 - Higher, 4 - Lower or Higher 4004 Temperature x10 R Temperature x 10 (np. 10,5 st.C to 105)","title":"Modbus Register Table"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#setup-modbuspal","text":"To simulate the sensor, do the following: Add mock device: Add registers according to the register table: Add the ModbusPal support value auto-generator, which can bind to the registers:","title":"Setup ModbusPal"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#run-the-simulator","text":"Enable the value generator and click the Run button.","title":"Run the Simulator"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#set-up-before-starting-services","text":"The following sections describe how to complete the set up before starting the services. If you prefer to start the services and then add the device, see Set Up After Starting Services","title":"Set Up Before Starting Services"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#create-a-custom-configuration-folder","text":"Run the following command: mkdir -p custom-config","title":"Create a Custom configuration folder"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#set-up-device-profile","text":"Run the following command to create your device profile: cd custom-config nano temperature.profile.yml Fill in the device profile according to the Modbus Register Table , as shown below: name : \"Ethernet-Temperature-Sensor\" manufacturer : \"Audon Electronics\" model : \"Temperature\" labels : - \"Web\" - \"Modbus TCP\" - \"SNMP\" description : \"The NANO_TEMP is a Ethernet Thermometer measuring from -55\u00b0C to 125\u00b0C with a web interface and Modbus TCP communications.\" deviceResources : - name : \"ThermostatL\" isHidden : true description : \"Lower alarm threshold of the temperature\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 3999 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"RW\" scale : \"0.1\" - name : \"ThermostatH\" isHidden : true description : \"Upper alarm threshold of the temperature\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4000 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"RW\" scale : \"0.1\" - name : \"AlarmMode\" isHidden : true description : \"1 - OFF (disabled), 2 - Lower, 3 - Higher, 4 - Lower or Higher\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4001 } properties : valueType : \"Int16\" readWrite : \"RW\" - name : \"Temperature\" isHidden : false description : \"Temperature x 10 (np. 10,5 st.C to 105)\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4003 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.1\" deviceCommands : - name : \"AlarmThreshold\" readWrite : \"RW\" isHidden : false resourceOperations : - { deviceResource : \"ThermostatL\" } - { deviceResource : \"ThermostatH\" } - name : \"AlarmMode\" readWrite : \"RW\" isHidden : false resourceOperations : - { deviceResource : \"AlarmMode\" , mappings : { \"1\" : \"OFF\" , \"2\" : \"Lower\" , \"3\" : \"Higher\" , \"4\" : \"Lower or Higher\" } } In the Modbus protocol, we provide the following attributes: 1. primaryTable : HOLDING_REGISTERS, INPUT_REGISTERS, COILS, DISCRETES_INPUT 2. startingAddress This attribute defines the zero-based startingAddress in Modbus device. For example, the GET command requests data from the Modbus address 4004 to get the temperature data, so the starting register address should be 4003. Address Starting Address Name R/W Description 4004 4003 Temperature x10 R Temperature x 10 (np. 10,5 st.C to 105) 3. IS_BYTE_SWAP , IS_WORD_SWAP : To handle the different Modbus binary data order, we support Int32, Uint32, Float32 to do the swap operation before decoding the binary data. For example: { primaryTable: \"INPUT_REGISTERS\", startingAddress: \"4\", isByteSwap: \"false\", isWordSwap: \"true\" } 4. RAW_TYPE : This attribute defines the binary data read from the Modbus device, then we can use the value type to indicate the data type that the user wants to receive. We only support Int16 and Uint16 for rawType. The corresponding value type must be Float32 and Float64 . For example: deviceResources : - name : \"Temperature\" isHidden : false description : \"Temperature x 10 (np. 10,5 st.C to 105)\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : 4003 , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.1\" In the device-modbus, the Property valueType decides how many registers will be read. Like Holding registers, a register has 16 bits. If the Modbus device's user manual specifies that a value has two registers, define it as Float32 or Int32 or Uint32 in the deviceProfile. Once we execute a command, device-modbus knows its value type and register type, startingAddress, and register length. So it can read or write value using the modbus protocol.","title":"Set Up Device Profile"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#set-up-device-service-configuration","text":"Run the following command to create your device configuration: cd custom-config nano device.config.toml Fill in the device.config.toml file, as shown below: [[DeviceList]] Name = \"Modbus-TCP-Temperature-Sensor\" ProfileName = \"Ethernet-Temperature-Sensor\" Description = \"This device is a product for monitoring the temperature via the ethernet\" labels = [ \"temperature\" , \"modbus TCP\" ] [DeviceList.Protocols] [DeviceList.Protocols.modbus-tcp] Address = \"172.17.0.1\" Port = \"502\" UnitID = \"1\" Timeout = \"5\" IdleTimeout = \"5\" [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"Temperature\" The address 172.17.0.1 is point to the docker bridge network which means it can forward the request from docker network to the host. Use this configuration file to define devices and AutoEvent. Then the device-modbus will generate the relative instance on startup. The device-modbus offers two types of protocol, Modbus TCP and Modbus RTU, which can be defined as shown below: protocol Name Protocol Address Port UnitID BaudRate DataBits StopBits Parity Timeout IdleTimeout Modbus TCP Gateway address TCP 10.211.55.6 502 1 5 5 Modbus RTU Gateway address RTU /tmp/slave 502 2 19200 8 1 N 5 5 In the RTU protocol, Parity can be: N - None is 0 O - Odd is 1 E - Even is 2, default is E","title":"Set Up Device Service Configuration"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#prepare-docker-compose-file","text":"Clone edgex-compose $ git clone git@github.com:edgexfoundry/edgex-compose.git Generate the docker-compose.yml file $ cd edgex-compose/compose-builder $ make gen ds-modbus","title":"Prepare docker-compose file"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#add-custom-configuration-to-docker-compose-file","text":"Add prepared configuration files to docker-compose file, you can mount them using volumes and change the environment for device-modbus internal use. Open the docker-compose.yml file and then add volumes path and environment as shown below: device-modbus : ... environment : ... DEVICE_DEVICESDIR : /custom-config DEVICE_PROFILESDIR : /custom-config volumes : ... - /path/to/custom-config:/custom-config","title":"Add Custom Configuration to docker-compose File"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#start-edgex-foundry-on-docker","text":"Since we generate the docker-compose.yml file at the previous step, we can deploy EdgeX as shown below: $ cd edgex-compose/compose-builder $ docker-compose up -d Creating network \"compose-builder_edgex-network\" with driver \"bridge\" Creating volume \"compose-builder_consul-acl-token\" with default driver ... Creating edgex-core-metadata ... done Creating edgex-core-command ... done Creating edgex-core-data ... done Creating edgex-device-modbus ... done Creating edgex-app-rules-engine ... done Creating edgex-sys-mgmt-agent ... done","title":"Start EdgeX Foundry on Docker"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#set-up-after-starting-services","text":"If the services are already running and you want to add a device, you can use the Core Metadata API as outlined in this section. If you set up the device profile and Service as described in Set Up Before Starting Services , you can skip this section. To add a device after starting the services, complete the following steps: Upload the device profile above to metadata with a POST to http://localhost:59881/api/v2/deviceprofile/uploadfile and add the file as key \"file\" to the body in form-data format, and the created ID will be returned. The following example command uses curl to send the request: $ curl http://localhost:59881/api/v2/deviceprofile/uploadfile \\ -F \"file=@temperature.profile.yml\" Ensure the Modbus device service is running, adjust the service name below to match if necessary or if using other device services. Add the device with a POST to http://localhost:59881/api/v2/device , the body will look something like: $ curl http://localhost:59881/api/v2/device -H \"Content-Type:application/json\" -X POST \\ -d '[ { \"apiVersion\": \"v2\", \"device\": { \"name\" :\"Modbus-TCP-Temperature-Sensor\", \"description\":\"This device is a product for monitoring the temperature via the ethernet\", \"labels\":[ \"Temperature\", \"Modbus TCP\" ], \"serviceName\": \"device-modbus\", \"profileName\": \"Ethernet-Temperature-Sensor\", \"protocols\":{ \"modbus-tcp\":{ \"Address\" : \"172.17.0.1\", \"Port\" : \"502\", \"UnitID\" : \"1\", \"Timeout\" : \"5\", \"IdleTimeout\" : \"5\" } }, \"autoEvents\":[ { \"Interval\":\"30s\", \"onChange\":false, \"SourceName\":\"Temperature\" } ], \"adminState\":\"UNLOCKED\", \"operatingState\":\"UP\" } } ]' The service name must match/refer to the target device service, and the profile name must match the device profile name from the previous steps.","title":"Set Up After Starting Services"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#execute-commands","text":"Now we're ready to run some commands.","title":"Execute Commands"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#find-executable-commands","text":"Use the following query to find executable commands: $ curl h tt p : //localhost:59882/api/v2/device/all | json_pp { \"apiVersion\" : \"v2\" , \"deviceCoreCommands\" : [ { \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"coreCommands\" : [ { \"url\" : \"http://edgex-core-command:59882\" , \"name\" : \"AlarmThreshold\" , \"get\" : true , \"set\" : true , \"parameters\" : [ { \"valueType\" : \"Float32\" , \"resourceName\" : \"ThermostatL\" }, { \"valueType\" : \"Float32\" , \"resourceName\" : \"ThermostatH\" } ], \"path\" : \"/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmThreshold\" }, { \"get\" : true , \"url\" : \"http://edgex-core-command:59882\" , \"name\" : \"AlarmMode\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmMode\" , \"parameters\" : [ { \"resourceName\" : \"AlarmMode\" , \"valueType\" : \"Int16\" } ] }, { \"get\" : true , \"url\" : \"http://edgex-core-command:59882\" , \"name\" : \"Temperature\" , \"path\" : \"/api/v2/device/name/Modbus-TCP-Temperature-Sensor/Temperature\" , \"parameters\" : [ { \"valueType\" : \"Float32\" , \"resourceName\" : \"Temperature\" } ] } ] } ], \"statusCode\" : 200 }","title":"Find Executable Commands"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#execute-set-command","text":"Execute SET command according to url and parameterNames , replacing [host] with the server IP when running the SET command. $ curl http://localhost:59882/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmThreshold \\ -H \"Content-Type:application/json\" -X PUT \\ -d '{\"ThermostatL\":\"15\",\"ThermostatH\":\"100\"}'","title":"Execute SET command"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#execute-get-command","text":"Replace \\ with the server IP when running the GET command. $ curl h tt p : //localhost:59882/api/v2/device/name/Modbus-TCP-Temperature-Sensor/AlarmThreshold | json_pp { \"statusCode\" : 200 , \"apiVersion\" : \"v2\" , \"event\" : { \"origin\" : 1624324686964377495 , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"f3d44a0f-d2c3-4ef6-9441-ad6b1bfb8a9e\" , \"sourceName\" : \"AlarmThreshold\" , \"readings\" : [ { \"resourceName\" : \"ThermostatL\" , \"value\" : \"1.500000e+01\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"9aa879a0-c184-476b-8124-34d35a2a51f3\" , \"valueType\" : \"Float32\" , \"mediaType\" : \"\" , \"binaryValue\" : null , \"origin\" : 1624324686963970614 , \"profileName\" : \"Ethernet-Temperature-Sensor\" }, { \"value\" : \"1.000000e+02\" , \"resourceName\" : \"ThermostatH\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"bf7df23b-4338-4b93-a8bd-7abd5e848379\" , \"valueType\" : \"Float32\" , \"mediaType\" : \"\" , \"binaryValue\" : null , \"origin\" : 1624324686964343768 , \"profileName\" : \"Ethernet-Temperature-Sensor\" } ], \"apiVersion\" : \"v2\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" } }","title":"Execute GET command"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#autoevent","text":"The AutoEvent is defined in the [[DeviceList.AutoEvents]] section of the device configuration file: [[DeviceList.AutoEvents]] Interval = \"30s\" OnChange = false SourceName = \"Temperature\" After service startup, query core-data's API. The results show that the service auto-executes the command every 30 seconds. $ curl h tt p : //localhost:59880/api/v2/event/device/name/Modbus-TCP-Temperature-Sensor | json_pp { \"events\" : [ { \"readings\" : [ { \"value\" : \"5.300000e+01\" , \"binaryValue\" : null , \"origin\" : 1624325219186870396 , \"id\" : \"68a66a35-d3cf-48a2-9bf0-09578267a3f7\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"mediaType\" : \"\" , \"valueType\" : \"Float32\" , \"resourceName\" : \"Temperature\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" } ], \"apiVersion\" : \"v2\" , \"origin\" : 1624325219186977564 , \"id\" : \"4b235616-7304-419e-97ae-17a244911b1c\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"sourceName\" : \"Temperature\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" }, { \"readings\" : [ { \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"resourceName\" : \"Temperature\" , \"valueType\" : \"Float32\" , \"id\" : \"56b7e8be-7ce8-4fa9-89e2-3a1a7ef09050\" , \"origin\" : 1624325189184675483 , \"value\" : \"5.300000e+01\" , \"binaryValue\" : null , \"mediaType\" : \"\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" } ], \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"sourceName\" : \"Temperature\" , \"deviceName\" : \"Modbus-TCP-Temperature-Sensor\" , \"id\" : \"fbab44f5-9775-4c09-84bd-cbfb00001115\" , \"origin\" : 1624325189184721223 , \"apiVersion\" : \"v2\" }, ... ], \"apiVersion\" : \"v2\" , \"statusCode\" : 200 }","title":"AutoEvent"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#set-up-the-modbus-rtu-device","text":"This section describes how to connect the Modbus RTU device. We use Ubuntu OS and a Modbus RTU device for this example. Modbus RTU device: http://www.icpdas.com/root/product/solutions/remote_io/rs-485/i-7000_m-7000/i-7055.html User manual: http://ftp.icpdas.com/pub/cd/8000cd/napdos/7000/manual/7000dio.pdf","title":"Set up the Modbus RTU Device"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#connect-the-device","text":"Connect the device to your machine(laptop or gateway,etc.) via RS485/USB adaptor and power on. Execute a command on the machine, and you can find a message like the following: $ dmesg | grep tty ... ... [18006.167625] usb 1-1: FTDI USB Serial Device converter now attached to ttyUSB0 It shows the USB attach to ttyUSB0, then you can check whether the device path exists: $ ls /dev/ttyUSB0 /dev/ttyUSB0","title":"Connect the device"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#change-the-owner-of-the-device","text":"For security reason, the EdgeX set up the user permission as below: device-modbus : ... user : 2002:2001 # UID:GID So we need to change the owner for the specified group by the following command: sudo chown :2001 /dev/ttyUSB0 # Or change the permissions for multiple files sudo chown :2001 /dev/tty* Note Since the owner will reset after the system reboot, we can add this script to the startup script. For Raspberry Pi as example, add script to /etc/rc.local , then the Pi will run this script at bootup.","title":"Change the Owner of the Device"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#mont-the-device-path-to-the-docker-container","text":"Modify the docker-compose.yml file to mount the device path to the device-modbus, and here are two ways to mount the device path: Using devices : device-modbus : ... devices : - /dev/ttyUSB0 Or using volumes and device_cgroup_rules : device-modbus : ... volumes : ... - /dev:/dev device_cgroup_rules : - 'c 188:* rw' c: character device 188: device major number(188=USB) *: device minor number rw: read/write","title":"Mont the Device Path to the Docker Container"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#deploy-the-edgex","text":"$ docker-compose up -d","title":"Deploy the EdgeX"},{"location":"examples/Ch-ExamplesAddingModbusDevice/#add-device-to-edgex","text":"Create the device profile according to the register table $ nano modbus.rtu.demo.profile.yml name : \"Modbus-RTU-IO-Module\" manufacturer : \"icpdas\" model : \"M-7055\" labels : - \"Modbus RTU\" - \"IO Module\" description : \"This IO module offers 8 isolated channels for digital input and 8 isolated channels for digital output.\" deviceResources : - name : \"DO0\" isHidden : true description : \"On/Off , 0-OFF 1-ON\" attributes : { primaryTable : \"COILS\" , startingAddress : 0 } properties : valueType : \"Bool\" readWrite : \"RW\" - name : \"DO1\" isHidden : true description : \"On/Off , 0-OFF 1-ON\" attributes : { primaryTable : \"COILS\" , startingAddress : 1 } properties : valueType : \"Bool\" readWrite : \"RW\" - name : \"DO2\" isHidden : true description : \"On/Off , 0-OFF 1-ON\" attributes : { primaryTable : \"COILS\" , startingAddress : 2 } properties : valueType : \"Bool\" readWrite : \"RW\" deviceCommands : - name : \"DO\" readWrite : \"RW\" isHidden : false resourceOperations : - { deviceResource : \"DO0\" } - { deviceResource : \"DO1\" } - { deviceResource : \"DO2\" } Upload the device profile $ curl http://localhost:59881/api/v2/deviceprofile/uploadfile \\ -F \"file=@modbus.rtu.demo.profile.yml\" Create the device entity to the EdgeX. You can find the Modbus RTU setting on the device or the user manual. $ curl h tt p : //localhost:59881/api/v2/device -H \"Content-Type:application/json\" -X POST \\ - d ' [ { \"apiVersion\" : \"v2\" , \"device\" : { \"name\" : \"Modbus-RTU-IO-Module\" , \"description\" : \"The device can be used to monitor the status of the digital input and digital output channels.\" , \"labels\" :[ \"IO Module\" , \"Modbus RTU\" ], \"serviceName\" : \"device-modbus\" , \"profileName\" : \"Ethernet-Temperature-Sensor\" , \"protocols\" :{ \"modbus-tcp\" :{ \"Address\" : \"/dev/ttyUSB0\" , \"BaudRate\" : \"19200\" , \"DataBits\" : \"8\" , \"StopBits\" : \"1\" , \"Parity\" : \"N\" , \"UnitID\" : \"1\" , \"Timeout\" : \"5\" , \"IdleTimeout\" : \"5\" } }, \"adminState\" : \"UNLOCKED\" , \"operatingState\" : \"UP\" } } ] ' Test the GET or SET command","title":"Add device to EdgeX"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/","text":"SNMP EdgeX - Ireland Release Overview In this example, you add a new Patlite Signal Tower which communicates via SNMP. This example demonstrates how to connect a device through the SNMP Device Service. Patlite Signal Tower, model NHL-FB2 Setup Hardware needed In order to exercise this example, you will need the following hardware A computer able to run EdgeX Foundry A Patlite Signal Tower (NHL-FB2 model) Both the computer and Patlite must be connected to the same ethernet network Software needed In addition to the hardware, you will need the following software Docker Docker Compose EdgeX Foundry V2 (Ireland release) curl to run REST commands (you can also use a tool like Postman) If you have not already done so, proceed to Getting Started using Docker for how to get these tools and run EdgeX Foundry. Add the SNMP Device Service to your docker-compose.yml The EdgeX docker-compose.yml file used to run EdgeX must include the SNMP device service for this example. You can either: download and use the docker-compose.yml file provided with this example or use the EdgeX Compose Builder tool to create your own custom docker-compose.yml file adding device-snmp. See Getting Started using Docker if you need assistance running EdgeX once you have your Docker Compose file. Add the SNMP Device Profile and Device SNMP devices, like the Patlite Signal Tower, provide a set of managed objects to get and set property information on the associated device. Each managed object has an address call an object identifier (or OID) that you use to interact with the SNMP device's managed object. You use the OID to query the state of the device or to set properties on the device. In the case of the Patlite, there are managed object for the colored lights and the buzzer of the device. You can read the current state of a colored light (get) or turn the light on (set) by making a call to the proper OIDs for the associated managed object. For example, on the NH series signal towers used in this example, a \"get\" call to the 1.3.6.1.4.1.20440.4.1.5.1.2.1.4.1 OID returns the current state of the Red signal light. A return value of 1 would signal the light is off. A return value of 2 says the light is on. A return value of 3 says the light is flashing. Read this SNMP tutorial to learn more about the basics of the SNMP protocol. See the Patlite NH Series User's Manual for more information on the SNMP OIDs and function calls and parameters needed for some requests. Add the Patlite Device Profile A device profile has been created for you to get and set the signal tower's three colored lights and to get and set the buzzer. The patlite-snmp device profile defines three device resources for each of the lights and the buzzer. Current State, a read request device resource to get the current state of the requested light or buzzer Control State, a write request device resource to set the current state of the light or buzzer Timer, a write request device resource used in combination with the control state to set the state after the number of seconds provided by the timer resource Note that the attributes of each device resource specify the SNMP OID that the device service will use to make a request of the signal tower. For example, the device resource YAML below (taken from the profile) provides the means to get the current Red light state. Note that a specific OID is provided that is unique to the RED light, current state property. - name : \"RedLightCurrentState\" isHidden : false description : \"red light current state\" attributes : { oid : \"1.3.6.1.4.1.20440.4.1.5.1.2.1.4.1\" , community : \"private\" } properties : valueType : \"Int32\" readWrite : \"R\" defaultValue : \"1\" Below is the device resource definitions for the Red light control state and timer. Again, unique OIDs are provided as attributes for each property. - name : \"RedLightControlState\" isHidden : true description : \"red light state\" attributes : { oid : \"1.3.6.1.4.1.20440.4.1.5.1.2.1.2.1\" , community : \"private\" } properties : valueType : \"Int32\" readWrite : \"W\" defaultValue : \"1\" - name : \"RedLightTimer\" isHidden : true description : \"red light timer\" attributes : { oid : \"1.3.6.1.4.1.20440.4.1.5.1.2.1.3.1\" , community : \"private\" } properties : valueType : \"Int32\" readWrite : \"W\" defaultValue : \"1\" In order to set the Red light on, one would need to send an SNMP request to set OID 1.3.6.1.4.1.20440.4.1.5.1.2.1.2.1 to a value of 2 (on state) along with a number of seconds delay to the time at OID 1.3.6.1.4.1.20440.4.1.5.1.2.1.3.1 . Sending a zero value (0) to the timer would say you want to turn the light on immediately. Because setting a light or buzzer requires both of the control state and timer OIDs to be set together (simultaneously), the device profile contains deviceCommands to set the light and timer device resources (and therefore their SNMP property OIDs) in a single operation. Here is the device command to set the Red light. - name : \"RedLight\" readWrite : \"W\" isHidden : false resourceOperations : - { deviceResource : \"RedLightControlState\" } - { deviceResource : \"RedLightTimer\" } You will need to upload this profile into core metadata. Download the Patlite device profile to a convenient directory. Then, using the following curl command, request the profile be uploaded into core metadata. curl -X 'POST' 'http://localhost:59881/api/v2/deviceprofile/uploadfile' --form 'file=@\"/home/yourfilelocationhere/patlite-snmp.yml\"' Alert Note that the curl command above assumes that core metadata is available at localhost . Change localhost to the host address of your core metadata service. Also note that you will need to replace the /home/yourfilelocationhere path with the path where the profile resides. Add the Patlite Device With the Patlite device profile now in metadata, you can add the Patlite device in metadata. When adding the device, you typically need to provide the name, description, labels and admin/op states of the device when creating it. You will also need to associate the device to a device service (in this case the device-snmp device service). You will ned to associate the new device to a profile - the patlite profile just added in the step above. And you will need to provide the protocol information (such as the address and port of the device) to tell the device service where it can find the physical device. If you wish the device service to automatically get readings from the device, you will also need to provide AutoEvent properties when creating the device. The curl command to POST the new Patlite device (named patlite1 ) into metadata is provide below. You will need to change the protocol Address (currently 10.0.0.14 ) and Port (currently 161 ) to point to your Patlite on your network. In this request to add a new device, AutoEvents are setup to collect the current state of the 3 lights and buzzer every 10 seconds. Notice the reference to the current state device resources in setting up the AutoEvents. curl -X 'POST' 'http://localhost:59881/api/v2/device' -d '[{\"apiVersion\": \"v2\", \"device\": {\"name\": \"patlite1\",\"description\": \"patlite #1\",\"adminState\": \"UNLOCKED\",\"operatingState\": \"UP\",\"labels\": [\"patlite\"],\"serviceName\": \"device-snmp\",\"profileName\": \"patlite-snmp-profile\",\"protocols\": {\"TCP\": {\"Address\": \"10.0.0.14\",\"Port\": \"161\"}}, \"AutoEvents\":[{\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"RedLightCurrentState\"}, {\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"GreenLightCurrentState\"}, {\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"AmberLightCurrentState\"}, {\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"BuzzerCurrentState\"}]}}]' Info Rather than making a REST API call into metadata to add the device, you could alternately provide device configuration files that define the device. These device configuration files would then have to be provided to the service when it starts up. Since you did not create a new Docker image containing the device configuration and just used the existing SNMP device service Docker image, it was easier to make simple API calls to add the profile and device. However, this would mean the profile and device would need to be added each time metadata's database is cleaned out and reset. Test If the device service is up and running and the profile and device have been added correctly, you should now be able to interact with the Patlite via the core command service (and SNMP under the covers via the SNMP device service). Get the Current State To get the current state of a light (in the example below the Green light), make a curl request like the following of the command service. curl 'http://localhost:59882/api/v2/device/name/patlite1/GreenLightCurrentState' | json_pp Alert Note that the curl command above assumes that the core command service is available at localhost . Change the host address of your core command service if it is not available at localhost . The results should look something like that below. { \"statusCode\" : 200 , \"apiVersion\" : \"v2\" , \"event\" : { \"origin\" : 1632188382048586660 , \"deviceName\" : \"patlite1\" , \"sourceName\" : \"GreenLightCurrentState\" , \"id\" : \"1e2a7ba1-c273-46d1-b919-207aafbc60ba\" , \"profileName\" : \"patlite-snmp-profile\" , \"apiVersion\" : \"v2\" , \"readings\" : [ { \"origin\" : 1632188382048586660 , \"resourceName\" : \"GreenLightCurrentState\" , \"deviceName\" : \"patlite1\" , \"id\" : \"a41ac1cf-703b-4572-bdef-8487e9a7100e\" , \"valueType\" : \"Int32\" , \"value\" : \"1\" , \"profileName\" : \"patlite-snmp-profile\" } ] } } Info Note the value will be one of 4 numbers indicating the current state of the light Value Description 1 Off 2 On - solid and not flashing 3 Flashing on 4 Flashing quickly on Set a light or buzzer on To turn a signal tower light or the buzzer on, you can issue a PUT device command via the core command service. The example below turns on the Green light. curl --location --request PUT 'http://localhost:59882/api/v2/device/name/patlite1/GreenLight' --header 'cont: application/json' --data-raw '{\"GreenLightControlState\":\"2\",\"GreenLightTimer\":\"0\"}' This command sets the light on (solid versus flashing) immediate (as denoted by the GreenLightTimer parameter is set to 0). The timer value is the number of seconds delay in making the request to the light or buzzer. Again, the control state can be set to one of four values as listed in the table above. Alert Again note that the curl command above assumes that the core command service is available at localhost . Change the host address of your core command service if it is not available at localhost . Observations Did you notice that EdgeX obfuscates almost all information about SNMP, and managed objects and OIDs? The power of EdgeX is to abstract away protocol differences so that to a user, getting data from a device or setting properties on a device such as this Patlite signal tower is as easy as making simple REST calls into the command service. The only place that protocol information is really seen is in the device profile (where the attributes specify the SNMP OIDs). Of course, the device service must be coded to deal with the protocol specifics and it must know how to translate the simple command REST calls into protocol specific requests of the device. But even device service creation is made easier with the use of the SDKs which provide much of the boilerplate code found in almost every device service regardless of the underlying device protocol.","title":"SNMP"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#snmp","text":"EdgeX - Ireland Release","title":"SNMP"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#overview","text":"In this example, you add a new Patlite Signal Tower which communicates via SNMP. This example demonstrates how to connect a device through the SNMP Device Service. Patlite Signal Tower, model NHL-FB2","title":"Overview"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#setup","text":"","title":"Setup"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#hardware-needed","text":"In order to exercise this example, you will need the following hardware A computer able to run EdgeX Foundry A Patlite Signal Tower (NHL-FB2 model) Both the computer and Patlite must be connected to the same ethernet network","title":"Hardware needed"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#software-needed","text":"In addition to the hardware, you will need the following software Docker Docker Compose EdgeX Foundry V2 (Ireland release) curl to run REST commands (you can also use a tool like Postman) If you have not already done so, proceed to Getting Started using Docker for how to get these tools and run EdgeX Foundry.","title":"Software needed"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#add-the-snmp-device-service-to-your-docker-composeyml","text":"The EdgeX docker-compose.yml file used to run EdgeX must include the SNMP device service for this example. You can either: download and use the docker-compose.yml file provided with this example or use the EdgeX Compose Builder tool to create your own custom docker-compose.yml file adding device-snmp. See Getting Started using Docker if you need assistance running EdgeX once you have your Docker Compose file.","title":"Add the SNMP Device Service to your docker-compose.yml"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#add-the-snmp-device-profile-and-device","text":"SNMP devices, like the Patlite Signal Tower, provide a set of managed objects to get and set property information on the associated device. Each managed object has an address call an object identifier (or OID) that you use to interact with the SNMP device's managed object. You use the OID to query the state of the device or to set properties on the device. In the case of the Patlite, there are managed object for the colored lights and the buzzer of the device. You can read the current state of a colored light (get) or turn the light on (set) by making a call to the proper OIDs for the associated managed object. For example, on the NH series signal towers used in this example, a \"get\" call to the 1.3.6.1.4.1.20440.4.1.5.1.2.1.4.1 OID returns the current state of the Red signal light. A return value of 1 would signal the light is off. A return value of 2 says the light is on. A return value of 3 says the light is flashing. Read this SNMP tutorial to learn more about the basics of the SNMP protocol. See the Patlite NH Series User's Manual for more information on the SNMP OIDs and function calls and parameters needed for some requests.","title":"Add the SNMP Device Profile and Device"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#add-the-patlite-device-profile","text":"A device profile has been created for you to get and set the signal tower's three colored lights and to get and set the buzzer. The patlite-snmp device profile defines three device resources for each of the lights and the buzzer. Current State, a read request device resource to get the current state of the requested light or buzzer Control State, a write request device resource to set the current state of the light or buzzer Timer, a write request device resource used in combination with the control state to set the state after the number of seconds provided by the timer resource Note that the attributes of each device resource specify the SNMP OID that the device service will use to make a request of the signal tower. For example, the device resource YAML below (taken from the profile) provides the means to get the current Red light state. Note that a specific OID is provided that is unique to the RED light, current state property. - name : \"RedLightCurrentState\" isHidden : false description : \"red light current state\" attributes : { oid : \"1.3.6.1.4.1.20440.4.1.5.1.2.1.4.1\" , community : \"private\" } properties : valueType : \"Int32\" readWrite : \"R\" defaultValue : \"1\" Below is the device resource definitions for the Red light control state and timer. Again, unique OIDs are provided as attributes for each property. - name : \"RedLightControlState\" isHidden : true description : \"red light state\" attributes : { oid : \"1.3.6.1.4.1.20440.4.1.5.1.2.1.2.1\" , community : \"private\" } properties : valueType : \"Int32\" readWrite : \"W\" defaultValue : \"1\" - name : \"RedLightTimer\" isHidden : true description : \"red light timer\" attributes : { oid : \"1.3.6.1.4.1.20440.4.1.5.1.2.1.3.1\" , community : \"private\" } properties : valueType : \"Int32\" readWrite : \"W\" defaultValue : \"1\" In order to set the Red light on, one would need to send an SNMP request to set OID 1.3.6.1.4.1.20440.4.1.5.1.2.1.2.1 to a value of 2 (on state) along with a number of seconds delay to the time at OID 1.3.6.1.4.1.20440.4.1.5.1.2.1.3.1 . Sending a zero value (0) to the timer would say you want to turn the light on immediately. Because setting a light or buzzer requires both of the control state and timer OIDs to be set together (simultaneously), the device profile contains deviceCommands to set the light and timer device resources (and therefore their SNMP property OIDs) in a single operation. Here is the device command to set the Red light. - name : \"RedLight\" readWrite : \"W\" isHidden : false resourceOperations : - { deviceResource : \"RedLightControlState\" } - { deviceResource : \"RedLightTimer\" } You will need to upload this profile into core metadata. Download the Patlite device profile to a convenient directory. Then, using the following curl command, request the profile be uploaded into core metadata. curl -X 'POST' 'http://localhost:59881/api/v2/deviceprofile/uploadfile' --form 'file=@\"/home/yourfilelocationhere/patlite-snmp.yml\"' Alert Note that the curl command above assumes that core metadata is available at localhost . Change localhost to the host address of your core metadata service. Also note that you will need to replace the /home/yourfilelocationhere path with the path where the profile resides.","title":"Add the Patlite Device Profile"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#add-the-patlite-device","text":"With the Patlite device profile now in metadata, you can add the Patlite device in metadata. When adding the device, you typically need to provide the name, description, labels and admin/op states of the device when creating it. You will also need to associate the device to a device service (in this case the device-snmp device service). You will ned to associate the new device to a profile - the patlite profile just added in the step above. And you will need to provide the protocol information (such as the address and port of the device) to tell the device service where it can find the physical device. If you wish the device service to automatically get readings from the device, you will also need to provide AutoEvent properties when creating the device. The curl command to POST the new Patlite device (named patlite1 ) into metadata is provide below. You will need to change the protocol Address (currently 10.0.0.14 ) and Port (currently 161 ) to point to your Patlite on your network. In this request to add a new device, AutoEvents are setup to collect the current state of the 3 lights and buzzer every 10 seconds. Notice the reference to the current state device resources in setting up the AutoEvents. curl -X 'POST' 'http://localhost:59881/api/v2/device' -d '[{\"apiVersion\": \"v2\", \"device\": {\"name\": \"patlite1\",\"description\": \"patlite #1\",\"adminState\": \"UNLOCKED\",\"operatingState\": \"UP\",\"labels\": [\"patlite\"],\"serviceName\": \"device-snmp\",\"profileName\": \"patlite-snmp-profile\",\"protocols\": {\"TCP\": {\"Address\": \"10.0.0.14\",\"Port\": \"161\"}}, \"AutoEvents\":[{\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"RedLightCurrentState\"}, {\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"GreenLightCurrentState\"}, {\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"AmberLightCurrentState\"}, {\"Interval\":\"10s\",\"OnChange\":true,\"SourceName\":\"BuzzerCurrentState\"}]}}]' Info Rather than making a REST API call into metadata to add the device, you could alternately provide device configuration files that define the device. These device configuration files would then have to be provided to the service when it starts up. Since you did not create a new Docker image containing the device configuration and just used the existing SNMP device service Docker image, it was easier to make simple API calls to add the profile and device. However, this would mean the profile and device would need to be added each time metadata's database is cleaned out and reset.","title":"Add the Patlite Device"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#test","text":"If the device service is up and running and the profile and device have been added correctly, you should now be able to interact with the Patlite via the core command service (and SNMP under the covers via the SNMP device service).","title":"Test"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#get-the-current-state","text":"To get the current state of a light (in the example below the Green light), make a curl request like the following of the command service. curl 'http://localhost:59882/api/v2/device/name/patlite1/GreenLightCurrentState' | json_pp Alert Note that the curl command above assumes that the core command service is available at localhost . Change the host address of your core command service if it is not available at localhost . The results should look something like that below. { \"statusCode\" : 200 , \"apiVersion\" : \"v2\" , \"event\" : { \"origin\" : 1632188382048586660 , \"deviceName\" : \"patlite1\" , \"sourceName\" : \"GreenLightCurrentState\" , \"id\" : \"1e2a7ba1-c273-46d1-b919-207aafbc60ba\" , \"profileName\" : \"patlite-snmp-profile\" , \"apiVersion\" : \"v2\" , \"readings\" : [ { \"origin\" : 1632188382048586660 , \"resourceName\" : \"GreenLightCurrentState\" , \"deviceName\" : \"patlite1\" , \"id\" : \"a41ac1cf-703b-4572-bdef-8487e9a7100e\" , \"valueType\" : \"Int32\" , \"value\" : \"1\" , \"profileName\" : \"patlite-snmp-profile\" } ] } } Info Note the value will be one of 4 numbers indicating the current state of the light Value Description 1 Off 2 On - solid and not flashing 3 Flashing on 4 Flashing quickly on","title":"Get the Current State"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#set-a-light-or-buzzer-on","text":"To turn a signal tower light or the buzzer on, you can issue a PUT device command via the core command service. The example below turns on the Green light. curl --location --request PUT 'http://localhost:59882/api/v2/device/name/patlite1/GreenLight' --header 'cont: application/json' --data-raw '{\"GreenLightControlState\":\"2\",\"GreenLightTimer\":\"0\"}' This command sets the light on (solid versus flashing) immediate (as denoted by the GreenLightTimer parameter is set to 0). The timer value is the number of seconds delay in making the request to the light or buzzer. Again, the control state can be set to one of four values as listed in the table above. Alert Again note that the curl command above assumes that the core command service is available at localhost . Change the host address of your core command service if it is not available at localhost .","title":"Set a light or buzzer on"},{"location":"examples/Ch-ExamplesAddingSNMPDevice/#observations","text":"Did you notice that EdgeX obfuscates almost all information about SNMP, and managed objects and OIDs? The power of EdgeX is to abstract away protocol differences so that to a user, getting data from a device or setting properties on a device such as this Patlite signal tower is as easy as making simple REST calls into the command service. The only place that protocol information is really seen is in the device profile (where the attributes specify the SNMP OIDs). Of course, the device service must be coded to deal with the protocol specifics and it must know how to translate the simple command REST calls into protocol specific requests of the device. But even device service creation is made easier with the use of the SDKs which provide much of the boilerplate code found in almost every device service regardless of the underlying device protocol.","title":"Observations"},{"location":"examples/Ch-ExamplesModbusdatatypeconversion/","text":"Modbus - Data Type Conversion In use cases where the device resource uses an integer data type with a float scale, precision can be lost following transformation. For example, a Modbus device stores the temperature and humidity in an Int16 data type with a float scale of 0.01. If the temperature is 26.53, the read value is 2653. However, following transformation, the value is 26. To avoid this scenario, the device resource data type must differ from the value descriptor data type. This is achieved using the optional rawType attribute in the device profile to define the binary data read from the Modbus device, and a valueType to indicate what data type the user wants to receive. If the rawType attribute exists, the device service parses the binary data according to the defined rawType , then casts the value according to the valueType defined in the properties of the device resources. The following extract from a device profile defines the rawType as Int16 and the valueType as Float32: EdgeX 2.0 For EdgeX 2.0 the device profile has many changes. Please see Device Profile section for more details. Example - Device Profile deviceResources : - name : \"humidity\" description : \"The response value is the result of the original value multiplied by 100.\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : \"1\" , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.01\" units : \"%RH\" - name : \"temperature\" description : \"The response value is the result of the original value multiplied by 100.\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : \"2\" , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.01\" units : \"degrees Celsius\" Read Command A Read command is executed as follows: The device service executes the Read command to read binary data The binary reading data is parsed as an Int16 data type The integer value is cast to a Float32 value Write Command A Write command is executed as follows: The device service cast the requested Float32 value to an integer value The integer value is converted to binary data The device service executes the Write command When to Transform Data You generally need to transform data when scaling readings between a 16-bit integer and a float value. The following limitations apply: rawType supports only Int16 and Uint16 data types The corresponding valueType must be Float32 or Float64 If an unsupported data type is defined for the rawType attribute, the device service throws an exception similar to the following: Read command failed. Cmd:temperature err:the raw type Int32 is not supported Supported Transformations The supported transformations are as follows: From rawType To valueType Int16 Float32 Int16 Float64 Uint16 Float32 Uint16 Float64","title":"Modbus - Data Type Conversion"},{"location":"examples/Ch-ExamplesModbusdatatypeconversion/#modbus-data-type-conversion","text":"In use cases where the device resource uses an integer data type with a float scale, precision can be lost following transformation. For example, a Modbus device stores the temperature and humidity in an Int16 data type with a float scale of 0.01. If the temperature is 26.53, the read value is 2653. However, following transformation, the value is 26. To avoid this scenario, the device resource data type must differ from the value descriptor data type. This is achieved using the optional rawType attribute in the device profile to define the binary data read from the Modbus device, and a valueType to indicate what data type the user wants to receive. If the rawType attribute exists, the device service parses the binary data according to the defined rawType , then casts the value according to the valueType defined in the properties of the device resources. The following extract from a device profile defines the rawType as Int16 and the valueType as Float32: EdgeX 2.0 For EdgeX 2.0 the device profile has many changes. Please see Device Profile section for more details. Example - Device Profile deviceResources : - name : \"humidity\" description : \"The response value is the result of the original value multiplied by 100.\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : \"1\" , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.01\" units : \"%RH\" - name : \"temperature\" description : \"The response value is the result of the original value multiplied by 100.\" attributes : { primaryTable : \"HOLDING_REGISTERS\" , startingAddress : \"2\" , rawType : \"Int16\" } properties : valueType : \"Float32\" readWrite : \"R\" scale : \"0.01\" units : \"degrees Celsius\"","title":"Modbus - Data Type Conversion"},{"location":"examples/Ch-ExamplesModbusdatatypeconversion/#read-command","text":"A Read command is executed as follows: The device service executes the Read command to read binary data The binary reading data is parsed as an Int16 data type The integer value is cast to a Float32 value","title":"Read Command"},{"location":"examples/Ch-ExamplesModbusdatatypeconversion/#write-command","text":"A Write command is executed as follows: The device service cast the requested Float32 value to an integer value The integer value is converted to binary data The device service executes the Write command","title":"Write Command"},{"location":"examples/Ch-ExamplesModbusdatatypeconversion/#when-to-transform-data","text":"You generally need to transform data when scaling readings between a 16-bit integer and a float value. The following limitations apply: rawType supports only Int16 and Uint16 data types The corresponding valueType must be Float32 or Float64 If an unsupported data type is defined for the rawType attribute, the device service throws an exception similar to the following: Read command failed. Cmd:temperature err:the raw type Int32 is not supported","title":"When to Transform Data"},{"location":"examples/Ch-ExamplesModbusdatatypeconversion/#supported-transformations","text":"The supported transformations are as follows: From rawType To valueType Int16 Float32 Int16 Float64 Uint16 Float32 Uint16 Float64","title":"Supported Transformations"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/","text":"Sending and Consuming Binary Data From EdgeX Device Services EdgeX - Ireland Release Overview In this example, we will demonstrate how to send EdgeX Events and Readings that contain arbitrary binary data. DeviceService Implementation Device Profile To indicate that a deviceResource represents a Binary type, the following format is used: deviceResources : - name : \"camera_snapshot\" isHidden : false description : \"snapshot from camera\" properties : valueType : \"Binary\" readWrite : \"R\" mediaType : \"image/jpeg\" deviceCommands : - name : \"OnvifSnapshot\" isHidden : false readWrite : \"R\" resourceOperations : - { deviceResource : \"camera_snapshot\" } Device Service Here is a snippet from a hypothetical Device Service's HandleReadCommands() method that produces an event that represents a JPEG image captured from a camera: if req . DeviceResourceName == \"camera_snapshot\" { data , err := cameraClient . GetSnapshot () // returns ([]byte, error) check ( err ) cv , err := sdkModels . NewCommandValue ( reqs [ i ]. DeviceResourceName , common . ValueTypeBinary , data ) check ( err ) responses [ i ] = cv } Calling Device Service Command Querying core-metadata for the Device's Commands and DeviceName provides the following as the URL to request a reading from the snapshot command: http://localhost:59990/api/v2/device/name/camera-device/OnvifSnapshot Unlike with non-binary Events, making a request to this URL will return an event in CBOR representation. CBOR is a representation of binary data loosely based off of the JSON data model. This Event will not be human-readable. Parsing CBOR Encoded Events To access the data enclosed in these Events and Readings, they will first need to be decoded from CBOR. The following is a simple Go program that reads in the CBOR response from a file containing the response from the previous HTTP request. The Go library recommended for parsing these events can be found at https://github.com/fxamacker/cbor/ package main import ( \"io/ioutil\" \"github.com/edgexfoundry/go-mod-core-contracts/v2/dtos/requests\" \"github.com/fxamacker/cbor/v2\" ) func check ( e error ) { if e != nil { panic ( e ) } } func main () { // Read in our cbor data fileBytes , err := ioutil . ReadFile ( \"/Users/johndoe/Desktop/image.cbor\" ) check ( err ) // Decode into an EdgeX Event eventRequest := & requests . AddEventRequest {} err = cbor . Unmarshal ( fileBytes , eventRequest ) check ( err ) // Grab binary data and write to a file imgBytes := eventRequest . Event . Readings [ 0 ]. BinaryValue ioutil . WriteFile ( \"/Users/johndoe/Desktop/image.jpeg\" , imgBytes , 0644 ) } In the code above, the CBOR data is read into a byte array , an EdgeX Event struct is created, and cbor.Unmarshal parses the CBOR-encoded data and stores the result in the Event struct. Finally, the binary payload is written to a file from the BinaryValue field of the Reading. This method would work as well for decoding Events off the EdgeX message bus. Encoding Arbitrary Structures in Events The Device SDK's NewCommandValue() function above only accepts a byte slice as binary data. Any arbitrary Go structure can be encoded in a binary reading by first encoding the structure into a byte slice using CBOR. The following illustrates this method: // DeviceService HandleReadCommands() code: foo := struct { X int Y int Z int Bar string } { X : 7 , Y : 3 , Z : 100 , Bar : \"Hello world!\" , } data , err := cbor . Marshal ( & foo ) check ( err ) cv , err := sdkModels . NewCommandValue ( reqs [ i ]. DeviceResourceName , common . ValueTypeBinary , data ) responses [ i ] = cv This code takes the anonymous struct with fields X, Y, Z, and Bar (of different types) and serializes it into a byte slice using the same cbor library, and passing the output to NewCommandValue() . When consuming these events, another level of decoding will need to take place to get the structure out of the binary payload. func main () { // Read in our cbor data fileBytes , err := ioutil . ReadFile ( \"/Users/johndoe/Desktop/foo.cbor\" ) check ( err ) // Decode into an EdgeX Event eventRequest := & requests . AddEventRequest {} err = cbor . Unmarshal ( fileBytes , eventRequest ) check ( err ) // Decode into arbitrary type foo := struct { X int Y int Z int Bar string }{} err = cbor . Unmarshal ( eventRequest . Event . Readings [ 0 ]. BinaryValue , & foo ) check ( err ) fmt . Println ( foo ) } This code takes a command response in the same format as the previous example, but uses the cbor library to decode the CBOR data inside the EdgeX Reading's BinaryValue field. Using this approach, an Event can be sent containing data containing an arbitrary, flexible structure. Use cases could be a Reading containing multiple images, a variable length list of integer read-outs, etc.","title":"Sending and Consuming Binary Data From EdgeX Device Services"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#sending-and-consuming-binary-data-from-edgex-device-services","text":"EdgeX - Ireland Release","title":"Sending and Consuming Binary Data From EdgeX Device Services"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#overview","text":"In this example, we will demonstrate how to send EdgeX Events and Readings that contain arbitrary binary data.","title":"Overview"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#deviceservice-implementation","text":"","title":"DeviceService Implementation"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#device-profile","text":"To indicate that a deviceResource represents a Binary type, the following format is used: deviceResources : - name : \"camera_snapshot\" isHidden : false description : \"snapshot from camera\" properties : valueType : \"Binary\" readWrite : \"R\" mediaType : \"image/jpeg\" deviceCommands : - name : \"OnvifSnapshot\" isHidden : false readWrite : \"R\" resourceOperations : - { deviceResource : \"camera_snapshot\" }","title":"Device Profile"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#device-service","text":"Here is a snippet from a hypothetical Device Service's HandleReadCommands() method that produces an event that represents a JPEG image captured from a camera: if req . DeviceResourceName == \"camera_snapshot\" { data , err := cameraClient . GetSnapshot () // returns ([]byte, error) check ( err ) cv , err := sdkModels . NewCommandValue ( reqs [ i ]. DeviceResourceName , common . ValueTypeBinary , data ) check ( err ) responses [ i ] = cv }","title":"Device Service"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#calling-device-service-command","text":"Querying core-metadata for the Device's Commands and DeviceName provides the following as the URL to request a reading from the snapshot command: http://localhost:59990/api/v2/device/name/camera-device/OnvifSnapshot Unlike with non-binary Events, making a request to this URL will return an event in CBOR representation. CBOR is a representation of binary data loosely based off of the JSON data model. This Event will not be human-readable.","title":"Calling Device Service Command"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#parsing-cbor-encoded-events","text":"To access the data enclosed in these Events and Readings, they will first need to be decoded from CBOR. The following is a simple Go program that reads in the CBOR response from a file containing the response from the previous HTTP request. The Go library recommended for parsing these events can be found at https://github.com/fxamacker/cbor/ package main import ( \"io/ioutil\" \"github.com/edgexfoundry/go-mod-core-contracts/v2/dtos/requests\" \"github.com/fxamacker/cbor/v2\" ) func check ( e error ) { if e != nil { panic ( e ) } } func main () { // Read in our cbor data fileBytes , err := ioutil . ReadFile ( \"/Users/johndoe/Desktop/image.cbor\" ) check ( err ) // Decode into an EdgeX Event eventRequest := & requests . AddEventRequest {} err = cbor . Unmarshal ( fileBytes , eventRequest ) check ( err ) // Grab binary data and write to a file imgBytes := eventRequest . Event . Readings [ 0 ]. BinaryValue ioutil . WriteFile ( \"/Users/johndoe/Desktop/image.jpeg\" , imgBytes , 0644 ) } In the code above, the CBOR data is read into a byte array , an EdgeX Event struct is created, and cbor.Unmarshal parses the CBOR-encoded data and stores the result in the Event struct. Finally, the binary payload is written to a file from the BinaryValue field of the Reading. This method would work as well for decoding Events off the EdgeX message bus.","title":"Parsing CBOR Encoded Events"},{"location":"examples/Ch-ExamplesSendingAndConsumingBinary/#encoding-arbitrary-structures-in-events","text":"The Device SDK's NewCommandValue() function above only accepts a byte slice as binary data. Any arbitrary Go structure can be encoded in a binary reading by first encoding the structure into a byte slice using CBOR. The following illustrates this method: // DeviceService HandleReadCommands() code: foo := struct { X int Y int Z int Bar string } { X : 7 , Y : 3 , Z : 100 , Bar : \"Hello world!\" , } data , err := cbor . Marshal ( & foo ) check ( err ) cv , err := sdkModels . NewCommandValue ( reqs [ i ]. DeviceResourceName , common . ValueTypeBinary , data ) responses [ i ] = cv This code takes the anonymous struct with fields X, Y, Z, and Bar (of different types) and serializes it into a byte slice using the same cbor library, and passing the output to NewCommandValue() . When consuming these events, another level of decoding will need to take place to get the structure out of the binary payload. func main () { // Read in our cbor data fileBytes , err := ioutil . ReadFile ( \"/Users/johndoe/Desktop/foo.cbor\" ) check ( err ) // Decode into an EdgeX Event eventRequest := & requests . AddEventRequest {} err = cbor . Unmarshal ( fileBytes , eventRequest ) check ( err ) // Decode into arbitrary type foo := struct { X int Y int Z int Bar string }{} err = cbor . Unmarshal ( eventRequest . Event . Readings [ 0 ]. BinaryValue , & foo ) check ( err ) fmt . Println ( foo ) } This code takes a command response in the same format as the previous example, but uses the cbor library to decode the CBOR data inside the EdgeX Reading's BinaryValue field. Using this approach, an Event can be sent containing data containing an arbitrary, flexible structure. Use cases could be a Reading containing multiple images, a variable length list of integer read-outs, etc.","title":"Encoding Arbitrary Structures in Events"},{"location":"examples/Ch-ExamplesVirtualDeviceService/","text":"Using the Virtual Device Service Overview The Virtual Device Service GO can simulate different kinds of devices to generate Events and Readings to the Core Data Micro Service. Furthermore, users can send commands and get responses through the Command and Control Micro Service. The Virtual Device Service allows you to execute functional or performance tests without any real devices. This version of the Virtual Device Service is implemented based on Device SDK GO , and uses ql (an embedded SQL database engine) to simulate virtual resources. Introduction For information on the virtual device service see virtual device under the Microservices tab. Working with the Virtual Device Service Running the Virtual Device Service Container The virtual device service depends on the EdgeX core services. By default, the virtual device service is part of the EdgeX community provided Docker Compose files. If you use one of the community provide Compose files , you can pull and run EdgeX inclusive of the virtual device service without having to make any changes. Running the Virtual Device Service Natively (in development mode) If you're going to download the source code and run the virtual device service in development mode, make sure that the EdgeX core service containers are up before starting the virtual device service. See how to work with EdgeX in a hybrid environment in order to run the virtual device service outside of containers. This same file will instruct you on how to get and run the virtual device service code . GET command example The virtual device service is configured to send simulated data to core data every few seconds (from 10-30 seconds depending on device - see the device configuration file for AutoEvent details). You can exercise the GET request on the command service to see the generated value produced by any of the virtual device's simulated devices. Use the curl command below to exercise the virtual device service API (via core command service). curl -X GET localhost:59882/api/v2/device/name/Random-Integer-Device/Int8 ` Warning The example above assumes your core command service is available on localhost at the default service port of 59882. Also, you must replace your device name and command name in the example above with your virtual device service's identifiers. If you are not sure of the identifiers to use, query the command service for the full list of commands and devices at http://localhost:59882/api/v2/device/all . The virtual device should respond (via the core command service) with event/reading JSON similar to that below. { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"event\" : { \"apiVersion\" : \"v2\" , \"id\" : \"3beb5b83-d923-4c8a-b949-c1708b6611c1\" , \"deviceName\" : \"Random-Integer-Device\" , \"profileName\" : \"Random-Integer-Device\" , \"sourceName\" : \"Int8\" , \"origin\" : 1626227770833093400 , \"readings\" : [ { \"id\" : \"baf42bc7-307a-4647-8876-4e84759fd2ba\" , \"origin\" : 1626227770833093400 , \"deviceName\" : \"Random-Integer-Device\" , \"resourceName\" : \"Int8\" , \"profileName\" : \"Random-Integer-Device\" , \"valueType\" : \"Int8\" , \"binaryValue\" : null , \"mediaType\" : \"\" , \"value\" : \"-5\" } ] } } PUT command example - Assign a value to a resource The virtual devices managed by the virtual device can also be actuated. The virtual device can be told to enable or disable random number generation. When disabled, the virtual device services can be told what value to respond with for all GET operations. When setting the fixed value, the value must be valid for the data type of the virtual device. For example, the minimum value of Int8 cannot be less than -128 and the maximum value cannot be greater than 127. Below is example actuation of one of the virtual devices. In this example, it sets the fixed GET return value to 123 and turns off random generation. curl -X PUT -d '{\"Int8\": \"123\", \"EnableRandomization_Int8\": \"false\"}' localhost:59882/api/v2/device/name/Random-Integer-Device/Int8 Note The value of the resource's EnableRandomization property is simultaneously updated to false when sending a put command to assign a specified value to the resource. Therefore, the need to set EnableRandomization_Int8 to false is not actually required in the call above Return the virtual device to randomly generating numbers with another PUT call. curl -X PUT -d '{\"EnableRandomization_Int8\": \"true\"}' localhost:59882/api/v2/device/name/Random-Integer-Device/Int8 Reference Architectural Diagram Sequence Diagram Virtual Resource Table Schema Column Type DEVICE_NAME STRING COMMAND_NAME STRING DEVICE_RESOURCE_NAME STRING ENABLE_RANDOMIZATION BOOL DATA_TYPE STRING VALUE STRING","title":"Using the Virtual Device Service"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#using-the-virtual-device-service","text":"","title":"Using the Virtual Device Service"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#overview","text":"The Virtual Device Service GO can simulate different kinds of devices to generate Events and Readings to the Core Data Micro Service. Furthermore, users can send commands and get responses through the Command and Control Micro Service. The Virtual Device Service allows you to execute functional or performance tests without any real devices. This version of the Virtual Device Service is implemented based on Device SDK GO , and uses ql (an embedded SQL database engine) to simulate virtual resources.","title":"Overview"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#introduction","text":"For information on the virtual device service see virtual device under the Microservices tab.","title":"Introduction"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#working-with-the-virtual-device-service","text":"","title":"Working with the Virtual Device Service"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#running-the-virtual-device-service-container","text":"The virtual device service depends on the EdgeX core services. By default, the virtual device service is part of the EdgeX community provided Docker Compose files. If you use one of the community provide Compose files , you can pull and run EdgeX inclusive of the virtual device service without having to make any changes.","title":"Running the Virtual Device Service Container"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#running-the-virtual-device-service-natively-in-development-mode","text":"If you're going to download the source code and run the virtual device service in development mode, make sure that the EdgeX core service containers are up before starting the virtual device service. See how to work with EdgeX in a hybrid environment in order to run the virtual device service outside of containers. This same file will instruct you on how to get and run the virtual device service code .","title":"Running the Virtual Device Service Natively (in development mode)"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#get-command-example","text":"The virtual device service is configured to send simulated data to core data every few seconds (from 10-30 seconds depending on device - see the device configuration file for AutoEvent details). You can exercise the GET request on the command service to see the generated value produced by any of the virtual device's simulated devices. Use the curl command below to exercise the virtual device service API (via core command service). curl -X GET localhost:59882/api/v2/device/name/Random-Integer-Device/Int8 ` Warning The example above assumes your core command service is available on localhost at the default service port of 59882. Also, you must replace your device name and command name in the example above with your virtual device service's identifiers. If you are not sure of the identifiers to use, query the command service for the full list of commands and devices at http://localhost:59882/api/v2/device/all . The virtual device should respond (via the core command service) with event/reading JSON similar to that below. { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"event\" : { \"apiVersion\" : \"v2\" , \"id\" : \"3beb5b83-d923-4c8a-b949-c1708b6611c1\" , \"deviceName\" : \"Random-Integer-Device\" , \"profileName\" : \"Random-Integer-Device\" , \"sourceName\" : \"Int8\" , \"origin\" : 1626227770833093400 , \"readings\" : [ { \"id\" : \"baf42bc7-307a-4647-8876-4e84759fd2ba\" , \"origin\" : 1626227770833093400 , \"deviceName\" : \"Random-Integer-Device\" , \"resourceName\" : \"Int8\" , \"profileName\" : \"Random-Integer-Device\" , \"valueType\" : \"Int8\" , \"binaryValue\" : null , \"mediaType\" : \"\" , \"value\" : \"-5\" } ] } }","title":"GET command example"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#put-command-example-assign-a-value-to-a-resource","text":"The virtual devices managed by the virtual device can also be actuated. The virtual device can be told to enable or disable random number generation. When disabled, the virtual device services can be told what value to respond with for all GET operations. When setting the fixed value, the value must be valid for the data type of the virtual device. For example, the minimum value of Int8 cannot be less than -128 and the maximum value cannot be greater than 127. Below is example actuation of one of the virtual devices. In this example, it sets the fixed GET return value to 123 and turns off random generation. curl -X PUT -d '{\"Int8\": \"123\", \"EnableRandomization_Int8\": \"false\"}' localhost:59882/api/v2/device/name/Random-Integer-Device/Int8 Note The value of the resource's EnableRandomization property is simultaneously updated to false when sending a put command to assign a specified value to the resource. Therefore, the need to set EnableRandomization_Int8 to false is not actually required in the call above Return the virtual device to randomly generating numbers with another PUT call. curl -X PUT -d '{\"EnableRandomization_Int8\": \"true\"}' localhost:59882/api/v2/device/name/Random-Integer-Device/Int8","title":"PUT command example - Assign a value to a resource"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#reference","text":"","title":"Reference"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#architectural-diagram","text":"","title":"Architectural Diagram"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#sequence-diagram","text":"","title":"Sequence Diagram"},{"location":"examples/Ch-ExamplesVirtualDeviceService/#virtual-resource-table-schema","text":"Column Type DEVICE_NAME STRING COMMAND_NAME STRING DEVICE_RESOURCE_NAME STRING ENABLE_RANDOMIZATION BOOL DATA_TYPE STRING VALUE STRING","title":"Virtual Resource Table Schema"},{"location":"general/ContainerNames/","text":"EdgeX Container Names The following table provides the list of the default EdgeX Docker image names to the Docker container name and Docker Compose names. EdgeX 2.0 For EdgeX 2.0 the EdgeX docker image names have been simplified and made consistent across all EdgeX services. Core Supporting Application & Analytics Device Security Miscellaneous Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/core-data edgex-core-data edgex-core-data data edgexfoundry/core-metadata edgex-core-metadata edgex-core-metadata metadata edgexfoundry/core-command edgex-core-command edgex-core-command command Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/support-notifications edgex-support-notifications edgex-support-notifications notifications edgexfoundry/support-scheduler edgex-support-scheduler edgex-support-scheduler scheduler Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/app-service-configurable edgex-app-rules-engine edgex-app-rules-engine app-service-rules edgexfoundry/app-service-configurable edgex-app-http-export edgex-app-http-export app-service-http-export edgexfoundry/app-service-configurable edgex-app-mqtt-export edgex-app-mqtt-export app-service-mqtt-export emqx/kuiper edgex-kuiper edgex-kuiper rulesengine Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/device-virtual edgex-device-virtual edgex-device-virtual device-virtual edgexfoundry/device-mqtt edgex-device-mqtt edgex-device-mqtt device-mqtt edgexfoundry/device-rest edgex-device-rest edgex-device-rest device-rest edgexfoundry/device-modbus edgex-device-modbus edgex-device-modbus device-modbus edgexfoundry/device-snmp edgex-device-snmp edgex-device-snmp device-snmp edgexfoundry/device-bacnet edgex-device-bacnet edgex-device-bacnet device-bacnet edgexfoundry/device-camera edgex-device-camera edgex-device-camera device-camera edgexfoundry/device-grove edgex-device-grove edgex-device-grove device-grove edgexfoundry/device-coap edgex-device-coap edgex-device-coap device-coap Docker image name Docker container name Docker network hostname Docker Compose service name vault edgex-vault edgex-vault vault postgress edgex-kong-db edgex-kong-db kong-db kong edgex-kong edgex-kong kong edgexfoundry/security-proxy-setup edgex-security-proxy-setup edgex-security-proxy-setup proxy-setup edgexfoundry/security-secretstore-setup edgex-security-secretstore-setup edgex-security-secretstore-setup secretstore-setup edgexfoundry/security-bootstrapper edgex-security-bootstrapper edgex-security-bootstrapper security-bootstrapper Docker image name Docker container name Docker network hostname Docker Compose service name consul edgex-core-consul edgex-core-consul consul redis edgex-redis edgex-redis database edgexfoundry/sys-mgmt-agent edgex-sys-mgmt-agent edgex-sys-mgmt-agent system","title":"EdgeX Container Names"},{"location":"general/ContainerNames/#edgex-container-names","text":"The following table provides the list of the default EdgeX Docker image names to the Docker container name and Docker Compose names. EdgeX 2.0 For EdgeX 2.0 the EdgeX docker image names have been simplified and made consistent across all EdgeX services. Core Supporting Application & Analytics Device Security Miscellaneous Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/core-data edgex-core-data edgex-core-data data edgexfoundry/core-metadata edgex-core-metadata edgex-core-metadata metadata edgexfoundry/core-command edgex-core-command edgex-core-command command Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/support-notifications edgex-support-notifications edgex-support-notifications notifications edgexfoundry/support-scheduler edgex-support-scheduler edgex-support-scheduler scheduler Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/app-service-configurable edgex-app-rules-engine edgex-app-rules-engine app-service-rules edgexfoundry/app-service-configurable edgex-app-http-export edgex-app-http-export app-service-http-export edgexfoundry/app-service-configurable edgex-app-mqtt-export edgex-app-mqtt-export app-service-mqtt-export emqx/kuiper edgex-kuiper edgex-kuiper rulesengine Docker image name Docker container name Docker network hostname Docker Compose service name edgexfoundry/device-virtual edgex-device-virtual edgex-device-virtual device-virtual edgexfoundry/device-mqtt edgex-device-mqtt edgex-device-mqtt device-mqtt edgexfoundry/device-rest edgex-device-rest edgex-device-rest device-rest edgexfoundry/device-modbus edgex-device-modbus edgex-device-modbus device-modbus edgexfoundry/device-snmp edgex-device-snmp edgex-device-snmp device-snmp edgexfoundry/device-bacnet edgex-device-bacnet edgex-device-bacnet device-bacnet edgexfoundry/device-camera edgex-device-camera edgex-device-camera device-camera edgexfoundry/device-grove edgex-device-grove edgex-device-grove device-grove edgexfoundry/device-coap edgex-device-coap edgex-device-coap device-coap Docker image name Docker container name Docker network hostname Docker Compose service name vault edgex-vault edgex-vault vault postgress edgex-kong-db edgex-kong-db kong-db kong edgex-kong edgex-kong kong edgexfoundry/security-proxy-setup edgex-security-proxy-setup edgex-security-proxy-setup proxy-setup edgexfoundry/security-secretstore-setup edgex-security-secretstore-setup edgex-security-secretstore-setup secretstore-setup edgexfoundry/security-bootstrapper edgex-security-bootstrapper edgex-security-bootstrapper security-bootstrapper Docker image name Docker container name Docker network hostname Docker Compose service name consul edgex-core-consul edgex-core-consul consul redis edgex-redis edgex-redis database edgexfoundry/sys-mgmt-agent edgex-sys-mgmt-agent edgex-sys-mgmt-agent system","title":"EdgeX Container Names"},{"location":"general/Definitions/","text":"Definitions The following glossary provides terms used in EdgeX Foundry. The definition are based on how EdgeX and its community use the term versus any strict technical or industry definition. Actuate To cause a machine or device to operate. In EdgeX terms, to command a device or sensor under management of EdgeX to do something (example: stop a motor) or to reconfigure itself (example: set a thermostat's cooling point). Brownfield and Greenfield Brownfield refers to older legacy equipment (nodes, devices, sensors) in an edge/IoT deployment, which typically uses older protocols. Greenfield refers to, typically, new equipment with modern protocols. CBOR An acronym for \"concise binary object representation.\" A binary data serialization format used by EdgeX to transport binary sensed data (like an image). The user can also choose to send all data via CBOR for efficiency purposes, but at the expense of having EdgeX convert the CBOR into another format whenever the data needs to be understood and inspected or to persist the data. Containerized EdgeX micro services and infrastructure (i.e. databases, registry, etc.) are built as executable programs, put into Docker images, and made available via Docker Hub (and Nexus repository for nightly builds). A service (or infrastructure element) that is available in Docker Hub (or Nexus) is said to be containerized. Docker images can be quickly downloaded and new Docker containers created from the images. Contributor/Developer If you want to change, add to or at least build the existing EdgeX code base, then you are a \"Developer\". \"Contributors\" are developers that further wish to contribute their code back into the EdgeX open source effort. Created time stamp The Created time stamp is the time the data was created in the database and is unchangeable. The Origin time stamp is the time the data is created on the device, device services, sensor, or object that collected the data before the data was sent to EdgeX Foundry and the database. Usually, the Origin and Created time stamps are the same, or very close to being the same. On occasion the sensor may be a long way from the gateway or even in a different time zone, and the Origin and Created time stamps may be quite different. If persistence is disable in core-data, the time stamp will default to 0. Device In EdgeX parlance, \"device\" is used to refer to a sensor, actuator, or IoT \"thing\". A sensor generally collects information from the physical world - like a temperature or vibration sensor. Actuators are machines that can be told to do something. Actuators move or otherwise control a mechanism or system - like a value on a pump. While there may be some technical differences, for the purposes of EdgeX documentation, device will refer to a sensor, actuator or \"thing\". Edge Analytics The terms edge or local analytics (the terms are used interchangeably and have the same meaning in this context) for the purposes of edge computing (and EdgeX), refers to an \u201canalytics\u201d service is that: - Receives and interprets the EdgeX sensor data to some degree; some analytics services are more sophisticated and able to provide more insights than others - Make determinations on what actions and actuations need to occur based on the insights it has achieved, thereby driving actuation requests to EdgeX associated devices or other services (like notifications) The analytics service could be some simple logic built into an app service, a rules engine package, or an agent of some artificial intelligence/machine learning system. From an EdgeX perspective, actionable intelligence generation is all the same. From an EdgeX perspective, edge analytics = seeing the edge data and be able to make requests to act on what is seen. While EdgeX provides a rules engine service as its reference implementation of local analytics, app services and its data preparation capability allow sensor data to be streamed to any analytics package. Because of EdgeX\u2019s micro service architecture and distributed nature, the analytics service would not necessarily have to run local to the devices / sensors. In other words, it would not have to run at the edge. App services could deliver the edge data to analytics living in the cloud. However, in these scenarios, the insight intelligence would not be considered local or edge in context. Because of latency concerns, data security and privacy needs, intermittent connectivity of edge systems, and other reasons, it is often vital for edge platforms to retain an analytic capability at the edge or local. Gateway An IoT gateway is a compute platform at the farthest ends of an edge or IoT network. It is the host or \u201cbox\u201d to which physical sensors and devices connect and that is, in turn, connected to the networks (wired or wirelessly) of the information technology realm. IoT or edge gateways are compute platforms that connect \u201cthings\u201d (sensors and devices) to IT networks and systems. Micro service In a micro service architecture, each component has its own process. This is in contrast to a monolithic architecture in which all components of the application run in the same process. Benefits of micro service architectures include: - Allow any one service to be replaced and upgraded more easily - Allow services to be programmed using different programming languages and underlying technical solutions (use the best technology for each specific service) - Ex: services written in C can communicate and work with services written in Go - This allows organizations building solutions to maximize available developer resources and some legacy code - Allow services to be distributed across host compute platforms - allowing better utilization of available compute resources - Allow for more scalable solutions by adding copies of services when needed Origin time stamp The Origin time stamp is the time the data is created on the device, device services, sensor, or object that collected the data before the data is sent to EdgeX Foundry and the database. The Created time stamp is the time the data was created in the database. Usually, the Origin and Created time stamps are the same or very close to the same. On occasion the sensor may be a long way from the gateway or even in a different time zone, and the Origin and Created time stamps may be quite different. Reference Implementation Default and example implementation(s) offered by the EdgeX community. Other implementations may be offered by 3rd parties or for specialization. Resource A piece of information or data available from a sensor or \"thing\". For example, a thermostat would have temperature and humidity resources. A resource has a name (ResourceName) to identify it (\"temperature\" or \"humidity\" in this example) and a value (the sensed data - like 72 degrees). A resource may also have additional properties or attributes associated with it. The data type of the value (e.g., integer, float, string, etc.) would be an example of a resource property. Rules Engine Rules engines are important to the IoT edge system. A rules engine is a software system that is connected to a collection of data (either database or data stream). The rules engine examines various elements of the data and monitors the data, and then triggers some action based on the results of the monitoring of the data it. A rules engine is a collection of \"If-Then\" conditional statements. The \"If\" informs the rules engine what data to look at and what ranges or values of data must match in order to trigger the \"Then\" part of the statement, which then informs the rules engine what action to take or what external resource to call on, when the data is a match to the \"If\" statement. Most rules engines can be dynamically programmed meaning that new \"If-Then\" statements or rules, can be provided while the engine is running. The rules are often defined by some type of rule language with simple syntax to enable non-Developers to provide the new rules. Rules engines are one of the simplest forms of \"edge analytics\" provided in IoT systems. Rules engines enable data picked up by IoT sensors to be monitored and acted upon (actuated). Typically, the actuation is accomplished on another IoT device or sensor. For example, a temperature sensor in an equipment enclosure may be monitored by a rules engine to detect when the temperature is getting too warm (or too cold) for safe or optimum operation of the equipment. The rules engine, upon detecting temperatures outside of the acceptable range, shuts off the equipment in the enclosure. Software Development Kit In EdgeX, a software development kit (or SDK) is a library or module to be incorporated into a new micro service. It provides a lot of the boilerplate code and scaffolding associated with the type of service being created. The SDK allows the developer to focus on the details of the service functionality and not have to worry about the mundane tasks associated with EdgeX services. South and North Side South Side: All IoT objects, within the physical realm, and the edge of the network that communicates directly with those devices, sensors, actuators, and other IoT objects, and collects the data from them, is known collectively as the \"south side.\" North Side: The cloud (or enterprise system) where data is collected, stored, aggregated, analyzed, and turned into information, and the part of the network that communicates with the cloud, is referred to as the \"north side\" of the network. EdgeX enables data to be sent \"north, \" \"south, \" or laterally as needed and as directed. \"Snappy\" / Ubuntu Core & Snaps A Linux-based Operating System provided by Ubuntu - formally called Ubuntu Core but often referred to as \"Snappy\". The packages are called 'snaps' and the tool for using them 'snapd', and works for phone, cloud, internet of things, and desktop computers. The \"Snap\" packages are self-contained and have no dependency on external stores. \"Snaps\" can be used to create command line tools, background services, and desktop applications. User If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\".","title":"Definitions"},{"location":"general/Definitions/#definitions","text":"The following glossary provides terms used in EdgeX Foundry. The definition are based on how EdgeX and its community use the term versus any strict technical or industry definition.","title":"Definitions"},{"location":"general/Definitions/#actuate","text":"To cause a machine or device to operate. In EdgeX terms, to command a device or sensor under management of EdgeX to do something (example: stop a motor) or to reconfigure itself (example: set a thermostat's cooling point).","title":"Actuate"},{"location":"general/Definitions/#brownfield-and-greenfield","text":"Brownfield refers to older legacy equipment (nodes, devices, sensors) in an edge/IoT deployment, which typically uses older protocols. Greenfield refers to, typically, new equipment with modern protocols.","title":"Brownfield and Greenfield"},{"location":"general/Definitions/#cbor","text":"An acronym for \"concise binary object representation.\" A binary data serialization format used by EdgeX to transport binary sensed data (like an image). The user can also choose to send all data via CBOR for efficiency purposes, but at the expense of having EdgeX convert the CBOR into another format whenever the data needs to be understood and inspected or to persist the data.","title":"CBOR"},{"location":"general/Definitions/#containerized","text":"EdgeX micro services and infrastructure (i.e. databases, registry, etc.) are built as executable programs, put into Docker images, and made available via Docker Hub (and Nexus repository for nightly builds). A service (or infrastructure element) that is available in Docker Hub (or Nexus) is said to be containerized. Docker images can be quickly downloaded and new Docker containers created from the images.","title":"Containerized"},{"location":"general/Definitions/#contributordeveloper","text":"If you want to change, add to or at least build the existing EdgeX code base, then you are a \"Developer\". \"Contributors\" are developers that further wish to contribute their code back into the EdgeX open source effort.","title":"Contributor/Developer"},{"location":"general/Definitions/#created-time-stamp","text":"The Created time stamp is the time the data was created in the database and is unchangeable. The Origin time stamp is the time the data is created on the device, device services, sensor, or object that collected the data before the data was sent to EdgeX Foundry and the database. Usually, the Origin and Created time stamps are the same, or very close to being the same. On occasion the sensor may be a long way from the gateway or even in a different time zone, and the Origin and Created time stamps may be quite different. If persistence is disable in core-data, the time stamp will default to 0.","title":"Created time stamp"},{"location":"general/Definitions/#device","text":"In EdgeX parlance, \"device\" is used to refer to a sensor, actuator, or IoT \"thing\". A sensor generally collects information from the physical world - like a temperature or vibration sensor. Actuators are machines that can be told to do something. Actuators move or otherwise control a mechanism or system - like a value on a pump. While there may be some technical differences, for the purposes of EdgeX documentation, device will refer to a sensor, actuator or \"thing\".","title":"Device"},{"location":"general/Definitions/#edge-analytics","text":"The terms edge or local analytics (the terms are used interchangeably and have the same meaning in this context) for the purposes of edge computing (and EdgeX), refers to an \u201canalytics\u201d service is that: - Receives and interprets the EdgeX sensor data to some degree; some analytics services are more sophisticated and able to provide more insights than others - Make determinations on what actions and actuations need to occur based on the insights it has achieved, thereby driving actuation requests to EdgeX associated devices or other services (like notifications) The analytics service could be some simple logic built into an app service, a rules engine package, or an agent of some artificial intelligence/machine learning system. From an EdgeX perspective, actionable intelligence generation is all the same. From an EdgeX perspective, edge analytics = seeing the edge data and be able to make requests to act on what is seen. While EdgeX provides a rules engine service as its reference implementation of local analytics, app services and its data preparation capability allow sensor data to be streamed to any analytics package. Because of EdgeX\u2019s micro service architecture and distributed nature, the analytics service would not necessarily have to run local to the devices / sensors. In other words, it would not have to run at the edge. App services could deliver the edge data to analytics living in the cloud. However, in these scenarios, the insight intelligence would not be considered local or edge in context. Because of latency concerns, data security and privacy needs, intermittent connectivity of edge systems, and other reasons, it is often vital for edge platforms to retain an analytic capability at the edge or local.","title":"Edge Analytics"},{"location":"general/Definitions/#gateway","text":"An IoT gateway is a compute platform at the farthest ends of an edge or IoT network. It is the host or \u201cbox\u201d to which physical sensors and devices connect and that is, in turn, connected to the networks (wired or wirelessly) of the information technology realm. IoT or edge gateways are compute platforms that connect \u201cthings\u201d (sensors and devices) to IT networks and systems.","title":"Gateway"},{"location":"general/Definitions/#micro-service","text":"In a micro service architecture, each component has its own process. This is in contrast to a monolithic architecture in which all components of the application run in the same process. Benefits of micro service architectures include: - Allow any one service to be replaced and upgraded more easily - Allow services to be programmed using different programming languages and underlying technical solutions (use the best technology for each specific service) - Ex: services written in C can communicate and work with services written in Go - This allows organizations building solutions to maximize available developer resources and some legacy code - Allow services to be distributed across host compute platforms - allowing better utilization of available compute resources - Allow for more scalable solutions by adding copies of services when needed","title":"Micro service"},{"location":"general/Definitions/#origin-time-stamp","text":"The Origin time stamp is the time the data is created on the device, device services, sensor, or object that collected the data before the data is sent to EdgeX Foundry and the database. The Created time stamp is the time the data was created in the database. Usually, the Origin and Created time stamps are the same or very close to the same. On occasion the sensor may be a long way from the gateway or even in a different time zone, and the Origin and Created time stamps may be quite different.","title":"Origin time stamp"},{"location":"general/Definitions/#reference-implementation","text":"Default and example implementation(s) offered by the EdgeX community. Other implementations may be offered by 3rd parties or for specialization.","title":"Reference Implementation"},{"location":"general/Definitions/#resource","text":"A piece of information or data available from a sensor or \"thing\". For example, a thermostat would have temperature and humidity resources. A resource has a name (ResourceName) to identify it (\"temperature\" or \"humidity\" in this example) and a value (the sensed data - like 72 degrees). A resource may also have additional properties or attributes associated with it. The data type of the value (e.g., integer, float, string, etc.) would be an example of a resource property.","title":"Resource"},{"location":"general/Definitions/#rules-engine","text":"Rules engines are important to the IoT edge system. A rules engine is a software system that is connected to a collection of data (either database or data stream). The rules engine examines various elements of the data and monitors the data, and then triggers some action based on the results of the monitoring of the data it. A rules engine is a collection of \"If-Then\" conditional statements. The \"If\" informs the rules engine what data to look at and what ranges or values of data must match in order to trigger the \"Then\" part of the statement, which then informs the rules engine what action to take or what external resource to call on, when the data is a match to the \"If\" statement. Most rules engines can be dynamically programmed meaning that new \"If-Then\" statements or rules, can be provided while the engine is running. The rules are often defined by some type of rule language with simple syntax to enable non-Developers to provide the new rules. Rules engines are one of the simplest forms of \"edge analytics\" provided in IoT systems. Rules engines enable data picked up by IoT sensors to be monitored and acted upon (actuated). Typically, the actuation is accomplished on another IoT device or sensor. For example, a temperature sensor in an equipment enclosure may be monitored by a rules engine to detect when the temperature is getting too warm (or too cold) for safe or optimum operation of the equipment. The rules engine, upon detecting temperatures outside of the acceptable range, shuts off the equipment in the enclosure.","title":"Rules Engine"},{"location":"general/Definitions/#software-development-kit","text":"In EdgeX, a software development kit (or SDK) is a library or module to be incorporated into a new micro service. It provides a lot of the boilerplate code and scaffolding associated with the type of service being created. The SDK allows the developer to focus on the details of the service functionality and not have to worry about the mundane tasks associated with EdgeX services.","title":"Software Development Kit"},{"location":"general/Definitions/#south-and-north-side","text":"South Side: All IoT objects, within the physical realm, and the edge of the network that communicates directly with those devices, sensors, actuators, and other IoT objects, and collects the data from them, is known collectively as the \"south side.\" North Side: The cloud (or enterprise system) where data is collected, stored, aggregated, analyzed, and turned into information, and the part of the network that communicates with the cloud, is referred to as the \"north side\" of the network. EdgeX enables data to be sent \"north, \" \"south, \" or laterally as needed and as directed.","title":"South and North Side"},{"location":"general/Definitions/#snappy-ubuntu-core-snaps","text":"A Linux-based Operating System provided by Ubuntu - formally called Ubuntu Core but often referred to as \"Snappy\". The packages are called 'snaps' and the tool for using them 'snapd', and works for phone, cloud, internet of things, and desktop computers. The \"Snap\" packages are self-contained and have no dependency on external stores. \"Snaps\" can be used to create command line tools, background services, and desktop applications.","title":"\"Snappy\" / Ubuntu Core & Snaps"},{"location":"general/Definitions/#user","text":"If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\".","title":"User"},{"location":"general/PlatformRequirements/","text":"Platform Requirements EdgeX Foundry is an operating system (OS)-agnostic and hardware (HW)-agnostic IoT edge platform. At this time the following platform minimums are recommended: Memory Storage Operating Systems Memory: minimum of 1 GB When considering memory for your EdgeX platform consider your use of database - Redis is the current default. Redis is an open source (BSD licensed), in-memory data structure store, used as a database and message broker in EdgeX. Redis is durable and uses persistence only for recovering state; the only data Redis operates on is in-memory. Redis uses a number of techniques to optimize memory utilization. Antirez and Redis Labs have written a number of articles on the underlying details (see list below). Those strategies has continued to evolve . When thinking about your system architecture, consider how long data will be living at the edge and consuming memory (physical or physical + virtual). Antirez Redis RAM Ramifications Redis IO Memory Optimization Hard drive space: minimum of 3 GB of space to run the EdgeX Foundry containers, but you may want more depending on how long sensor and device data is to be retained. Approximately 32GB of storage is minimally recommended to start. EdgeX Foundry has been run successfully on many systems, including, but not limited to the following systems Windows (ver 7 - 10) Ubuntu Desktop (ver 14-20) Ubuntu Server (ver 14-20) Ubuntu Core (ver 16-18) Mac OS X 10 Info EdgeX is agnostic with regards to hardware (x86 and ARM), but only release artifacts for x86 and ARM 64 systems. EdgeX has been successfully run on ARM 32 platforms but has required users to build their own executable from source. EdgeX does not officially support ARM 32.","title":"Platform Requirements"},{"location":"general/PlatformRequirements/#platform-requirements","text":"EdgeX Foundry is an operating system (OS)-agnostic and hardware (HW)-agnostic IoT edge platform. At this time the following platform minimums are recommended: Memory Storage Operating Systems Memory: minimum of 1 GB When considering memory for your EdgeX platform consider your use of database - Redis is the current default. Redis is an open source (BSD licensed), in-memory data structure store, used as a database and message broker in EdgeX. Redis is durable and uses persistence only for recovering state; the only data Redis operates on is in-memory. Redis uses a number of techniques to optimize memory utilization. Antirez and Redis Labs have written a number of articles on the underlying details (see list below). Those strategies has continued to evolve . When thinking about your system architecture, consider how long data will be living at the edge and consuming memory (physical or physical + virtual). Antirez Redis RAM Ramifications Redis IO Memory Optimization Hard drive space: minimum of 3 GB of space to run the EdgeX Foundry containers, but you may want more depending on how long sensor and device data is to be retained. Approximately 32GB of storage is minimally recommended to start. EdgeX Foundry has been run successfully on many systems, including, but not limited to the following systems Windows (ver 7 - 10) Ubuntu Desktop (ver 14-20) Ubuntu Server (ver 14-20) Ubuntu Core (ver 16-18) Mac OS X 10 Info EdgeX is agnostic with regards to hardware (x86 and ARM), but only release artifacts for x86 and ARM 64 systems. EdgeX has been successfully run on ARM 32 platforms but has required users to build their own executable from source. EdgeX does not officially support ARM 32.","title":"Platform Requirements"},{"location":"general/ServiceConfiguration/","text":"Service Configuration Each EdgeX micro service requires configuration (i.e. - a repository of initialization and operating values). The configuration is initially provided by a TOML file but a service can utilize the centralized configuration management provided by EdgeX for its configuration. See the Configuration and Registry documentation for more details about initialization of services and the use of the configuration service. Please refer to the EdgeX Foundry architectural decision record for details (and design decisions) behind the configuration in EdgeX. Please refer to the general Common Configuration documentation for configuration properties common to all services. Find service specific configuration references in the tabs below. EdgeX 2.0 For EdgeX 2.0 the Service configuration section has been standardized across all EdgeX services. Core Supporting Application & Analytics Device Security System Management Service Name Configuration Reference core-data Core Data Configuration core-metadata Core Metadata Configuration core-command Core Command Configuration Service Name Configuration Reference support-notifications Support Notifications Configuration support-scheduler Support Scheduler Configuration Services Name Configuration Reference app-service General Application Service Configuration app-service-configurable Configurable Application Service Configuration eKuiper rules engine/eKuiper Basic eKuiper Configuration Services Name Configuration Reference device-service General Device Service Configuration device-virtual Virtual Device Service Configuration Services Name Configuration Reference API Gateway Kong Configuration Add-on Services Configuring Add-on Service Services Name Configuration Reference system management System Management Agent Configuration","title":"Service Configuration"},{"location":"general/ServiceConfiguration/#service-configuration","text":"Each EdgeX micro service requires configuration (i.e. - a repository of initialization and operating values). The configuration is initially provided by a TOML file but a service can utilize the centralized configuration management provided by EdgeX for its configuration. See the Configuration and Registry documentation for more details about initialization of services and the use of the configuration service. Please refer to the EdgeX Foundry architectural decision record for details (and design decisions) behind the configuration in EdgeX. Please refer to the general Common Configuration documentation for configuration properties common to all services. Find service specific configuration references in the tabs below. EdgeX 2.0 For EdgeX 2.0 the Service configuration section has been standardized across all EdgeX services. Core Supporting Application & Analytics Device Security System Management Service Name Configuration Reference core-data Core Data Configuration core-metadata Core Metadata Configuration core-command Core Command Configuration Service Name Configuration Reference support-notifications Support Notifications Configuration support-scheduler Support Scheduler Configuration Services Name Configuration Reference app-service General Application Service Configuration app-service-configurable Configurable Application Service Configuration eKuiper rules engine/eKuiper Basic eKuiper Configuration Services Name Configuration Reference device-service General Device Service Configuration device-virtual Virtual Device Service Configuration Services Name Configuration Reference API Gateway Kong Configuration Add-on Services Configuring Add-on Service Services Name Configuration Reference system management System Management Agent Configuration","title":"Service Configuration"},{"location":"general/ServicePorts/","text":"Default Service Ports The following tables (organized by type of service) capture the default service ports. These default ports are also used in the EdgeX provided service routes defined in the Kong API Gateway for access control. Core Supporting Application & Analytics Device Security Miscellaneous Services Name Port Definition core-data 59880 ZMQ - to be deprecated in a future release 5563 core-metadata 59881 core-command 59882 Services Name Port Definition support-notifications 59860 support-scheduler 59861 Services Name Port Definition app-sample 59700 app-service-rules 59701 app-push-to-core 59702 app-mqtt-export 59703 app-http-export 59704 app-functional-tests 59705 app-rfid-llrp-inventory 59711 rules engine/eKuiper 59720 Services Name Port Definition device-virtual 59900 device-modbus 59901 device-bacnet 59980 device-mqtt 59982 device-camera 59985 device-rest 59986 device-coap 59988 device-llrp 59989 device-grove 59992 device-snmp 59993 device-gpio 59994 Services Name Port Definition kong-db 5432 vault 8200 kong 8000 8100 8443 security-spire-server 59840 security-spiffe-token-provider 59841 Services Name Port Definition Modbus simulator 1502 MQTT broker 1883 redis 6379 consul 8500 system management 58890","title":"Default Service Ports"},{"location":"general/ServicePorts/#default-service-ports","text":"The following tables (organized by type of service) capture the default service ports. These default ports are also used in the EdgeX provided service routes defined in the Kong API Gateway for access control. Core Supporting Application & Analytics Device Security Miscellaneous Services Name Port Definition core-data 59880 ZMQ - to be deprecated in a future release 5563 core-metadata 59881 core-command 59882 Services Name Port Definition support-notifications 59860 support-scheduler 59861 Services Name Port Definition app-sample 59700 app-service-rules 59701 app-push-to-core 59702 app-mqtt-export 59703 app-http-export 59704 app-functional-tests 59705 app-rfid-llrp-inventory 59711 rules engine/eKuiper 59720 Services Name Port Definition device-virtual 59900 device-modbus 59901 device-bacnet 59980 device-mqtt 59982 device-camera 59985 device-rest 59986 device-coap 59988 device-llrp 59989 device-grove 59992 device-snmp 59993 device-gpio 59994 Services Name Port Definition kong-db 5432 vault 8200 kong 8000 8100 8443 security-spire-server 59840 security-spiffe-token-provider 59841 Services Name Port Definition Modbus simulator 1502 MQTT broker 1883 redis 6379 consul 8500 system management 58890","title":"Default Service Ports"},{"location":"getting-started/","text":"Getting Started Attention Supported Architectures EdgeX Foundry is a hardware and operating system agnostic IoT / edge platform. It is was built to run on Intel and ARM hardware. It runs on various distributions and / or versions of Linux, Unix, MacOS, Windows, etc. However, the EdgeX Foundry community only supports the platform on Intel (x86, x86_64) and ARM64 hardware. Adopters may build EdgeX for ARM32, but the community does not support or provide pre-built artifacts such as Docker containers, snaps, etc. for ARM32. Building EdgeX for ARM32 will typically require some modifications to the build process. Some services may not compile on ARM32 and/or require removal of components such as ZMQ that are not available or compile on ARM32. To get started you need to get EdgeX Foundry either as a User or as a Developer/Contributor. User If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". You will want to follow the Getting Started as a User guide which takes you through the process of deploying the latest EdgeX releases. For demo purposes and to run EdgeX on your machine in just a few minutes, please refer to the Quick Start guide. Developer and Contributor If you want to change, add to or at least build the existing EdgeX code base, then you are a \"Developer\". \"Contributors\" are developers that further wish to contribute their code back into the EdgeX open source effort. You will want to follow the Getting Started for Developers guide. Hybrid See Getting Started Hybrid if you are developing or working on a particular micro service, but want to run the other micro services via Docker Containers. When working on something like an analytics service (as a developer or contributor) you may not wish to download, build and run all the EdgeX code - you only want to work with the code of your service. Your new service may still need to communicate with other services while you test your new service. Unless you want to get and build all the services, developers will often get and run the containers for the other EdgeX micro services and run only their service natively in a development environment. The EdgeX community refers to this as \"Hybrid\" development. Device Service Developer As a developer, if you intend to connect IoT objects (device, sensor or other \"thing\") that are not currently connected to EdgeX Foundry, you may also want to obtain the Device Service Software Development Kit (DS SDK) and create new device services. The DS SDK creates all the scaffolding code for a new EdgeX Foundry device service; allowing you to focus on the details of interfacing with the device in its native protocol. See Getting Started with Device SDK for help on using the DS SDK to create a new device service. Learn more about Device Services and the Device Service SDK at Device Services . Application Service Developer As a developer, if you intend to get EdgeX sensor data to external systems (be that an enterprise application, on-prem server or Cloud platform like Azure IoT Hub, AWS IoT, Google Cloud IOT, etc.), you will likely want to obtain the Application Functions SDK (App Func SDK) and create new application services. The App Func SDK creates all the scaffolding code for a new EdgeX Foundry application service; allowing you to focus on the details of data transformation, filtering, and otherwise prepare the sensor data for the external endpoint. Learn more about Application Services and the Application Functions SDK at Application Services . Versioning Please refer to the EdgeX Foundry versioning policy for information on how EdgeX services are released and how EdgeX services are compatible with one another. Specifically, device services (and the associated SDK), application services (and the associated app functions SDK), and client tools (like the EdgeX CLI and UI) can have independent minor releases, but these services must be compatible with the latest major release of EdgeX. Long Term Support Please refer to the EdgeX Foundry LTS policy for information on support of EdgeX releases. The EdgeX community does not offer support on any non-LTS release outside of the latest release.","title":"Getting Started"},{"location":"getting-started/#getting-started","text":"Attention","title":"Getting Started"},{"location":"getting-started/#supported-architectures","text":"EdgeX Foundry is a hardware and operating system agnostic IoT / edge platform. It is was built to run on Intel and ARM hardware. It runs on various distributions and / or versions of Linux, Unix, MacOS, Windows, etc. However, the EdgeX Foundry community only supports the platform on Intel (x86, x86_64) and ARM64 hardware. Adopters may build EdgeX for ARM32, but the community does not support or provide pre-built artifacts such as Docker containers, snaps, etc. for ARM32. Building EdgeX for ARM32 will typically require some modifications to the build process. Some services may not compile on ARM32 and/or require removal of components such as ZMQ that are not available or compile on ARM32. To get started you need to get EdgeX Foundry either as a User or as a Developer/Contributor.","title":"Supported Architectures"},{"location":"getting-started/#user","text":"If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". You will want to follow the Getting Started as a User guide which takes you through the process of deploying the latest EdgeX releases. For demo purposes and to run EdgeX on your machine in just a few minutes, please refer to the Quick Start guide.","title":"User"},{"location":"getting-started/#developer-and-contributor","text":"If you want to change, add to or at least build the existing EdgeX code base, then you are a \"Developer\". \"Contributors\" are developers that further wish to contribute their code back into the EdgeX open source effort. You will want to follow the Getting Started for Developers guide.","title":"Developer and Contributor"},{"location":"getting-started/#hybrid","text":"See Getting Started Hybrid if you are developing or working on a particular micro service, but want to run the other micro services via Docker Containers. When working on something like an analytics service (as a developer or contributor) you may not wish to download, build and run all the EdgeX code - you only want to work with the code of your service. Your new service may still need to communicate with other services while you test your new service. Unless you want to get and build all the services, developers will often get and run the containers for the other EdgeX micro services and run only their service natively in a development environment. The EdgeX community refers to this as \"Hybrid\" development.","title":"Hybrid"},{"location":"getting-started/#device-service-developer","text":"As a developer, if you intend to connect IoT objects (device, sensor or other \"thing\") that are not currently connected to EdgeX Foundry, you may also want to obtain the Device Service Software Development Kit (DS SDK) and create new device services. The DS SDK creates all the scaffolding code for a new EdgeX Foundry device service; allowing you to focus on the details of interfacing with the device in its native protocol. See Getting Started with Device SDK for help on using the DS SDK to create a new device service. Learn more about Device Services and the Device Service SDK at Device Services .","title":"Device Service Developer"},{"location":"getting-started/#application-service-developer","text":"As a developer, if you intend to get EdgeX sensor data to external systems (be that an enterprise application, on-prem server or Cloud platform like Azure IoT Hub, AWS IoT, Google Cloud IOT, etc.), you will likely want to obtain the Application Functions SDK (App Func SDK) and create new application services. The App Func SDK creates all the scaffolding code for a new EdgeX Foundry application service; allowing you to focus on the details of data transformation, filtering, and otherwise prepare the sensor data for the external endpoint. Learn more about Application Services and the Application Functions SDK at Application Services .","title":"Application Service Developer"},{"location":"getting-started/#versioning","text":"Please refer to the EdgeX Foundry versioning policy for information on how EdgeX services are released and how EdgeX services are compatible with one another. Specifically, device services (and the associated SDK), application services (and the associated app functions SDK), and client tools (like the EdgeX CLI and UI) can have independent minor releases, but these services must be compatible with the latest major release of EdgeX.","title":"Versioning"},{"location":"getting-started/#long-term-support","text":"Please refer to the EdgeX Foundry LTS policy for information on support of EdgeX releases. The EdgeX community does not offer support on any non-LTS release outside of the latest release.","title":"Long Term Support"},{"location":"getting-started/ApplicationFunctionsSDK/","text":"Getting Started The Application Functions SDK The SDK is built around the idea of a \"Functions Pipeline\". A functions pipeline is a collection of various functions that process the data in the order that you've specified. The functions pipeline is executed by the specified trigger in the configuration.toml . The first function in the pipeline is called with the event that triggered the pipeline (ex. dtos.Event ). Each successive call in the pipeline is called with the return result of the previous function. Let's take a look at a simple example that creates a pipeline to filter particular device ids and subsequently transform the data to XML: package main import ( \"errors\" \"fmt\" \"os\" \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg\" \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg/interfaces\" \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg/transforms\" ) const ( serviceKey = \"app-simple-filter-xml\" ) func main () { // turn off secure mode for examples. Not recommended for production _ = os . Setenv ( \"EDGEX_SECURITY_SECRET_STORE\" , \"false\" ) // 1) First thing to do is to create an new instance of an EdgeX Application Service. service , ok := pkg . NewAppService ( serviceKey ) if ! ok { os . Exit ( - 1 ) } // Leverage the built in logging service in EdgeX lc := service . LoggingClient () // 2) shows how to access the application's specific configuration settings. deviceNames , err := service . GetAppSettingStrings ( \"DeviceNames\" ) if err != nil { lc . Error ( err . Error ()) os . Exit ( - 1 ) } lc . Info ( fmt . Sprintf ( \"Filtering for devices %v\" , deviceNames )) // 3) This is our pipeline configuration, the collection of functions to // execute every time an event is triggered. if err := service . SetFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , transforms . NewConversion (). TransformToXML ); err != nil { lc . Errorf ( \"SetFunctionsPipeline returned error: %s\" , err . Error ()) os . Exit ( - 1 ) } // 4) Lastly, we'll go ahead and tell the SDK to \"start\" and begin listening for events // to trigger the pipeline. err = service . MakeItRun () if err != nil { lc . Errorf ( \"MakeItRun returned error: %s\" , err . Error ()) os . Exit ( - 1 ) } // Do any required cleanup here os . Exit ( 0 ) } The above example is meant to merely demonstrate the structure of your application. Notice that the output of the last function is not available anywhere inside this application. You must provide a function in order to work with the data from the previous function. Let's go ahead and add the following function that prints the output to the console. func printXMLToConsole ( ctx interfaces . AppFunctionContext , data interface {}) ( bool , interface {}) { // Leverage the built in logging service in EdgeX lc := ctx . LoggingClient () if data == nil { return false , errors . New ( \"printXMLToConsole: No data received\" ) } xml , ok := data .( string ) if ! ok { return false , errors . New ( \"printXMLToConsole: Data received is not the expected 'string' type\" ) } println ( xml ) return true , nil } After placing the above function in your code, the next step is to modify the pipeline to call this function: if err := service . SetFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , transforms . NewConversion (). TransformToXML , printXMLToConsole //notice this is not a function call, but simply a function pointer. ); err != nil { ... } Set the Trigger type to http in res/configuration.toml [Trigger] Type = \"http\" Using PostMan or curl send the following JSON to localhost:/api/v2/trigger { \"requestId\" : \"82eb2e26-0f24-48ba-ae4c-de9dac3fb9bc\" , \"apiVersion\" : \"v2\" , \"event\" : { \"apiVersion\" : \"v2\" , \"deviceName\" : \"Random-Float-Device\" , \"profileName\" : \"Random-Float-Device\" , \"sourceName\" : \"Float32\" , \"origin\" : 1540855006456 , \"id\" : \"94eb2e26-0f24-5555-2222-de9dac3fb228\" , \"readings\" : [ { \"apiVersion\" : \"v2\" , \"resourceName\" : \"Float32\" , \"profileName\" : \"Random-Float-Device\" , \"deviceName\" : \"Random-Float-Device\" , \"value\" : \"76677\" , \"origin\" : 1540855006469 , \"ValueType\" : \"Float32\" , \"id\" : \"82eb2e36-0f24-48aa-ae4c-de9dac3fb920\" } ] } } After making the above modifications, you should now see data printing out to the console in XML when an event is triggered. Note You can find this complete example \" Simple Filter XML \" and more examples located in the examples section. Up until this point, the pipeline has been triggered by an event over HTTP and the data at the end of that pipeline lands in the last function specified. In the example, data ends up printed to the console. Perhaps we'd like to send the data back to where it came from. In the case of an HTTP trigger, this would be the HTTP response. In the case of EdgeX MessageBus, this could be a new topic to send the data back to the MessageBus for other applications that wish to receive it. To do this, simply call ctx.SetResponseData(data []byte) passing in the data you wish to \"respond\" with. In the above printXMLToConsole(...) function, replace println(xml) with ctx.SetResponseData([]byte(xml)) . You should now see the response in your postman window when testing the pipeline.","title":"Application Functions SDK"},{"location":"getting-started/ApplicationFunctionsSDK/#getting-started","text":"","title":"Getting Started"},{"location":"getting-started/ApplicationFunctionsSDK/#the-application-functions-sdk","text":"The SDK is built around the idea of a \"Functions Pipeline\". A functions pipeline is a collection of various functions that process the data in the order that you've specified. The functions pipeline is executed by the specified trigger in the configuration.toml . The first function in the pipeline is called with the event that triggered the pipeline (ex. dtos.Event ). Each successive call in the pipeline is called with the return result of the previous function. Let's take a look at a simple example that creates a pipeline to filter particular device ids and subsequently transform the data to XML: package main import ( \"errors\" \"fmt\" \"os\" \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg\" \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg/interfaces\" \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg/transforms\" ) const ( serviceKey = \"app-simple-filter-xml\" ) func main () { // turn off secure mode for examples. Not recommended for production _ = os . Setenv ( \"EDGEX_SECURITY_SECRET_STORE\" , \"false\" ) // 1) First thing to do is to create an new instance of an EdgeX Application Service. service , ok := pkg . NewAppService ( serviceKey ) if ! ok { os . Exit ( - 1 ) } // Leverage the built in logging service in EdgeX lc := service . LoggingClient () // 2) shows how to access the application's specific configuration settings. deviceNames , err := service . GetAppSettingStrings ( \"DeviceNames\" ) if err != nil { lc . Error ( err . Error ()) os . Exit ( - 1 ) } lc . Info ( fmt . Sprintf ( \"Filtering for devices %v\" , deviceNames )) // 3) This is our pipeline configuration, the collection of functions to // execute every time an event is triggered. if err := service . SetFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , transforms . NewConversion (). TransformToXML ); err != nil { lc . Errorf ( \"SetFunctionsPipeline returned error: %s\" , err . Error ()) os . Exit ( - 1 ) } // 4) Lastly, we'll go ahead and tell the SDK to \"start\" and begin listening for events // to trigger the pipeline. err = service . MakeItRun () if err != nil { lc . Errorf ( \"MakeItRun returned error: %s\" , err . Error ()) os . Exit ( - 1 ) } // Do any required cleanup here os . Exit ( 0 ) } The above example is meant to merely demonstrate the structure of your application. Notice that the output of the last function is not available anywhere inside this application. You must provide a function in order to work with the data from the previous function. Let's go ahead and add the following function that prints the output to the console. func printXMLToConsole ( ctx interfaces . AppFunctionContext , data interface {}) ( bool , interface {}) { // Leverage the built in logging service in EdgeX lc := ctx . LoggingClient () if data == nil { return false , errors . New ( \"printXMLToConsole: No data received\" ) } xml , ok := data .( string ) if ! ok { return false , errors . New ( \"printXMLToConsole: Data received is not the expected 'string' type\" ) } println ( xml ) return true , nil } After placing the above function in your code, the next step is to modify the pipeline to call this function: if err := service . SetFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , transforms . NewConversion (). TransformToXML , printXMLToConsole //notice this is not a function call, but simply a function pointer. ); err != nil { ... } Set the Trigger type to http in res/configuration.toml [Trigger] Type = \"http\" Using PostMan or curl send the following JSON to localhost:/api/v2/trigger { \"requestId\" : \"82eb2e26-0f24-48ba-ae4c-de9dac3fb9bc\" , \"apiVersion\" : \"v2\" , \"event\" : { \"apiVersion\" : \"v2\" , \"deviceName\" : \"Random-Float-Device\" , \"profileName\" : \"Random-Float-Device\" , \"sourceName\" : \"Float32\" , \"origin\" : 1540855006456 , \"id\" : \"94eb2e26-0f24-5555-2222-de9dac3fb228\" , \"readings\" : [ { \"apiVersion\" : \"v2\" , \"resourceName\" : \"Float32\" , \"profileName\" : \"Random-Float-Device\" , \"deviceName\" : \"Random-Float-Device\" , \"value\" : \"76677\" , \"origin\" : 1540855006469 , \"ValueType\" : \"Float32\" , \"id\" : \"82eb2e36-0f24-48aa-ae4c-de9dac3fb920\" } ] } } After making the above modifications, you should now see data printing out to the console in XML when an event is triggered. Note You can find this complete example \" Simple Filter XML \" and more examples located in the examples section. Up until this point, the pipeline has been triggered by an event over HTTP and the data at the end of that pipeline lands in the last function specified. In the example, data ends up printed to the console. Perhaps we'd like to send the data back to where it came from. In the case of an HTTP trigger, this would be the HTTP response. In the case of EdgeX MessageBus, this could be a new topic to send the data back to the MessageBus for other applications that wish to receive it. To do this, simply call ctx.SetResponseData(data []byte) passing in the data you wish to \"respond\" with. In the above printXMLToConsole(...) function, replace println(xml) with ctx.SetResponseData([]byte(xml)) . You should now see the response in your postman window when testing the pipeline.","title":"The Application Functions SDK"},{"location":"getting-started/Ch-GettingStartedCDevelopers/","text":"Getting Started - C Developers Introduction These instructions are for C Developers and Contributors to get, run and otherwise work with C-based EdgeX Foundry micro services. Before reading this guide, review the general developer requirements . If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". Users should read: Getting Started as a User ) What You Need For C Development Many of EdgeX device services are built in C. In the future, other services could be built in C. In additional to the hardware and software listed in the Developers guide , to build EdgeX C services, you will need the following: libmicrohttpd libcurl libyaml libcbor paho libuuid hiredis You can install these on Debian 11 (Bullseye) by running: sudo apt-get install libcurl4-openssl-dev libmicrohttpd-dev libyaml-dev libcbor-dev libpaho-mqtt-dev uuid-dev libhiredis-dev Some of these supporting packages have dependencies of their own, which will be automatically installed when using package managers such as APT , DNF etc. libpaho-mqtt-dev is not included in Ubuntu prior to Groovy (20.10). IOTech provides a package for Focal (20.04 LTS) which may be installed as follows: sudo curl -fsSL https://iotech.jfrog.io/artifactory/api/gpg/key/public -o /etc/apt/trusted.gpg.d/iotech-public.asc sudo echo \"deb https://iotech.jfrog.io/iotech/debian-release $( lsb_release -cs ) main\" | tee -a /etc/apt/sources.list.d/iotech.list sudo apt-get update sudo apt-get install libpaho-mqtt EdgeX 2.0 For EdgeX 2.0 the C SDK now supports MQTT and Redis implementations of the EdgeX MessageBus CMake is required to build the SDKs. Version 3 or better is required. You can install CMake on Debian by running: sudo apt-get install cmake Check that your C development environment includes the following: a version of GCC supporting C11 CMake version 3 or greater Development libraries and headers for: curl (version 7.56 or later) microhttpd (version 0.9) libyaml (version 0.1.6 or later) libcbor (version 0.5) libuuid (from util-linux v2.x) paho (version 1.3.x) hiredis (version 0.14) Next Steps To explore how to create and build EdgeX device services in C, head to the Device Services, C SDK guide .","title":"Getting Started - C Developers"},{"location":"getting-started/Ch-GettingStartedCDevelopers/#getting-started-c-developers","text":"","title":"Getting Started - C Developers"},{"location":"getting-started/Ch-GettingStartedCDevelopers/#introduction","text":"These instructions are for C Developers and Contributors to get, run and otherwise work with C-based EdgeX Foundry micro services. Before reading this guide, review the general developer requirements . If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". Users should read: Getting Started as a User )","title":"Introduction"},{"location":"getting-started/Ch-GettingStartedCDevelopers/#what-you-need-for-c-development","text":"Many of EdgeX device services are built in C. In the future, other services could be built in C. In additional to the hardware and software listed in the Developers guide , to build EdgeX C services, you will need the following: libmicrohttpd libcurl libyaml libcbor paho libuuid hiredis You can install these on Debian 11 (Bullseye) by running: sudo apt-get install libcurl4-openssl-dev libmicrohttpd-dev libyaml-dev libcbor-dev libpaho-mqtt-dev uuid-dev libhiredis-dev Some of these supporting packages have dependencies of their own, which will be automatically installed when using package managers such as APT , DNF etc. libpaho-mqtt-dev is not included in Ubuntu prior to Groovy (20.10). IOTech provides a package for Focal (20.04 LTS) which may be installed as follows: sudo curl -fsSL https://iotech.jfrog.io/artifactory/api/gpg/key/public -o /etc/apt/trusted.gpg.d/iotech-public.asc sudo echo \"deb https://iotech.jfrog.io/iotech/debian-release $( lsb_release -cs ) main\" | tee -a /etc/apt/sources.list.d/iotech.list sudo apt-get update sudo apt-get install libpaho-mqtt EdgeX 2.0 For EdgeX 2.0 the C SDK now supports MQTT and Redis implementations of the EdgeX MessageBus CMake is required to build the SDKs. Version 3 or better is required. You can install CMake on Debian by running: sudo apt-get install cmake Check that your C development environment includes the following: a version of GCC supporting C11 CMake version 3 or greater Development libraries and headers for: curl (version 7.56 or later) microhttpd (version 0.9) libyaml (version 0.1.6 or later) libcbor (version 0.5) libuuid (from util-linux v2.x) paho (version 1.3.x) hiredis (version 0.14)","title":"What You Need For C Development"},{"location":"getting-started/Ch-GettingStartedCDevelopers/#next-steps","text":"To explore how to create and build EdgeX device services in C, head to the Device Services, C SDK guide .","title":"Next Steps"},{"location":"getting-started/Ch-GettingStartedDevelopers/","text":"Getting Started as a Developer Introduction These instructions are for Developers and Contributors to get and run EdgeX Foundry. If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". Users should read: Getting Started as a User ) EdgeX is a collection of more than a dozen micro services that are deployed to provide a minimal edge platform capability. EdgeX consists of a collection of reference implementation services and SDK tools. The micro services and SDKs are written in Go or C. These documentation pages provide a developer with the information and instructions to get and run EdgeX Foundry in development mode - that is running natively outside of containers and with the intent of adding to or changing the existing code base. What You Need Hardware EdgeX Foundry is an operating system (OS) and hardware (HW)-agnostic edge software platform. See the reference page for platform requirements . These provide guidance on a minimal platform to run the EdgeX platform. However, as a developer, you may find that additional memory, disk space, and improved CPU are essential to building and debugging. Software Developers need to install the following software to get, run and develop EdgeX Foundry micro services: Git Use this free and open source version control (SVC) system to download (and upload) the EdgeX Foundry source code from the project's GitHub repositories. See https://git-scm.com/downloads for download and install instructions. Alternative tools (Easy Git for example) could be used, but this document assumes use of git and leaves how to use alternative SVC tools to the reader. Redis By default, EdgeX Foundry uses Redis (version 5 starting with the Geneva release) as the persistence mechanism for sensor data as well as metadata about the devices/sensors that are connected. See Redis Documentation for download and installation instructions. MongoDB As an alternative, EdgeX Foundry allows use of MongoDB (version 4.2 as of Geneva) as the alternative persistence mechanism in place of Redis for sensor data as well as metadata about the connected devices/sensors. See Mongo's Documentation for download and installation instructions. Warning Use of MongoDB is deprecated with the Geneva release. EdgeX will remove MongoDB support in a future release. Developers should start to migrate to Redis in all development efforts targeting future EdgeX releases. ZeroMQ Several EdgeX Foundry services depend on ZeroMQ for communications by default. See the installation for your OS. Linux/Unix MacOS Windows The easiest way to get and install ZeroMQ on Linux is to use this setup script: https://gist.github.com/katopz/8b766a5cb0ca96c816658e9407e83d00 . Note The 0MQ install script above assumes bash is available on your system and the bash executable is in /usr/bin. Before running the script at the link, run which bash at your Linux terminal to insure that bash is in /usr/bin. If not, change the first line of the script so that it points to the correct location of bash. For MacOS, use brew to install ZeroMQ. brew install zeromq For directions installing ZeroMQ on Windows, please see the Windows documentation: https://github.com/edgexfoundry/edgex-go/blob/master/ZMQWindows.md Docker (Optional) If you intend to create Docker images for your updated or newly created EdgeX services, you need to install Docker. See https://docs.docker.com/install/ to learn how to install Docker. If you are new to Docker, the same web site provides you educational information. Additional Programming Tools and Next Steps Depending on which part of EdgeX you work on, you need to install one or more programming languages (Go, C, etc.) and associated tooling. These tools are covered under the documentation specific to each type of development. Go (Golang) C Versioning Please refer to the EdgeX Foundry versioning policy for information on how EdgeX services are released and how EdgeX services are compatible with one another. Specifically, device services (and the associated SDK), application services (and the associated app functions SDK), and client tools (like the EdgeX CLI and UI) can have independent minor releases, but these services must be compatible with the latest major release of EdgeX. Long Term Support Please refer to the EdgeX Foundry LTS policy for information on support of EdgeX releases. The EdgeX community does not offer support on any non-LTS release outside of the latest release.","title":"Getting Started as a Developer"},{"location":"getting-started/Ch-GettingStartedDevelopers/#getting-started-as-a-developer","text":"","title":"Getting Started as a Developer"},{"location":"getting-started/Ch-GettingStartedDevelopers/#introduction","text":"These instructions are for Developers and Contributors to get and run EdgeX Foundry. If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". Users should read: Getting Started as a User ) EdgeX is a collection of more than a dozen micro services that are deployed to provide a minimal edge platform capability. EdgeX consists of a collection of reference implementation services and SDK tools. The micro services and SDKs are written in Go or C. These documentation pages provide a developer with the information and instructions to get and run EdgeX Foundry in development mode - that is running natively outside of containers and with the intent of adding to or changing the existing code base.","title":"Introduction"},{"location":"getting-started/Ch-GettingStartedDevelopers/#what-you-need","text":"","title":"What You Need"},{"location":"getting-started/Ch-GettingStartedDevelopers/#hardware","text":"EdgeX Foundry is an operating system (OS) and hardware (HW)-agnostic edge software platform. See the reference page for platform requirements . These provide guidance on a minimal platform to run the EdgeX platform. However, as a developer, you may find that additional memory, disk space, and improved CPU are essential to building and debugging.","title":"Hardware"},{"location":"getting-started/Ch-GettingStartedDevelopers/#software","text":"Developers need to install the following software to get, run and develop EdgeX Foundry micro services:","title":"Software"},{"location":"getting-started/Ch-GettingStartedDevelopers/#git","text":"Use this free and open source version control (SVC) system to download (and upload) the EdgeX Foundry source code from the project's GitHub repositories. See https://git-scm.com/downloads for download and install instructions. Alternative tools (Easy Git for example) could be used, but this document assumes use of git and leaves how to use alternative SVC tools to the reader.","title":"Git"},{"location":"getting-started/Ch-GettingStartedDevelopers/#redis","text":"By default, EdgeX Foundry uses Redis (version 5 starting with the Geneva release) as the persistence mechanism for sensor data as well as metadata about the devices/sensors that are connected. See Redis Documentation for download and installation instructions.","title":"Redis"},{"location":"getting-started/Ch-GettingStartedDevelopers/#mongodb","text":"As an alternative, EdgeX Foundry allows use of MongoDB (version 4.2 as of Geneva) as the alternative persistence mechanism in place of Redis for sensor data as well as metadata about the connected devices/sensors. See Mongo's Documentation for download and installation instructions. Warning Use of MongoDB is deprecated with the Geneva release. EdgeX will remove MongoDB support in a future release. Developers should start to migrate to Redis in all development efforts targeting future EdgeX releases.","title":"MongoDB"},{"location":"getting-started/Ch-GettingStartedDevelopers/#zeromq","text":"Several EdgeX Foundry services depend on ZeroMQ for communications by default. See the installation for your OS. Linux/Unix MacOS Windows The easiest way to get and install ZeroMQ on Linux is to use this setup script: https://gist.github.com/katopz/8b766a5cb0ca96c816658e9407e83d00 . Note The 0MQ install script above assumes bash is available on your system and the bash executable is in /usr/bin. Before running the script at the link, run which bash at your Linux terminal to insure that bash is in /usr/bin. If not, change the first line of the script so that it points to the correct location of bash. For MacOS, use brew to install ZeroMQ. brew install zeromq For directions installing ZeroMQ on Windows, please see the Windows documentation: https://github.com/edgexfoundry/edgex-go/blob/master/ZMQWindows.md","title":"ZeroMQ"},{"location":"getting-started/Ch-GettingStartedDevelopers/#docker-optional","text":"If you intend to create Docker images for your updated or newly created EdgeX services, you need to install Docker. See https://docs.docker.com/install/ to learn how to install Docker. If you are new to Docker, the same web site provides you educational information.","title":"Docker (Optional)"},{"location":"getting-started/Ch-GettingStartedDevelopers/#additional-programming-tools-and-next-steps","text":"Depending on which part of EdgeX you work on, you need to install one or more programming languages (Go, C, etc.) and associated tooling. These tools are covered under the documentation specific to each type of development. Go (Golang) C","title":"Additional Programming Tools and Next Steps"},{"location":"getting-started/Ch-GettingStartedDevelopers/#versioning","text":"Please refer to the EdgeX Foundry versioning policy for information on how EdgeX services are released and how EdgeX services are compatible with one another. Specifically, device services (and the associated SDK), application services (and the associated app functions SDK), and client tools (like the EdgeX CLI and UI) can have independent minor releases, but these services must be compatible with the latest major release of EdgeX.","title":"Versioning"},{"location":"getting-started/Ch-GettingStartedDevelopers/#long-term-support","text":"Please refer to the EdgeX Foundry LTS policy for information on support of EdgeX releases. The EdgeX community does not offer support on any non-LTS release outside of the latest release.","title":"Long Term Support"},{"location":"getting-started/Ch-GettingStartedDockerUsers/","text":"Getting Started using Docker Introduction These instructions are for users to get and run EdgeX Foundry using the latest stable Docker images. If you wish to get the latest builds of EdgeX Docker images (prior to releases), then see the EdgeX Nexus Repository guide. Get & Run EdgeX Foundry Install Docker & Docker Compose To run Dockerized EdgeX, you need to install Docker. See https://docs.docker.com/install/ to learn how to install Docker. If you are new to Docker, the same web site provides you educational information. The following short video is also very informative https://www.youtube.com/watch?time_continue=3&v=VhabrYF1nms Use Docker Compose to orchestrate the fetch (or pull), install, and start the EdgeX micro service containers. Also use Docker Compose to stop the micro service containers. See: https://docs.docker.com/compose/ to learn more about Docker Compose. You do not need to be an expert with Docker (or Docker Compose) to get and run EdgeX. This guide provides the steps to get EdgeX running in your environment. Some knowledge of Docker and Docker Compose are nice to have, but not required. Basic Docker and Docker Compose commands provided here enable you to run, update, and diagnose issues within EdgeX. Select a EdgeX Foundry Compose File After installing Docker and Docker Compose, you need a EdgeX Docker Compose file. EdgeX Foundry has over a dozen micro services, each deployed in its own Docker container. This file is a manifest of all the EdgeX Foundry micro services to run. The Docker Compose file provides details about how to run each of the services. Specifically, a Docker Compose file is a manifest file, which lists: The Docker container images that should be downloaded, The order in which the containers should be started, The parameters (such as ports) under which the containers should be run The EdgeX development team provides Docker Compose files for each release. Visit the project's GitHub and find the edgex-compose repository . This repository holds all of the EdgeX Docker Compose files for each of the EdgeX releases/versions. The Compose files for each release are found in separate branches. Click on the main button to see all the branches. The edgex-compose repositor contains branches for each release. Select the release branch to locate the Docker Compose files for each release. Locate the branch containing the EdgeX Docker Compose file for the version of EdgeX you want to run. Note The main branch contains the Docker Compose files that use artifacts created from the latest code submitted by contributors (from the night builds). Most end users should avoid using these Docker Compose files. They are work-in-progress. Users should use the Docker Compose files for the latest version of EdgeX. In each edgex-compose branch, you will find several Docker Compose files (all with a .yml extension). The name of the file will suggest the type of EdgeX instance the Compose file will help setup. The table below provides a list of the Docker Compose filenames for the latest release (Ireland). Find the Docker Compose file that matches: your hardware (x86 or ARM) your desire to have security services on or off filename Docker Compose contents docker-compose-arm64.yml Specifies x86 containers, uses Redis database for persistence, and includes security services docker-compose-no-secty-arm64.yml Specifies ARM 64 containers, uses Redis database for persistence, but does not include security services docker-compose-no-secty.yml Specifies x86 containers, uses Redis database for persistence, but does not include security services docker-compose.yml Specifies x86 containers, uses Redis database for persistence, and includes security services docker-compose-no-secty-with-ui-arm64. Same as docker-compose-no-secty-arm64.yml but also includes EdgeX user interface docker-compose-no-secty-with-ui.yml Same as docker-compose-no-secty.yml but also includes EdgeX user interface docker-compose-portainer.yml Specifies the Portainer user interface extension (to be used with the x86 or ARM EdgeX platform) Download a EdgeX Foundry Compose File Once you have selected the release branch of edgex-compose you want to use, download it using your favorite tool. The examples below uses wget to fetch Docker Compose for the Ireland release with no security. x86 ARM wget https://raw.githubusercontent.com/edgexfoundry/edgex-compose/ireland/docker-compose-no-secty.yml -O docker-compose.yml wget https://raw.githubusercontent.com/edgexfoundry/edgex-compose/ireland/docker-compose-no-secty-arm64.yml -O docker-compose.yml Note The commands above fetch the Docker Compose to a file named 'docker-compose.yml' in the current directory. Docker Compose commands look for a file named 'docker-compose.yml' by default. You can use an alternate file name but then must specify that file name when issuing Docker Compose commands. See Compose reference documentation for help. Generate a custom Docker Compose file The Docker Compose files in the ireland branch contain the standard set of EdgeX services configured to use Redis message bus and include only the Virtual and REST device services. If you need to have different device services running or use MQTT for the message bus, you need a modified version of one of the standard Docker Compose files. You could manually add the device services to one of the existing EdgeX Compose files or, use the EdgeX Compose Builder tool to generate a new custom Compose file that contains the services you would like included. When you use Compose Builder, you don't have to worry about adding all the necessary ports, variables, etc. as the tool will generate the service elements in the file for you. The Compose Builder tool was added with the Hanoi release. You will find the Compose Builder tool in each of the release branches since Hanoi under the compose-builder folder of those branches. You will also find a compose-builder folder on the main branch for creating custom Compose files for the nightly builds. Do the following to use this tool to generate a custom Compose file: Clone the edgex-compose repository. git clone https://github.com/edgexfoundry/edgex-compose.git 2. Change directories to the clone and checkout the appropriate release branch. Checkout of the Ireland release branch is shown here. cd edgex-compose/ git checkout ireland 3. Change directories to the compose-builder folder and then use the make gen command to generate your custom compose file. The generated Docker Compose file is named docker-compose.yaml . Here are some examples: cd compose-builder/ make gen ds-mqtt mqtt-broker - Generates secure Compose file configured to use MQTT for the message bus, adds then MQTT broker and the Device MQTT services. make gen no-secty ds-modbus - Generates non-secure compose file with just the Device Modbus device service. make gen no-secty arm64 ds-grove - Generates non-secure compose file for ARM64 with just the Device Grove device service. \u200b See the README document in the compose-builder directory for details on all the available options. The Compose Builder is different per release, so make sure to consult the README in the appropriate release branch. See Ireland's Compose Builder README for details on the lastest release Compose Builder options for make gen . Note The generated Docker Compose file may require addition customizations for your specific needs, such as environment override(s) to set appropriate Host IP address, etc. Run EdgeX Foundry Now that you have the EdgeX Docker Compose file, you are ready to run EdgeX. Follow these steps to get the container images and start EdgeX! In a command terminal, change directories to the location of your docker-compose.yml. Run the following command in the terminal to pull (fetch) and then start the EdgeX containers. docker-compose up -d Info If you wish, you can fetch the images first and then run them. This allows you to make sure the EdgeX images you need are all available before trying to run. docker-compose pull docker-compose up -d Note The -d option indicates you want Docker Compose to run the EdgeX containers in detached mode - that is to run the containers in the background. Without -d, the containers will all start in the terminal and in order to use the terminal further you have to stop the containers. Verify EdgeX Foundry Running In the same terminal, run the process status command shown below to confirm that all the containers downloaded and started. docker-compose ps If all EdgeX containers pulled and started correctly and without error, you should see a process status (ps) that looks similar to the image above. If you are using a custom Compose file, your containers list may vary. Also note that some \"setup\" containers are designed to start and then exit after configuring your EdgeX instance. Checking the Status of EdgeX Foundry In addition to the process status of the EdgeX containers, there are a number of other tools to check on the health and status of your EdgeX instance. EdgeX Foundry Container Logs Use the command below to see the log of any service. # see the logs of a service docker-compose logs -f [ compose-service-name ] # example - core data docker-compose logs -f data See EdgeX Container Names for a list of the EdgeX Docker Compose service names. A check of an EdgeX service log usually indicates if the service is running normally or has errors. When you are done reviewing the content of the log, select Control-c to stop the output to your terminal. Ping Check Each EdgeX micro service has a built-in response to a \"ping\" HTTP request. In networking environments, use a ping request to check the reach-ability of a network resource. EdgeX uses the same concept to check the availability or reach-ability of a micro service. After the EdgeX micro service containers are running, you can \"ping\" any one of the micro services to check that it is running. Open a browser or HTTP REST client tool and use the service's ping address (outlined below) to check that is available. http://localhost:[service port]/api/v2/ping See EdgeX Default Service Ports for a list of the EdgeX default service ports. \"Pinging\" an EdgeX micro service allows you to check on its availability. If the service does not respond to ping, the service is down or having issues. Consul Registry Check EdgeX uses the open source Consul project as its registry service. All EdgeX micro services are expected to register with Consul as they start. Going to Consul's dashboard UI enables you to see which services are up. Find the Consul UI at http://localhost:8500/ui . EdgeX 2.0 Please note that as of EdgeX 2.0, Consul can be secured. When EdgeX is running in secure mode with secure Consul , you must provide Consul's access token to get to the dashboard UI referenced above. See How to get Consul ACL token for details.","title":"Getting Started using Docker"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#getting-started-using-docker","text":"","title":"Getting Started using Docker"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#introduction","text":"These instructions are for users to get and run EdgeX Foundry using the latest stable Docker images. If you wish to get the latest builds of EdgeX Docker images (prior to releases), then see the EdgeX Nexus Repository guide.","title":"Introduction"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#get-run-edgex-foundry","text":"","title":"Get & Run EdgeX Foundry"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#install-docker-docker-compose","text":"To run Dockerized EdgeX, you need to install Docker. See https://docs.docker.com/install/ to learn how to install Docker. If you are new to Docker, the same web site provides you educational information. The following short video is also very informative https://www.youtube.com/watch?time_continue=3&v=VhabrYF1nms Use Docker Compose to orchestrate the fetch (or pull), install, and start the EdgeX micro service containers. Also use Docker Compose to stop the micro service containers. See: https://docs.docker.com/compose/ to learn more about Docker Compose. You do not need to be an expert with Docker (or Docker Compose) to get and run EdgeX. This guide provides the steps to get EdgeX running in your environment. Some knowledge of Docker and Docker Compose are nice to have, but not required. Basic Docker and Docker Compose commands provided here enable you to run, update, and diagnose issues within EdgeX.","title":"Install Docker & Docker Compose"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#select-a-edgex-foundry-compose-file","text":"After installing Docker and Docker Compose, you need a EdgeX Docker Compose file. EdgeX Foundry has over a dozen micro services, each deployed in its own Docker container. This file is a manifest of all the EdgeX Foundry micro services to run. The Docker Compose file provides details about how to run each of the services. Specifically, a Docker Compose file is a manifest file, which lists: The Docker container images that should be downloaded, The order in which the containers should be started, The parameters (such as ports) under which the containers should be run The EdgeX development team provides Docker Compose files for each release. Visit the project's GitHub and find the edgex-compose repository . This repository holds all of the EdgeX Docker Compose files for each of the EdgeX releases/versions. The Compose files for each release are found in separate branches. Click on the main button to see all the branches. The edgex-compose repositor contains branches for each release. Select the release branch to locate the Docker Compose files for each release. Locate the branch containing the EdgeX Docker Compose file for the version of EdgeX you want to run. Note The main branch contains the Docker Compose files that use artifacts created from the latest code submitted by contributors (from the night builds). Most end users should avoid using these Docker Compose files. They are work-in-progress. Users should use the Docker Compose files for the latest version of EdgeX. In each edgex-compose branch, you will find several Docker Compose files (all with a .yml extension). The name of the file will suggest the type of EdgeX instance the Compose file will help setup. The table below provides a list of the Docker Compose filenames for the latest release (Ireland). Find the Docker Compose file that matches: your hardware (x86 or ARM) your desire to have security services on or off filename Docker Compose contents docker-compose-arm64.yml Specifies x86 containers, uses Redis database for persistence, and includes security services docker-compose-no-secty-arm64.yml Specifies ARM 64 containers, uses Redis database for persistence, but does not include security services docker-compose-no-secty.yml Specifies x86 containers, uses Redis database for persistence, but does not include security services docker-compose.yml Specifies x86 containers, uses Redis database for persistence, and includes security services docker-compose-no-secty-with-ui-arm64. Same as docker-compose-no-secty-arm64.yml but also includes EdgeX user interface docker-compose-no-secty-with-ui.yml Same as docker-compose-no-secty.yml but also includes EdgeX user interface docker-compose-portainer.yml Specifies the Portainer user interface extension (to be used with the x86 or ARM EdgeX platform)","title":"Select a EdgeX Foundry Compose File"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#download-a-edgex-foundry-compose-file","text":"Once you have selected the release branch of edgex-compose you want to use, download it using your favorite tool. The examples below uses wget to fetch Docker Compose for the Ireland release with no security. x86 ARM wget https://raw.githubusercontent.com/edgexfoundry/edgex-compose/ireland/docker-compose-no-secty.yml -O docker-compose.yml wget https://raw.githubusercontent.com/edgexfoundry/edgex-compose/ireland/docker-compose-no-secty-arm64.yml -O docker-compose.yml Note The commands above fetch the Docker Compose to a file named 'docker-compose.yml' in the current directory. Docker Compose commands look for a file named 'docker-compose.yml' by default. You can use an alternate file name but then must specify that file name when issuing Docker Compose commands. See Compose reference documentation for help.","title":"Download a EdgeX Foundry Compose File"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#generate-a-custom-docker-compose-file","text":"The Docker Compose files in the ireland branch contain the standard set of EdgeX services configured to use Redis message bus and include only the Virtual and REST device services. If you need to have different device services running or use MQTT for the message bus, you need a modified version of one of the standard Docker Compose files. You could manually add the device services to one of the existing EdgeX Compose files or, use the EdgeX Compose Builder tool to generate a new custom Compose file that contains the services you would like included. When you use Compose Builder, you don't have to worry about adding all the necessary ports, variables, etc. as the tool will generate the service elements in the file for you. The Compose Builder tool was added with the Hanoi release. You will find the Compose Builder tool in each of the release branches since Hanoi under the compose-builder folder of those branches. You will also find a compose-builder folder on the main branch for creating custom Compose files for the nightly builds. Do the following to use this tool to generate a custom Compose file: Clone the edgex-compose repository. git clone https://github.com/edgexfoundry/edgex-compose.git 2. Change directories to the clone and checkout the appropriate release branch. Checkout of the Ireland release branch is shown here. cd edgex-compose/ git checkout ireland 3. Change directories to the compose-builder folder and then use the make gen command to generate your custom compose file. The generated Docker Compose file is named docker-compose.yaml . Here are some examples: cd compose-builder/ make gen ds-mqtt mqtt-broker - Generates secure Compose file configured to use MQTT for the message bus, adds then MQTT broker and the Device MQTT services. make gen no-secty ds-modbus - Generates non-secure compose file with just the Device Modbus device service. make gen no-secty arm64 ds-grove - Generates non-secure compose file for ARM64 with just the Device Grove device service. \u200b See the README document in the compose-builder directory for details on all the available options. The Compose Builder is different per release, so make sure to consult the README in the appropriate release branch. See Ireland's Compose Builder README for details on the lastest release Compose Builder options for make gen . Note The generated Docker Compose file may require addition customizations for your specific needs, such as environment override(s) to set appropriate Host IP address, etc.","title":"Generate a custom Docker Compose file"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#run-edgex-foundry","text":"Now that you have the EdgeX Docker Compose file, you are ready to run EdgeX. Follow these steps to get the container images and start EdgeX! In a command terminal, change directories to the location of your docker-compose.yml. Run the following command in the terminal to pull (fetch) and then start the EdgeX containers. docker-compose up -d Info If you wish, you can fetch the images first and then run them. This allows you to make sure the EdgeX images you need are all available before trying to run. docker-compose pull docker-compose up -d Note The -d option indicates you want Docker Compose to run the EdgeX containers in detached mode - that is to run the containers in the background. Without -d, the containers will all start in the terminal and in order to use the terminal further you have to stop the containers.","title":"Run EdgeX Foundry"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#verify-edgex-foundry-running","text":"In the same terminal, run the process status command shown below to confirm that all the containers downloaded and started. docker-compose ps If all EdgeX containers pulled and started correctly and without error, you should see a process status (ps) that looks similar to the image above. If you are using a custom Compose file, your containers list may vary. Also note that some \"setup\" containers are designed to start and then exit after configuring your EdgeX instance.","title":"Verify EdgeX Foundry Running"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#checking-the-status-of-edgex-foundry","text":"In addition to the process status of the EdgeX containers, there are a number of other tools to check on the health and status of your EdgeX instance.","title":"Checking the Status of EdgeX Foundry"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#edgex-foundry-container-logs","text":"Use the command below to see the log of any service. # see the logs of a service docker-compose logs -f [ compose-service-name ] # example - core data docker-compose logs -f data See EdgeX Container Names for a list of the EdgeX Docker Compose service names. A check of an EdgeX service log usually indicates if the service is running normally or has errors. When you are done reviewing the content of the log, select Control-c to stop the output to your terminal.","title":"EdgeX Foundry Container Logs"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#ping-check","text":"Each EdgeX micro service has a built-in response to a \"ping\" HTTP request. In networking environments, use a ping request to check the reach-ability of a network resource. EdgeX uses the same concept to check the availability or reach-ability of a micro service. After the EdgeX micro service containers are running, you can \"ping\" any one of the micro services to check that it is running. Open a browser or HTTP REST client tool and use the service's ping address (outlined below) to check that is available. http://localhost:[service port]/api/v2/ping See EdgeX Default Service Ports for a list of the EdgeX default service ports. \"Pinging\" an EdgeX micro service allows you to check on its availability. If the service does not respond to ping, the service is down or having issues.","title":"Ping Check"},{"location":"getting-started/Ch-GettingStartedDockerUsers/#consul-registry-check","text":"EdgeX uses the open source Consul project as its registry service. All EdgeX micro services are expected to register with Consul as they start. Going to Consul's dashboard UI enables you to see which services are up. Find the Consul UI at http://localhost:8500/ui . EdgeX 2.0 Please note that as of EdgeX 2.0, Consul can be secured. When EdgeX is running in secure mode with secure Consul , you must provide Consul's access token to get to the dashboard UI referenced above. See How to get Consul ACL token for details.","title":"Consul Registry Check"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/","text":"Getting Started - Go Developers Introduction These instructions are for Go Lang Developers and Contributors to get, run and otherwise work with Go-based EdgeX Foundry micro services. Before reading this guide, review the general developer requirements . If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". Users should read: Getting Started as a User ) What You Need For Go Development In additional to the hardware and software listed in the Developers guide , you will need the following to work with the EdgeX Go-based micro services. Go The open sourced micro services of EdgeX Foundry are written in Go 1.16. See https://golang.org/dl/ for download and installation instructions. Newer versions of Go are available and may work, but the project has not built and tested to these newer versions of the language. Older versions of Go, especially 1.10 or older, are likely to cause issues (EdgeX now uses Go Modules which were introduced with Go Lang 1.11). Build Essentials In order to compile and build some elements of EdgeX, Gnu C compiler, utilities (like make), and associated librarires need to be installed. Some IDEs may already come with these tools. Some OS environments may already come with these tools. Others environments may require you install them. For Ubuntu environments, you can install a convenience package called Build Essentials . Note If you are installing Build Essentials, note that there is a build-essential package for each Ubuntu release. Search for 'build-essential' associated to your Ubuntu version via Ubuntu Packages Search . IDE (Optional) There are many tool options for writing and editing Go Lang code. You could use a simple text editor. For more convenience, you may choose to use an integrated development environment (IDE). The list below highlights IDEs used by some of the EdgeX community (without any project endorsement). GoLand GoLand is a popular, although subscription-fee based, Go specific IDE. Learn how to purchase and download Go Land here: https://www.jetbrains.com/go/ . Visual Studio Code Visual Studio Code is a free, open source IDE developed by Microsoft. Find and download Visual Studio Code here: https://code.visualstudio.com/ . Atom Atom is also a free, open source IDE used with many languages. Find and download Atom here: https://ide.atom.io/ . Get the code This part of the documentation assumes you wish to get and work with the key EdgeX services. This includes but is not limited to Core, Supporting, some security, and system management services. To work with other Go-based security services, device services, application services, SDKs, user interface, or other service you may need to pull in other EdgeX repository code. See other getting started guides for working with other Go-based services. As you will see below, you do not need to explicitly pull in dependency modules (whether EdgeX or 3rd party provided). Dependencies will automatically be pulled through the building process. To work with the key services, you will need to download the source code from the EdgeX Go repository . The EdgeX Go-based micro services are all available in a single GitHub repository download. Once the code is pulled, the Go micro services are built and packaged as platform dependent executables. If Docker is installed, the executable can also be containerized for end user deployment/use. To download the EdgeX Go code, first change directories to the location where you want to download the code (to edgex in the image below). Then use your git tool and request to clone this repository with the following command: git clone https://github.com/edgexfoundry/edgex-go.git Note If you plan to contribute code back to the EdgeX project (as a Contributor), you are going to want to fork the repositories you plan to work with and then pull your fork versus the EdgeX repositories directly. This documentation does not address the process and procedures for working with an EdgeX fork, committing changes and submitting contribution pull requests (PRs). See some of the links below in the EdgeX Wiki for help on how to fork and contribute EdgeX code. https://wiki.edgexfoundry.org/display/FA/Contributor%27s+Guide https://wiki.edgexfoundry.org/display/FA/Contributor%27s+Process Furthermore, this pulls and works with the latest code from the main branch. The main branch contains code that is \"work in progress\" for the upcoming release. If you want to work with a specific release, checkout code from the specific release branch or tag(e.g. v2.0.0 , hanoi , v1.3.11 , etc.) Build EdgeX Foundry To build the Go Lang services found in edgex-go, first change directories to the root of the edgex-go code cd edgex-go Second, use the community provided Makefile to build all the services in a single call make build Info The first time EdgeX builds, it will take longer than other builds as it has to download all dependencies. Depending on the size of your host machine, an initial build can take several minutes. Make sure the build completes and has no errors. If it does build, you should find new service executables in each of the service folders under the service directories found in the /edgex-go/cmd folder. Run EdgeX Foundry Run the Database Several of the EdgeX Foundry micro services use a database. This includes core-data, core-metadata, support-scheduler, among others. Therefore, when working with EdgeX Foundry its a good idea to have the database up and running as a general rule. See the Redis Quick Start Guide for how to run Redis in a Linux environment (or find similar documentation for other environments). Run EdgeX Services With the services built, and the database up and running, you can now run each of the services. In this example, the services will run without security services turned on. If you wish to run with security, you will need to clone, build and run the security services. In order to turn security off, first set the EDGEX_SECURITY_SECRET_STORE environment variable to false with an export call. Simply call export EDGEX_SECURITY_SECRET_STORE = false Next, move to the cmd folder and then change folders to the service folder for the service you want to run. Start the executable (with default configuration) that is in that folder. For example, to start Core Metadata, enter the cmd/core-metadata folder and start core-metadata. cd cmd/core-metadata/ ./core-metadata & Note When running the services from the command line, you will usually want to start the service with the & character after the command. This makes the command run in the background. If you do not run the service in the background, then you will need to leave the service running in the terminal and open another terminal to start the other services. This will start the EdgeX go service and leave it running in the background until you kill it. The log entries from the service will still display in the terminal. Watch the log entries for any ERROR indicators. Info To kill a service there are several options, but an easy means is to use pkill with the service name. pkill core-metadata Start as many services as you need in order to carry out your development, testing, etc. As an absolute minimal set, you will typically need to run core-metadata, core-data, core-command and a device service. Selection of the device service will depend on which physical sensor or device you want to use (or use the virtual device to simulate a sensor). Here are the set of commands to launch core-data and core-command (in addition to core-metadata above) cd ../core-data/ ./core-data & cd ../core-command/ ./core-command & Tip You can run some services via Docker containers while working on specific services in Go. See Working in a Hybrid Environment for more details. While the EdgeX services are running you can make EdgeX API calls to localhost . Info No sensor data will flow yet as this just gets the key services up and running. To get sensor data flowing into EdgeX, you will need to get, build and run an EdgeX device service in a similar fashion. The community provides a virtual device service to test and experiment with ( https://github.com/edgexfoundry/device-virtual-go ). Verify EdgeX is Working Each EdgeX micro service has a built-in respond to a \"ping\" HTTP request. In networking environments, use a ping request to check the reach-ability of a network resource. EdgeX uses the same concept to check the availability or reach-ability of a micro service. After the EdgeX micro services are running, you can \"ping\" any one of the micro services to check that it is running. Open a browser or HTTP REST client tool and use the service's ping address (outlined below) to check that is available. http://localhost:[port]/api/v2/ping See EdgeX Default Service Ports for a list of the EdgeX default service ports. \"Pinging\" an EdgeX micro service allows you to check on its availability. If the service does not respond to ping, the service is down or having issues. The example above shows the ping of core-data. Next Steps Application services and some device services are also built in Go. To explore how to create and build EdgeX application and devices services in Go, head to SDK documentation covering these EdgeX elements. Application Services and the Application Functions SDK Device Services in Go EdgeX Foundry in GoLand IDEs offer many code editing conveniences. Go Land was specifically built to edit and work with Go code. So if you are doing any significant code work with the EdgeX Go micro services, you will likely find it convenient to edit, build, run, test, etc. from GoLand or other IDE. Import EdgeX To bring in the EdgeX repository code into Go Land, use the File \u2192 Open... menu option in Go Land to open the Open File or Project Window. In the \"Open File or Project\" popup, select the location of the folder containing your cloned edgex-go repo. Open the Terminal From the View menu in Go Land, select the Terminal menu option. This will open a command terminal from which you can issue commands to install the dependencies, build the micro services, run the micro services, etc. Build the EdgeX Micro Services Run \"make build\" in the Terminal view (as shown below) to build the services. This can take a few minutes to build all the services. Just as when running make build from the command line in a terminal, the micro service executables that get built in Go Land's terminal will be created in each of the service folders under the service directories found in the /edgex-go/cmd folder.. Run EdgeX With all the micro services built, you can now run EdgeX services. You may first want to make sure the database is running. Then, set any environment variables, change directories to the /cmd and service subfolder, and run the service right from the the terminal (same as in Run EdgeX Services ). You can now call on the service APIs to make sure they are running correctly. Namely, call on http://localhost:\\[service port\\]/api/v2/ping to see each service respond to the simplest of requests.","title":"Getting Started - Go Developers"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#getting-started-go-developers","text":"","title":"Getting Started - Go Developers"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#introduction","text":"These instructions are for Go Lang Developers and Contributors to get, run and otherwise work with Go-based EdgeX Foundry micro services. Before reading this guide, review the general developer requirements . If you want to get the EdgeX platform and run it (but do not intend to change or add to the existing code base now) then you are considered a \"User\". Users should read: Getting Started as a User )","title":"Introduction"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#what-you-need-for-go-development","text":"In additional to the hardware and software listed in the Developers guide , you will need the following to work with the EdgeX Go-based micro services.","title":"What You Need For Go Development"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#go","text":"The open sourced micro services of EdgeX Foundry are written in Go 1.16. See https://golang.org/dl/ for download and installation instructions. Newer versions of Go are available and may work, but the project has not built and tested to these newer versions of the language. Older versions of Go, especially 1.10 or older, are likely to cause issues (EdgeX now uses Go Modules which were introduced with Go Lang 1.11).","title":"Go"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#build-essentials","text":"In order to compile and build some elements of EdgeX, Gnu C compiler, utilities (like make), and associated librarires need to be installed. Some IDEs may already come with these tools. Some OS environments may already come with these tools. Others environments may require you install them. For Ubuntu environments, you can install a convenience package called Build Essentials . Note If you are installing Build Essentials, note that there is a build-essential package for each Ubuntu release. Search for 'build-essential' associated to your Ubuntu version via Ubuntu Packages Search .","title":"Build Essentials"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#ide-optional","text":"There are many tool options for writing and editing Go Lang code. You could use a simple text editor. For more convenience, you may choose to use an integrated development environment (IDE). The list below highlights IDEs used by some of the EdgeX community (without any project endorsement).","title":"IDE (Optional)"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#goland","text":"GoLand is a popular, although subscription-fee based, Go specific IDE. Learn how to purchase and download Go Land here: https://www.jetbrains.com/go/ .","title":"GoLand"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#visual-studio-code","text":"Visual Studio Code is a free, open source IDE developed by Microsoft. Find and download Visual Studio Code here: https://code.visualstudio.com/ .","title":"Visual Studio Code"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#atom","text":"Atom is also a free, open source IDE used with many languages. Find and download Atom here: https://ide.atom.io/ .","title":"Atom"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#get-the-code","text":"This part of the documentation assumes you wish to get and work with the key EdgeX services. This includes but is not limited to Core, Supporting, some security, and system management services. To work with other Go-based security services, device services, application services, SDKs, user interface, or other service you may need to pull in other EdgeX repository code. See other getting started guides for working with other Go-based services. As you will see below, you do not need to explicitly pull in dependency modules (whether EdgeX or 3rd party provided). Dependencies will automatically be pulled through the building process. To work with the key services, you will need to download the source code from the EdgeX Go repository . The EdgeX Go-based micro services are all available in a single GitHub repository download. Once the code is pulled, the Go micro services are built and packaged as platform dependent executables. If Docker is installed, the executable can also be containerized for end user deployment/use. To download the EdgeX Go code, first change directories to the location where you want to download the code (to edgex in the image below). Then use your git tool and request to clone this repository with the following command: git clone https://github.com/edgexfoundry/edgex-go.git Note If you plan to contribute code back to the EdgeX project (as a Contributor), you are going to want to fork the repositories you plan to work with and then pull your fork versus the EdgeX repositories directly. This documentation does not address the process and procedures for working with an EdgeX fork, committing changes and submitting contribution pull requests (PRs). See some of the links below in the EdgeX Wiki for help on how to fork and contribute EdgeX code. https://wiki.edgexfoundry.org/display/FA/Contributor%27s+Guide https://wiki.edgexfoundry.org/display/FA/Contributor%27s+Process Furthermore, this pulls and works with the latest code from the main branch. The main branch contains code that is \"work in progress\" for the upcoming release. If you want to work with a specific release, checkout code from the specific release branch or tag(e.g. v2.0.0 , hanoi , v1.3.11 , etc.)","title":"Get the code"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#build-edgex-foundry","text":"To build the Go Lang services found in edgex-go, first change directories to the root of the edgex-go code cd edgex-go Second, use the community provided Makefile to build all the services in a single call make build Info The first time EdgeX builds, it will take longer than other builds as it has to download all dependencies. Depending on the size of your host machine, an initial build can take several minutes. Make sure the build completes and has no errors. If it does build, you should find new service executables in each of the service folders under the service directories found in the /edgex-go/cmd folder.","title":"Build EdgeX Foundry"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#run-edgex-foundry","text":"","title":"Run EdgeX Foundry"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#run-the-database","text":"Several of the EdgeX Foundry micro services use a database. This includes core-data, core-metadata, support-scheduler, among others. Therefore, when working with EdgeX Foundry its a good idea to have the database up and running as a general rule. See the Redis Quick Start Guide for how to run Redis in a Linux environment (or find similar documentation for other environments).","title":"Run the Database"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#run-edgex-services","text":"With the services built, and the database up and running, you can now run each of the services. In this example, the services will run without security services turned on. If you wish to run with security, you will need to clone, build and run the security services. In order to turn security off, first set the EDGEX_SECURITY_SECRET_STORE environment variable to false with an export call. Simply call export EDGEX_SECURITY_SECRET_STORE = false Next, move to the cmd folder and then change folders to the service folder for the service you want to run. Start the executable (with default configuration) that is in that folder. For example, to start Core Metadata, enter the cmd/core-metadata folder and start core-metadata. cd cmd/core-metadata/ ./core-metadata & Note When running the services from the command line, you will usually want to start the service with the & character after the command. This makes the command run in the background. If you do not run the service in the background, then you will need to leave the service running in the terminal and open another terminal to start the other services. This will start the EdgeX go service and leave it running in the background until you kill it. The log entries from the service will still display in the terminal. Watch the log entries for any ERROR indicators. Info To kill a service there are several options, but an easy means is to use pkill with the service name. pkill core-metadata Start as many services as you need in order to carry out your development, testing, etc. As an absolute minimal set, you will typically need to run core-metadata, core-data, core-command and a device service. Selection of the device service will depend on which physical sensor or device you want to use (or use the virtual device to simulate a sensor). Here are the set of commands to launch core-data and core-command (in addition to core-metadata above) cd ../core-data/ ./core-data & cd ../core-command/ ./core-command & Tip You can run some services via Docker containers while working on specific services in Go. See Working in a Hybrid Environment for more details. While the EdgeX services are running you can make EdgeX API calls to localhost . Info No sensor data will flow yet as this just gets the key services up and running. To get sensor data flowing into EdgeX, you will need to get, build and run an EdgeX device service in a similar fashion. The community provides a virtual device service to test and experiment with ( https://github.com/edgexfoundry/device-virtual-go ).","title":"Run EdgeX Services"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#verify-edgex-is-working","text":"Each EdgeX micro service has a built-in respond to a \"ping\" HTTP request. In networking environments, use a ping request to check the reach-ability of a network resource. EdgeX uses the same concept to check the availability or reach-ability of a micro service. After the EdgeX micro services are running, you can \"ping\" any one of the micro services to check that it is running. Open a browser or HTTP REST client tool and use the service's ping address (outlined below) to check that is available. http://localhost:[port]/api/v2/ping See EdgeX Default Service Ports for a list of the EdgeX default service ports. \"Pinging\" an EdgeX micro service allows you to check on its availability. If the service does not respond to ping, the service is down or having issues. The example above shows the ping of core-data.","title":"Verify EdgeX is Working"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#next-steps","text":"Application services and some device services are also built in Go. To explore how to create and build EdgeX application and devices services in Go, head to SDK documentation covering these EdgeX elements. Application Services and the Application Functions SDK Device Services in Go","title":"Next Steps"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#edgex-foundry-in-goland","text":"IDEs offer many code editing conveniences. Go Land was specifically built to edit and work with Go code. So if you are doing any significant code work with the EdgeX Go micro services, you will likely find it convenient to edit, build, run, test, etc. from GoLand or other IDE.","title":"EdgeX Foundry in GoLand"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#import-edgex","text":"To bring in the EdgeX repository code into Go Land, use the File \u2192 Open... menu option in Go Land to open the Open File or Project Window. In the \"Open File or Project\" popup, select the location of the folder containing your cloned edgex-go repo.","title":"Import EdgeX"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#open-the-terminal","text":"From the View menu in Go Land, select the Terminal menu option. This will open a command terminal from which you can issue commands to install the dependencies, build the micro services, run the micro services, etc.","title":"Open the Terminal"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#build-the-edgex-micro-services","text":"Run \"make build\" in the Terminal view (as shown below) to build the services. This can take a few minutes to build all the services. Just as when running make build from the command line in a terminal, the micro service executables that get built in Go Land's terminal will be created in each of the service folders under the service directories found in the /edgex-go/cmd folder..","title":"Build the EdgeX Micro Services"},{"location":"getting-started/Ch-GettingStartedGoDevelopers/#run-edgex","text":"With all the micro services built, you can now run EdgeX services. You may first want to make sure the database is running. Then, set any environment variables, change directories to the /cmd and service subfolder, and run the service right from the the terminal (same as in Run EdgeX Services ). You can now call on the service APIs to make sure they are running correctly. Namely, call on http://localhost:\\[service port\\]/api/v2/ping to see each service respond to the simplest of requests.","title":"Run EdgeX"},{"location":"getting-started/Ch-GettingStartedHybrid/","text":"Working in a Hybrid Environment In some cases, as a developer or contributor , you want to work on a particular micro service. Yet, you don't want to have to download all the source code, and then build and run all the micro services. There is an alternative approach! You can download and run the EdgeX Docker containers for all the micro services you need and run your single micro service (the one you are presumably working on) natively or from a developer tool of choice outside of a container. Within EdgeX, we call this a \"hybrid\" environment - where part of your EdgeX platform is running from a development environment, while other parts are running from Docker containers. This page outlines how to work in a hybrid development environment. As an example of this process, let's say you want to do coding work with/on the Virtual Device service. You want the rest of the EdgeX environment up and running via Docker containers. How would you set up this hybrid environment? Let's take a look. Get and Run the EdgeX Docker Containers If you haven't already, follow the Getting Started using Docker guide to set up your environment (Docker, Docker Compose, etc.) before continuing. Since we plan to work with the virtual device service in this example, you don't need or want to run the virtual device service. You will run all the other services via Docker Compose. Based on the instructions found in the Getting Started using Docker , locate and download the appropriate Docker Compose file for your development environment. Next, issue the following commands to start the EdgeX containers and then stop the virtual device service (which is the service you are working on in this example). docker-compose up -d docker-compose stop device-virtual Run the EdgeX containers and then stop the service container that you are going to work on - in this case the virtual device service container. Note These notes assume you are working with the EdgeX Ireland release. It also assumes you have downloaded the appropriate Docker Compose file and have named it docker-compose.yml so you don't have to specify the file name each time you run a Docker Compose command. Some versions of EdgeX may require other or additional containers to run. Tip You can also use the EdgeX Compose Builder tool to create a custom Docker Compose file with just the services you want. See the Compose Builder documentation on and checkout the Compose Builder tool in GitHub . Run the command below to confirm that all the containers have started and that the virtual device container is no longer running. docker-compose ps Get, Build and Run the (non-Docker) Service With the EdgeX containers running, you can now download, build and run natively (outside of a container) the service you want to work on. In this example, the virtual device service is used to exemplify the steps necessary to get, build and run the native service with the EdgeX containerized services. However, the practice could be applied to any service. Get the service code Per Getting Started Go Developers , pull the micro service code you want to work on from GitHub. In this example, we use the device-virtual-go as the micro service that is going to be worked on. git clone https://github.com/edgexfoundry/device-virtual-go.git Build the service code At this time, you can add or modify the code to make the service changes you need. Once ready, you must compile and build the service into an executable. Change folders to the cloned micro service directory and build the service. cd device-virtual-go/ make build Clone the service from Github, make your code changes and then build the service locally. Change the configuration Depending on the service you are working on, you may need to change the configuration of the service to point to and use the other services that are containerized (running in Docker). In particular, if the service you are working on is not on the same host as the Docker Engine running the containerized services, you will likely need to change the configuration. Examine the configuration.toml file in the cmd/res folder of the device-virtual-go. Note that the Service (located in the [Service] section of the configuration), Registry (located in the [Registry] section) and all the \"Clients\" (located in the [Clients] section) suggest that the Host of these services is \"localhost\". These and other host configuration elements need to change when the services are not running on the same host - specifically the localhost. When your service is running on a different host than the rest of EdgeX, change the [Service] Host to be the address of the machine hosting your service. Change the [Registry] and [Clients] Host configuration to specify the location of the machine hosting these services. If you do have to change the configuration, save the configuration.toml file after making changes. Run the service code natively. The executable created by the make build command is found in the cmd folder of the service. Change folders to the location of the executable. Set any environment variables needed depending on your EdgeX setup. In this example, we did not start the security elements so we need to set EDGEX_SECURITY_SECRET_STORE to false in order to turn off security. Finally, run the service right from a terminal. cd cmd export EDGEX_SECURITY_SECRET_STORE = false ./device-virtual Change folders to the service's cmd/ folder, set env vars, and then execute the service executable in the cmd folder. Check the results At this time, your virtual device micro service should be communicating with the other EdgeX micro services running in their Docker containers. Because Core Metadata callbacks do not work in the hybrid environment, the virtual device service will not receive the Add Device callbacks on the inital run after creating them in Core Metadata. The simple work around for this issue is to stop ( Ctrl-c from the terminal) and restart the virtual device service (again with ./device-virtual execution). The virtual device service log after stopping and restarting. Give the virtual device a few seconds or so to initialize itself and start sending data to Core Data. To check that it is working properly, open a browser and point your browser to Core Data to check that events are being deposited. You can do this by calling on the Core Data API that checks the count of events in Core Data. http://localhost:59880/api/v2/event/count For this example, you can check that the virtual device service is sending data into Core Data by checking the event count. Note If you choose, you can also import the service into GoLand and then code and run the service from GoLand. Follow the instructions in the Getting Started - Go Developers to learn how to import, build and run a service in GoLand.","title":"Working in a Hybrid Environment"},{"location":"getting-started/Ch-GettingStartedHybrid/#working-in-a-hybrid-environment","text":"In some cases, as a developer or contributor , you want to work on a particular micro service. Yet, you don't want to have to download all the source code, and then build and run all the micro services. There is an alternative approach! You can download and run the EdgeX Docker containers for all the micro services you need and run your single micro service (the one you are presumably working on) natively or from a developer tool of choice outside of a container. Within EdgeX, we call this a \"hybrid\" environment - where part of your EdgeX platform is running from a development environment, while other parts are running from Docker containers. This page outlines how to work in a hybrid development environment. As an example of this process, let's say you want to do coding work with/on the Virtual Device service. You want the rest of the EdgeX environment up and running via Docker containers. How would you set up this hybrid environment? Let's take a look.","title":"Working in a Hybrid Environment"},{"location":"getting-started/Ch-GettingStartedHybrid/#get-and-run-the-edgex-docker-containers","text":"If you haven't already, follow the Getting Started using Docker guide to set up your environment (Docker, Docker Compose, etc.) before continuing. Since we plan to work with the virtual device service in this example, you don't need or want to run the virtual device service. You will run all the other services via Docker Compose. Based on the instructions found in the Getting Started using Docker , locate and download the appropriate Docker Compose file for your development environment. Next, issue the following commands to start the EdgeX containers and then stop the virtual device service (which is the service you are working on in this example). docker-compose up -d docker-compose stop device-virtual Run the EdgeX containers and then stop the service container that you are going to work on - in this case the virtual device service container. Note These notes assume you are working with the EdgeX Ireland release. It also assumes you have downloaded the appropriate Docker Compose file and have named it docker-compose.yml so you don't have to specify the file name each time you run a Docker Compose command. Some versions of EdgeX may require other or additional containers to run. Tip You can also use the EdgeX Compose Builder tool to create a custom Docker Compose file with just the services you want. See the Compose Builder documentation on and checkout the Compose Builder tool in GitHub . Run the command below to confirm that all the containers have started and that the virtual device container is no longer running. docker-compose ps","title":"Get and Run the EdgeX Docker Containers"},{"location":"getting-started/Ch-GettingStartedHybrid/#get-build-and-run-the-non-docker-service","text":"With the EdgeX containers running, you can now download, build and run natively (outside of a container) the service you want to work on. In this example, the virtual device service is used to exemplify the steps necessary to get, build and run the native service with the EdgeX containerized services. However, the practice could be applied to any service.","title":"Get, Build and Run the (non-Docker) Service"},{"location":"getting-started/Ch-GettingStartedHybrid/#get-the-service-code","text":"Per Getting Started Go Developers , pull the micro service code you want to work on from GitHub. In this example, we use the device-virtual-go as the micro service that is going to be worked on. git clone https://github.com/edgexfoundry/device-virtual-go.git","title":"Get the service code"},{"location":"getting-started/Ch-GettingStartedHybrid/#build-the-service-code","text":"At this time, you can add or modify the code to make the service changes you need. Once ready, you must compile and build the service into an executable. Change folders to the cloned micro service directory and build the service. cd device-virtual-go/ make build Clone the service from Github, make your code changes and then build the service locally.","title":"Build the service code"},{"location":"getting-started/Ch-GettingStartedHybrid/#change-the-configuration","text":"Depending on the service you are working on, you may need to change the configuration of the service to point to and use the other services that are containerized (running in Docker). In particular, if the service you are working on is not on the same host as the Docker Engine running the containerized services, you will likely need to change the configuration. Examine the configuration.toml file in the cmd/res folder of the device-virtual-go. Note that the Service (located in the [Service] section of the configuration), Registry (located in the [Registry] section) and all the \"Clients\" (located in the [Clients] section) suggest that the Host of these services is \"localhost\". These and other host configuration elements need to change when the services are not running on the same host - specifically the localhost. When your service is running on a different host than the rest of EdgeX, change the [Service] Host to be the address of the machine hosting your service. Change the [Registry] and [Clients] Host configuration to specify the location of the machine hosting these services. If you do have to change the configuration, save the configuration.toml file after making changes.","title":"Change the configuration"},{"location":"getting-started/Ch-GettingStartedHybrid/#run-the-service-code-natively","text":"The executable created by the make build command is found in the cmd folder of the service. Change folders to the location of the executable. Set any environment variables needed depending on your EdgeX setup. In this example, we did not start the security elements so we need to set EDGEX_SECURITY_SECRET_STORE to false in order to turn off security. Finally, run the service right from a terminal. cd cmd export EDGEX_SECURITY_SECRET_STORE = false ./device-virtual Change folders to the service's cmd/ folder, set env vars, and then execute the service executable in the cmd folder.","title":"Run the service code natively."},{"location":"getting-started/Ch-GettingStartedHybrid/#check-the-results","text":"At this time, your virtual device micro service should be communicating with the other EdgeX micro services running in their Docker containers. Because Core Metadata callbacks do not work in the hybrid environment, the virtual device service will not receive the Add Device callbacks on the inital run after creating them in Core Metadata. The simple work around for this issue is to stop ( Ctrl-c from the terminal) and restart the virtual device service (again with ./device-virtual execution). The virtual device service log after stopping and restarting. Give the virtual device a few seconds or so to initialize itself and start sending data to Core Data. To check that it is working properly, open a browser and point your browser to Core Data to check that events are being deposited. You can do this by calling on the Core Data API that checks the count of events in Core Data. http://localhost:59880/api/v2/event/count For this example, you can check that the virtual device service is sending data into Core Data by checking the event count. Note If you choose, you can also import the service into GoLand and then code and run the service from GoLand. Follow the instructions in the Getting Started - Go Developers to learn how to import, build and run a service in GoLand.","title":"Check the results"},{"location":"getting-started/Ch-GettingStartedSDK-C/","text":"C SDK In this guide, you create a simple device service that generates a random number as a means to simulate getting data from an actual device. In this way, you explore some of the SDK framework and work necessary to complete a device service without actually having a device to talk to. Install dependencies See the Getting Started - C Developers guide to install the necessary tools and infrastructure needed to develop a C service. Get the EdgeX Device SDK for C The next step is to download and build the EdgeX device service SDK for C. First, clone the device-sdk-c from Github: git clone -b v2.0.0 https://github.com/edgexfoundry/device-sdk-c.git cd ./device-sdk-c Note The clone command above has you pull v2.0.0 of the C SDK which is the version compatible with the Ireland release. Then, build the device-sdk-c: make Starting a new Device Service For this guide, you use the example template provided by the C SDK as a starting point for a new device service. You modify the device service to generate random integer values. Begin by copying the template example source into a new directory named example-device-c : mkdir -p ../example-device-c/res/profiles mkdir -p ../example-device-c/res/devices cp ./src/c/examples/template.c ../example-device-c cd ../example-device-c EdgeX 2.0 In EdgeX 2.0 the profiles have been moved to their own res/profiles directory and device definitions have been moved out of the configuration file into the res/devices directory. Build your Device Service Now you are ready to build your new device service using the C SDK you compiled in an earlier step. Tell the compiler where to find the C SDK files: export CSDK_DIR = ../device-sdk-c/build/release/_CPack_Packages/Linux/TGZ/csdk-2.0.0 Note The exact path to your compiled CSDK_DIR may differ depending on the tagged version number on the SDK. The version of the SDK can be found in the VERSION file located in the ./device-sdk-c/VERSION file. In the example above, the Ireland release of 2.0.0 is used. Now build your device service executable: gcc -I $CSDK_DIR /include -L $CSDK_DIR /lib -o device-example-c template.c -lcsdk If everything is working properly, a device-example-c executable will be created in the directory. Customize your Device Service Up to now you've been building the example device service provided by the C SDK. In order to change it to a device service that generates random numbers, you need to modify your template.c method template_get_handler . Replace the following code: for ( uint32_t i = 0 ; i < nreadings ; i ++ ) { /* Log the attributes for each requested resource */ iot_log_debug ( driver -> lc , \" Requested reading %u:\" , i ); dump_attributes ( driver -> lc , requests [ i ]. resource -> attrs ); /* Fill in a result regardless */ readings [ i ]. value = iot_data_alloc_string ( \"Template result\" , IOT_DATA_REF ); } return true ; with this code: for ( uint32_t i = 0 ; i < nreadings ; i ++ ) { const char * rdtype = iot_data_string_map_get_string ( requests [ i ]. resource -> attrs , \"type\" ); if ( rdtype ) { if ( strcmp ( rdtype , \"random\" ) == 0 ) { /* Set the reading as a random value between 0 and 100 */ readings [ i ]. value = iot_data_alloc_i32 ( rand () % 100 ); } else { * exception = iot_data_alloc_string ( \"Unknown sensor type requested\" , IOT_DATA_REF ); return false ; } } else { * exception = iot_data_alloc_string ( \"Unable to read value, no \\\" type \\\" attribute given\" , IOT_DATA_REF ); return false ; } } return true ; Here the reading value is set to a random signed integer. Various iot_data_alloc_ functions are defined in the iot/data.h header allowing readings of different types to be generated. Creating your Device Profile A device profile is a YAML file that describes a class of device to EdgeX. General characteristics about the type of device, the data these devices provide, and how to command the device are all in a device profile. The device profile tells the device service what data gets collected from the the device and how to get it. Follow these steps to create a device profile for the simple random number generating device service. Explore the files in the device-sdk-c/src/c/examples/res/profiles folder. Note the example TemplateProfile.json device profile that is already in this folder. Open the file with your favorite editor and explore its contents. Note how deviceResources in the file represent properties of a device (properties like SensorOne, SensorTwo and Switch). A pre-created device profile for the random number device is provided in this documentation. This is supplied in the alternative file format .yaml. Download random-generator-device.yaml and save the file to the ./res/profiles folder. Open the random-generator-device.yaml file in a text editor. In this device profile, the device described has a deviceResource: RandomNumber . Note how the association of a type to the deviceResource. In this case, the device profile informs EdgeX that RandomNumber will be a Int32. In real world IoT situations, this deviceResource list could be extensive and filled with many deviceResources all different types of data. Creating your Device Device Service accepts pre-defined devices to be added to EdgeX during device service startup. Follow these steps to create a pre-defined device for the simple random number generating device service. Explore the files in the cmd/device-simple/res/devices folder. Note the example simple-device.json that is already in this folder. Open the file with your favorite editor and explore its contents. Note how the file contents represent an actual device with its properties (properties like Name, ProfileName, AutoEvents). A pre-created device for the random number device is provided in this documentation. Download random-generator-device.json and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res/devices folder. Open the random-generator-device.json file in a text editor. In this example, the device described has a profileName: RandNum-Device . In this case, the device informs EdgeX that it will be using the device profile we created in Creating your Device Profile Configuring your Device Service Now update the configuration for the new device service. This documentation provides a new configuration.toml file. This configuration file: - changes the port the service operates on so as not to conflict with other device services Download configuration.toml and save the file to the ./res folder. Custom Structured Configuration C Device Services support structured custom configuration as part of the [Driver] section in the configuration.toml file. View the main function of template.c . The confparams variable is initialized with default values for three test parameters. These values may be overridden by entries in the configuration file or by environment variables in the usual way. The resulting configuration is passed to the init function when the service starts. Configuration parameters X , Y/Z and Writable/Q correspond to configuration file entries as follows: [Writable] [Writable.Driver] Q = \"foo\" [Driver] X = \"bar\" [Driver.Y] Z = \"baz\" Entries in the writable section can be changed dynamically if using the registry; the reconfigure callback will be invoked with the new configuration when changes are made. In addition to strings, configuration entries may be integer, float or boolean typed. Use the different iot_data_alloc_ functions when setting up the defaults as appropriate. Rebuild your Device Service Now you have your new device service, modified to return a random number, a device profile that will tell EdgeX how to read that random number, as well as a configuration file that will let your device service register itself and its device profile with EdgeX, and begin taking readings every 10 seconds. Rebuild your Device Service to reflect the changes that you have made: gcc -I $CSDK_DIR /include -L $CSDK_DIR /lib -o device-example-c template.c -lcsdk Run your Device Service Allow your newly created Device Service, which was formed out of the Device Service C SDK, to create sensor mimicking data which it then sends to EdgeX. Follow the Getting Started using Docker guide to start all of EdgeX. From the folder containing the docker-compose file, start EdgeX with the following call: docker-compose up -d Back in your custom device service directory, tell your device service where to find the libcsdk.so : export LD_LIBRARY_PATH = $CSDK_DIR /lib Run your device service: ./device-example-c You should now see your device service having its /Random command called every 10 seconds. You can verify that it is sending data into EdgeX by watching the logs of the edgex-core-data service: docker logs -f edgex-core-data Which would print an event record every time your device service is called. You can manually generate an event using curl to query the device service directly: curl 0 :59992/api/v2/device/name/RandNum-Device01/RandomNumber Using a browser, enter the following URL to see the event/reading data that the service is generating and sending to EdgeX: http://localhost:59880/api/v2/event/device/name/RandNum-Device01?limit=100 This request asks core data to provide the last 100 events/readings associated to the RandNum-Device-01.","title":"C SDK"},{"location":"getting-started/Ch-GettingStartedSDK-C/#c-sdk","text":"In this guide, you create a simple device service that generates a random number as a means to simulate getting data from an actual device. In this way, you explore some of the SDK framework and work necessary to complete a device service without actually having a device to talk to.","title":"C SDK"},{"location":"getting-started/Ch-GettingStartedSDK-C/#install-dependencies","text":"See the Getting Started - C Developers guide to install the necessary tools and infrastructure needed to develop a C service.","title":"Install dependencies"},{"location":"getting-started/Ch-GettingStartedSDK-C/#get-the-edgex-device-sdk-for-c","text":"The next step is to download and build the EdgeX device service SDK for C. First, clone the device-sdk-c from Github: git clone -b v2.0.0 https://github.com/edgexfoundry/device-sdk-c.git cd ./device-sdk-c Note The clone command above has you pull v2.0.0 of the C SDK which is the version compatible with the Ireland release. Then, build the device-sdk-c: make","title":"Get the EdgeX Device SDK for C"},{"location":"getting-started/Ch-GettingStartedSDK-C/#starting-a-new-device-service","text":"For this guide, you use the example template provided by the C SDK as a starting point for a new device service. You modify the device service to generate random integer values. Begin by copying the template example source into a new directory named example-device-c : mkdir -p ../example-device-c/res/profiles mkdir -p ../example-device-c/res/devices cp ./src/c/examples/template.c ../example-device-c cd ../example-device-c EdgeX 2.0 In EdgeX 2.0 the profiles have been moved to their own res/profiles directory and device definitions have been moved out of the configuration file into the res/devices directory.","title":"Starting a new Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-C/#build-your-device-service","text":"Now you are ready to build your new device service using the C SDK you compiled in an earlier step. Tell the compiler where to find the C SDK files: export CSDK_DIR = ../device-sdk-c/build/release/_CPack_Packages/Linux/TGZ/csdk-2.0.0 Note The exact path to your compiled CSDK_DIR may differ depending on the tagged version number on the SDK. The version of the SDK can be found in the VERSION file located in the ./device-sdk-c/VERSION file. In the example above, the Ireland release of 2.0.0 is used. Now build your device service executable: gcc -I $CSDK_DIR /include -L $CSDK_DIR /lib -o device-example-c template.c -lcsdk If everything is working properly, a device-example-c executable will be created in the directory.","title":"Build your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-C/#customize-your-device-service","text":"Up to now you've been building the example device service provided by the C SDK. In order to change it to a device service that generates random numbers, you need to modify your template.c method template_get_handler . Replace the following code: for ( uint32_t i = 0 ; i < nreadings ; i ++ ) { /* Log the attributes for each requested resource */ iot_log_debug ( driver -> lc , \" Requested reading %u:\" , i ); dump_attributes ( driver -> lc , requests [ i ]. resource -> attrs ); /* Fill in a result regardless */ readings [ i ]. value = iot_data_alloc_string ( \"Template result\" , IOT_DATA_REF ); } return true ; with this code: for ( uint32_t i = 0 ; i < nreadings ; i ++ ) { const char * rdtype = iot_data_string_map_get_string ( requests [ i ]. resource -> attrs , \"type\" ); if ( rdtype ) { if ( strcmp ( rdtype , \"random\" ) == 0 ) { /* Set the reading as a random value between 0 and 100 */ readings [ i ]. value = iot_data_alloc_i32 ( rand () % 100 ); } else { * exception = iot_data_alloc_string ( \"Unknown sensor type requested\" , IOT_DATA_REF ); return false ; } } else { * exception = iot_data_alloc_string ( \"Unable to read value, no \\\" type \\\" attribute given\" , IOT_DATA_REF ); return false ; } } return true ; Here the reading value is set to a random signed integer. Various iot_data_alloc_ functions are defined in the iot/data.h header allowing readings of different types to be generated.","title":"Customize your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-C/#creating-your-device-profile","text":"A device profile is a YAML file that describes a class of device to EdgeX. General characteristics about the type of device, the data these devices provide, and how to command the device are all in a device profile. The device profile tells the device service what data gets collected from the the device and how to get it. Follow these steps to create a device profile for the simple random number generating device service. Explore the files in the device-sdk-c/src/c/examples/res/profiles folder. Note the example TemplateProfile.json device profile that is already in this folder. Open the file with your favorite editor and explore its contents. Note how deviceResources in the file represent properties of a device (properties like SensorOne, SensorTwo and Switch). A pre-created device profile for the random number device is provided in this documentation. This is supplied in the alternative file format .yaml. Download random-generator-device.yaml and save the file to the ./res/profiles folder. Open the random-generator-device.yaml file in a text editor. In this device profile, the device described has a deviceResource: RandomNumber . Note how the association of a type to the deviceResource. In this case, the device profile informs EdgeX that RandomNumber will be a Int32. In real world IoT situations, this deviceResource list could be extensive and filled with many deviceResources all different types of data.","title":"Creating your Device Profile"},{"location":"getting-started/Ch-GettingStartedSDK-C/#creating-your-device","text":"Device Service accepts pre-defined devices to be added to EdgeX during device service startup. Follow these steps to create a pre-defined device for the simple random number generating device service. Explore the files in the cmd/device-simple/res/devices folder. Note the example simple-device.json that is already in this folder. Open the file with your favorite editor and explore its contents. Note how the file contents represent an actual device with its properties (properties like Name, ProfileName, AutoEvents). A pre-created device for the random number device is provided in this documentation. Download random-generator-device.json and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res/devices folder. Open the random-generator-device.json file in a text editor. In this example, the device described has a profileName: RandNum-Device . In this case, the device informs EdgeX that it will be using the device profile we created in Creating your Device Profile","title":"Creating your Device"},{"location":"getting-started/Ch-GettingStartedSDK-C/#configuring-your-device-service","text":"Now update the configuration for the new device service. This documentation provides a new configuration.toml file. This configuration file: - changes the port the service operates on so as not to conflict with other device services Download configuration.toml and save the file to the ./res folder.","title":"Configuring your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-C/#custom-structured-configuration","text":"C Device Services support structured custom configuration as part of the [Driver] section in the configuration.toml file. View the main function of template.c . The confparams variable is initialized with default values for three test parameters. These values may be overridden by entries in the configuration file or by environment variables in the usual way. The resulting configuration is passed to the init function when the service starts. Configuration parameters X , Y/Z and Writable/Q correspond to configuration file entries as follows: [Writable] [Writable.Driver] Q = \"foo\" [Driver] X = \"bar\" [Driver.Y] Z = \"baz\" Entries in the writable section can be changed dynamically if using the registry; the reconfigure callback will be invoked with the new configuration when changes are made. In addition to strings, configuration entries may be integer, float or boolean typed. Use the different iot_data_alloc_ functions when setting up the defaults as appropriate.","title":"Custom Structured Configuration"},{"location":"getting-started/Ch-GettingStartedSDK-C/#rebuild-your-device-service","text":"Now you have your new device service, modified to return a random number, a device profile that will tell EdgeX how to read that random number, as well as a configuration file that will let your device service register itself and its device profile with EdgeX, and begin taking readings every 10 seconds. Rebuild your Device Service to reflect the changes that you have made: gcc -I $CSDK_DIR /include -L $CSDK_DIR /lib -o device-example-c template.c -lcsdk","title":"Rebuild your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-C/#run-your-device-service","text":"Allow your newly created Device Service, which was formed out of the Device Service C SDK, to create sensor mimicking data which it then sends to EdgeX. Follow the Getting Started using Docker guide to start all of EdgeX. From the folder containing the docker-compose file, start EdgeX with the following call: docker-compose up -d Back in your custom device service directory, tell your device service where to find the libcsdk.so : export LD_LIBRARY_PATH = $CSDK_DIR /lib Run your device service: ./device-example-c You should now see your device service having its /Random command called every 10 seconds. You can verify that it is sending data into EdgeX by watching the logs of the edgex-core-data service: docker logs -f edgex-core-data Which would print an event record every time your device service is called. You can manually generate an event using curl to query the device service directly: curl 0 :59992/api/v2/device/name/RandNum-Device01/RandomNumber Using a browser, enter the following URL to see the event/reading data that the service is generating and sending to EdgeX: http://localhost:59880/api/v2/event/device/name/RandNum-Device01?limit=100 This request asks core data to provide the last 100 events/readings associated to the RandNum-Device-01.","title":"Run your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-Go/","text":"Golang SDK In this guide, you create a simple device service that generates a random number as a means to simulate getting data from an actual device. In this way, you explore some SDK framework and work necessary to complete a device service without actually having a device to talk to. Install dependencies See the Getting Started - Go Developers guide to install the necessary tools and infrastructure needed to develop a GoLang service. Get the EdgeX Device SDK for Go Follow these steps to create a folder on your file system, download the Device SDK , and get the GoLang device service SDK on your system. Create a collection of nested folders, ~/edgexfoundry on your file system. This folder will hold your new Device Service. In Linux, create a directory with a single mkdir command mkdir -p ~/edgexfoundry In a terminal window, change directories to the folder just created and pull down the SDK in Go with the commands as shown. cd ~/edgexfoundry git clone --depth 1 --branch v2.0.0 https://github.com/edgexfoundry/device-sdk-go.git Note The clone command above has you pull v2.0.0 of the Go SDK which is the version associated to Ireland. There are later releases of EdgeX, and it is always a good idea to pull and use the latest version associated with the major version of EdgeX you are using. You may want to check for the latest released version by going to https://github.com/edgexfoundry/device-sdk-go and look for the latest release. Create a folder that will hold the new device service. The name of the folder is also the name you want to give your new device service. Standard practice in EdgeX is to prefix the name of a device service with device- . In this example, the name 'device-simple' is used. mkdir -p ~/edgexfoundry/device-simple Copy the example code from device-sdk-go to device-simple : cd ~/edgexfoundry cp -rf ./device-sdk-go/example/* ./device-simple/ Copy Makefile to device-simple: cp ./device-sdk-go/Makefile ./device-simple Copy version.go to device-simple: cp ./device-sdk-go/version.go ./device-simple/ After completing these steps, your device-simple folder should look like the listing below. Start a new Device Service With the device service application structure in place, time now to program the service to act like a sensor data fetching service. Change folders to the device-simple directory. cd ~/edgexfoundry/device-simple Open main.go file in the cmd/device-simple folder with your favorite text editor. Modify the import statements. Replace github.com/edgexfoundry/device-sdk-go/v2/example/driver with github.com/edgexfoundry/device-simple/driver in the import statements. Also replace github.com/edgexfoundry/device-sdk-go/v2 with github.com/edgexfoundry/device-simple . Save the file when you have finished editing. Open Makefile found in the base folder (~/edgexfoundry/device-simple) in your favorite text editor and make the following changes Replace: MICROSERVICES=example/cmd/device-simple/device-simple with: MICROSERVICES=cmd/device-simple/device-simple Change: GOFLAGS = -ldflags \"-X github.com/edgexfoundry/device-sdk-go/v2.Version= $( VERSION ) \" to refer to the new service with: GOFLAGS = -ldflags \"-X github.com/edgexfoundry/device-simple.Version= $( VERSION ) \" Change: example/cmd/device-simple/device-simple : go mod tidy $( GOCGO ) build $( GOFLAGS ) -o $@ ./example/cmd/device-simple to: cmd/device-simple/device-simple : go mod tidy $( GOCGO ) build $( GOFLAGS ) -o $@ ./cmd/device-simple Save the file. Enter the following command to create the initial module definition and write it to the go.mod file: GO111MODULE = on go mod init github . com / edgexfoundry / device - simple Use an editor to open and edit the go.mod file created in ~/edgexfoundry/device-simple. Add the code highlighted below to the bottom of the file. This code indicates which version of the device service SDK and the associated EdgeX contracts module to use. require ( github . com / edgexfoundry / device - sdk - go / v2 v2 .0.0 github . com / edgexfoundry / go - mod - core - contracts / v2 v2 .0.0 ) Note You should always check the go.mod file in the latest released version SDK for the correct versions of the Go SDK and go-mod-contracts to use in your go.mod. Build your Device Service To ensure that the code you have moved and updated still works, build the device service. In a terminal window, make sure you are still in the device-simple folder (the folder containing the Makefile). Build the service by issuing the following command: make build If there are no errors, your service is ready for you to add custom code to generate data values as if there was a sensor attached. Customize your Device Service The device service you are creating isn't going to talk to a real device. Instead, it is going to generate a random number where the service would ordinarily make a call to get sensor data from the actual device. Locate the simpledriver.go file in the /driver folder and open it with your favorite editor. In the import() area at the top of the file, add \"math/rand\" under \"time\". Locate the HandleReadCommands() function in this same file (simpledriver.go). Find the following lines of code in this file (around line 139): if reqs [ 0 ]. DeviceResourceName == \"SwitchButton\" { cv , _ := sdkModels . NewCommandValue ( reqs [ 0 ]. DeviceResourceName , common . ValueTypeBool , s . switchButton ) res [ 0 ] = cv } Add the conditional (if-else) code in front of the above conditional: if reqs [ 0 ]. DeviceResourceName == \"randomnumber\" { cv , _ := sdkModels . NewCommandValue ( reqs [ 0 ]. DeviceResourceName , common . ValueTypeInt32 , int32 ( rand . Intn ( 100 ))) res [ 0 ] = cv } else The first line of code checks that the current request is for a resource called \"RandomNumber\". The second line of code generates an integer (between 0 and 100) and uses that as the value the device service sends to EdgeX -- mimicking the collection of data from a real device. It is here that the device service would normally capture some sensor reading from a device and send the data to EdgeX. The HandleReadCommands is where you'd need to do some customization work to talk to the device, get the latest sensor values and send them into EdgeX. Save the simpledriver.go file Creating your Device Profile A device profile is a YAML file that describes a class of device to EdgeX. General characteristics about the type of device, the data these devices provide, and how to command the device are all in a device profile. The device profile tells the device service what data gets collected from the the device and how to get it. Follow these steps to create a device profile for the simple random number generating device service. Explore the files in the cmd/device-simple/res/profiles folder. Note the example Simple-Driver.yaml device profile that is already in this folder. Open the file with your favorite editor and explore its contents. Note how deviceResources in the file represent properties of a device (properties like SwitchButton, X, Y and Z rotation). A pre-created device profile for the random number device is provided in this documentation. Download random-generator-device.yaml and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res/profiles folder. Open the random-generator-device.yaml file in a text editor. In this device profile, the device described has a deviceResource: RandomNumber . Note how the association of a type to the deviceResource. In this case, the device profile informs EdgeX that RandomNumber will be a INT32. In real world IoT situations, this deviceResource list could be extensive. Rather than a single deviceResource, you might find this section filled with many deviceResources and each deviceResource associated to a different type. Creating your Device Device Service accepts pre-defined devices to be added to EdgeX during device service startup. Follow these steps to create a pre-defined device for the simple random number generating device service. Explore the files in the cmd/device-simple/res/devices folder. Note the example simple-device.toml that is already in this folder. Open the file with your favorite editor and explore its contents. Note how DeviceList in the file represent an actual device with its properties (properties like Name, ProfileName, AutoEvents). A pre-created device for the random number device is provided in this documentation. Download random-generator-device.toml and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res/devices folder. Open the random-generator-device.toml file in a text editor. In this example, the device described has a ProfileName: RandNum-Device . In this case, the device informs EdgeX that it will be using the device profile we created in Creating your Device Profile Validating your Device Go Device Services provide /api/v2/validate/device API to validate device's ProtocolProperties. This feature allows Device Services whose protocol has strict rule to validate their devices before adding them into EdgeX. Go SDK provides DeviceValidator interface: // DeviceValidator is a low-level device-specific interface implemented // by device services that validate device's protocol properties. type DeviceValidator interface { // ValidateDevice triggers device's protocol properties validation, returns error // if validation failed and the incoming device will not be added into EdgeX. ValidateDevice ( device models . Device ) error } By implementing DeviceValidator interface whenever a device is added or updated, ValidateDevice function will be called to validate incoming device's ProtocolProperties and reject the request if validation failed. Configuring your Device Service Now update the configuration for the new device service. This documentation provides a new configuration.toml file. This configuration file: changes the port the service operates on so as not to conflict with other device services Download configuration.toml and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res folder (overwrite the existing configuration file). Change the host address of the device service to your system's IP address. Warning In the configuration.toml, change the host address (around line 14) to the IP address of the system host. This allows core metadata to callback to your new device service when a new device is created. Because the rest of EdgeX, to include core metadata, will be running in Docker, the IP address of the host system on the Docker network must be provided to allow metadata in Docker to call out from Docker to the new device service running on your host system. Custom Structured Configuration EdgeX 2.0 New for EdgeX 2.0 Go Device Services can now define their own custom structured configuration section in the configuration.toml file. Any additional sections in the TOML are ignored by the SDK when it parses the file for the SDK defined sections. This feature allows a Device Service to define and watch it's own structured section in the service's TOML configuration file. The SDK API provides the follow APIs to enable structured custom configuration: LoadCustomConfig(config UpdatableConfig, sectionName string) error Loads the service's custom configuration from local file or the Configuration Provider (if enabled). The Configuration Provider will also be seeded with the custom configuration the first time the service is started, if service is using the Configuration Provider. The UpdateFromRaw interface will be called on the custom configuration when the configuration is loaded from the Configuration Provider. ListenForCustomConfigChanges(configToWatch interface{}, sectionName string, changedCallback func(interface{})) error Starts a listener on the Configuration Provider for changes to the specified section of the custom configuration. When changes are received from the Configuration Provider the UpdateWritableFromRaw interface will be called on the custom configuration to apply the updates and then signal that the changes occurred via changedCallback. See the Device MQTT Service for an example of using the new Structured Custom Configuration capability. See here for defining the structured custom configuration See here for custom section on the configuration.toml file See here for loading, validating and watching the configuration Retrieving Secrets The Go Device SDK provides the SecretProvider.GetSecret() API to retrieve the Device Services secrets. See the Device MQTT Service for an example of using the SecretProvider.GetSecret() API. Note that this code implements a retry loop allowing time for the secret(s) to be push into the service's SecretStore via the /secret endpoint. See Storing Secrets section for more details. Rebuild your Device Service Just as you did in the Build your Device Service step above, build the device-simple service, which creates the executable program that is your device service. In a terminal window, make sure you are in the device-simple folder (the folder containing the Makefile). Build the service by issuing the following command: cd ~/edgexfoundry/device-simple make build If there are no errors, your service is created and put in the ~/edgexfoundry/device-simple/cmd/device-simple folder. Look for the device-simple executable in the folder. Run your Device Service Allow the newly created device service, which was formed out of the Device Service Go SDK, to create sensor-mimicking data that it then sends to EdgeX: Follow the Getting Started using Docker guide to start all of EdgeX. From the folder containing the docker-compose file, start EdgeX with the following call (we're using non-security EdgeX in this example): docker-compose -f docker-compose-no-secty.yml up -d In a terminal window, change directories to the device-simple's cmd/device-simple folder and run the new device-simple service. cd ~/edgexfoundry/device-simple/cmd/device-simple ./device-simple This starts the service and immediately displays log entries in the terminal. Using a browser, enter the following URL to see the event/reading data that the service is generating and sending to EdgeX: http://localhost:59880/api/v2/event/device/name/RandNum-Device01 This request asks core data to provide the events associated to the RandNum-Device-01.","title":"Golang SDK"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#golang-sdk","text":"In this guide, you create a simple device service that generates a random number as a means to simulate getting data from an actual device. In this way, you explore some SDK framework and work necessary to complete a device service without actually having a device to talk to.","title":"Golang SDK"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#install-dependencies","text":"See the Getting Started - Go Developers guide to install the necessary tools and infrastructure needed to develop a GoLang service.","title":"Install dependencies"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#get-the-edgex-device-sdk-for-go","text":"Follow these steps to create a folder on your file system, download the Device SDK , and get the GoLang device service SDK on your system. Create a collection of nested folders, ~/edgexfoundry on your file system. This folder will hold your new Device Service. In Linux, create a directory with a single mkdir command mkdir -p ~/edgexfoundry In a terminal window, change directories to the folder just created and pull down the SDK in Go with the commands as shown. cd ~/edgexfoundry git clone --depth 1 --branch v2.0.0 https://github.com/edgexfoundry/device-sdk-go.git Note The clone command above has you pull v2.0.0 of the Go SDK which is the version associated to Ireland. There are later releases of EdgeX, and it is always a good idea to pull and use the latest version associated with the major version of EdgeX you are using. You may want to check for the latest released version by going to https://github.com/edgexfoundry/device-sdk-go and look for the latest release. Create a folder that will hold the new device service. The name of the folder is also the name you want to give your new device service. Standard practice in EdgeX is to prefix the name of a device service with device- . In this example, the name 'device-simple' is used. mkdir -p ~/edgexfoundry/device-simple Copy the example code from device-sdk-go to device-simple : cd ~/edgexfoundry cp -rf ./device-sdk-go/example/* ./device-simple/ Copy Makefile to device-simple: cp ./device-sdk-go/Makefile ./device-simple Copy version.go to device-simple: cp ./device-sdk-go/version.go ./device-simple/ After completing these steps, your device-simple folder should look like the listing below.","title":"Get the EdgeX Device SDK for Go"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#start-a-new-device-service","text":"With the device service application structure in place, time now to program the service to act like a sensor data fetching service. Change folders to the device-simple directory. cd ~/edgexfoundry/device-simple Open main.go file in the cmd/device-simple folder with your favorite text editor. Modify the import statements. Replace github.com/edgexfoundry/device-sdk-go/v2/example/driver with github.com/edgexfoundry/device-simple/driver in the import statements. Also replace github.com/edgexfoundry/device-sdk-go/v2 with github.com/edgexfoundry/device-simple . Save the file when you have finished editing. Open Makefile found in the base folder (~/edgexfoundry/device-simple) in your favorite text editor and make the following changes Replace: MICROSERVICES=example/cmd/device-simple/device-simple with: MICROSERVICES=cmd/device-simple/device-simple Change: GOFLAGS = -ldflags \"-X github.com/edgexfoundry/device-sdk-go/v2.Version= $( VERSION ) \" to refer to the new service with: GOFLAGS = -ldflags \"-X github.com/edgexfoundry/device-simple.Version= $( VERSION ) \" Change: example/cmd/device-simple/device-simple : go mod tidy $( GOCGO ) build $( GOFLAGS ) -o $@ ./example/cmd/device-simple to: cmd/device-simple/device-simple : go mod tidy $( GOCGO ) build $( GOFLAGS ) -o $@ ./cmd/device-simple Save the file. Enter the following command to create the initial module definition and write it to the go.mod file: GO111MODULE = on go mod init github . com / edgexfoundry / device - simple Use an editor to open and edit the go.mod file created in ~/edgexfoundry/device-simple. Add the code highlighted below to the bottom of the file. This code indicates which version of the device service SDK and the associated EdgeX contracts module to use. require ( github . com / edgexfoundry / device - sdk - go / v2 v2 .0.0 github . com / edgexfoundry / go - mod - core - contracts / v2 v2 .0.0 ) Note You should always check the go.mod file in the latest released version SDK for the correct versions of the Go SDK and go-mod-contracts to use in your go.mod.","title":"Start a new Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#build-your-device-service","text":"To ensure that the code you have moved and updated still works, build the device service. In a terminal window, make sure you are still in the device-simple folder (the folder containing the Makefile). Build the service by issuing the following command: make build If there are no errors, your service is ready for you to add custom code to generate data values as if there was a sensor attached.","title":"Build your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#customize-your-device-service","text":"The device service you are creating isn't going to talk to a real device. Instead, it is going to generate a random number where the service would ordinarily make a call to get sensor data from the actual device. Locate the simpledriver.go file in the /driver folder and open it with your favorite editor. In the import() area at the top of the file, add \"math/rand\" under \"time\". Locate the HandleReadCommands() function in this same file (simpledriver.go). Find the following lines of code in this file (around line 139): if reqs [ 0 ]. DeviceResourceName == \"SwitchButton\" { cv , _ := sdkModels . NewCommandValue ( reqs [ 0 ]. DeviceResourceName , common . ValueTypeBool , s . switchButton ) res [ 0 ] = cv } Add the conditional (if-else) code in front of the above conditional: if reqs [ 0 ]. DeviceResourceName == \"randomnumber\" { cv , _ := sdkModels . NewCommandValue ( reqs [ 0 ]. DeviceResourceName , common . ValueTypeInt32 , int32 ( rand . Intn ( 100 ))) res [ 0 ] = cv } else The first line of code checks that the current request is for a resource called \"RandomNumber\". The second line of code generates an integer (between 0 and 100) and uses that as the value the device service sends to EdgeX -- mimicking the collection of data from a real device. It is here that the device service would normally capture some sensor reading from a device and send the data to EdgeX. The HandleReadCommands is where you'd need to do some customization work to talk to the device, get the latest sensor values and send them into EdgeX. Save the simpledriver.go file","title":"Customize your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#creating-your-device-profile","text":"A device profile is a YAML file that describes a class of device to EdgeX. General characteristics about the type of device, the data these devices provide, and how to command the device are all in a device profile. The device profile tells the device service what data gets collected from the the device and how to get it. Follow these steps to create a device profile for the simple random number generating device service. Explore the files in the cmd/device-simple/res/profiles folder. Note the example Simple-Driver.yaml device profile that is already in this folder. Open the file with your favorite editor and explore its contents. Note how deviceResources in the file represent properties of a device (properties like SwitchButton, X, Y and Z rotation). A pre-created device profile for the random number device is provided in this documentation. Download random-generator-device.yaml and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res/profiles folder. Open the random-generator-device.yaml file in a text editor. In this device profile, the device described has a deviceResource: RandomNumber . Note how the association of a type to the deviceResource. In this case, the device profile informs EdgeX that RandomNumber will be a INT32. In real world IoT situations, this deviceResource list could be extensive. Rather than a single deviceResource, you might find this section filled with many deviceResources and each deviceResource associated to a different type.","title":"Creating your Device Profile"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#creating-your-device","text":"Device Service accepts pre-defined devices to be added to EdgeX during device service startup. Follow these steps to create a pre-defined device for the simple random number generating device service. Explore the files in the cmd/device-simple/res/devices folder. Note the example simple-device.toml that is already in this folder. Open the file with your favorite editor and explore its contents. Note how DeviceList in the file represent an actual device with its properties (properties like Name, ProfileName, AutoEvents). A pre-created device for the random number device is provided in this documentation. Download random-generator-device.toml and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res/devices folder. Open the random-generator-device.toml file in a text editor. In this example, the device described has a ProfileName: RandNum-Device . In this case, the device informs EdgeX that it will be using the device profile we created in Creating your Device Profile","title":"Creating your Device"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#validating-your-device","text":"Go Device Services provide /api/v2/validate/device API to validate device's ProtocolProperties. This feature allows Device Services whose protocol has strict rule to validate their devices before adding them into EdgeX. Go SDK provides DeviceValidator interface: // DeviceValidator is a low-level device-specific interface implemented // by device services that validate device's protocol properties. type DeviceValidator interface { // ValidateDevice triggers device's protocol properties validation, returns error // if validation failed and the incoming device will not be added into EdgeX. ValidateDevice ( device models . Device ) error } By implementing DeviceValidator interface whenever a device is added or updated, ValidateDevice function will be called to validate incoming device's ProtocolProperties and reject the request if validation failed.","title":"Validating your Device"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#configuring-your-device-service","text":"Now update the configuration for the new device service. This documentation provides a new configuration.toml file. This configuration file: changes the port the service operates on so as not to conflict with other device services Download configuration.toml and save the file to the ~/edgexfoundry/device-simple/cmd/device-simple/res folder (overwrite the existing configuration file). Change the host address of the device service to your system's IP address. Warning In the configuration.toml, change the host address (around line 14) to the IP address of the system host. This allows core metadata to callback to your new device service when a new device is created. Because the rest of EdgeX, to include core metadata, will be running in Docker, the IP address of the host system on the Docker network must be provided to allow metadata in Docker to call out from Docker to the new device service running on your host system.","title":"Configuring your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#custom-structured-configuration","text":"EdgeX 2.0 New for EdgeX 2.0 Go Device Services can now define their own custom structured configuration section in the configuration.toml file. Any additional sections in the TOML are ignored by the SDK when it parses the file for the SDK defined sections. This feature allows a Device Service to define and watch it's own structured section in the service's TOML configuration file. The SDK API provides the follow APIs to enable structured custom configuration: LoadCustomConfig(config UpdatableConfig, sectionName string) error Loads the service's custom configuration from local file or the Configuration Provider (if enabled). The Configuration Provider will also be seeded with the custom configuration the first time the service is started, if service is using the Configuration Provider. The UpdateFromRaw interface will be called on the custom configuration when the configuration is loaded from the Configuration Provider. ListenForCustomConfigChanges(configToWatch interface{}, sectionName string, changedCallback func(interface{})) error Starts a listener on the Configuration Provider for changes to the specified section of the custom configuration. When changes are received from the Configuration Provider the UpdateWritableFromRaw interface will be called on the custom configuration to apply the updates and then signal that the changes occurred via changedCallback. See the Device MQTT Service for an example of using the new Structured Custom Configuration capability. See here for defining the structured custom configuration See here for custom section on the configuration.toml file See here for loading, validating and watching the configuration","title":"Custom Structured Configuration"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#retrieving-secrets","text":"The Go Device SDK provides the SecretProvider.GetSecret() API to retrieve the Device Services secrets. See the Device MQTT Service for an example of using the SecretProvider.GetSecret() API. Note that this code implements a retry loop allowing time for the secret(s) to be push into the service's SecretStore via the /secret endpoint. See Storing Secrets section for more details.","title":"Retrieving Secrets"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#rebuild-your-device-service","text":"Just as you did in the Build your Device Service step above, build the device-simple service, which creates the executable program that is your device service. In a terminal window, make sure you are in the device-simple folder (the folder containing the Makefile). Build the service by issuing the following command: cd ~/edgexfoundry/device-simple make build If there are no errors, your service is created and put in the ~/edgexfoundry/device-simple/cmd/device-simple folder. Look for the device-simple executable in the folder.","title":"Rebuild your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK-Go/#run-your-device-service","text":"Allow the newly created device service, which was formed out of the Device Service Go SDK, to create sensor-mimicking data that it then sends to EdgeX: Follow the Getting Started using Docker guide to start all of EdgeX. From the folder containing the docker-compose file, start EdgeX with the following call (we're using non-security EdgeX in this example): docker-compose -f docker-compose-no-secty.yml up -d In a terminal window, change directories to the device-simple's cmd/device-simple folder and run the new device-simple service. cd ~/edgexfoundry/device-simple/cmd/device-simple ./device-simple This starts the service and immediately displays log entries in the terminal. Using a browser, enter the following URL to see the event/reading data that the service is generating and sending to EdgeX: http://localhost:59880/api/v2/event/device/name/RandNum-Device01 This request asks core data to provide the events associated to the RandNum-Device-01.","title":"Run your Device Service"},{"location":"getting-started/Ch-GettingStartedSDK/","text":"Device Service SDK The EdgeX device service software development kits (SDKs) help developers create new device connectors for EdgeX. An SDK provides the common scaffolding that each device service needs. This allows developers to create new device/sensor connectors more quickly. The EdgeX community already provides many device services. However, there is no way the community can provide for every protocol and every sensor. Even if the EdgeX community provided a device service for every protocol, your use case, sensor, or security infrastructure might require customization. Thus, the device service SDKs provide the means to extend or customize EdgeX\u2019s device connectivity. EdgeX provides two SDKs to help developers create new device services. Most of EdgeX is written in Go and C. Thus, there's a device service SDK written in both Go and C to support the more popular languages used in EdgeX today. In the future, the community may offer alternate language SDKs. The SDKs are libraries that get incorporated into a new micro services. They make writing a new device service much easier. By importing the SDK library into your new device service project, developers are left to focus on the code that is specific to the communications with the device via the protocol of the device. The code in the SDK handles the other details, such as: - initialization of the device service - getting the service configured - sending sensor data to core data - managing communications with core metadata - and much more. The code in the SDK also helps to ensure your device service adheres to rules and standards of EdgeX. For example, it makes sure the service registers with the EdgeX registry service when it starts. Use the GoLang SDK Use the C SDK","title":"Device Service SDK"},{"location":"getting-started/Ch-GettingStartedSDK/#device-service-sdk","text":"The EdgeX device service software development kits (SDKs) help developers create new device connectors for EdgeX. An SDK provides the common scaffolding that each device service needs. This allows developers to create new device/sensor connectors more quickly. The EdgeX community already provides many device services. However, there is no way the community can provide for every protocol and every sensor. Even if the EdgeX community provided a device service for every protocol, your use case, sensor, or security infrastructure might require customization. Thus, the device service SDKs provide the means to extend or customize EdgeX\u2019s device connectivity. EdgeX provides two SDKs to help developers create new device services. Most of EdgeX is written in Go and C. Thus, there's a device service SDK written in both Go and C to support the more popular languages used in EdgeX today. In the future, the community may offer alternate language SDKs. The SDKs are libraries that get incorporated into a new micro services. They make writing a new device service much easier. By importing the SDK library into your new device service project, developers are left to focus on the code that is specific to the communications with the device via the protocol of the device. The code in the SDK handles the other details, such as: - initialization of the device service - getting the service configured - sending sensor data to core data - managing communications with core metadata - and much more. The code in the SDK also helps to ensure your device service adheres to rules and standards of EdgeX. For example, it makes sure the service registers with the EdgeX registry service when it starts. Use the GoLang SDK Use the C SDK","title":"Device Service SDK"},{"location":"getting-started/Ch-GettingStartedSnapUsers/","text":"Getting Started using Snaps Introduction Snaps are application packages that are easy to install and update while being secure, cross\u2010platform and self-contained. Snaps can be installed on any Linux distribution with snap support . Installation When using the snap CLI, the installation is possible by simply executing: snap install This is similar to setting --channel=latest/stable or shorthand --stable and will install the latest stable release of a snap. In this case, latest/stable is the channel , composed of latest track and stable risk level. To install a specific version with long term support (e.g. 2.1), or to install a beta or development release, refer to the store page for the snap, choose install, and then pick the desired channel. The store page also provides instructions for installation on different Linux distributions as well as the list of supported CPU architectures. For the list of EdgeX snaps, please refer here . Configuration EdgeX snaps are packaged with default service configuration files. In certain cases, few configuration fields are overridden within the snap for snap-specific deployment requirements. Config files The default configuration files are typically placed at /var/snap//current/config . Upon startup, the server configurations files are uploaded to the registry by default. Once the service starts without errors, the local configurations become obsolete and will no longer be read. Any modifications after the initial startup will not be applied. For device services, the Device and Device Profile files are submitted to Core Metadata upon initial startup. Refer to the documentation of Device Services for details. Config registry The configurations that are uploaded to Consul can be modified using Consul's UI or kv REST API . Changes to configurations in Consul are loaded by the service at startup. If the service has already started, a restart is required to load new configurations. Configurations that are in the writable section get loaded not only at startup, but also during the runtime. In other words, changes to the writable configurations are loaded automatically without a restart. Please refer to Common Configuration and Configuration and Registry Providers for more information. Config provider snap Most EdgeX snaps have a content interface which allows another snap to seed the snap with configuration files. This is useful when replacing entire configuration files via another snap, packaged with the deployment-specific configurations. Please refer to edgex-config-provider , for an example. Config overrides EdgeX 2.2 - app options This version of EdgeX snaps introduce a new scheme for the snap configuration options: apps... where: is the name of the app (service, executable) is the type of option with respect to the app is key for the option. It could contain a path to set a value inside an object, e.g. x.y=z sets {\"x\": {\"y\": \"z\"}} . The app options are disabled by default . This is for usability purposes and to avoid confusion with legacy options. The app options may be enabled by default in the next major release of EdgeX. Refer to edgexfoundry/edgex-go#3986 for details. EdgeX 2.2 - config options The snaps now provide an interface to set any environment variable for supported apps. We call these the config options because they use a config prefix for the variable names. The config options are a subset of app options, which are disabled by default . This functionality supersedes for the snap env options (prefix env. ) which allow setting a pre-defined set of configurations. Existing env options will continue to work until the next major EdgeX release. Please refer to snap READMEs ( jakarta branch) for the documentation of the deprecated options. Enabling app options will migrate internal env options and persistently remove any newly set env option. The EdgeX services allow overriding server configurations using environment variables. Moreover, the services read EdgeX Common Environment Variables to change configurations that aren't defined in config files. The EdgeX snaps provide an interface via snap configuration options to set environment variables for the services. We call these the config options because they a have config prefix for the variable names. The snap options for setting environment variable uses the the following format: apps..config. : setting an app-specific value (e.g. apps.core-data.config.service-port=1000 ). config. : setting a global value (e.g. config.service-host=localhost or config.writable-loglevel=DEBUG where: is the name of the app (service, executable) is a lowercase, dash-separated mapping to uppercase, underscore-separate environment variable name (e.g. x-y -> X_Y ). The reason for such mapping is that uppercase and underscore characters are not supported as config keys for snaps. Mapping examples: Snap config key Environment Variable Service configuration TOML service-port SERVICE_PORT [Service] Port clients-core-data-host CLIENTS_CORE_DATA_HOST [Clients] --[Clients.core-data] --Host edgex-startup-duration EDGEX_STARTUP_DURATION - add-secretstore-tokens ADD_SECRETSTORE_TOKENS - Note The config options are supported as of EdgeX 2.2 and are disabled by default! Setting app-options=true is necessary to enable their support. Enabling app options will migrate internal env options and persistently remove any newly set env option. For example, to change the service port of the core-data service on edgexfoundry snap to 8080: snap set edgexfoundry app-options = true snap set edgexfoundry apps.core-data.config.service-port = 8080 The services load the set config options on startup. If the service has already started, a restart is necessary to load them. Disabling security Warning Disabling security is NOT recommended, unless for demonstration purposes, or when there are other means to secure the services. The snap will NOT allow the Secret Store to be re-enabled. The only way to re-enable the Secret Store is to re-install the snap. The Secret Store is used by EdgeX for secret management (e.g. certificates, keys, passwords). Use of the Secret Store by all services can be disabled globally. Note that doing so will also disable the API Gateway, as it depends on the Secret Store. The following command disables the Secret Store and in turn the API Gateway: sudo snap set edgexfoundry security-secret-store = off All services in the snap except for the API Gateway are restricted by default to listening on localhost (127.0.0.1). The API Gateway proxies external requests to internal services. Since disabling the Secret Store also disables the API Gateway, the service endpoint will no longer be accessible from other systems. They will be still accessible on the local machine for demonstration and testing. If you really need to make an insecure service accessible remotely, set the bind address of the service to the IP address of that networking interface on the local machine. If you trust all your interfaces and want the services to accept connections from all, set it to 0.0.0.0 . After disabling the Secret Store, the external services should be configured such that they don't attempt to initialize the security. For this purpose, EDGEX_SECURITY_SECRET_STORE should be set to false, using the corresponding snap option: config.edgex-security-secret-store . Managing services The services of a snap can be started/stopped/restarted using the snap CLI. When starting/stopping, you can additionally set them to enable/disable which configures whether or not the service should also start on boot. To list the services and check their status: snap services To start and optionally enable services: # all services snap start --enable # one service snap start --enable . Similarly, a service can be stopped and optionally disabled using snap stop --disable . Seeding custom service startup using snap options To spin up an EdgeX instance with a different startup configuration (e.g. enabled instead of disabled), the edgexfoundry snap provides the following config options that accept values \"on\" / \"off\" to enable/disable a service by default: consul redis core-metadata core-command core-data support-notifications support-scheduler device-virtual security-secret-store security-proxy Device and app service snaps provide a similar functionality using the auto-start option. This is particularly useful when seeding the snap from a Gadget on an Ubuntu Core system. To restart services, e.g. to load the configurations: # all services snap restart # one service snap restart . Debugging The service logs can be queried using the snap log command. For example, to query 100 lines and follow: # all services snap logs -n = 100 -f # one service snap logs -n = 100 -f . Check snap logs --help for details. To query not only the service logs, but also the snap logs (incl. hook apps such as install and configure), use journalctl : sudo journalctl -n 100 -f | grep Info The verbosity of service logs is INFO by default. This can be changed by overriding the log level using the WRITABLE_LOGLEVEL environment variable using snap config overrides apps..config.writable-loglevel or globally as config.writable-loglevel . EdgeX Snaps The following snaps are maintained by the EdgeX working groups. To find all EdgeX snaps on the public Snap Store, search by keyword . Platform Snap | Installation | Configuration | Managing Services | Debugging | Source | The main platform snap, simply called edgexfoundry contains all reference core services along with several other security, supporting, application, and device services. Upon installation, the following EdgeX services are automatically started: consul (Registry) core-command core-data core-metadata kong-daemon (API Gateway / Reverse Proxy) postgres (kong's database) redis (default Message Bus and database backend for core-data and core-metadata) security-bootstrapper-redis (oneshot service to setup secure Redis) security-consul-bootstrapper (oneshot service to setup secure Consul) security-proxy-setup (oneshot service to setup Kong) security-secretstore-setup (oneshot service to setup Vault) vault (Secret Store) The following services are disabled by default: support-notifications support-scheduler sys-mgmt-agent - deprecated EdgeX component device-virtual kuiper (Rules Engine / eKuiper) - deprecated; use the standalone EdgeX eKuiper snap app-service-configurable (used to filter events for kuiper) - deprecated; use the standalone App Service Configurable snap The disabled services can be manually enabled and started; see managing services . For the configuration of services, refer to configuration . Most services are exposed and accessible on localhost without access control. However, the access from outside is restricted to authorized users. Adding API Gateway users The service endpoints can be accessed securely through the API Gateway. The API Gateway requires a JSON Web Token (JWT) to authorize requests. Please refer to Adding EdgeX API Gateway Users Remotely and use the snapped edgexfoundry.secrets-config utility. To get the usage help: edgexfoundry.secrets-config proxy adduser -h You may also refer to the secrets-config proxy documentation. Creating an example user Create private and public keys: openssl ecparam -genkey -name prime256v1 -noout -out private.pem openssl ec -in private.pem -pubout -out public.pem Read the API Gateway token: KONG_ADMIN_JWT = ` sudo cat /var/snap/edgexfoundry/current/secrets/security-proxy-setup/kong-admin-jwt ` Use secrets-config to add a user example with id 1000 : edgexfoundry.secrets-config proxy adduser --token-type jwt --user example --algorithm ES256 --public_key public.pem --id 1000 --jwt $KONG_ADMIN_JWT On success, the above command prints the user id. Seeding an admin user using snap options To spin up a pre-configured and securely accessible EdgeX instance, the snap provides a way to pass the public key of a single user with snap options. When requested, the user is created with user admin , id 1 and JWT signing algorithm ES256 . The snap option for passing the public key is: apps.secrets-config.proxy.admin.public-key . This is particularly useful when seeding the snap from a Gadget on an Ubuntu Core system. Generating a JWT token for the example user On success, a JWT token is printed out and written to user-jwt.txt file. We use the user id 1000 as set in the previous example. edgexfoundry.secrets-config proxy jwt --algorithm ES256 --private_key private.pem --id 1000 --expiration = 1h | tee user-jwt.txt It is also possible to create the JWT token using bash and openssl. But that is beyond the scope of this guide. Once you have the token, you can access the services via the API Gateway. Calling an API on behalf of example user curl --insecure https://localhost:8443/core-data/api/v2/ping -H \"Authorization: Bearer $( cat user-jwt.txt ) \" Output: {\"apiVersion\":\"v2\",\"timestamp\":\"Mon May 2 12:14:17 CEST 2022\",\"serviceName\":\"core-data\"} Accessing Consul Consul API and UI can be accessed using the consul token (Secret ID). For the snap, token is the value of SecretID typically placed in a JSON file at /var/snap/edgexfoundry/current/secrets/consul-acl-token/bootstrap_token.json . Example To get the token: $ sudo cat /var/snap/edgexfoundry/current/secrets/consul-acl-token/bootstrap_token.json | jq -r '.SecretID' | tee consul-token.txt The output is printed out and written to consul-token.txt . Example output: ee3964d0-505f-6b62-4c88-0d29a8226daa Try it out locally: curl --insecure --silent http://localhost:8500/v1/kv/edgex/core/2.0/core-data/Service/Port -H \"X-Consul-Token: $( cat consul-token.txt ) \" Through the API Gateway: We need to pass both the Consul token and Secret Store token obtained in Adding API Gateway users examples. curl --insecure --silent https://localhost:8443/consul/v1/kv/edgex/core/2.0/core-data/Service/Port -H \"X-Consul-Token: $( cat consul-token.txt ) \" -H \"Authorization: Bearer $( cat user-jwt.txt ) \" Changing TLS certificates The API Gateway setup generates a self-signed certificate by default. To replace that with your own certificate, refer to API Gateway guide: Using a bring-your-own external TLS certificate for API gateway and use the snapped edgexfoundry.secrets-config utility. To get the usage help: edgexfoundry.secrets-config proxy tls -h You may also refer to the secrets-config proxy documentation. Example Given the following files created outside the scope of this document: cert.pem certificate privkey.pem private key ca.pem certificate authority file (if not available in root certificates) Read the API Gateway token: KONG_ADMIN_JWT = ` sudo cat /var/snap/edgexfoundry/current/secrets/security-proxy-setup/kong-admin-jwt ` Add the certificate, using Kong Admin JWT to authenticate: edgexfoundry.secrets-config proxy tls --incert cert.pem --inkey privkey.pem --admin_api_jwt $KONG_ADMIN_JWT Try it out: curl --cacert ca.pem https://localhost:8443/core-data/api/v2/ping Output: {\"message\":\"Unauthorized\"} This means that TLS is setup correctly, but the request is not authorized. Set the -v command for diagnosing TLS issues. The --cacert can be omitted if the CA is available in root certificates (e.g. CA-signed or pre-installed CA certificate). Seeding a custom TLS certificate using snap options To spin up an EdgeX instance with a custom certificate, the snap provides the following configuration options: apps.secrets-config.proxy.tls.cert apps.secrets-config.proxy.tls.key apps.secrets-config.proxy.tls.snis (comma-separated values) This is particularly useful when seeding the snap from a Gadget on an Ubuntu Core system. Secret Store token The services inside standalone snaps (e.g. device, app snaps) automatically receive a Secret Store token when: The standalone snap is downloaded and installed from the store The platform snap is downloaded and installed from the store Both snaps are installed on the same machine The service is registered as an add-on service The edgex-secretstore-token content interface provides the mechanism to automatically supply tokens to connected snaps. Execute the following command to check the status of connections: sudo snap connections edgexfoundry To manually connect the edgexfoundry's plug to a standalone snap's slot: snap connect edgexfoundry:edgex-secretstore-token :edgex-secretstore-token Note that the token has a limited expiry time of 1h by default. The connection and service startup should happen within the validity period. To better understand the snap connections, read the interface management Extend the default Secret Store token TTL Set TOKENFILEPROVIDER_DEFAULTTOKENTTL for security-secretstore-setup and restart: sudo snap set edgexfoundry app-options = true sudo snap set edgexfoundry apps.security-secretstore-setup.config.tokenfileprovider-defaulttokenttl = 72h sudo snap restart edgexfoundry.security-secretstore-setup EdgeX UI | Installation | Managing Services | Debugging | Source | For usage instructions, please refer to the Graphical User Interface (GUI) guide. By default, the UI is reachable locally at: http://localhost:4000 A valid JWT token is required to access the UI; follow Adding API Gateway users steps to generate a token. To enable all the functionalities of the UI, the following support services should be running: Support Scheduler Support Notifications System Management Agent EdgeX eKuiper For example, to start/install all: sudo snap start edgexfoundry.support-scheduler sudo snap start edgexfoundry.support-notifications sudo snap start edgexfoundry.sys-mgmt-agent sudo snap install edgex-ekuiper EdgeX CLI | Installation | Source | For usage instructions, refer to Command Line Interface (CLI) guide. EdgeX eKuiper | Installation | Managing Services | Debugging | Source | EdgeX 2.2 This version of EdgeX introduces a standalone EdgeX eKuiper snap. The new snap is the supported way of using eKuiper with other EdgeX snaps. For the documentation of the standalone EdgeX eKuiper snap, visit the README . Note The standalone EdgeX eKuiper snap documented here should not be confused with the deprecated edgexfoundry.kuiper and edgexfoundry.kuiper-cli apps built into the platform. The standalone snap can provide similar functionality. App Service Configurable | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-app-service-configurable/current/config/ \u2514\u2500\u2500 res \u251c\u2500\u2500 external-mqtt-trigger \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 functional-tests \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 http-export \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 metrics-influxdb \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 mqtt-export \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 push-to-core \u2502 \u2514\u2500\u2500 configuration.toml \u2514\u2500\u2500 rules-engine \u2514\u2500\u2500 configuration.toml Please refer to App Service Configurable guide for detailed usage instructions. Profile Before you can start the service, you must select one of available profiles, using snap options. For example, to set mqtt-export profile using the snap CLI: sudo snap set edgex-app-service-configurable profile = mqtt-export Note The standalone App Service Configurable snap documented above should not be confused with the deprecated edgexfoundry.app-service-configurable , built into the platform snap. The standalone snap can serve the same functionality of filtering events for eKuiper by using the rules-engine profile. App RFID LLRP Inventory | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-app-rfid-llrp-inventory/current/config/ \u2514\u2500\u2500 app-rfid-llrp-inventory \u2514\u2500\u2500 res \u2514\u2500\u2500 configuration.toml Aliases The aliases need to be provided for the service to work. See Setting the Aliases . For the snap, this can either be by: using a config-provider-snap to provide a configuration.toml file with the correct aliases, before startup setting the values manually in Consul during or after deployment Device Camera | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-camera/current/config/ \u2514\u2500\u2500 device-camera \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 camera.toml \u2514\u2500\u2500 profiles \u251c\u2500\u2500 camera-axis.yaml \u251c\u2500\u2500 camera-bosch.yaml \u2514\u2500\u2500 camera.yaml Device GPIO | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-gpio/current/config \u2514\u2500\u2500 device-gpio \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 device.custom.gpio.toml \u2514\u2500\u2500 profiles \u2514\u2500\u2500 device.custom.gpio.yaml GPIO Access This snap is strictly confined which means that the access to interfaces are subject to various security measures. On a Linux distribution without snap confinement for GPIO (e.g. Raspberry Pi OS 11), the snap may be able to access the GPIO directly, without any snap interface and manual connections. On Linux distributions with snap confinement for GPIO such as Ubuntu Core, the GPIO access is possible via the gpio interface , provided by a gadget snap. The official Raspberry Pi Ubuntu Core image includes that gadget. It is NOT possible to use this snap on Linux distributions that have the GPIO confinement but not the interface (e.g. Ubuntu Server 20.04), unless for development purposes. In development environments, it is possible to install the snap in dev mode (using --devmode flag which disables security confinement and automatic upgrades) to allow direct GPIO access. The gpio interface provides slots for each GPIO channel. The slots can be listed using: $ sudo snap interface gpio name: gpio summary: allows access to specific GPIO pin plugs: - edgex-device-gpio slots: - pi:bcm-gpio-0 - pi:bcm-gpio-1 - pi:bcm-gpio-10 ... The slots are not connected automatically. For example, to connect GPIO-17: $ sudo snap connect edgex-device-gpio:gpio pi:bcm-gpio-17 Check the list of connections: $ sudo snap connections Interface Plug Slot Notes gpio edgex-device-gpio:gpio pi:bcm-gpio-17 manual \u2026 Device Grove | Installation | Source | beta Device Grove snap is released as beta for arm64 . It is compatible with EdgeX 1.3 only. It does not support the snap configurations described above. The default configuration files are under /var/snap/edgex-device-grove/current/config/ . This device service is started by default. Changes to the configuration files require a restart to take effect: sudo snap restart edgex-device-grove Device Modbus | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-modbus/current/config/ \u2514\u2500\u2500 device-modbus \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 modbus.test.devices.toml \u2514\u2500\u2500 profiles \u2514\u2500\u2500 modbus.test.device.profile.yml Device MQTT | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-mqtt/current/config/ \u2514\u2500\u2500 device-mqtt \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 mqtt.test.device.toml \u2514\u2500\u2500 profiles \u2514\u2500\u2500 mqtt.test.device.profile.yml Device REST | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-rest/current/config/ \u2514\u2500\u2500 device-rest \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 sample-devices.toml \u2514\u2500\u2500 profiles \u251c\u2500\u2500 sample-image-device.yaml \u251c\u2500\u2500 sample-json-device.yaml \u2514\u2500\u2500 sample-numeric-device.yaml Device RFID LLRP | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-rfid-llrp/current/config/ \u2514\u2500\u2500 device-rfid-llrp \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u251c\u2500\u2500 profiles \u2502 \u251c\u2500\u2500 llrp.device.profile.yaml \u2502 \u2514\u2500\u2500 llrp.impinj.profile.yaml \u2514\u2500\u2500 provision_watchers \u251c\u2500\u2500 impinj.provision.watcher.json \u2514\u2500\u2500 llrp.provision.watcher.json Subnet setup The DiscoverySubnets setting needs to be provided before a device discovery can occur. This can be done in a number of ways: Using snap set to set your local subnet information. Example: sudo snap set edgex-device-rfid-llrp apps.device-rfid-llrp.config.app-custom.discovery-subnets = \"192.168.10.0/24\" curl -X POST http://localhost:59989/api/v2/discovery Using a config-provider-snap to set device configuration Using the auto-configure command. This command finds all local network interfaces which are online and non-virtual and sets the value of DiscoverySubnets in Consul. When running with security enabled, it requires a Consul token, so it needs to be run as follows: # get Consul ACL token CONSUL_TOKEN = $( sudo cat /var/snap/edgexfoundry/current/secrets/consul-acl-token/bootstrap_token.json | jq \".SecretID\" | tr -d '\"' ) echo $CONSUL_TOKEN # start the device service and connect the interfaces required for network interface discovery sudo snap start edgex-device-rfid-llrp.device-rfid-llrp sudo snap connect edgex-device-rfid-llrp:network-control sudo snap connect edgex-device-rfid-llrp:network-observe # run the nework interface discovery, providing the Consul token edgex-device-rfid-llrp.auto-configure $CONSUL_TOKEN Device SNMP | Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-snmp/current/config/ \u2514\u2500\u2500 device-snmp \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 device.snmp.trendnet.TPE082WS.toml \u2514\u2500\u2500 profiles \u251c\u2500\u2500 device.snmp.patlite.yaml \u251c\u2500\u2500 device.snmp.switch.dell.N1108P-ON.yaml \u2514\u2500\u2500 device.snmp.trendnet.TPE082WS.yaml","title":"Getting Started using Snaps"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#getting-started-using-snaps","text":"","title":"Getting Started using Snaps"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#introduction","text":"Snaps are application packages that are easy to install and update while being secure, cross\u2010platform and self-contained. Snaps can be installed on any Linux distribution with snap support .","title":"Introduction"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#installation","text":"When using the snap CLI, the installation is possible by simply executing: snap install This is similar to setting --channel=latest/stable or shorthand --stable and will install the latest stable release of a snap. In this case, latest/stable is the channel , composed of latest track and stable risk level. To install a specific version with long term support (e.g. 2.1), or to install a beta or development release, refer to the store page for the snap, choose install, and then pick the desired channel. The store page also provides instructions for installation on different Linux distributions as well as the list of supported CPU architectures. For the list of EdgeX snaps, please refer here .","title":"Installation"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#configuration","text":"EdgeX snaps are packaged with default service configuration files. In certain cases, few configuration fields are overridden within the snap for snap-specific deployment requirements.","title":"Configuration"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#config-files","text":"The default configuration files are typically placed at /var/snap//current/config . Upon startup, the server configurations files are uploaded to the registry by default. Once the service starts without errors, the local configurations become obsolete and will no longer be read. Any modifications after the initial startup will not be applied. For device services, the Device and Device Profile files are submitted to Core Metadata upon initial startup. Refer to the documentation of Device Services for details.","title":"Config files"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#config-registry","text":"The configurations that are uploaded to Consul can be modified using Consul's UI or kv REST API . Changes to configurations in Consul are loaded by the service at startup. If the service has already started, a restart is required to load new configurations. Configurations that are in the writable section get loaded not only at startup, but also during the runtime. In other words, changes to the writable configurations are loaded automatically without a restart. Please refer to Common Configuration and Configuration and Registry Providers for more information.","title":"Config registry"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#config-provider-snap","text":"Most EdgeX snaps have a content interface which allows another snap to seed the snap with configuration files. This is useful when replacing entire configuration files via another snap, packaged with the deployment-specific configurations. Please refer to edgex-config-provider , for an example.","title":"Config provider snap"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#config-overrides","text":"EdgeX 2.2 - app options This version of EdgeX snaps introduce a new scheme for the snap configuration options: apps... where: is the name of the app (service, executable) is the type of option with respect to the app is key for the option. It could contain a path to set a value inside an object, e.g. x.y=z sets {\"x\": {\"y\": \"z\"}} . The app options are disabled by default . This is for usability purposes and to avoid confusion with legacy options. The app options may be enabled by default in the next major release of EdgeX. Refer to edgexfoundry/edgex-go#3986 for details. EdgeX 2.2 - config options The snaps now provide an interface to set any environment variable for supported apps. We call these the config options because they use a config prefix for the variable names. The config options are a subset of app options, which are disabled by default . This functionality supersedes for the snap env options (prefix env. ) which allow setting a pre-defined set of configurations. Existing env options will continue to work until the next major EdgeX release. Please refer to snap READMEs ( jakarta branch) for the documentation of the deprecated options. Enabling app options will migrate internal env options and persistently remove any newly set env option. The EdgeX services allow overriding server configurations using environment variables. Moreover, the services read EdgeX Common Environment Variables to change configurations that aren't defined in config files. The EdgeX snaps provide an interface via snap configuration options to set environment variables for the services. We call these the config options because they a have config prefix for the variable names. The snap options for setting environment variable uses the the following format: apps..config. : setting an app-specific value (e.g. apps.core-data.config.service-port=1000 ). config. : setting a global value (e.g. config.service-host=localhost or config.writable-loglevel=DEBUG where: is the name of the app (service, executable) is a lowercase, dash-separated mapping to uppercase, underscore-separate environment variable name (e.g. x-y -> X_Y ). The reason for such mapping is that uppercase and underscore characters are not supported as config keys for snaps. Mapping examples: Snap config key Environment Variable Service configuration TOML service-port SERVICE_PORT [Service] Port clients-core-data-host CLIENTS_CORE_DATA_HOST [Clients] --[Clients.core-data] --Host edgex-startup-duration EDGEX_STARTUP_DURATION - add-secretstore-tokens ADD_SECRETSTORE_TOKENS - Note The config options are supported as of EdgeX 2.2 and are disabled by default! Setting app-options=true is necessary to enable their support. Enabling app options will migrate internal env options and persistently remove any newly set env option. For example, to change the service port of the core-data service on edgexfoundry snap to 8080: snap set edgexfoundry app-options = true snap set edgexfoundry apps.core-data.config.service-port = 8080 The services load the set config options on startup. If the service has already started, a restart is necessary to load them.","title":"Config overrides"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#disabling-security","text":"Warning Disabling security is NOT recommended, unless for demonstration purposes, or when there are other means to secure the services. The snap will NOT allow the Secret Store to be re-enabled. The only way to re-enable the Secret Store is to re-install the snap. The Secret Store is used by EdgeX for secret management (e.g. certificates, keys, passwords). Use of the Secret Store by all services can be disabled globally. Note that doing so will also disable the API Gateway, as it depends on the Secret Store. The following command disables the Secret Store and in turn the API Gateway: sudo snap set edgexfoundry security-secret-store = off All services in the snap except for the API Gateway are restricted by default to listening on localhost (127.0.0.1). The API Gateway proxies external requests to internal services. Since disabling the Secret Store also disables the API Gateway, the service endpoint will no longer be accessible from other systems. They will be still accessible on the local machine for demonstration and testing. If you really need to make an insecure service accessible remotely, set the bind address of the service to the IP address of that networking interface on the local machine. If you trust all your interfaces and want the services to accept connections from all, set it to 0.0.0.0 . After disabling the Secret Store, the external services should be configured such that they don't attempt to initialize the security. For this purpose, EDGEX_SECURITY_SECRET_STORE should be set to false, using the corresponding snap option: config.edgex-security-secret-store .","title":"Disabling security"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#managing-services","text":"The services of a snap can be started/stopped/restarted using the snap CLI. When starting/stopping, you can additionally set them to enable/disable which configures whether or not the service should also start on boot. To list the services and check their status: snap services To start and optionally enable services: # all services snap start --enable # one service snap start --enable . Similarly, a service can be stopped and optionally disabled using snap stop --disable . Seeding custom service startup using snap options To spin up an EdgeX instance with a different startup configuration (e.g. enabled instead of disabled), the edgexfoundry snap provides the following config options that accept values \"on\" / \"off\" to enable/disable a service by default: consul redis core-metadata core-command core-data support-notifications support-scheduler device-virtual security-secret-store security-proxy Device and app service snaps provide a similar functionality using the auto-start option. This is particularly useful when seeding the snap from a Gadget on an Ubuntu Core system. To restart services, e.g. to load the configurations: # all services snap restart # one service snap restart .","title":"Managing services"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#debugging","text":"The service logs can be queried using the snap log command. For example, to query 100 lines and follow: # all services snap logs -n = 100 -f # one service snap logs -n = 100 -f . Check snap logs --help for details. To query not only the service logs, but also the snap logs (incl. hook apps such as install and configure), use journalctl : sudo journalctl -n 100 -f | grep Info The verbosity of service logs is INFO by default. This can be changed by overriding the log level using the WRITABLE_LOGLEVEL environment variable using snap config overrides apps..config.writable-loglevel or globally as config.writable-loglevel .","title":"Debugging"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#edgex-snaps","text":"The following snaps are maintained by the EdgeX working groups. To find all EdgeX snaps on the public Snap Store, search by keyword .","title":"EdgeX Snaps"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#platform-snap","text":"| Installation | Configuration | Managing Services | Debugging | Source | The main platform snap, simply called edgexfoundry contains all reference core services along with several other security, supporting, application, and device services. Upon installation, the following EdgeX services are automatically started: consul (Registry) core-command core-data core-metadata kong-daemon (API Gateway / Reverse Proxy) postgres (kong's database) redis (default Message Bus and database backend for core-data and core-metadata) security-bootstrapper-redis (oneshot service to setup secure Redis) security-consul-bootstrapper (oneshot service to setup secure Consul) security-proxy-setup (oneshot service to setup Kong) security-secretstore-setup (oneshot service to setup Vault) vault (Secret Store) The following services are disabled by default: support-notifications support-scheduler sys-mgmt-agent - deprecated EdgeX component device-virtual kuiper (Rules Engine / eKuiper) - deprecated; use the standalone EdgeX eKuiper snap app-service-configurable (used to filter events for kuiper) - deprecated; use the standalone App Service Configurable snap The disabled services can be manually enabled and started; see managing services . For the configuration of services, refer to configuration . Most services are exposed and accessible on localhost without access control. However, the access from outside is restricted to authorized users.","title":"Platform Snap"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#adding-api-gateway-users","text":"The service endpoints can be accessed securely through the API Gateway. The API Gateway requires a JSON Web Token (JWT) to authorize requests. Please refer to Adding EdgeX API Gateway Users Remotely and use the snapped edgexfoundry.secrets-config utility. To get the usage help: edgexfoundry.secrets-config proxy adduser -h You may also refer to the secrets-config proxy documentation. Creating an example user Create private and public keys: openssl ecparam -genkey -name prime256v1 -noout -out private.pem openssl ec -in private.pem -pubout -out public.pem Read the API Gateway token: KONG_ADMIN_JWT = ` sudo cat /var/snap/edgexfoundry/current/secrets/security-proxy-setup/kong-admin-jwt ` Use secrets-config to add a user example with id 1000 : edgexfoundry.secrets-config proxy adduser --token-type jwt --user example --algorithm ES256 --public_key public.pem --id 1000 --jwt $KONG_ADMIN_JWT On success, the above command prints the user id. Seeding an admin user using snap options To spin up a pre-configured and securely accessible EdgeX instance, the snap provides a way to pass the public key of a single user with snap options. When requested, the user is created with user admin , id 1 and JWT signing algorithm ES256 . The snap option for passing the public key is: apps.secrets-config.proxy.admin.public-key . This is particularly useful when seeding the snap from a Gadget on an Ubuntu Core system. Generating a JWT token for the example user On success, a JWT token is printed out and written to user-jwt.txt file. We use the user id 1000 as set in the previous example. edgexfoundry.secrets-config proxy jwt --algorithm ES256 --private_key private.pem --id 1000 --expiration = 1h | tee user-jwt.txt It is also possible to create the JWT token using bash and openssl. But that is beyond the scope of this guide. Once you have the token, you can access the services via the API Gateway. Calling an API on behalf of example user curl --insecure https://localhost:8443/core-data/api/v2/ping -H \"Authorization: Bearer $( cat user-jwt.txt ) \" Output: {\"apiVersion\":\"v2\",\"timestamp\":\"Mon May 2 12:14:17 CEST 2022\",\"serviceName\":\"core-data\"}","title":"Adding API Gateway users"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#accessing-consul","text":"Consul API and UI can be accessed using the consul token (Secret ID). For the snap, token is the value of SecretID typically placed in a JSON file at /var/snap/edgexfoundry/current/secrets/consul-acl-token/bootstrap_token.json . Example To get the token: $ sudo cat /var/snap/edgexfoundry/current/secrets/consul-acl-token/bootstrap_token.json | jq -r '.SecretID' | tee consul-token.txt The output is printed out and written to consul-token.txt . Example output: ee3964d0-505f-6b62-4c88-0d29a8226daa Try it out locally: curl --insecure --silent http://localhost:8500/v1/kv/edgex/core/2.0/core-data/Service/Port -H \"X-Consul-Token: $( cat consul-token.txt ) \" Through the API Gateway: We need to pass both the Consul token and Secret Store token obtained in Adding API Gateway users examples. curl --insecure --silent https://localhost:8443/consul/v1/kv/edgex/core/2.0/core-data/Service/Port -H \"X-Consul-Token: $( cat consul-token.txt ) \" -H \"Authorization: Bearer $( cat user-jwt.txt ) \"","title":"Accessing Consul"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#changing-tls-certificates","text":"The API Gateway setup generates a self-signed certificate by default. To replace that with your own certificate, refer to API Gateway guide: Using a bring-your-own external TLS certificate for API gateway and use the snapped edgexfoundry.secrets-config utility. To get the usage help: edgexfoundry.secrets-config proxy tls -h You may also refer to the secrets-config proxy documentation. Example Given the following files created outside the scope of this document: cert.pem certificate privkey.pem private key ca.pem certificate authority file (if not available in root certificates) Read the API Gateway token: KONG_ADMIN_JWT = ` sudo cat /var/snap/edgexfoundry/current/secrets/security-proxy-setup/kong-admin-jwt ` Add the certificate, using Kong Admin JWT to authenticate: edgexfoundry.secrets-config proxy tls --incert cert.pem --inkey privkey.pem --admin_api_jwt $KONG_ADMIN_JWT Try it out: curl --cacert ca.pem https://localhost:8443/core-data/api/v2/ping Output: {\"message\":\"Unauthorized\"} This means that TLS is setup correctly, but the request is not authorized. Set the -v command for diagnosing TLS issues. The --cacert can be omitted if the CA is available in root certificates (e.g. CA-signed or pre-installed CA certificate). Seeding a custom TLS certificate using snap options To spin up an EdgeX instance with a custom certificate, the snap provides the following configuration options: apps.secrets-config.proxy.tls.cert apps.secrets-config.proxy.tls.key apps.secrets-config.proxy.tls.snis (comma-separated values) This is particularly useful when seeding the snap from a Gadget on an Ubuntu Core system.","title":"Changing TLS certificates"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#secret-store-token","text":"The services inside standalone snaps (e.g. device, app snaps) automatically receive a Secret Store token when: The standalone snap is downloaded and installed from the store The platform snap is downloaded and installed from the store Both snaps are installed on the same machine The service is registered as an add-on service The edgex-secretstore-token content interface provides the mechanism to automatically supply tokens to connected snaps. Execute the following command to check the status of connections: sudo snap connections edgexfoundry To manually connect the edgexfoundry's plug to a standalone snap's slot: snap connect edgexfoundry:edgex-secretstore-token :edgex-secretstore-token Note that the token has a limited expiry time of 1h by default. The connection and service startup should happen within the validity period. To better understand the snap connections, read the interface management Extend the default Secret Store token TTL Set TOKENFILEPROVIDER_DEFAULTTOKENTTL for security-secretstore-setup and restart: sudo snap set edgexfoundry app-options = true sudo snap set edgexfoundry apps.security-secretstore-setup.config.tokenfileprovider-defaulttokenttl = 72h sudo snap restart edgexfoundry.security-secretstore-setup","title":"Secret Store token"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#edgex-ui","text":"| Installation | Managing Services | Debugging | Source | For usage instructions, please refer to the Graphical User Interface (GUI) guide. By default, the UI is reachable locally at: http://localhost:4000 A valid JWT token is required to access the UI; follow Adding API Gateway users steps to generate a token. To enable all the functionalities of the UI, the following support services should be running: Support Scheduler Support Notifications System Management Agent EdgeX eKuiper For example, to start/install all: sudo snap start edgexfoundry.support-scheduler sudo snap start edgexfoundry.support-notifications sudo snap start edgexfoundry.sys-mgmt-agent sudo snap install edgex-ekuiper","title":"EdgeX UI"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#edgex-cli","text":"| Installation | Source | For usage instructions, refer to Command Line Interface (CLI) guide.","title":"EdgeX CLI"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#edgex-ekuiper","text":"| Installation | Managing Services | Debugging | Source | EdgeX 2.2 This version of EdgeX introduces a standalone EdgeX eKuiper snap. The new snap is the supported way of using eKuiper with other EdgeX snaps. For the documentation of the standalone EdgeX eKuiper snap, visit the README . Note The standalone EdgeX eKuiper snap documented here should not be confused with the deprecated edgexfoundry.kuiper and edgexfoundry.kuiper-cli apps built into the platform. The standalone snap can provide similar functionality.","title":"EdgeX eKuiper"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#app-service-configurable","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-app-service-configurable/current/config/ \u2514\u2500\u2500 res \u251c\u2500\u2500 external-mqtt-trigger \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 functional-tests \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 http-export \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 metrics-influxdb \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 mqtt-export \u2502 \u2514\u2500\u2500 configuration.toml \u251c\u2500\u2500 push-to-core \u2502 \u2514\u2500\u2500 configuration.toml \u2514\u2500\u2500 rules-engine \u2514\u2500\u2500 configuration.toml Please refer to App Service Configurable guide for detailed usage instructions. Profile Before you can start the service, you must select one of available profiles, using snap options. For example, to set mqtt-export profile using the snap CLI: sudo snap set edgex-app-service-configurable profile = mqtt-export Note The standalone App Service Configurable snap documented above should not be confused with the deprecated edgexfoundry.app-service-configurable , built into the platform snap. The standalone snap can serve the same functionality of filtering events for eKuiper by using the rules-engine profile.","title":"App Service Configurable"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#app-rfid-llrp-inventory","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-app-rfid-llrp-inventory/current/config/ \u2514\u2500\u2500 app-rfid-llrp-inventory \u2514\u2500\u2500 res \u2514\u2500\u2500 configuration.toml Aliases The aliases need to be provided for the service to work. See Setting the Aliases . For the snap, this can either be by: using a config-provider-snap to provide a configuration.toml file with the correct aliases, before startup setting the values manually in Consul during or after deployment","title":"App RFID LLRP Inventory"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-camera","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-camera/current/config/ \u2514\u2500\u2500 device-camera \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 camera.toml \u2514\u2500\u2500 profiles \u251c\u2500\u2500 camera-axis.yaml \u251c\u2500\u2500 camera-bosch.yaml \u2514\u2500\u2500 camera.yaml","title":"Device Camera"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-gpio","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-gpio/current/config \u2514\u2500\u2500 device-gpio \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 device.custom.gpio.toml \u2514\u2500\u2500 profiles \u2514\u2500\u2500 device.custom.gpio.yaml GPIO Access This snap is strictly confined which means that the access to interfaces are subject to various security measures. On a Linux distribution without snap confinement for GPIO (e.g. Raspberry Pi OS 11), the snap may be able to access the GPIO directly, without any snap interface and manual connections. On Linux distributions with snap confinement for GPIO such as Ubuntu Core, the GPIO access is possible via the gpio interface , provided by a gadget snap. The official Raspberry Pi Ubuntu Core image includes that gadget. It is NOT possible to use this snap on Linux distributions that have the GPIO confinement but not the interface (e.g. Ubuntu Server 20.04), unless for development purposes. In development environments, it is possible to install the snap in dev mode (using --devmode flag which disables security confinement and automatic upgrades) to allow direct GPIO access. The gpio interface provides slots for each GPIO channel. The slots can be listed using: $ sudo snap interface gpio name: gpio summary: allows access to specific GPIO pin plugs: - edgex-device-gpio slots: - pi:bcm-gpio-0 - pi:bcm-gpio-1 - pi:bcm-gpio-10 ... The slots are not connected automatically. For example, to connect GPIO-17: $ sudo snap connect edgex-device-gpio:gpio pi:bcm-gpio-17 Check the list of connections: $ sudo snap connections Interface Plug Slot Notes gpio edgex-device-gpio:gpio pi:bcm-gpio-17 manual \u2026","title":"Device GPIO"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-grove","text":"| Installation | Source | beta Device Grove snap is released as beta for arm64 . It is compatible with EdgeX 1.3 only. It does not support the snap configurations described above. The default configuration files are under /var/snap/edgex-device-grove/current/config/ . This device service is started by default. Changes to the configuration files require a restart to take effect: sudo snap restart edgex-device-grove","title":"Device Grove"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-modbus","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-modbus/current/config/ \u2514\u2500\u2500 device-modbus \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 modbus.test.devices.toml \u2514\u2500\u2500 profiles \u2514\u2500\u2500 modbus.test.device.profile.yml","title":"Device Modbus"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-mqtt","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-mqtt/current/config/ \u2514\u2500\u2500 device-mqtt \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 mqtt.test.device.toml \u2514\u2500\u2500 profiles \u2514\u2500\u2500 mqtt.test.device.profile.yml","title":"Device MQTT"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-rest","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-rest/current/config/ \u2514\u2500\u2500 device-rest \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 sample-devices.toml \u2514\u2500\u2500 profiles \u251c\u2500\u2500 sample-image-device.yaml \u251c\u2500\u2500 sample-json-device.yaml \u2514\u2500\u2500 sample-numeric-device.yaml","title":"Device REST"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-rfid-llrp","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-rfid-llrp/current/config/ \u2514\u2500\u2500 device-rfid-llrp \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u251c\u2500\u2500 profiles \u2502 \u251c\u2500\u2500 llrp.device.profile.yaml \u2502 \u2514\u2500\u2500 llrp.impinj.profile.yaml \u2514\u2500\u2500 provision_watchers \u251c\u2500\u2500 impinj.provision.watcher.json \u2514\u2500\u2500 llrp.provision.watcher.json Subnet setup The DiscoverySubnets setting needs to be provided before a device discovery can occur. This can be done in a number of ways: Using snap set to set your local subnet information. Example: sudo snap set edgex-device-rfid-llrp apps.device-rfid-llrp.config.app-custom.discovery-subnets = \"192.168.10.0/24\" curl -X POST http://localhost:59989/api/v2/discovery Using a config-provider-snap to set device configuration Using the auto-configure command. This command finds all local network interfaces which are online and non-virtual and sets the value of DiscoverySubnets in Consul. When running with security enabled, it requires a Consul token, so it needs to be run as follows: # get Consul ACL token CONSUL_TOKEN = $( sudo cat /var/snap/edgexfoundry/current/secrets/consul-acl-token/bootstrap_token.json | jq \".SecretID\" | tr -d '\"' ) echo $CONSUL_TOKEN # start the device service and connect the interfaces required for network interface discovery sudo snap start edgex-device-rfid-llrp.device-rfid-llrp sudo snap connect edgex-device-rfid-llrp:network-control sudo snap connect edgex-device-rfid-llrp:network-observe # run the nework interface discovery, providing the Consul token edgex-device-rfid-llrp.auto-configure $CONSUL_TOKEN","title":"Device RFID LLRP"},{"location":"getting-started/Ch-GettingStartedSnapUsers/#device-snmp","text":"| Installation | Configuration | Managing Services | Debugging | Source | The service is not started by default. Please refer to configuration and managing services . The default configuration files are installed at: /var/snap/edgex-device-snmp/current/config/ \u2514\u2500\u2500 device-snmp \u2514\u2500\u2500 res \u251c\u2500\u2500 configuration.toml \u251c\u2500\u2500 devices \u2502 \u2514\u2500\u2500 device.snmp.trendnet.TPE082WS.toml \u2514\u2500\u2500 profiles \u251c\u2500\u2500 device.snmp.patlite.yaml \u251c\u2500\u2500 device.snmp.switch.dell.N1108P-ON.yaml \u2514\u2500\u2500 device.snmp.trendnet.TPE082WS.yaml","title":"Device SNMP"},{"location":"getting-started/Ch-GettingStartedUsers/","text":"Getting Started as a User This section provides instructions for Users to get EdgeX up and running. If you are a Developer, you should read Getting Started as a Developer . EdgeX is a collection of more than a dozen micro services that are deployed to provide a minimal edge platform capability. You can download EdgeX micro service source code and build your own micro services. However, if you do not have a need to change or add to EdgeX, then you do not need to download source code. Instead, you can download and run the pre-built EdgeX micro service artifacts. The EdgeX community builds and creates Docker images as well as Snap packages with each release. The community also provides the latest unstable builds (prior to releases). Please continue by referring to: Getting Started using Docker Getting Started using Snaps","title":"Getting Started as a User"},{"location":"getting-started/Ch-GettingStartedUsers/#getting-started-as-a-user","text":"This section provides instructions for Users to get EdgeX up and running. If you are a Developer, you should read Getting Started as a Developer . EdgeX is a collection of more than a dozen micro services that are deployed to provide a minimal edge platform capability. You can download EdgeX micro service source code and build your own micro services. However, if you do not have a need to change or add to EdgeX, then you do not need to download source code. Instead, you can download and run the pre-built EdgeX micro service artifacts. The EdgeX community builds and creates Docker images as well as Snap packages with each release. The community also provides the latest unstable builds (prior to releases). Please continue by referring to: Getting Started using Docker Getting Started using Snaps","title":"Getting Started as a User"},{"location":"getting-started/Ch-GettingStartedUsersNexus/","text":"Getting Docker Images from EdgeX Nexus Repository Released EdgeX Docker container images are available from Docker Hub . Please refer to the Getting Started using Docker for instructions related to stable releases. In some cases, it may be necessary to get your EdgeX container images from the Nexus repository. The Linux Foundation manages the Nexus repository for the project. Warning Containers used from Nexus are considered \"work in progress\". There is no guarantee that these containers will function properly or function properly with other containers from the current release. Nexus contains the EdgeX project staging and development container images. In other words, Nexus contains work-in-progress or pre-release images. These, pre-release/work-in-progress Docker images are built nightly and made available at the following Nexus location: nexus3.edgexfoundry.org:10004 Rationale To Use Nexus Images Reasons you might want to use container images from Nexus include: The container is not available from Docker Hub (or Docker Hub is down temporarily) You need the latest development container image (the work in progress) You are working in a Windows or non-Linux environment and you are unable to build a container without some issues. A set of Docker Compose files have been created to allow you to get and use the latest EdgeX service images from Nexus. Find these Nexus \"Nightly Build\" Compose files in the main branch of the edgex-compose respository in GitHub. The EdgeX development team provides these Docker Compose files. As with the EdgeX release Compose files, you will find several different Docker Compose files that allow you to get the type of EdgeX instance setup based on: your hardware (x86 or ARM) your desire to have security services on or off your desire to run with the EdgeX GUI included Warning The \"Nightly Build\" images are provided as-is and may not always function properly or with other EdgeX services. Use with caution and typically only if you are a developer/contributor to EdgeX. These images represent the latest development work and may not have been thoroughly tested or integrated. Using Nexus Images The operations to pull the images and run the Nexus Repository containers are the same as when using EdgeX images from Docker Hub (see Getting Started using Docker ). To get container images from the Nexus Repository, in a command terminal, change directories to the location of your downloaded Nexus Docker Compose yaml. Rename the file to docker-compose.yml. Then run the following command in the terminal to pull (fetch) and then start the EdgeX Nexus-image containers. docker-compose up -d Using a Single Nexus Image In some cases, you may only need to use a single image from Nexus while other EdgeX services are created from the Docker Hub images. In this case, you can simply replace the image location for the selected image in your original Docker Compose file. The address of Nexus is nexus3.edgexfoundry.org at port 10004 . So, if you wished to use the EdgeX core data image from Nexus, you would replace the name and location of the core data image edgexfoundry/core-data:2.0.0 with nexus3.edgexfoundry.org:10004/core-data:latest in the Compose file. Note The example above replaces the Ireland core data service from Docker Hub with the latest core data image in Nexus.","title":"Getting Docker Images from EdgeX Nexus Repository"},{"location":"getting-started/Ch-GettingStartedUsersNexus/#getting-docker-images-from-edgex-nexus-repository","text":"Released EdgeX Docker container images are available from Docker Hub . Please refer to the Getting Started using Docker for instructions related to stable releases. In some cases, it may be necessary to get your EdgeX container images from the Nexus repository. The Linux Foundation manages the Nexus repository for the project. Warning Containers used from Nexus are considered \"work in progress\". There is no guarantee that these containers will function properly or function properly with other containers from the current release. Nexus contains the EdgeX project staging and development container images. In other words, Nexus contains work-in-progress or pre-release images. These, pre-release/work-in-progress Docker images are built nightly and made available at the following Nexus location: nexus3.edgexfoundry.org:10004","title":"Getting Docker Images from EdgeX Nexus Repository"},{"location":"getting-started/Ch-GettingStartedUsersNexus/#rationale-to-use-nexus-images","text":"Reasons you might want to use container images from Nexus include: The container is not available from Docker Hub (or Docker Hub is down temporarily) You need the latest development container image (the work in progress) You are working in a Windows or non-Linux environment and you are unable to build a container without some issues. A set of Docker Compose files have been created to allow you to get and use the latest EdgeX service images from Nexus. Find these Nexus \"Nightly Build\" Compose files in the main branch of the edgex-compose respository in GitHub. The EdgeX development team provides these Docker Compose files. As with the EdgeX release Compose files, you will find several different Docker Compose files that allow you to get the type of EdgeX instance setup based on: your hardware (x86 or ARM) your desire to have security services on or off your desire to run with the EdgeX GUI included Warning The \"Nightly Build\" images are provided as-is and may not always function properly or with other EdgeX services. Use with caution and typically only if you are a developer/contributor to EdgeX. These images represent the latest development work and may not have been thoroughly tested or integrated.","title":"Rationale To Use Nexus Images"},{"location":"getting-started/Ch-GettingStartedUsersNexus/#using-nexus-images","text":"The operations to pull the images and run the Nexus Repository containers are the same as when using EdgeX images from Docker Hub (see Getting Started using Docker ). To get container images from the Nexus Repository, in a command terminal, change directories to the location of your downloaded Nexus Docker Compose yaml. Rename the file to docker-compose.yml. Then run the following command in the terminal to pull (fetch) and then start the EdgeX Nexus-image containers. docker-compose up -d","title":"Using Nexus Images"},{"location":"getting-started/Ch-GettingStartedUsersNexus/#using-a-single-nexus-image","text":"In some cases, you may only need to use a single image from Nexus while other EdgeX services are created from the Docker Hub images. In this case, you can simply replace the image location for the selected image in your original Docker Compose file. The address of Nexus is nexus3.edgexfoundry.org at port 10004 . So, if you wished to use the EdgeX core data image from Nexus, you would replace the name and location of the core data image edgexfoundry/core-data:2.0.0 with nexus3.edgexfoundry.org:10004/core-data:latest in the Compose file. Note The example above replaces the Ireland core data service from Docker Hub with the latest core data image in Nexus.","title":"Using a Single Nexus Image"},{"location":"getting-started/quick-start/","text":"Quick Start This guide will get EdgeX up and running on your machine in as little as 5 minutes using Docker containers. We will skip over lengthy descriptions for now. The goal here is to get you a working IoT Edge stack, from device to cloud, as simply as possible. When you need more detailed instructions or a breakdown of some of the commands you see in this quick start, see either the Getting Started as a User or Getting Started as a Developer guides. Setup The fastest way to start running EdgeX is by using our pre-built Docker images. To use them you'll need to install the following: Docker https://docs.docker.com/install/ Docker Compose https://docs.docker.com/compose/install/ Running EdgeX Info Jakarta (v 2.1) is the latest version of EdgeX and used by example in this guide. Once you have Docker and Docker Compose installed, you need to: download / save the latest docker-compose file issue command to download and run the EdgeX Foundry Docker images from Docker Hub This can be accomplished with a single command as shown below (please note the tabs for x86 vs ARM architectures). x86 ARM curl https://raw.githubusercontent.com/edgexfoundry/edgex-compose/jakarta/docker-compose-no-secty.yml -o docker-compose.yml; docker-compose up -d curl https://raw.githubusercontent.com/edgexfoundry/edgex-compose/Jakarta/docker-compose-no-secty-arm64.yml -o docker-compose.yml; docker-compose up -d Verify that the EdgeX containers have started: docker-compose ps If all EdgeX containers pulled and started correctly and without error, you should see a process status (ps) that looks similar to the image above. Connected Devices EdgeX Foundry provides a Virtual device service which is useful for testing and development. It simulates a number of devices , each randomly generating data of various types and within configurable parameters. For example, the Random-Integer-Device will generate random integers. The Virtual Device (also known as Device Virtual) service is already a service pulled and running as part of the default EdgeX configuration. You can verify that Virtual Device readings are already being sent by querying the EdgeX core data service for the event records sent for Random-Integer-Device: curl http://localhost:59880/api/v2/event/device/name/Random-Integer-Device Verify the virtual device service is operating correctly by requesting the last event records received by core data for the Random-Integer-Device. Note By default, the maximum number of events returned will be 20 (the default limit). You can pass a limit parameter to get more or less event records. curl http://localhost:59880/api/v2/event/device/name/Random-Integer-Device?limit=50 Controlling the Device Reading data from devices is only part of what EdgeX is capable of. You can also use it to control your devices - this is termed 'actuating' the device. When a device registers with the EdgeX services, it provides a Device Profile that describes both the data readings available from that device, and also the commands that control it. When our Virtual Device service registered the device Random-Integer-Device , it used a profile to also define commands that allow you to tell the service not to generate random integers, but to always return a value you set. You won't call commands on devices directly, instead you use the EdgeX Foundry Command Service to do that. The first step is to check what commands are available to call by asking the Command service about your device: curl http://localhost:59882/api/v2/device/name/Random-Integer-Device This will return a lot of JSON, because there are a number of commands you can call on this device, but the commands we're going to use in this guide are Int16 (the comand to get the current integer 16 value) and WriteInt16Value (the command to disable the generation of the random integer 16 number and specify the integer value to return). Look for the Int16 and WriteInt16Value commands like those shown in the JSON as below: { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"deviceCoreCommand\" : { \"deviceName\" : \"Random-Integer-Device\" , \"profileName\" : \"Random-Integer-Device\" , \"coreCommands\" : [ { \"name\" : \"WriteInt16Value\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Integer-Device/WriteInt16Value\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Int16\" , \"valueType\" : \"Int16\" }, { \"resourceName\" : \"EnableRandomization_Int16\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"Int16\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Integer-Device/Int16\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Int16\" , \"valueType\" : \"Int16\" } ] } ... ] } } You'll notice that the commands have get or set (or both) options. A get call will return a random number (integer 16), and is what is being called automatically to send data into the rest of EdgeX (specifically core data). You can also call get manually using the URL provided (with no additinal parameters needed): curl http://localhost:59882/api/v2/device/name/Random-Integer-Device/Int16 Warning Notice that localhost replaces edgex-core-command here. That's because the EdgeX Foundry services are running in Docker. Docker recognizes the internal hostname edgex-core-command , but when calling the service from outside of Docker, you have to use localhost to reach it. This command will return a JSON result that looks like this: { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"event\" : { \"apiVersion\" : \"v2\" , \"id\" : \"6d829637-730c-4b70-9208-dc179070003f\" , \"deviceName\" : \"Random-Integer-Device\" , \"profileName\" : \"Random-Integer-Device\" , \"sourceName\" : \"Int16\" , \"origin\" : 1625605672073875500 , \"readings\" : [ { \"id\" : \"545b7add-683b-4745-84f1-d859f3d839e0\" , \"origin\" : 1625605672073875500 , \"deviceName\" : \"Random-Integer-Device\" , \"resourceName\" : \"Int16\" , \"profileName\" : \"Random-Integer-Device\" , \"valueType\" : \"Int16\" , \"binaryValue\" : null , \"mediaType\" : \"\" , \"value\" : \"-8146\" } ] } } A call to GET of the Int16 device's Random-Integer-Device operation through the command service results in the next random value produced by the device in JSON format. The default range for this reading is -32,768 to 32,767. In the example above, a value of -8146 was returned as the reading value. With the service set up to randomly return values, the value returned will be different each time the Int16 command is sent. However, we can use the WriteInt16Value command to disable random values from being returned and instead specify a value to return. Use the curl command below to call the set command to disable random values and return the value 42 each time. curl -X PUT -d '{\"Int16\":\"42\", \"EnableRandomization_Int16\":\"false\"}' http://localhost:59882/api/v2/device/name/Random-Integer-Device/WriteInt16Value Warning Again, also notice that localhost replaces edgex-core-command . If successful, the service will confirm your setting of the value to be returned with a 200 status code. A call to the device's SET command through the command service will return the API version and a status code (200 for success). Now every time we call get on the Int16 command, the returned value will be 42 . A call to GET of the Int16 device's Random-Integer-Device operation after setting the Int16 value to 42 and disabling randomization will always return a value of 42. Exporting Data EdgeX provides exporters (called application services) for a variety of cloud services and applications. To keep this guide simple, we're going to use the community provided 'application service configurable' to send the EdgeX data to a public MQTT broker hosted by HiveMQ. You can then watch for the EdgeX event data via HiveMQ provided MQTT browser client. First add the following application service to your docker-compose.yml file right after the 'app-service-rules' service (the first service in the file). Spacing is important in YAML, so make sure to copy and paste it correctly. app-service-mqtt : container_name : edgex-app-mqtt depends_on : - consul - data environment : CLIENTS_CORE_COMMAND_HOST : edgex-core-command CLIENTS_CORE_DATA_HOST : edgex-core-data CLIENTS_CORE_METADATA_HOST : edgex-core-metadata CLIENTS_SUPPORT_NOTIFICATIONS_HOST : edgex-support-notifications CLIENTS_SUPPORT_SCHEDULER_HOST : edgex-support-scheduler DATABASES_PRIMARY_HOST : edgex-redis EDGEX_PROFILE : mqtt-export EDGEX_SECURITY_SECRET_STORE : \"false\" MESSAGEQUEUE_HOST : edgex-redis REGISTRY_HOST : edgex-core-consul SERVICE_HOST : edgex-app-mqtt TRIGGER_EDGEXMESSAGEBUS_PUBLISHHOST_HOST : edgex-redis TRIGGER_EDGEXMESSAGEBUS_SUBSCRIBEHOST_HOST : edgex-redis WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_BROKERADDRESS : tcp://broker.mqttdashboard.com:1883 WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_TOPIC : EdgeXEvents hostname : edgex-app-mqtt image : edgexfoundry/app-service-configurable:2.0.0 networks : edgex-network : {} ports : - 127.0.0.1:59702:59702/tcp read_only : true security_opt : - no-new-privileges:true user : 2002:2001 Note This adds the application service configurable to your EdgeX system. The application service configurable allows you to configure (versus program) new exports - in this case exporting the EdgeX sensor data to the HiveMQ broker at tcp://broker.mqttdashboard.com:1883 . You will be publishing to the EdgeXEvents topic. For convenience, see documentation on the EdgeX Compose Builder to create custom Docker Compose files. Save the compose file and then execute another compose up command to have Docker Compose pull and start the configurable application service. docker-compose up -d You can connect to this broker with any MQTT client to watch the sent data. HiveMQ provides a web-based client that you can use. Use a browser to go to the client's URL. Once there, hit the Connect button to connect to the HiveMQ public broker. Using the HiveMQ provided client tool, connect to the same public HiveMQ broker your configurable application service is sending EdgeX data to. Then, use the Subscriptions area to subscribe to the \"EdgeXEvents\" topic. You must subscribe to the same topic - EdgeXEvents - to see the EdgeX data sent by the configurable application service. You will begin seeing your random number readings appear in the Messages area on the screen. Once subscribed, the EdgeX event data will begin to appear in the Messages area on the browser screen. Next Steps Congratulations! You now have a full EdgeX deployment reading data from a (virtual) device and publishing it to an MQTT broker in the cloud, and you were able to control your device through commands into EdgeX. It's time to continue your journey by reading the Introduction to EdgeX Foundry, what it is and how it's built. From there you can take the Walkthrough to learn how the micro services work together to control devices and read data from them as you just did.","title":"Quick Start"},{"location":"getting-started/quick-start/#quick-start","text":"This guide will get EdgeX up and running on your machine in as little as 5 minutes using Docker containers. We will skip over lengthy descriptions for now. The goal here is to get you a working IoT Edge stack, from device to cloud, as simply as possible. When you need more detailed instructions or a breakdown of some of the commands you see in this quick start, see either the Getting Started as a User or Getting Started as a Developer guides.","title":"Quick Start"},{"location":"getting-started/quick-start/#setup","text":"The fastest way to start running EdgeX is by using our pre-built Docker images. To use them you'll need to install the following: Docker https://docs.docker.com/install/ Docker Compose https://docs.docker.com/compose/install/","title":"Setup"},{"location":"getting-started/quick-start/#running-edgex","text":"Info Jakarta (v 2.1) is the latest version of EdgeX and used by example in this guide. Once you have Docker and Docker Compose installed, you need to: download / save the latest docker-compose file issue command to download and run the EdgeX Foundry Docker images from Docker Hub This can be accomplished with a single command as shown below (please note the tabs for x86 vs ARM architectures). x86 ARM curl https://raw.githubusercontent.com/edgexfoundry/edgex-compose/jakarta/docker-compose-no-secty.yml -o docker-compose.yml; docker-compose up -d curl https://raw.githubusercontent.com/edgexfoundry/edgex-compose/Jakarta/docker-compose-no-secty-arm64.yml -o docker-compose.yml; docker-compose up -d Verify that the EdgeX containers have started: docker-compose ps If all EdgeX containers pulled and started correctly and without error, you should see a process status (ps) that looks similar to the image above.","title":"Running EdgeX"},{"location":"getting-started/quick-start/#connected-devices","text":"EdgeX Foundry provides a Virtual device service which is useful for testing and development. It simulates a number of devices , each randomly generating data of various types and within configurable parameters. For example, the Random-Integer-Device will generate random integers. The Virtual Device (also known as Device Virtual) service is already a service pulled and running as part of the default EdgeX configuration. You can verify that Virtual Device readings are already being sent by querying the EdgeX core data service for the event records sent for Random-Integer-Device: curl http://localhost:59880/api/v2/event/device/name/Random-Integer-Device Verify the virtual device service is operating correctly by requesting the last event records received by core data for the Random-Integer-Device. Note By default, the maximum number of events returned will be 20 (the default limit). You can pass a limit parameter to get more or less event records. curl http://localhost:59880/api/v2/event/device/name/Random-Integer-Device?limit=50","title":"Connected Devices"},{"location":"getting-started/quick-start/#controlling-the-device","text":"Reading data from devices is only part of what EdgeX is capable of. You can also use it to control your devices - this is termed 'actuating' the device. When a device registers with the EdgeX services, it provides a Device Profile that describes both the data readings available from that device, and also the commands that control it. When our Virtual Device service registered the device Random-Integer-Device , it used a profile to also define commands that allow you to tell the service not to generate random integers, but to always return a value you set. You won't call commands on devices directly, instead you use the EdgeX Foundry Command Service to do that. The first step is to check what commands are available to call by asking the Command service about your device: curl http://localhost:59882/api/v2/device/name/Random-Integer-Device This will return a lot of JSON, because there are a number of commands you can call on this device, but the commands we're going to use in this guide are Int16 (the comand to get the current integer 16 value) and WriteInt16Value (the command to disable the generation of the random integer 16 number and specify the integer value to return). Look for the Int16 and WriteInt16Value commands like those shown in the JSON as below: { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"deviceCoreCommand\" : { \"deviceName\" : \"Random-Integer-Device\" , \"profileName\" : \"Random-Integer-Device\" , \"coreCommands\" : [ { \"name\" : \"WriteInt16Value\" , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Integer-Device/WriteInt16Value\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Int16\" , \"valueType\" : \"Int16\" }, { \"resourceName\" : \"EnableRandomization_Int16\" , \"valueType\" : \"Bool\" } ] }, { \"name\" : \"Int16\" , \"get\" : true , \"set\" : true , \"path\" : \"/api/v2/device/name/Random-Integer-Device/Int16\" , \"url\" : \"http://edgex-core-command:59882\" , \"parameters\" : [ { \"resourceName\" : \"Int16\" , \"valueType\" : \"Int16\" } ] } ... ] } } You'll notice that the commands have get or set (or both) options. A get call will return a random number (integer 16), and is what is being called automatically to send data into the rest of EdgeX (specifically core data). You can also call get manually using the URL provided (with no additinal parameters needed): curl http://localhost:59882/api/v2/device/name/Random-Integer-Device/Int16 Warning Notice that localhost replaces edgex-core-command here. That's because the EdgeX Foundry services are running in Docker. Docker recognizes the internal hostname edgex-core-command , but when calling the service from outside of Docker, you have to use localhost to reach it. This command will return a JSON result that looks like this: { \"apiVersion\" : \"v2\" , \"statusCode\" : 200 , \"event\" : { \"apiVersion\" : \"v2\" , \"id\" : \"6d829637-730c-4b70-9208-dc179070003f\" , \"deviceName\" : \"Random-Integer-Device\" , \"profileName\" : \"Random-Integer-Device\" , \"sourceName\" : \"Int16\" , \"origin\" : 1625605672073875500 , \"readings\" : [ { \"id\" : \"545b7add-683b-4745-84f1-d859f3d839e0\" , \"origin\" : 1625605672073875500 , \"deviceName\" : \"Random-Integer-Device\" , \"resourceName\" : \"Int16\" , \"profileName\" : \"Random-Integer-Device\" , \"valueType\" : \"Int16\" , \"binaryValue\" : null , \"mediaType\" : \"\" , \"value\" : \"-8146\" } ] } } A call to GET of the Int16 device's Random-Integer-Device operation through the command service results in the next random value produced by the device in JSON format. The default range for this reading is -32,768 to 32,767. In the example above, a value of -8146 was returned as the reading value. With the service set up to randomly return values, the value returned will be different each time the Int16 command is sent. However, we can use the WriteInt16Value command to disable random values from being returned and instead specify a value to return. Use the curl command below to call the set command to disable random values and return the value 42 each time. curl -X PUT -d '{\"Int16\":\"42\", \"EnableRandomization_Int16\":\"false\"}' http://localhost:59882/api/v2/device/name/Random-Integer-Device/WriteInt16Value Warning Again, also notice that localhost replaces edgex-core-command . If successful, the service will confirm your setting of the value to be returned with a 200 status code. A call to the device's SET command through the command service will return the API version and a status code (200 for success). Now every time we call get on the Int16 command, the returned value will be 42 . A call to GET of the Int16 device's Random-Integer-Device operation after setting the Int16 value to 42 and disabling randomization will always return a value of 42.","title":"Controlling the Device"},{"location":"getting-started/quick-start/#exporting-data","text":"EdgeX provides exporters (called application services) for a variety of cloud services and applications. To keep this guide simple, we're going to use the community provided 'application service configurable' to send the EdgeX data to a public MQTT broker hosted by HiveMQ. You can then watch for the EdgeX event data via HiveMQ provided MQTT browser client. First add the following application service to your docker-compose.yml file right after the 'app-service-rules' service (the first service in the file). Spacing is important in YAML, so make sure to copy and paste it correctly. app-service-mqtt : container_name : edgex-app-mqtt depends_on : - consul - data environment : CLIENTS_CORE_COMMAND_HOST : edgex-core-command CLIENTS_CORE_DATA_HOST : edgex-core-data CLIENTS_CORE_METADATA_HOST : edgex-core-metadata CLIENTS_SUPPORT_NOTIFICATIONS_HOST : edgex-support-notifications CLIENTS_SUPPORT_SCHEDULER_HOST : edgex-support-scheduler DATABASES_PRIMARY_HOST : edgex-redis EDGEX_PROFILE : mqtt-export EDGEX_SECURITY_SECRET_STORE : \"false\" MESSAGEQUEUE_HOST : edgex-redis REGISTRY_HOST : edgex-core-consul SERVICE_HOST : edgex-app-mqtt TRIGGER_EDGEXMESSAGEBUS_PUBLISHHOST_HOST : edgex-redis TRIGGER_EDGEXMESSAGEBUS_SUBSCRIBEHOST_HOST : edgex-redis WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_BROKERADDRESS : tcp://broker.mqttdashboard.com:1883 WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_TOPIC : EdgeXEvents hostname : edgex-app-mqtt image : edgexfoundry/app-service-configurable:2.0.0 networks : edgex-network : {} ports : - 127.0.0.1:59702:59702/tcp read_only : true security_opt : - no-new-privileges:true user : 2002:2001 Note This adds the application service configurable to your EdgeX system. The application service configurable allows you to configure (versus program) new exports - in this case exporting the EdgeX sensor data to the HiveMQ broker at tcp://broker.mqttdashboard.com:1883 . You will be publishing to the EdgeXEvents topic. For convenience, see documentation on the EdgeX Compose Builder to create custom Docker Compose files. Save the compose file and then execute another compose up command to have Docker Compose pull and start the configurable application service. docker-compose up -d You can connect to this broker with any MQTT client to watch the sent data. HiveMQ provides a web-based client that you can use. Use a browser to go to the client's URL. Once there, hit the Connect button to connect to the HiveMQ public broker. Using the HiveMQ provided client tool, connect to the same public HiveMQ broker your configurable application service is sending EdgeX data to. Then, use the Subscriptions area to subscribe to the \"EdgeXEvents\" topic. You must subscribe to the same topic - EdgeXEvents - to see the EdgeX data sent by the configurable application service. You will begin seeing your random number readings appear in the Messages area on the screen. Once subscribed, the EdgeX event data will begin to appear in the Messages area on the browser screen.","title":"Exporting Data"},{"location":"getting-started/quick-start/#next-steps","text":"Congratulations! You now have a full EdgeX deployment reading data from a (virtual) device and publishing it to an MQTT broker in the cloud, and you were able to control your device through commands into EdgeX. It's time to continue your journey by reading the Introduction to EdgeX Foundry, what it is and how it's built. From there you can take the Walkthrough to learn how the micro services work together to control devices and read data from them as you just did.","title":"Next Steps"},{"location":"getting-started/tools/Ch-CommandLineInterface/","text":"Command Line Interface (CLI) What is EdgeX CLI? EdgeX CLI is a command-line interface tool for developers, used for interacting with EdgeX Foundry microservices. Installing EdgeX CLI The client can be installed using a snap sudo snap install edgex-cli You can also download the appropriate binary for your operating system from GitHub . If you want to build EdgeX CLI from source, do the following: git clone http://github.com/edgexfoundry/edgex-cli.git cd edgex-cli make tidy make build ./bin/edgex-cli For more information, see the EdgeX CLI README . Features EdgeX CLI provides access to most of the core and support APIs. The commands map directly to the REST API structure. Running edgex-cli with no arguments shows a list of the available commands and information for each of them, including the name of the service implementing the command. Use the -h or --help flag to get more information about each command. $ edgex-cli EdgeX-CLI Usage: edgex-cli [command] Available Commands: command Read, write and list commands [Core Command] config Return the current configuration of all EdgeX core/support microservices device Add, remove, get, list and modify devices [Core Metadata] deviceprofile Add, remove, get and list device profiles [Core Metadata] deviceservice Add, remove, get, list and modify device services [Core Metadata] event Add, remove and list events help Help about any command interval Add, get and list intervals [Support Scheduler] intervalaction Get, list, update and remove interval actions [Support Scheduler] metrics Output the CPU/memory usage stats for all EdgeX core/support microservices notification Add, remove and list notifications [Support Notifications] ping Ping (health check) all EdgeX core/support microservices provisionwatcher Add, remove, get, list and modify provison watchers [Core Metadata] reading Count and list readings subscription Add, remove and list subscriptions [Support Notificationss] transmission Remove and list transmissions [Support Notifications] version Output the current version of EdgeX CLI and EdgeX microservices Flags: -h, --help help for edgex-cli Use \"edgex-cli [command] --help\" for more information about a command. Commands implemented by all microservices The ping , config , metrics and version work with more than one microservice. By default these commands will return values from all core and support services: $ edgex-cli metrics Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-metadata 13 1878936 38262 9445 47707 75318280 5967608 core-data 13 1716256 40200 8997 49197 75580424 5949504 core-command 13 1737288 31367 8582 39949 75318280 5380584 support-scheduler 10 2612296 20754 20224 40978 74728456 4146800 support-notifications 10 2714480 21199 20678 41877 74728456 4258640 To only return information for one service, specify the service to use: -c, --command use core-command service endpoint -d, --data use core-data service endpoint -m, --metadata use core-metadata service endpoint -n, --notifications use support-notifications service endpoint -s, --scheduler use support-scheduler service endpoint Example: $ edgex-cli metrics -d Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-data 14 1917712 870037 12258 882295 75580424 64148880 $ edgex-cli metrics -c Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-command 13 1618424 90890 8328 99218 75580424 22779448 $ edgex-cli metrics --metadata Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-metadata 12 1704256 39606 8870 48476 75318280 6139912 The -j/--json flag can be used with most of edgex-go commands to return the JSON output: $ edgex-cli metrics --metadata --json {\"apiVersion\":\"v2\",\"metrics\":{\"memAlloc\":1974544,\"memFrees\":39625,\"memLiveObjects\":9780,\"memMallocs\":49405,\"memSys\":75318280,\"memTotalAlloc\":6410200,\"cpuBusyAvg\":13}} This could then be formatted and filtered using jq : $ edgex-cli metrics --metadata --json | jq '.' { \"apiVersion\": \"v2\", \"metrics\": { \"memAlloc\": 1684176, \"memFrees\": 41142, \"memLiveObjects\": 8679, \"memMallocs\": 49821, \"memSys\": 75318280, \"memTotalAlloc\": 6530824, \"cpuBusyAvg\": 12 } } Core-command service edgex-cli command list Return a list of all supported device commands, optionally filtered by device name. Example: $ edgex-cli command list Name Device Name Profile Name Methods URL BoolArray Random-Boolean-Device Random-Boolean-Device Get, Put http://localhost:59882/api/v2/device/name/Random-Boolean-Device/BoolArray WriteBoolValue Random-Boolean-Device Random-Boolean-Device Put http://localhost:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue WriteBoolArrayValue Random-Boolean-Device Random-Boolean-Device Put http://localhost:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolArrayValue edgex-cli command read Issue a read command to the specified device. Example: $ edgex-cli command read -c Int16 -d Random-Integer-Device -j | jq '.' { \"apiVersion\": \"v2\", \"statusCode\": 200, \"event\": { \"apiVersion\": \"v2\", \"id\": \"e19f417e-3130-485f-8212-64b593b899f9\", \"deviceName\": \"Random-Integer-Device\", \"profileName\": \"Random-Integer-Device\", \"sourceName\": \"Int16\", \"origin\": 1641484109458647300, \"readings\": [ { \"id\": \"dc1f212d-148a-457c-ab13-48aa0fa58dd1\", \"origin\": 1641484109458647300, \"deviceName\": \"Random-Integer-Device\", \"resourceName\": \"Int16\", \"profileName\": \"Random-Integer-Device\", \"valueType\": \"Int16\", \"binaryValue\": null, \"mediaType\": \"\", \"value\": \"587\" } ] } } edgex-cli command write Issue a write command to the specified device. Example using in-line request body: $ edgex-cli command write -d Random-Integer-Device -c Int8 -b \"{\\\"Int8\\\": \\\"99\\\"}\" $ edgex-cli command read -d Random-Integer-Device -c Int8 apiVersion: v2,statusCode: 200 Command Name Device Name Profile Name Value Type Value Int8 Random-Integer-Device Random-Integer-Device Int8 99 Example using a file containing the request: $ echo \"{ \\\"Int8\\\":\\\"88\\\" }\" > file.txt $ edgex-cli command write -d Random-Integer-Device -c Int8 -f file.txt apiVersion: v2,statusCode: 200 $ edgex-cli command read -d Random-Integer-Device -c Int8 Command Name Device Name Profile Name Value Type Value Int8 Random-Integer-Device Random-Integer-Device Int8 88 Core-metadata service edgex-cli deviceservice list List device services $ edgex-cli deviceservice list edgex-cli deviceservice add Add a device service $ edgex-cli deviceservice add -n TestDeviceService -b \"http://localhost:51234\" edgex-cli deviceservice name Shows information about a device service. Most edgex-cli commands support the -v/--verbose and -j/--json flags: $ edgex-cli deviceservice name -n TestDeviceService Name BaseAddress Description TestDeviceService http://localhost:51234 $ edgex-cli deviceservice name -n TestDeviceService -v Name BaseAddress Description AdminState Id Labels LastConnected LastReported Modified TestDeviceService http://localhost:51234 UNLOCKED 7f29ad45-65dc-46c0-a928-00147d328032 [] 0 0 10 Jan 22 17:26 GMT $ edgex-cli deviceservice name -n TestDeviceService -j | jq '.' { \"apiVersion\": \"v2\", \"statusCode\": 200, \"service\": { \"created\": 1641835585465, \"modified\": 1641835585465, \"id\": \"7f29ad45-65dc-46c0-a928-00147d328032\", \"name\": \"TestDeviceService\", \"baseAddress\": \"http://localhost:51234\", \"adminState\": \"UNLOCKED\" } } edgex-cli deviceservice rm Remove a device service $ edgex-cli deviceservice rm -n TestDeviceService edgex-cli deviceservice update Update the device service, getting the ID using jq and confirm that the labels were added $ edgex-cli deviceservice add -n TestDeviceService -b \"http://localhost:51234\" {{{v2} c2600ad2-6489-4c3f-9207-5bdffdb8d68f 201} 844473b1-551d-4545-9143-28cfdf68a539} $ ID=`edgex-cli deviceservice name -n TestDeviceService -j | jq -r '.service.id'` $ edgex-cli deviceservice update -n TestDeviceService -i $ID --labels \"label1,label2\" {{v2} 9f4a4758-48a1-43ce-a232-828f442c2e34 200} $ edgex-cli deviceservice name -n TestDeviceService -v Name BaseAddress Description AdminState Id Labels LastConnected LastReported Modified TestDeviceService http://localhost:51234 UNLOCKED 844473b1-551d-4545-9143-28cfdf68a539 [label1 label2] 0 0 28 Jan 22 12:00 GMT edgex-cli deviceprofile list List device profiles $ edgex-cli deviceprofile list edgex-cli deviceprofile add Add a device profile $ edgex-cli deviceprofile add -n TestProfile -r \"[{\\\"name\\\": \\\"SwitchButton\\\",\\\"description\\\": \\\"Switch On/Off.\\\",\\\"properties\\\": {\\\"valueType\\\": \\\"String\\\",\\\"readWrite\\\": \\\"RW\\\",\\\"defaultValue\\\": \\\"On\\\",\\\"units\\\": \\\"On/Off\\\" } }]\" -c \"[{\\\"name\\\": \\\"Switch\\\",\\\"readWrite\\\": \\\"RW\\\",\\\"resourceOperations\\\": [{\\\"deviceResource\\\": \\\"SwitchButton\\\",\\\"DefaultValue\\\": \\\"false\\\" }]} ]\" {{{v2} 65d083cc-b876-4744-af65-59a00c63fc25 201} 4c0af6b0-4e83-4f3c-a574-dcea5f42d3f0} edgex-cli deviceprofile name Show information about a specifed device profile $ edgex-cli deviceprofile name -n TestProfile Name Description Manufacturer Model Name TestProfile TestProfile edgex-cli deviceprofile rm Remove a device profile $ edgex-cli deviceprofile rm -n TestProfile edgex-cli device list List current devices $ edgex-cli device list Name Description ServiceName ProfileName Labels AutoEvents Random-Float-Device Example of Device Virtual device-virtual Random-Float-Device [device-virtual-example] [{30s false Float32} {30s false Float64}] Random-UnsignedInteger-Device Example of Device Virtual device-virtual Random-UnsignedInteger-Device [device-virtual-example] [{20s false Uint8} {20s false Uint16} {20s false Uint32} {20s false Uint64}] Random-Boolean-Device Example of Device Virtual device-virtual Random-Boolean-Device [device-virtual-example] [{10s false Bool}] TestDevice TestDeviceService TestProfile [] [] Random-Binary-Device Example of Device Virtual device-virtual Random-Binary-Device [device-virtual-example] [] Random-Integer-Device Example of Device Virtual device-virtual Random-Integer-Device [device-virtual-example] [{15s false Int8} {15s false Int16} {15s false Int32} {15s false Int64}] edgex-cli device add Add a new device. This needs a device service and device profile to be created first $ edgex-cli device add -n TestDevice -p TestProfile -s TestDeviceService --protocols \"{\\\"modbus-tcp\\\":{\\\"Address\\\": \\\"localhost\\\",\\\"Port\\\": \\\"1234\\\" }}\" {{{v2} e912aa16-af4a-491d-993b-b0aeb8cd9c67 201} ae0e8b95-52fc-4778-892d-ae7e1127ed39} edgex-cli device name Show information about a specified named device $ edgex-cli device name -n TestDevice Name Description ServiceName ProfileName Labels AutoEvents TestDevice TestDeviceService TestProfile [] [] edgex-cli device rm Remove a device edgex-cli device rm -n TestDevice edgex-cli device list edgex-cli device add -n TestDevice -p TestProfile -s TestDeviceService --protocols \"{\\\"modbus-tcp\\\":{\\\"Address\\\": \\\"localhost\\\",\\\"Port\\\": \\\"1234\\\" }}\" edgex-cli device list edgex-cli device update Update a device This example gets the ID of a device, updates it using that ID and then displays device information to confirm that the labels were added $ ID=`edgex-cli device name -n TestDevice -j | jq -r '.device.id'` $ edgex-cli device update -n TestDevice -i $ID --labels \"label1,label2\" {{v2} 73427492-1158-45b2-9a7c-491a474cecce 200} $ edgex-cli device name -n TestDevice Name Description ServiceName ProfileName Labels AutoEvents TestDevice TestDeviceService TestProfile [label1 label2] [] edgex-cli provisionwatcher add Add a new provision watcher $ edgex-cli provisionwatcher add -n TestWatcher --identifiers \"{\\\"address\\\":\\\"localhost\\\",\\\"port\\\":\\\"1234\\\"}\" -p TestProfile -s TestDeviceService {{{v2} 3f05f6e0-9d9b-4d96-96df-f394cc2ad6f4 201} ee76f4d8-46d4-454c-a4da-8ad9e06d8d7e} edgex-cli provisionwatcher list List provision watchers $ edgex-cli provisionwatcher list Name ServiceName ProfileName Labels Identifiers TestWatcher TestDeviceService TestProfile [] map[address:localhost port:1234] edgex-cli provisionwatcher name Show information about a specific named provision watcher $ edgex-cli provisionwatcher name -n TestWatcher Name ServiceName ProfileName Labels Identifiers TestWatcher TestDeviceService TestProfile [] map[address:localhost port:1234] edgex-cli provisionwatcher rm Remove a provision watcher $ edgex-cli provisionwatcher rm -n TestWatcher $ edgex-cli provisionwatcher list No provision watchers available edgex-cli provisionwatcher update Update a provision watcher This example gets the ID of a provision watcher, updates it using that ID and then displays information about it to confirm that the labels were added $ edgex-cli provisionwatcher add -n TestWatcher2 --identifiers \"{\\\"address\\\":\\\"localhost\\\",\\\"port\\\":\\\"1234\\\"}\" -p TestProfile -s TestDeviceService {{{v2} fb7b8bcf-8f58-477b-929e-8dac53cddc81 201} 7aadb7df-1ff1-4b3b-8986-b97e0ef53116} $ ID=`edgex-cli provisionwatcher name -n TestWatcher2 -j | jq -r '.provisionWatcher.id'` $ edgex-cli provisionwatcher update -n TestWatcher2 -i $ID --labels \"label1,label2\" {{v2} af1e70bf-4705-47f4-9046-c7b789799405 200} $ edgex-cli provisionwatcher name -n TestWatcher2 Name ServiceName ProfileName Labels Identifiers TestWatcher2 TestDeviceService TestProfile [label1 label2] map[address:localhost port:1234] Core-data service edgex-cli event add Create an event with a specified number of random readings $ edgex-cli event add -d Random-Integer-Device -p Random-Integer-Device -r 1 -s Int16 -t int16 Added event 75f06078-e8da-4671-8938-ab12ebb2c244 $ edgex-cli event list -v Origin Device Profile Source Id Versionable Readings 10 Jan 22 15:38 GMT Random-Integer-Device Random-Integer-Device Int16 75f06078-e8da-4671-8938-ab12ebb2c244 {v2} [{974a70fe-71ef-4a47-a008-c89f0e4e3bb6 1641829092129391876 Random-Integer-Device Int16 Random-Integer-Device Int16 {[] } {13342}}] edgex-cli event count Count the number of events in core data, optionally filtering by device name $ edgex-cli event count -d Random-Integer-Device Total Random-Integer-Device events: 54 edgex-cli event list List all events, optionally specifying a limit and offset $ edgex-cli event list To see two readings only, skipping the first 100 readings: $ edgex-cli reading list --limit 2 --offset 100 Origin Device ProfileName Value ValueType 28 Jan 22 12:55 GMT Random-Integer-Device Random-Integer-Device 22502 Int16 28 Jan 22 12:55 GMT Random-Integer-Device Random-Integer-Device 1878517239016780388 Int64 edgex-cli event rm Remove events, specifying either device name or maximum event age in milliseconds - edgex-cli event rm --device {devicename} removes all events for the specified device - edgex-cli event rm --age {ms} removes all events generated in the last {ms} milliseconds $ edgex-cli event rm -a 30000 $ edgex-cli event count Total events: 0 edgex-cli reading count Count the number of readings in core data, optionally filtering by device name $ edgex-cli reading count Total readings: 235 edgex-cli reading list List all readings, optionally specifying a limit and offset $ edgex-cli reading list Support-scheduler service edgex-cli interval add Add an interval $ edgex-cli interval add -n \"hourly\" -i \"1h\" {{{v2} c7c51f21-dab5-4307-a4c9-bc5d5f2194d9 201} 98a6d5f6-f4c4-4ec5-a00c-7fe24b9c9a18} edgex-cli interval name Return an interval by name $ edgex-cli interval name -n \"hourly\" Name Interval Start End hourly 1h edgex-cli interval list List all intervals $ edgex-cli interval list -j | jq '.' { \"apiVersion\": \"v2\", \"statusCode\": 200, \"intervals\": [ { \"created\": 1641830955058, \"modified\": 1641830955058, \"id\": \"98a6d5f6-f4c4-4ec5-a00c-7fe24b9c9a18\", \"name\": \"hourly\", \"interval\": \"1h\" }, { \"created\": 1641830953884, \"modified\": 1641830953884, \"id\": \"507a2a9a-82eb-41ea-afa8-79a9b0033665\", \"name\": \"midnight\", \"start\": \"20180101T000000\", \"interval\": \"24h\" } ] } edgex-cli interval update Update an interval, specifying either ID or name $ edgex-cli interval update -n \"hourly\" -i \"1m\" {{v2} 08239cc4-d4d7-4ea2-9915-d91b9557c742 200} $ edgex-cli interval name -n \"hourly\" -v Id Name Interval Start End 98a6d5f6-f4c4-4ec5-a00c-7fe24b9c9a18 hourly 1m edgex-cli interval rm Delete a named interval and associated interval actions $ edgex-cli interval rm -n \"hourly\" edgex-cli intervalaction add Add an interval action $ edgex-cli intervalaction add -n \"name01\" -i \"midnight\" -a \"{\\\"type\\\": \\\"REST\\\", \\\"host\\\": \\\"192.168.0.102\\\", \\\"port\\\": 8080, \\\"httpMethod\\\": \\\"GET\\\"}\" edgex-cli intervalaction name Return an interval action by name $ edgex-cli intervalaction name -n \"name01\" Name Interval Address Content ContentType name01 midnight {REST 192.168.0.102 8080 { GET} { 0 0 false false 0} {[]}} edgex-cli intervalaction list List all interval actions $ edgex-cli intervalaction list Name Interval Address Content ContentType name01 midnight {REST 192.168.0.102 8080 { GET} { 0 0 false false 0} {[]}} scrub-aged-events midnight {REST localhost 59880 {/api/v2/event/age/604800000000000 DELETE} { 0 0 false false 0} {[]}} edgex-cli intervalaction update Update an interval action, specifying either ID or name $ edgex-cli intervalaction update -n \"name01\" --admin-state \"LOCKED\" {{v2} afc7b08c-5dc6-4923-9786-30bfebc8a8b6 200} $ edgex-cli intervalaction name -n \"name01\" -j | jq '.action.adminState' \"LOCKED\" edgex-cli intervalaction rm Delete an interval action by name $ edgex-cli intervalaction rm -n \"name01\" Support-notifications service edgex-cli notification add Add a notification to be sent $ edgex-cli notification add -s \"sender01\" -c \"content\" --category \"category04\" --labels \"l3\" {{{v2} 13938e01-a560-47d8-bb50-060effdbe490 201} 6a1138c2-b58e-4696-afa7-2074e95165eb} edgex-cli notification list List notifications associated with a given label, category or time range $ edgex-cli notification list -c \"category04\" Category Content Description Labels Sender Severity Status category04 content [l3] sender01 NORMAL PROCESSED $ edgex-cli notification list --start \"01 jan 20 00:00 GMT\" --end \"01 dec 24 00:00 GMT\" Category Content Description Labels Sender Severity Status category04 content [l3] sender01 NORMAL PROCESSED edgex-cli notification rm Delete a notification and all of its associated transmissions $ ID=`edgex-cli notification list -c \"category04\" -v -j | jq -r '.notifications[0].id'` $ echo $ID 6a1138c2-b58e-4696-afa7-2074e95165eb $ edgex-cli notification rm -i $ID $ edgex-cli notification list -c \"category04\" No notifications available edgex-cli notification cleanup Delete all notifications and corresponding transmissions $ edgex-cli notification cleanup $ edgex-cli notification list --start \"01 jan 20 00:00 GMT\" --end \"01 dec 24 00:00 GMT\" No notifications available edgex-cli subscription add Add a new subscription $ edgex-cli subscription add -n \"name01\" --receiver \"receiver01\" -c \"[{\\\"type\\\": \\\"REST\\\", \\\"host\\\": \\\"localhost\\\", \\\"port\\\": 7770, \\\"httpMethod\\\": \\\"POST\\\"}]\" --labels \"l1,l2,l3\" {{{v2} 2bbfdac0-d2e1-4f08-8344-392b8e8ddc5e 201} 1ec08af0-5767-4505-82f7-581fada6006b} $ edgex-cli subscription add -n \"name02\" --receiver \"receiver01\" -c \"[{\\\"type\\\": \\\"EMAIL\\\", \\\"recipients\\\": [\\\"123@gmail.com\\\"]}]\" --labels \"l1,l2,l3\" {{{v2} f6b417ca-740c-4dee-bc1e-c721c0de4051 201} 156fc2b9-de60-423b-9bff-5312d8452c48} edgex-cli subscription name Return a subscription by its unique name $ edgex-cli subscription name -n \"name01\" Name Description Channels Receiver Categories Labels name01 [{REST localhost 7770 { POST} { 0 0 false false 0} {[]}}] receiver01 [] [l1 l2 l3] edgex-cli subscription list List all subscriptions, optionally filtered by a given category, label or receiver $ edgex-cli subscription list --label \"l1\" Name Description Channels Receiver Categories Labels name02 [{EMAIL 0 { } { 0 0 false false 0} {[123@gmail.com]}}] receiver01 [] [l1 l2 l3] name01 [{REST localhost 7770 { POST} { 0 0 false false 0} {[]}}] receiver01 [] [l1 l2 l3] edgex-cli subscription rm Delete the named subscription $ edgex-cli subscription rm -n \"name01\" edgex-cli transmission list To create a transmission, first create a subscription and notifications: $ edgex-cli subscription add -n \"Test-Subscription\" --description \"Test data for subscription\" --categories \"health-check\" --labels \"simple\" --receiver \"tafuser\" --resend-limit 0 --admin-state \"UNLOCKED\" -c \"[{\\\"type\\\": \\\"REST\\\", \\\"host\\\": \\\"localhost\\\", \\\"port\\\": 7770, \\\"httpMethod\\\": \\\"POST\\\"}]\" {{{v2} f281ec1a-876e-4a29-a14d-195b66d0506c 201} 3b489d23-b0c7-4791-b839-d9a578ebccb9} $ edgex-cli notification add -d \"Test data for notification 1\" --category \"health-check\" --labels \"simple\" --content-type \"string\" --content \"This is a test notification\" --sender \"taf-admin\" {{{v2} 8df79c7c-03fb-4626-b6e8-bf2d616fa327 201} 0be98b91-daf9-46e2-bcca-39f009d93866} $ edgex-cli notification add -d \"Test data for notification 2\" --category \"health-check\" --labels \"simple\" --content-type \"string\" --content \"This is a test notification\" --sender \"taf-admin\" {{{v2} ec0b2444-c8b0-45d0-bbd6-847dd007c2fd 201} a7c65d7d-0f9c-47e1-82c2-c8098c47c016} $ edgex-cli notification add -d \"Test data for notification 3\" --category \"health-check\" --labels \"simple\" --content-type \"string\" --content \"This is a test notification\" --sender \"taf-admin\" {{{v2} 45af7f94-c99e-4fb1-a632-fab5ff475be4 201} f982fc97-f53f-4154-bfce-3ef8666c3911} Then list the transmissions: $ edgex-cli transmission list SubscriptionName ResendCount Status Test-Subscription 0 FAILED Test-Subscription 0 FAILED Test-Subscription 0 FAILED edgex-cli transmission id Return a transmission by ID $ ID=`edgex-cli transmission list -j | jq -r '.transmissions[0].id'` $ edgex-cli transmission id -i $ID SubscriptionName ResendCount Status Test-Subscription 0 FAILED edgex-cli transmission rm Delete processed transmissions older than the specificed age (in milliseconds) $ edgex-cli transmission rm -a 100","title":"Command Line Interface (CLI)"},{"location":"getting-started/tools/Ch-CommandLineInterface/#command-line-interface-cli","text":"","title":"Command Line Interface (CLI)"},{"location":"getting-started/tools/Ch-CommandLineInterface/#what-is-edgex-cli","text":"EdgeX CLI is a command-line interface tool for developers, used for interacting with EdgeX Foundry microservices.","title":"What is EdgeX CLI?"},{"location":"getting-started/tools/Ch-CommandLineInterface/#installing-edgex-cli","text":"The client can be installed using a snap sudo snap install edgex-cli You can also download the appropriate binary for your operating system from GitHub . If you want to build EdgeX CLI from source, do the following: git clone http://github.com/edgexfoundry/edgex-cli.git cd edgex-cli make tidy make build ./bin/edgex-cli For more information, see the EdgeX CLI README .","title":"Installing EdgeX CLI"},{"location":"getting-started/tools/Ch-CommandLineInterface/#features","text":"EdgeX CLI provides access to most of the core and support APIs. The commands map directly to the REST API structure. Running edgex-cli with no arguments shows a list of the available commands and information for each of them, including the name of the service implementing the command. Use the -h or --help flag to get more information about each command. $ edgex-cli EdgeX-CLI Usage: edgex-cli [command] Available Commands: command Read, write and list commands [Core Command] config Return the current configuration of all EdgeX core/support microservices device Add, remove, get, list and modify devices [Core Metadata] deviceprofile Add, remove, get and list device profiles [Core Metadata] deviceservice Add, remove, get, list and modify device services [Core Metadata] event Add, remove and list events help Help about any command interval Add, get and list intervals [Support Scheduler] intervalaction Get, list, update and remove interval actions [Support Scheduler] metrics Output the CPU/memory usage stats for all EdgeX core/support microservices notification Add, remove and list notifications [Support Notifications] ping Ping (health check) all EdgeX core/support microservices provisionwatcher Add, remove, get, list and modify provison watchers [Core Metadata] reading Count and list readings subscription Add, remove and list subscriptions [Support Notificationss] transmission Remove and list transmissions [Support Notifications] version Output the current version of EdgeX CLI and EdgeX microservices Flags: -h, --help help for edgex-cli Use \"edgex-cli [command] --help\" for more information about a command.","title":"Features"},{"location":"getting-started/tools/Ch-CommandLineInterface/#commands-implemented-by-all-microservices","text":"The ping , config , metrics and version work with more than one microservice. By default these commands will return values from all core and support services: $ edgex-cli metrics Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-metadata 13 1878936 38262 9445 47707 75318280 5967608 core-data 13 1716256 40200 8997 49197 75580424 5949504 core-command 13 1737288 31367 8582 39949 75318280 5380584 support-scheduler 10 2612296 20754 20224 40978 74728456 4146800 support-notifications 10 2714480 21199 20678 41877 74728456 4258640 To only return information for one service, specify the service to use: -c, --command use core-command service endpoint -d, --data use core-data service endpoint -m, --metadata use core-metadata service endpoint -n, --notifications use support-notifications service endpoint -s, --scheduler use support-scheduler service endpoint Example: $ edgex-cli metrics -d Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-data 14 1917712 870037 12258 882295 75580424 64148880 $ edgex-cli metrics -c Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-command 13 1618424 90890 8328 99218 75580424 22779448 $ edgex-cli metrics --metadata Service CpuBusyAvg MemAlloc MemFrees MemLiveObjects MemMallocs MemSys MemTotalAlloc core-metadata 12 1704256 39606 8870 48476 75318280 6139912 The -j/--json flag can be used with most of edgex-go commands to return the JSON output: $ edgex-cli metrics --metadata --json {\"apiVersion\":\"v2\",\"metrics\":{\"memAlloc\":1974544,\"memFrees\":39625,\"memLiveObjects\":9780,\"memMallocs\":49405,\"memSys\":75318280,\"memTotalAlloc\":6410200,\"cpuBusyAvg\":13}} This could then be formatted and filtered using jq : $ edgex-cli metrics --metadata --json | jq '.' { \"apiVersion\": \"v2\", \"metrics\": { \"memAlloc\": 1684176, \"memFrees\": 41142, \"memLiveObjects\": 8679, \"memMallocs\": 49821, \"memSys\": 75318280, \"memTotalAlloc\": 6530824, \"cpuBusyAvg\": 12 } }","title":"Commands implemented by all microservices"},{"location":"getting-started/tools/Ch-CommandLineInterface/#core-command-service","text":"","title":"Core-command service"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-command-list","text":"Return a list of all supported device commands, optionally filtered by device name. Example: $ edgex-cli command list Name Device Name Profile Name Methods URL BoolArray Random-Boolean-Device Random-Boolean-Device Get, Put http://localhost:59882/api/v2/device/name/Random-Boolean-Device/BoolArray WriteBoolValue Random-Boolean-Device Random-Boolean-Device Put http://localhost:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolValue WriteBoolArrayValue Random-Boolean-Device Random-Boolean-Device Put http://localhost:59882/api/v2/device/name/Random-Boolean-Device/WriteBoolArrayValue","title":"edgex-cli command list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-command-read","text":"Issue a read command to the specified device. Example: $ edgex-cli command read -c Int16 -d Random-Integer-Device -j | jq '.' { \"apiVersion\": \"v2\", \"statusCode\": 200, \"event\": { \"apiVersion\": \"v2\", \"id\": \"e19f417e-3130-485f-8212-64b593b899f9\", \"deviceName\": \"Random-Integer-Device\", \"profileName\": \"Random-Integer-Device\", \"sourceName\": \"Int16\", \"origin\": 1641484109458647300, \"readings\": [ { \"id\": \"dc1f212d-148a-457c-ab13-48aa0fa58dd1\", \"origin\": 1641484109458647300, \"deviceName\": \"Random-Integer-Device\", \"resourceName\": \"Int16\", \"profileName\": \"Random-Integer-Device\", \"valueType\": \"Int16\", \"binaryValue\": null, \"mediaType\": \"\", \"value\": \"587\" } ] } }","title":"edgex-cli command read"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-command-write","text":"Issue a write command to the specified device. Example using in-line request body: $ edgex-cli command write -d Random-Integer-Device -c Int8 -b \"{\\\"Int8\\\": \\\"99\\\"}\" $ edgex-cli command read -d Random-Integer-Device -c Int8 apiVersion: v2,statusCode: 200 Command Name Device Name Profile Name Value Type Value Int8 Random-Integer-Device Random-Integer-Device Int8 99 Example using a file containing the request: $ echo \"{ \\\"Int8\\\":\\\"88\\\" }\" > file.txt $ edgex-cli command write -d Random-Integer-Device -c Int8 -f file.txt apiVersion: v2,statusCode: 200 $ edgex-cli command read -d Random-Integer-Device -c Int8 Command Name Device Name Profile Name Value Type Value Int8 Random-Integer-Device Random-Integer-Device Int8 88","title":"edgex-cli command write"},{"location":"getting-started/tools/Ch-CommandLineInterface/#core-metadata-service","text":"","title":"Core-metadata service"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceservice-list","text":"List device services $ edgex-cli deviceservice list","title":"edgex-cli deviceservice list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceservice-add","text":"Add a device service $ edgex-cli deviceservice add -n TestDeviceService -b \"http://localhost:51234\"","title":"edgex-cli deviceservice add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceservice-name","text":"Shows information about a device service. Most edgex-cli commands support the -v/--verbose and -j/--json flags: $ edgex-cli deviceservice name -n TestDeviceService Name BaseAddress Description TestDeviceService http://localhost:51234 $ edgex-cli deviceservice name -n TestDeviceService -v Name BaseAddress Description AdminState Id Labels LastConnected LastReported Modified TestDeviceService http://localhost:51234 UNLOCKED 7f29ad45-65dc-46c0-a928-00147d328032 [] 0 0 10 Jan 22 17:26 GMT $ edgex-cli deviceservice name -n TestDeviceService -j | jq '.' { \"apiVersion\": \"v2\", \"statusCode\": 200, \"service\": { \"created\": 1641835585465, \"modified\": 1641835585465, \"id\": \"7f29ad45-65dc-46c0-a928-00147d328032\", \"name\": \"TestDeviceService\", \"baseAddress\": \"http://localhost:51234\", \"adminState\": \"UNLOCKED\" } }","title":"edgex-cli deviceservice name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceservice-rm","text":"Remove a device service $ edgex-cli deviceservice rm -n TestDeviceService","title":"edgex-cli deviceservice rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceservice-update","text":"Update the device service, getting the ID using jq and confirm that the labels were added $ edgex-cli deviceservice add -n TestDeviceService -b \"http://localhost:51234\" {{{v2} c2600ad2-6489-4c3f-9207-5bdffdb8d68f 201} 844473b1-551d-4545-9143-28cfdf68a539} $ ID=`edgex-cli deviceservice name -n TestDeviceService -j | jq -r '.service.id'` $ edgex-cli deviceservice update -n TestDeviceService -i $ID --labels \"label1,label2\" {{v2} 9f4a4758-48a1-43ce-a232-828f442c2e34 200} $ edgex-cli deviceservice name -n TestDeviceService -v Name BaseAddress Description AdminState Id Labels LastConnected LastReported Modified TestDeviceService http://localhost:51234 UNLOCKED 844473b1-551d-4545-9143-28cfdf68a539 [label1 label2] 0 0 28 Jan 22 12:00 GMT","title":"edgex-cli deviceservice update"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceprofile-list","text":"List device profiles $ edgex-cli deviceprofile list","title":"edgex-cli deviceprofile list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceprofile-add","text":"Add a device profile $ edgex-cli deviceprofile add -n TestProfile -r \"[{\\\"name\\\": \\\"SwitchButton\\\",\\\"description\\\": \\\"Switch On/Off.\\\",\\\"properties\\\": {\\\"valueType\\\": \\\"String\\\",\\\"readWrite\\\": \\\"RW\\\",\\\"defaultValue\\\": \\\"On\\\",\\\"units\\\": \\\"On/Off\\\" } }]\" -c \"[{\\\"name\\\": \\\"Switch\\\",\\\"readWrite\\\": \\\"RW\\\",\\\"resourceOperations\\\": [{\\\"deviceResource\\\": \\\"SwitchButton\\\",\\\"DefaultValue\\\": \\\"false\\\" }]} ]\" {{{v2} 65d083cc-b876-4744-af65-59a00c63fc25 201} 4c0af6b0-4e83-4f3c-a574-dcea5f42d3f0}","title":"edgex-cli deviceprofile add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceprofile-name","text":"Show information about a specifed device profile $ edgex-cli deviceprofile name -n TestProfile Name Description Manufacturer Model Name TestProfile TestProfile","title":"edgex-cli deviceprofile name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-deviceprofile-rm","text":"Remove a device profile $ edgex-cli deviceprofile rm -n TestProfile","title":"edgex-cli deviceprofile rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-device-list","text":"List current devices $ edgex-cli device list Name Description ServiceName ProfileName Labels AutoEvents Random-Float-Device Example of Device Virtual device-virtual Random-Float-Device [device-virtual-example] [{30s false Float32} {30s false Float64}] Random-UnsignedInteger-Device Example of Device Virtual device-virtual Random-UnsignedInteger-Device [device-virtual-example] [{20s false Uint8} {20s false Uint16} {20s false Uint32} {20s false Uint64}] Random-Boolean-Device Example of Device Virtual device-virtual Random-Boolean-Device [device-virtual-example] [{10s false Bool}] TestDevice TestDeviceService TestProfile [] [] Random-Binary-Device Example of Device Virtual device-virtual Random-Binary-Device [device-virtual-example] [] Random-Integer-Device Example of Device Virtual device-virtual Random-Integer-Device [device-virtual-example] [{15s false Int8} {15s false Int16} {15s false Int32} {15s false Int64}]","title":"edgex-cli device list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-device-add","text":"Add a new device. This needs a device service and device profile to be created first $ edgex-cli device add -n TestDevice -p TestProfile -s TestDeviceService --protocols \"{\\\"modbus-tcp\\\":{\\\"Address\\\": \\\"localhost\\\",\\\"Port\\\": \\\"1234\\\" }}\" {{{v2} e912aa16-af4a-491d-993b-b0aeb8cd9c67 201} ae0e8b95-52fc-4778-892d-ae7e1127ed39}","title":"edgex-cli device add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-device-name","text":"Show information about a specified named device $ edgex-cli device name -n TestDevice Name Description ServiceName ProfileName Labels AutoEvents TestDevice TestDeviceService TestProfile [] []","title":"edgex-cli device name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-device-rm","text":"Remove a device edgex-cli device rm -n TestDevice edgex-cli device list edgex-cli device add -n TestDevice -p TestProfile -s TestDeviceService --protocols \"{\\\"modbus-tcp\\\":{\\\"Address\\\": \\\"localhost\\\",\\\"Port\\\": \\\"1234\\\" }}\" edgex-cli device list","title":"edgex-cli device rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-device-update","text":"Update a device This example gets the ID of a device, updates it using that ID and then displays device information to confirm that the labels were added $ ID=`edgex-cli device name -n TestDevice -j | jq -r '.device.id'` $ edgex-cli device update -n TestDevice -i $ID --labels \"label1,label2\" {{v2} 73427492-1158-45b2-9a7c-491a474cecce 200} $ edgex-cli device name -n TestDevice Name Description ServiceName ProfileName Labels AutoEvents TestDevice TestDeviceService TestProfile [label1 label2] []","title":"edgex-cli device update"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-provisionwatcher-add","text":"Add a new provision watcher $ edgex-cli provisionwatcher add -n TestWatcher --identifiers \"{\\\"address\\\":\\\"localhost\\\",\\\"port\\\":\\\"1234\\\"}\" -p TestProfile -s TestDeviceService {{{v2} 3f05f6e0-9d9b-4d96-96df-f394cc2ad6f4 201} ee76f4d8-46d4-454c-a4da-8ad9e06d8d7e}","title":"edgex-cli provisionwatcher add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-provisionwatcher-list","text":"List provision watchers $ edgex-cli provisionwatcher list Name ServiceName ProfileName Labels Identifiers TestWatcher TestDeviceService TestProfile [] map[address:localhost port:1234]","title":"edgex-cli provisionwatcher list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-provisionwatcher-name","text":"Show information about a specific named provision watcher $ edgex-cli provisionwatcher name -n TestWatcher Name ServiceName ProfileName Labels Identifiers TestWatcher TestDeviceService TestProfile [] map[address:localhost port:1234]","title":"edgex-cli provisionwatcher name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-provisionwatcher-rm","text":"Remove a provision watcher $ edgex-cli provisionwatcher rm -n TestWatcher $ edgex-cli provisionwatcher list No provision watchers available","title":"edgex-cli provisionwatcher rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-provisionwatcher-update","text":"Update a provision watcher This example gets the ID of a provision watcher, updates it using that ID and then displays information about it to confirm that the labels were added $ edgex-cli provisionwatcher add -n TestWatcher2 --identifiers \"{\\\"address\\\":\\\"localhost\\\",\\\"port\\\":\\\"1234\\\"}\" -p TestProfile -s TestDeviceService {{{v2} fb7b8bcf-8f58-477b-929e-8dac53cddc81 201} 7aadb7df-1ff1-4b3b-8986-b97e0ef53116} $ ID=`edgex-cli provisionwatcher name -n TestWatcher2 -j | jq -r '.provisionWatcher.id'` $ edgex-cli provisionwatcher update -n TestWatcher2 -i $ID --labels \"label1,label2\" {{v2} af1e70bf-4705-47f4-9046-c7b789799405 200} $ edgex-cli provisionwatcher name -n TestWatcher2 Name ServiceName ProfileName Labels Identifiers TestWatcher2 TestDeviceService TestProfile [label1 label2] map[address:localhost port:1234]","title":"edgex-cli provisionwatcher update"},{"location":"getting-started/tools/Ch-CommandLineInterface/#core-data-service","text":"","title":"Core-data service"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-event-add","text":"Create an event with a specified number of random readings $ edgex-cli event add -d Random-Integer-Device -p Random-Integer-Device -r 1 -s Int16 -t int16 Added event 75f06078-e8da-4671-8938-ab12ebb2c244 $ edgex-cli event list -v Origin Device Profile Source Id Versionable Readings 10 Jan 22 15:38 GMT Random-Integer-Device Random-Integer-Device Int16 75f06078-e8da-4671-8938-ab12ebb2c244 {v2} [{974a70fe-71ef-4a47-a008-c89f0e4e3bb6 1641829092129391876 Random-Integer-Device Int16 Random-Integer-Device Int16 {[] } {13342}}]","title":"edgex-cli event add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-event-count","text":"Count the number of events in core data, optionally filtering by device name $ edgex-cli event count -d Random-Integer-Device Total Random-Integer-Device events: 54","title":"edgex-cli event count"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-event-list","text":"List all events, optionally specifying a limit and offset $ edgex-cli event list To see two readings only, skipping the first 100 readings: $ edgex-cli reading list --limit 2 --offset 100 Origin Device ProfileName Value ValueType 28 Jan 22 12:55 GMT Random-Integer-Device Random-Integer-Device 22502 Int16 28 Jan 22 12:55 GMT Random-Integer-Device Random-Integer-Device 1878517239016780388 Int64","title":"edgex-cli event list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-event-rm","text":"Remove events, specifying either device name or maximum event age in milliseconds - edgex-cli event rm --device {devicename} removes all events for the specified device - edgex-cli event rm --age {ms} removes all events generated in the last {ms} milliseconds $ edgex-cli event rm -a 30000 $ edgex-cli event count Total events: 0","title":"edgex-cli event rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-reading-count","text":"Count the number of readings in core data, optionally filtering by device name $ edgex-cli reading count Total readings: 235","title":"edgex-cli reading count"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-reading-list","text":"List all readings, optionally specifying a limit and offset $ edgex-cli reading list","title":"edgex-cli reading list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#support-scheduler-service","text":"","title":"Support-scheduler service"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-interval-add","text":"Add an interval $ edgex-cli interval add -n \"hourly\" -i \"1h\" {{{v2} c7c51f21-dab5-4307-a4c9-bc5d5f2194d9 201} 98a6d5f6-f4c4-4ec5-a00c-7fe24b9c9a18}","title":"edgex-cli interval add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-interval-name","text":"Return an interval by name $ edgex-cli interval name -n \"hourly\" Name Interval Start End hourly 1h","title":"edgex-cli interval name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-interval-list","text":"List all intervals $ edgex-cli interval list -j | jq '.' { \"apiVersion\": \"v2\", \"statusCode\": 200, \"intervals\": [ { \"created\": 1641830955058, \"modified\": 1641830955058, \"id\": \"98a6d5f6-f4c4-4ec5-a00c-7fe24b9c9a18\", \"name\": \"hourly\", \"interval\": \"1h\" }, { \"created\": 1641830953884, \"modified\": 1641830953884, \"id\": \"507a2a9a-82eb-41ea-afa8-79a9b0033665\", \"name\": \"midnight\", \"start\": \"20180101T000000\", \"interval\": \"24h\" } ] }","title":"edgex-cli interval list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-interval-update","text":"Update an interval, specifying either ID or name $ edgex-cli interval update -n \"hourly\" -i \"1m\" {{v2} 08239cc4-d4d7-4ea2-9915-d91b9557c742 200} $ edgex-cli interval name -n \"hourly\" -v Id Name Interval Start End 98a6d5f6-f4c4-4ec5-a00c-7fe24b9c9a18 hourly 1m","title":"edgex-cli interval update"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-interval-rm","text":"Delete a named interval and associated interval actions $ edgex-cli interval rm -n \"hourly\"","title":"edgex-cli interval rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-intervalaction-add","text":"Add an interval action $ edgex-cli intervalaction add -n \"name01\" -i \"midnight\" -a \"{\\\"type\\\": \\\"REST\\\", \\\"host\\\": \\\"192.168.0.102\\\", \\\"port\\\": 8080, \\\"httpMethod\\\": \\\"GET\\\"}\"","title":"edgex-cli intervalaction add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-intervalaction-name","text":"Return an interval action by name $ edgex-cli intervalaction name -n \"name01\" Name Interval Address Content ContentType name01 midnight {REST 192.168.0.102 8080 { GET} { 0 0 false false 0} {[]}}","title":"edgex-cli intervalaction name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-intervalaction-list","text":"List all interval actions $ edgex-cli intervalaction list Name Interval Address Content ContentType name01 midnight {REST 192.168.0.102 8080 { GET} { 0 0 false false 0} {[]}} scrub-aged-events midnight {REST localhost 59880 {/api/v2/event/age/604800000000000 DELETE} { 0 0 false false 0} {[]}}","title":"edgex-cli intervalaction list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-intervalaction-update","text":"Update an interval action, specifying either ID or name $ edgex-cli intervalaction update -n \"name01\" --admin-state \"LOCKED\" {{v2} afc7b08c-5dc6-4923-9786-30bfebc8a8b6 200} $ edgex-cli intervalaction name -n \"name01\" -j | jq '.action.adminState' \"LOCKED\"","title":"edgex-cli intervalaction update"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-intervalaction-rm","text":"Delete an interval action by name $ edgex-cli intervalaction rm -n \"name01\"","title":"edgex-cli intervalaction rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#support-notifications-service","text":"","title":"Support-notifications service"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-notification-add","text":"Add a notification to be sent $ edgex-cli notification add -s \"sender01\" -c \"content\" --category \"category04\" --labels \"l3\" {{{v2} 13938e01-a560-47d8-bb50-060effdbe490 201} 6a1138c2-b58e-4696-afa7-2074e95165eb}","title":"edgex-cli notification add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-notification-list","text":"List notifications associated with a given label, category or time range $ edgex-cli notification list -c \"category04\" Category Content Description Labels Sender Severity Status category04 content [l3] sender01 NORMAL PROCESSED $ edgex-cli notification list --start \"01 jan 20 00:00 GMT\" --end \"01 dec 24 00:00 GMT\" Category Content Description Labels Sender Severity Status category04 content [l3] sender01 NORMAL PROCESSED","title":"edgex-cli notification list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-notification-rm","text":"Delete a notification and all of its associated transmissions $ ID=`edgex-cli notification list -c \"category04\" -v -j | jq -r '.notifications[0].id'` $ echo $ID 6a1138c2-b58e-4696-afa7-2074e95165eb $ edgex-cli notification rm -i $ID $ edgex-cli notification list -c \"category04\" No notifications available","title":"edgex-cli notification rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-notification-cleanup","text":"Delete all notifications and corresponding transmissions $ edgex-cli notification cleanup $ edgex-cli notification list --start \"01 jan 20 00:00 GMT\" --end \"01 dec 24 00:00 GMT\" No notifications available","title":"edgex-cli notification cleanup"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-subscription-add","text":"Add a new subscription $ edgex-cli subscription add -n \"name01\" --receiver \"receiver01\" -c \"[{\\\"type\\\": \\\"REST\\\", \\\"host\\\": \\\"localhost\\\", \\\"port\\\": 7770, \\\"httpMethod\\\": \\\"POST\\\"}]\" --labels \"l1,l2,l3\" {{{v2} 2bbfdac0-d2e1-4f08-8344-392b8e8ddc5e 201} 1ec08af0-5767-4505-82f7-581fada6006b} $ edgex-cli subscription add -n \"name02\" --receiver \"receiver01\" -c \"[{\\\"type\\\": \\\"EMAIL\\\", \\\"recipients\\\": [\\\"123@gmail.com\\\"]}]\" --labels \"l1,l2,l3\" {{{v2} f6b417ca-740c-4dee-bc1e-c721c0de4051 201} 156fc2b9-de60-423b-9bff-5312d8452c48}","title":"edgex-cli subscription add"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-subscription-name","text":"Return a subscription by its unique name $ edgex-cli subscription name -n \"name01\" Name Description Channels Receiver Categories Labels name01 [{REST localhost 7770 { POST} { 0 0 false false 0} {[]}}] receiver01 [] [l1 l2 l3]","title":"edgex-cli subscription name"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-subscription-list","text":"List all subscriptions, optionally filtered by a given category, label or receiver $ edgex-cli subscription list --label \"l1\" Name Description Channels Receiver Categories Labels name02 [{EMAIL 0 { } { 0 0 false false 0} {[123@gmail.com]}}] receiver01 [] [l1 l2 l3] name01 [{REST localhost 7770 { POST} { 0 0 false false 0} {[]}}] receiver01 [] [l1 l2 l3]","title":"edgex-cli subscription list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-subscription-rm","text":"Delete the named subscription $ edgex-cli subscription rm -n \"name01\"","title":"edgex-cli subscription rm"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-transmission-list","text":"To create a transmission, first create a subscription and notifications: $ edgex-cli subscription add -n \"Test-Subscription\" --description \"Test data for subscription\" --categories \"health-check\" --labels \"simple\" --receiver \"tafuser\" --resend-limit 0 --admin-state \"UNLOCKED\" -c \"[{\\\"type\\\": \\\"REST\\\", \\\"host\\\": \\\"localhost\\\", \\\"port\\\": 7770, \\\"httpMethod\\\": \\\"POST\\\"}]\" {{{v2} f281ec1a-876e-4a29-a14d-195b66d0506c 201} 3b489d23-b0c7-4791-b839-d9a578ebccb9} $ edgex-cli notification add -d \"Test data for notification 1\" --category \"health-check\" --labels \"simple\" --content-type \"string\" --content \"This is a test notification\" --sender \"taf-admin\" {{{v2} 8df79c7c-03fb-4626-b6e8-bf2d616fa327 201} 0be98b91-daf9-46e2-bcca-39f009d93866} $ edgex-cli notification add -d \"Test data for notification 2\" --category \"health-check\" --labels \"simple\" --content-type \"string\" --content \"This is a test notification\" --sender \"taf-admin\" {{{v2} ec0b2444-c8b0-45d0-bbd6-847dd007c2fd 201} a7c65d7d-0f9c-47e1-82c2-c8098c47c016} $ edgex-cli notification add -d \"Test data for notification 3\" --category \"health-check\" --labels \"simple\" --content-type \"string\" --content \"This is a test notification\" --sender \"taf-admin\" {{{v2} 45af7f94-c99e-4fb1-a632-fab5ff475be4 201} f982fc97-f53f-4154-bfce-3ef8666c3911} Then list the transmissions: $ edgex-cli transmission list SubscriptionName ResendCount Status Test-Subscription 0 FAILED Test-Subscription 0 FAILED Test-Subscription 0 FAILED","title":"edgex-cli transmission list"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-transmission-id","text":"Return a transmission by ID $ ID=`edgex-cli transmission list -j | jq -r '.transmissions[0].id'` $ edgex-cli transmission id -i $ID SubscriptionName ResendCount Status Test-Subscription 0 FAILED","title":"edgex-cli transmission id"},{"location":"getting-started/tools/Ch-CommandLineInterface/#edgex-cli-transmission-rm","text":"Delete processed transmissions older than the specificed age (in milliseconds) $ edgex-cli transmission rm -a 100","title":"edgex-cli transmission rm"},{"location":"getting-started/tools/Ch-GUI/","text":"Graphical User Interface (GUI) EdgeX's graphical user interface (GUI) is provided for demonstration and development use to manage and monitor a single instance of EdgeX Foundry. Setup You can quickly run the GUI in a Docker container or as a Snap. You can also download, build and run the GUI natively on your host. Docker Compose The EdgeX GUI is now incorporated into all the secure and non-sure Docker Compose files provided by the project. Locate and download the Docker Compose file that best suits your needs from https://github.com/edgexfoundry/edgex-compose. For example, in the Jakarta branch of edgex-compose the *-with-app-sample* compose files include the Sample App Service allowing the configurable pipeline to be manipulated from the UI. See the four Docker Compose files that include the Sample App Service circled below. Note The GUI can now be used in secure mode as well as non-secure mode. See the Getting Started using Docker guide for help on how to find, download and use a Docker Compose file to run EdgeX - in this case with the Sample App Service. Secure mode with API Gateway token When first running the UI in secure mode, you will be prompted to enter a token. Following the How to get access token? link to view the documentation how get an API Gateway access token. Once you enter the token the UI will have asses to the EdgeX service via the API Gateway. Note The UI is no longer restricted to access from localhost . It can now be accessed from any IP address that can access the host system. This is allowed because the UI is secured via API Gateway token when running in secure mode. Snaps Installing EdgeX UI as a snap The latest stable version of the snap can be installed using: $ sudo snap install edgex-ui A specific release of the snap can be installed from a dedicated channel. For example, to install the 2.1 (Jakarta) release: $ sudo snap install edgex-ui --channel=2.1 The latest development version of the edgex-ui snap can be installed using: $ sudo snap install edgex-ui --edge Generate token for entering UI secure mode A JWT access token is required to access the UI securely through the API Gateway. To do so: Generate a public/private keypair $ openssl ecparam -genkey -name prime256v1 -noout -out private.pem $ openssl ec -in private.pem -pubout -out public.pem Configure user and public-key $ sudo snap set edgexfoundry env.security-proxy.user=user01,USER_ID,ES256 $ sudo snap set edgexfoundry env.security-proxy.public-key=\"$(cat public.pem)\" Generate a token $ edgexfoundry.secrets-config proxy jwt --algorithm ES256 \\ --private_key private.pem --id USER_ID --expiration=1h This output is the JWT token for UI login in secure mode. Please keep the token in a safe place for future re-use as the same token cannot be regenerated or recovered from EdgeX's secret-config CLI. The token is required each time you reopen the web page. Using the edgex-ui snap Open your browser http://localhost:4000 Please log in to EdgeX with the JWT token we generated above. For more details please refer to edgex-ui Snap Native If you are running EdgeX natively (outside of Docker Compose or a Snap), you will find instructions on how to build and run the GUI on your platform in the GUI repository README General GUI Address Once the GUI is up and running, simply visit port 4000 on the GUI's host machine (ex: http://localhost:4000) to enter the GUI Dashboard (see below). The GUI does not require any login. Menu Bar The left side of the Dashboard holds a menu bar that allows you access to the GUI functionality. The \"hamburger\" icon on the menu bar allows you to shrink or expand the menu bar to icons vs icons and menu bar labels. Mobile Device Ready The EdgeX GUI can be used/displayed on a mobile device via the mobile device's browser if the GUI address is accessible to the device. The display may be skewed in order to fit the device screen. For example, the Dashboard menu will often change to icons over the expanded labeled menu bar when shown on a mobile device. Capability The GUI allows you to manage (add, remove, update) most of the EdgeX objects to include devices, device profiles, device services, rules, schedules, notifications, app services, etc. start, stop or restart the EdgeX services explore the memory, CPU and network traffic usage of EdgeX services monitor the data stream (the events and readings) collected by sensors and devices explore the configuration of an EdgeX service Dashboard The Dashboard page (the main page of the GUI) presents you with a set of clickable \"tiles\" that provide a quick view of the status of your EdgeX instance. That is, it provides some quick data points about the EdgeX instance and what the GUI is tracking. Specifically, the tiles in the Dashboard show you: the number of device services that it is aware of and their status (locked vs unlocked) the number of devices being managed by EdgeX (through the associated device services) the number of device profiles registered with core metadata the number of schedules (or intervals) EdgeX is managing the number of notifications EdgeX has seen the number of events and readings generated by device services and passing through core data the number of EdgeX micro services currently being monitored through the system management service If for some reason the GUI has an issue or difficulty getting the information it needs to display a tile in the Dashboard when it is displayed, a popup will be displayed over the screen indicating the issue. In the example below, the support scheduling service was down and the GUI Dashboard was unable to access the scheduler service. In this way, the Dashboard provides a quick and easy way to see whether the EdgeX instance is nominal or has underlying issues. You can click on each of the tiles in the Dashboard. Doing so provides more details about each. More precisely, clicking on a tile takes you to another part of the GUI where the details of that item can be found. For example, clicking on the Device Profiles tile takes you to the Metadata page and the Device Profile tab (covered below) System The EdgeX platform is comprised of a set of micro services. The system management service (and associated executors) tracks the micro services status (up or down), metrics of the running service (memory, CPU, network traffic), and configuration influencing the operation of the service. The system management service also provides the ability (through APIs) to start, stop and restart a service. Service information and the ability to call on the start, stop, restart APIs is surfaced through the System page. Warning The system management services are deprecated in EdgeX as of Ireland. Their full replacement has not been identified, but adopters should be aware that the service will be replaced in a future release. Please note that the System List display provides access to a static list of EdgeX services. As device services and application services (among other services) may be added or removed based on use case needs (often requireing new custom south and north side services), the GUI is not made aware of these and therefore will not display details on these services. Metrics From the System Service List, you can click on the Metric icon for any service to see the memory, CPU and network traffic telemetry for any service. The referesh rate can be adjusted on the display to have the GUI poll the system management service more or less frequently. Info The metrics are provided via an associated executor feeding the system management agent telemtry data. In the case of Docker, a Docker executor is capturing standard Docker stats and relaying them to the system management agent that in turn makes these available through its APIs to the GUI. Config The configuration of each service is made available for each service by clicking on the Config icon for any service from the System Service List. The configuration is displayed in JSON form and is read only. If running Consul, use the Consul Web UI to make changes to the configuration. Operation From the System Service List, you can request to stop, start or restart any of the listed services with the operation buttons in the far right column. Warning There is no confirmation popup or warning on these requests. When you push a stop, start, restart button, the request is immediately made to the system management service for that operation. The state of the service will change when these operations are invoked. When a service is stopped, the metric and config information for the service will be unavailable. After starting (or restarting) a service, you may need to hit the Refresh button on the page to get the state and metric/config icons to change. Metadata The Metadata page (available from the Metadata menu option) provides three tabs to be able to see and manage the basic elements of metadata: device services, device profiles and devices. Device Service Tab The Device Service tab displays the device services known to EdgeX (as device services registered in core metadata). Device services cannot be added or removed through the GUI, but information about the existing device services (i.e., port, admin state) and several actions on the existing device services can be accomplished on this tab. First note that for each device service listed, the number of associated devices are depicted. If you click on the Associated Devices button, it will take you to the Device tab to be able to get more information about or work with any of the associated devices. The Settings button on each device service allows you to change the description or the admin state of the device service. Alert Please note that you must hit the Save button after making any changes to the Device Service Settings. If you don't and move away from the page, your changes will be lost. Device Tab The Device Tab on the Metadata page offers you details about all the sensors/devices known to your EdgeX instance. Buttons at the top of the tab allow you to add, remove or edit a device (or collection of devices when deleting and using the selector checkbox in the device list). On the row of each device listed, links take you to the appropriate tabs to see the associated device profile or device service for the device. Icons on the row of each device listed cause editable areas to expand at the bottom of the tab to execute a device command or see/modify the device's AutoEvents. The command execution display allows you to select the specific device resource or device command (from the Command Name List ), and execute or try either a GET or SET command (depending on what the associated device profile for the device says is allowed). The response will be displayed in the ResponseRaw area after the try button is pushed. Add Device Wizard The Add button on the Device List tab will take you to the Add Device Wizard . This nice utility will assist you, entry screen by entry screen, in getting a new device setup in EdgeX. Specifically, it has you (in order): select the device service to which the new device will be associated select the device profile to which the new device will be templated or typed after enter general characteristics for the device (name, description, labels, etc.) and set its operating and admin states optionally setup auto events for scheduled data collection enter specific protocol properties for the device (based on known templates the GUI has at its disposal such as REST, MQTT, Modbus, etc.) Once all the information in the Add Device Wizard screens is entered, the Submit button at the end of the wizard causes your new device to be created in core metadata with all appropriate associations. Device Profile Tab The Device Profile Tab on the Metadata page displays the device profiles known to EdgeX and allows you to add new profiles or edit/remove existing profiles. The AssociatedDevice button on each row of the Device Profile List will take you to the Device tab and show you the list of devices currently associated to the device profile. Warning When deleting a profile, the system will popup an error if deices are still associated to the profile. Data Center (Seeing Event/Reading Data) From the Data Center option on the GUI's menu bar you can see the stream of Event/Readings coming from the device services into core data. The event/reading data will be displayed in JSON form. There are two tabs on the Data Stream page, both with Start and Pause buttons: Event (which allows incoming events to be displayed and the display will include the event's associated readings) Reading (allows incoming readings to be displayed, which will only show the reading and not its associated owning event) Hit the Start button on either tab to see the event or reading data displayed in the stream pane (events are shown in the example below). Push the Pause button to stop the display of event or reading data. Warning In actuality, the event and reading data is pulled from core data via REST call every three (3) seconds - so it is not a live stream display but a poll of data. Furthermore, if EdgeX is setup to have device services send data directly to application services via message bus and core data is not running or if core data is configured to have persistence turned off, there will be no data in core data to pull and so there will be no events or readings to see. Scheduler (Interval/Interval List) Interval and Interval Actions, which help define task management schedules in EdgeX, are managed via the Scheduler page from selecting Scheduler off the menu bar. Again, as with many of the EdgeX GUI pages, there are two tabs on the Scheduler page: Interval List to display, add, edit and delete Intervals Interval Action List to display, add, edit and delete Interval Actions which must be associated to an Interval Interval List When updating or adding an Interval, you must provide a name Interval duration string which takes an unsigned integer plus a unit of measure which must be one of \"ns\", \"us\" (or \"\u00b5s\"), \"ms\", \"s\", \"m\", \"h\" representing nanoseconds, microseconds, milliseconds, seconds, minutes or hours. Optionally provide a start/end dates and an indication that the interval runs only once (and thereby ignores the interval). Interval Action List Interval Actions define what happens when the Interval kicks off. Interval Actions can define REST, MQTT or Email actions that take place when an Interval timer hits. The GUI provides the means to edit or create any of these actions. Note that an Interval Action must be associated to an already defined Interval. Notifications Notifications are messages from EdgeX to external systems about something that has happened in EdgeX - for example that a new device has been created. Currently, notifications can be sent by email or REST call. The Notification Center page, available from the Notifications menu option, allows you to see new (not processed), processed or escalated (notifications that have failed to be sent within its resend limit) notifications. By default, the new notifications are displayed, but if you click on the Advanced >> link on the page (see below), you can select which type of notifications to display. The Subscriptions tab on the Notification Center page allows you to add, update or remove subscriptions to notifications. Subscribers are registered receivers of notifications - either via email or REST. When adding (or editing) a subscription, you must provide a name, category, label, receiver, and either an email address or REST endpoint. A template is provided to specify either the email or REST endpoint configuration data needed for the subscription. RuleEngine The Rule Engine page, from the RuleEngine menu option, provides the means to define streams and rules for the integrated eKuiper rules engine. Via the Stream tab, streams are defined by JSON. All that is really required is a stream name (EdgeXStream in the example below). The Rules tab allows eKuiper rules to be added, removed or updated/edited as well as started, stopped or restarted. When adding or editing a rule, you must provide a name, the rule SQL and action. The action can be one of the following (some requiring extra parameters): send the result to a REST HTTP Server (allowing an EdgeX command to be called) send the result to an MQTT broker send the result to the EdgeX message bus send the result to a log file See the eKuiper documentation for more information on how to define rules. Alert Once a rule is created, it is started by default. Return to the Rules tab on the RulesEngine page to stop a new rule. When creating or editing the rule, if the stream referenced in the rule is not already defined, the GUI will present an error when trying to submit the rule. AppService In the AppService page, you can configure existing configurable application services . The list of available configurable app services is determined by the UI automatically (based on a query for available app services from the registry service). Configurable When the application service is a configurable app service and is known to the GUI, the Configurable button on the App Service List allows you to change the triggers, functions, secrets and other configuration associated to the configurable app service. There are four tabs in the Configurable Setting editor: Trigger which defines how the configurable app service begins execution Pipeline Functions defining which functions are part of the configurable app service pipeline and in which order should they be executed Insecure Secrets - setting up secrets used by the configurable app service when running in non-secure mode (meaning Vault is not used to provide the secrets) Store and Forward which enables and configures the batch store and forward export capability Note When the Trigger is changed, the service must be restarted for the change to take effect. Why Demo and Developer Use Only The GUI is meant as a developer tool or to be used in EdgeX demonstration situations. It is not yet designed for production settings. There are several reasons for this restriction. The GUI is not designed to assist you in managing multiple EdgeX instances running in a deployment as would be typical in a production setting. It cannot be dynamically pointed to any running instance of EdgeX on multiple hosts. The GUI knows about a single instance of EdgeX running (by default, the instance that is on the same host as the GUI). The GUI provides no access controls. All functionality is open to anyone that can access the GUI URL. The GUI does not have the Kong token to negotiate through the API Gateway when the GUI is running outside of the Docker network - where the other EdgeX services are running. This would mean that the GUI would not be able to access any of the EdgeX service instance APIs. The EdgeX community is exploring efforts to make the GUI available in secure mode in a future release.","title":"Graphical User Interface (GUI)"},{"location":"getting-started/tools/Ch-GUI/#graphical-user-interface-gui","text":"EdgeX's graphical user interface (GUI) is provided for demonstration and development use to manage and monitor a single instance of EdgeX Foundry.","title":"Graphical User Interface (GUI)"},{"location":"getting-started/tools/Ch-GUI/#setup","text":"You can quickly run the GUI in a Docker container or as a Snap. You can also download, build and run the GUI natively on your host.","title":"Setup"},{"location":"getting-started/tools/Ch-GUI/#docker-compose","text":"The EdgeX GUI is now incorporated into all the secure and non-sure Docker Compose files provided by the project. Locate and download the Docker Compose file that best suits your needs from https://github.com/edgexfoundry/edgex-compose. For example, in the Jakarta branch of edgex-compose the *-with-app-sample* compose files include the Sample App Service allowing the configurable pipeline to be manipulated from the UI. See the four Docker Compose files that include the Sample App Service circled below. Note The GUI can now be used in secure mode as well as non-secure mode. See the Getting Started using Docker guide for help on how to find, download and use a Docker Compose file to run EdgeX - in this case with the Sample App Service.","title":"Docker Compose"},{"location":"getting-started/tools/Ch-GUI/#secure-mode-with-api-gateway-token","text":"When first running the UI in secure mode, you will be prompted to enter a token. Following the How to get access token? link to view the documentation how get an API Gateway access token. Once you enter the token the UI will have asses to the EdgeX service via the API Gateway. Note The UI is no longer restricted to access from localhost . It can now be accessed from any IP address that can access the host system. This is allowed because the UI is secured via API Gateway token when running in secure mode.","title":"Secure mode with API Gateway token"},{"location":"getting-started/tools/Ch-GUI/#snaps","text":"","title":"Snaps"},{"location":"getting-started/tools/Ch-GUI/#installing-edgex-ui-as-a-snap","text":"The latest stable version of the snap can be installed using: $ sudo snap install edgex-ui A specific release of the snap can be installed from a dedicated channel. For example, to install the 2.1 (Jakarta) release: $ sudo snap install edgex-ui --channel=2.1 The latest development version of the edgex-ui snap can be installed using: $ sudo snap install edgex-ui --edge","title":"Installing EdgeX UI as a snap"},{"location":"getting-started/tools/Ch-GUI/#generate-token-for-entering-ui-secure-mode","text":"A JWT access token is required to access the UI securely through the API Gateway. To do so: Generate a public/private keypair $ openssl ecparam -genkey -name prime256v1 -noout -out private.pem $ openssl ec -in private.pem -pubout -out public.pem Configure user and public-key $ sudo snap set edgexfoundry env.security-proxy.user=user01,USER_ID,ES256 $ sudo snap set edgexfoundry env.security-proxy.public-key=\"$(cat public.pem)\" Generate a token $ edgexfoundry.secrets-config proxy jwt --algorithm ES256 \\ --private_key private.pem --id USER_ID --expiration=1h This output is the JWT token for UI login in secure mode. Please keep the token in a safe place for future re-use as the same token cannot be regenerated or recovered from EdgeX's secret-config CLI. The token is required each time you reopen the web page.","title":"Generate token for entering UI secure mode"},{"location":"getting-started/tools/Ch-GUI/#using-the-edgex-ui-snap","text":"Open your browser http://localhost:4000 Please log in to EdgeX with the JWT token we generated above. For more details please refer to edgex-ui Snap","title":"Using the edgex-ui snap"},{"location":"getting-started/tools/Ch-GUI/#native","text":"If you are running EdgeX natively (outside of Docker Compose or a Snap), you will find instructions on how to build and run the GUI on your platform in the GUI repository README","title":"Native"},{"location":"getting-started/tools/Ch-GUI/#general","text":"","title":"General"},{"location":"getting-started/tools/Ch-GUI/#gui-address","text":"Once the GUI is up and running, simply visit port 4000 on the GUI's host machine (ex: http://localhost:4000) to enter the GUI Dashboard (see below). The GUI does not require any login.","title":"GUI Address"},{"location":"getting-started/tools/Ch-GUI/#menu-bar","text":"The left side of the Dashboard holds a menu bar that allows you access to the GUI functionality. The \"hamburger\" icon on the menu bar allows you to shrink or expand the menu bar to icons vs icons and menu bar labels.","title":"Menu Bar"},{"location":"getting-started/tools/Ch-GUI/#mobile-device-ready","text":"The EdgeX GUI can be used/displayed on a mobile device via the mobile device's browser if the GUI address is accessible to the device. The display may be skewed in order to fit the device screen. For example, the Dashboard menu will often change to icons over the expanded labeled menu bar when shown on a mobile device.","title":"Mobile Device Ready"},{"location":"getting-started/tools/Ch-GUI/#capability","text":"The GUI allows you to manage (add, remove, update) most of the EdgeX objects to include devices, device profiles, device services, rules, schedules, notifications, app services, etc. start, stop or restart the EdgeX services explore the memory, CPU and network traffic usage of EdgeX services monitor the data stream (the events and readings) collected by sensors and devices explore the configuration of an EdgeX service","title":"Capability"},{"location":"getting-started/tools/Ch-GUI/#dashboard","text":"The Dashboard page (the main page of the GUI) presents you with a set of clickable \"tiles\" that provide a quick view of the status of your EdgeX instance. That is, it provides some quick data points about the EdgeX instance and what the GUI is tracking. Specifically, the tiles in the Dashboard show you: the number of device services that it is aware of and their status (locked vs unlocked) the number of devices being managed by EdgeX (through the associated device services) the number of device profiles registered with core metadata the number of schedules (or intervals) EdgeX is managing the number of notifications EdgeX has seen the number of events and readings generated by device services and passing through core data the number of EdgeX micro services currently being monitored through the system management service If for some reason the GUI has an issue or difficulty getting the information it needs to display a tile in the Dashboard when it is displayed, a popup will be displayed over the screen indicating the issue. In the example below, the support scheduling service was down and the GUI Dashboard was unable to access the scheduler service. In this way, the Dashboard provides a quick and easy way to see whether the EdgeX instance is nominal or has underlying issues. You can click on each of the tiles in the Dashboard. Doing so provides more details about each. More precisely, clicking on a tile takes you to another part of the GUI where the details of that item can be found. For example, clicking on the Device Profiles tile takes you to the Metadata page and the Device Profile tab (covered below)","title":"Dashboard"},{"location":"getting-started/tools/Ch-GUI/#system","text":"The EdgeX platform is comprised of a set of micro services. The system management service (and associated executors) tracks the micro services status (up or down), metrics of the running service (memory, CPU, network traffic), and configuration influencing the operation of the service. The system management service also provides the ability (through APIs) to start, stop and restart a service. Service information and the ability to call on the start, stop, restart APIs is surfaced through the System page. Warning The system management services are deprecated in EdgeX as of Ireland. Their full replacement has not been identified, but adopters should be aware that the service will be replaced in a future release. Please note that the System List display provides access to a static list of EdgeX services. As device services and application services (among other services) may be added or removed based on use case needs (often requireing new custom south and north side services), the GUI is not made aware of these and therefore will not display details on these services.","title":"System"},{"location":"getting-started/tools/Ch-GUI/#metrics","text":"From the System Service List, you can click on the Metric icon for any service to see the memory, CPU and network traffic telemetry for any service. The referesh rate can be adjusted on the display to have the GUI poll the system management service more or less frequently. Info The metrics are provided via an associated executor feeding the system management agent telemtry data. In the case of Docker, a Docker executor is capturing standard Docker stats and relaying them to the system management agent that in turn makes these available through its APIs to the GUI.","title":"Metrics"},{"location":"getting-started/tools/Ch-GUI/#config","text":"The configuration of each service is made available for each service by clicking on the Config icon for any service from the System Service List. The configuration is displayed in JSON form and is read only. If running Consul, use the Consul Web UI to make changes to the configuration.","title":"Config"},{"location":"getting-started/tools/Ch-GUI/#operation","text":"From the System Service List, you can request to stop, start or restart any of the listed services with the operation buttons in the far right column. Warning There is no confirmation popup or warning on these requests. When you push a stop, start, restart button, the request is immediately made to the system management service for that operation. The state of the service will change when these operations are invoked. When a service is stopped, the metric and config information for the service will be unavailable. After starting (or restarting) a service, you may need to hit the Refresh button on the page to get the state and metric/config icons to change.","title":"Operation"},{"location":"getting-started/tools/Ch-GUI/#metadata","text":"The Metadata page (available from the Metadata menu option) provides three tabs to be able to see and manage the basic elements of metadata: device services, device profiles and devices.","title":"Metadata"},{"location":"getting-started/tools/Ch-GUI/#device-service-tab","text":"The Device Service tab displays the device services known to EdgeX (as device services registered in core metadata). Device services cannot be added or removed through the GUI, but information about the existing device services (i.e., port, admin state) and several actions on the existing device services can be accomplished on this tab. First note that for each device service listed, the number of associated devices are depicted. If you click on the Associated Devices button, it will take you to the Device tab to be able to get more information about or work with any of the associated devices. The Settings button on each device service allows you to change the description or the admin state of the device service. Alert Please note that you must hit the Save button after making any changes to the Device Service Settings. If you don't and move away from the page, your changes will be lost.","title":"Device Service Tab"},{"location":"getting-started/tools/Ch-GUI/#device-tab","text":"The Device Tab on the Metadata page offers you details about all the sensors/devices known to your EdgeX instance. Buttons at the top of the tab allow you to add, remove or edit a device (or collection of devices when deleting and using the selector checkbox in the device list). On the row of each device listed, links take you to the appropriate tabs to see the associated device profile or device service for the device. Icons on the row of each device listed cause editable areas to expand at the bottom of the tab to execute a device command or see/modify the device's AutoEvents. The command execution display allows you to select the specific device resource or device command (from the Command Name List ), and execute or try either a GET or SET command (depending on what the associated device profile for the device says is allowed). The response will be displayed in the ResponseRaw area after the try button is pushed.","title":"Device Tab"},{"location":"getting-started/tools/Ch-GUI/#add-device-wizard","text":"The Add button on the Device List tab will take you to the Add Device Wizard . This nice utility will assist you, entry screen by entry screen, in getting a new device setup in EdgeX. Specifically, it has you (in order): select the device service to which the new device will be associated select the device profile to which the new device will be templated or typed after enter general characteristics for the device (name, description, labels, etc.) and set its operating and admin states optionally setup auto events for scheduled data collection enter specific protocol properties for the device (based on known templates the GUI has at its disposal such as REST, MQTT, Modbus, etc.) Once all the information in the Add Device Wizard screens is entered, the Submit button at the end of the wizard causes your new device to be created in core metadata with all appropriate associations.","title":"Add Device Wizard"},{"location":"getting-started/tools/Ch-GUI/#device-profile-tab","text":"The Device Profile Tab on the Metadata page displays the device profiles known to EdgeX and allows you to add new profiles or edit/remove existing profiles. The AssociatedDevice button on each row of the Device Profile List will take you to the Device tab and show you the list of devices currently associated to the device profile. Warning When deleting a profile, the system will popup an error if deices are still associated to the profile.","title":"Device Profile Tab"},{"location":"getting-started/tools/Ch-GUI/#data-center-seeing-eventreading-data","text":"From the Data Center option on the GUI's menu bar you can see the stream of Event/Readings coming from the device services into core data. The event/reading data will be displayed in JSON form. There are two tabs on the Data Stream page, both with Start and Pause buttons: Event (which allows incoming events to be displayed and the display will include the event's associated readings) Reading (allows incoming readings to be displayed, which will only show the reading and not its associated owning event) Hit the Start button on either tab to see the event or reading data displayed in the stream pane (events are shown in the example below). Push the Pause button to stop the display of event or reading data. Warning In actuality, the event and reading data is pulled from core data via REST call every three (3) seconds - so it is not a live stream display but a poll of data. Furthermore, if EdgeX is setup to have device services send data directly to application services via message bus and core data is not running or if core data is configured to have persistence turned off, there will be no data in core data to pull and so there will be no events or readings to see.","title":"Data Center (Seeing Event/Reading Data)"},{"location":"getting-started/tools/Ch-GUI/#scheduler-intervalinterval-list","text":"Interval and Interval Actions, which help define task management schedules in EdgeX, are managed via the Scheduler page from selecting Scheduler off the menu bar. Again, as with many of the EdgeX GUI pages, there are two tabs on the Scheduler page: Interval List to display, add, edit and delete Intervals Interval Action List to display, add, edit and delete Interval Actions which must be associated to an Interval","title":"Scheduler (Interval/Interval List)"},{"location":"getting-started/tools/Ch-GUI/#interval-list","text":"When updating or adding an Interval, you must provide a name Interval duration string which takes an unsigned integer plus a unit of measure which must be one of \"ns\", \"us\" (or \"\u00b5s\"), \"ms\", \"s\", \"m\", \"h\" representing nanoseconds, microseconds, milliseconds, seconds, minutes or hours. Optionally provide a start/end dates and an indication that the interval runs only once (and thereby ignores the interval).","title":"Interval List"},{"location":"getting-started/tools/Ch-GUI/#interval-action-list","text":"Interval Actions define what happens when the Interval kicks off. Interval Actions can define REST, MQTT or Email actions that take place when an Interval timer hits. The GUI provides the means to edit or create any of these actions. Note that an Interval Action must be associated to an already defined Interval.","title":"Interval Action List"},{"location":"getting-started/tools/Ch-GUI/#notifications","text":"Notifications are messages from EdgeX to external systems about something that has happened in EdgeX - for example that a new device has been created. Currently, notifications can be sent by email or REST call. The Notification Center page, available from the Notifications menu option, allows you to see new (not processed), processed or escalated (notifications that have failed to be sent within its resend limit) notifications. By default, the new notifications are displayed, but if you click on the Advanced >> link on the page (see below), you can select which type of notifications to display. The Subscriptions tab on the Notification Center page allows you to add, update or remove subscriptions to notifications. Subscribers are registered receivers of notifications - either via email or REST. When adding (or editing) a subscription, you must provide a name, category, label, receiver, and either an email address or REST endpoint. A template is provided to specify either the email or REST endpoint configuration data needed for the subscription.","title":"Notifications"},{"location":"getting-started/tools/Ch-GUI/#ruleengine","text":"The Rule Engine page, from the RuleEngine menu option, provides the means to define streams and rules for the integrated eKuiper rules engine. Via the Stream tab, streams are defined by JSON. All that is really required is a stream name (EdgeXStream in the example below). The Rules tab allows eKuiper rules to be added, removed or updated/edited as well as started, stopped or restarted. When adding or editing a rule, you must provide a name, the rule SQL and action. The action can be one of the following (some requiring extra parameters): send the result to a REST HTTP Server (allowing an EdgeX command to be called) send the result to an MQTT broker send the result to the EdgeX message bus send the result to a log file See the eKuiper documentation for more information on how to define rules. Alert Once a rule is created, it is started by default. Return to the Rules tab on the RulesEngine page to stop a new rule. When creating or editing the rule, if the stream referenced in the rule is not already defined, the GUI will present an error when trying to submit the rule.","title":"RuleEngine"},{"location":"getting-started/tools/Ch-GUI/#appservice","text":"In the AppService page, you can configure existing configurable application services . The list of available configurable app services is determined by the UI automatically (based on a query for available app services from the registry service).","title":"AppService"},{"location":"getting-started/tools/Ch-GUI/#configurable","text":"When the application service is a configurable app service and is known to the GUI, the Configurable button on the App Service List allows you to change the triggers, functions, secrets and other configuration associated to the configurable app service. There are four tabs in the Configurable Setting editor: Trigger which defines how the configurable app service begins execution Pipeline Functions defining which functions are part of the configurable app service pipeline and in which order should they be executed Insecure Secrets - setting up secrets used by the configurable app service when running in non-secure mode (meaning Vault is not used to provide the secrets) Store and Forward which enables and configures the batch store and forward export capability Note When the Trigger is changed, the service must be restarted for the change to take effect.","title":"Configurable"},{"location":"getting-started/tools/Ch-GUI/#why-demo-and-developer-use-only","text":"The GUI is meant as a developer tool or to be used in EdgeX demonstration situations. It is not yet designed for production settings. There are several reasons for this restriction. The GUI is not designed to assist you in managing multiple EdgeX instances running in a deployment as would be typical in a production setting. It cannot be dynamically pointed to any running instance of EdgeX on multiple hosts. The GUI knows about a single instance of EdgeX running (by default, the instance that is on the same host as the GUI). The GUI provides no access controls. All functionality is open to anyone that can access the GUI URL. The GUI does not have the Kong token to negotiate through the API Gateway when the GUI is running outside of the Docker network - where the other EdgeX services are running. This would mean that the GUI would not be able to access any of the EdgeX service instance APIs. The EdgeX community is exploring efforts to make the GUI available in secure mode in a future release.","title":"Why Demo and Developer Use Only"},{"location":"microservices/application/AdvancedTopics/","text":"Advanced Topics The following items discuss topics that are a bit beyond the basic use cases of the Application Functions SDK when interacting with EdgeX. Configurable Functions Pipeline This SDK provides the capability to define the functions pipeline via configuration rather than code by using the app-service-configurable application service. See the App Service Configurable section for more details. Custom REST Endpoints It is not uncommon to require your own custom REST endpoints when building an Application Service. Rather than spin up your own webserver inside of your app (alongside the already existing running webserver), we've exposed a method that allows you add your own routes to the existing webserver. A few routes are reserved and cannot be used: /api/v2/version /api/v2/ping /api/v2/metrics /api/v2/config /api/v2/trigger /api/v2/secret To add your own route, use the AddRoute() API provided on the ApplicationService interface. Example - Add Custom REST route myhandler := func ( writer http . ResponseWriter , req * http . Request ) { service := req . Context (). Value ( interfaces . AppServiceContextKey ).( interfaces . ApplicationService ) service . LoggingClient (). Info ( \"TEST\" ) writer . Header (). Set ( \"Content-Type\" , \"text/plain\" ) writer . Write ([] byte ( \"hello\" )) writer . WriteHeader ( 200 ) } service := pkg . NewAppService ( serviceKey ) service . AddRoute ( \"/myroute\" , myHandler , \"GET\" ) Under the hood, this simply adds the provided route, handler, and method to the gorilla mux.Router used in the SDK. For more information on gorilla mux you can check out the github repo here . You can access the interfaces.ApplicationService API for resources such as the logging client by pulling it from the context as shown above -- this is useful for when your routes might not be defined in your main.go where you have access to the interfaces.ApplicationService instance. Target Type The target type is the object type of the incoming data that is sent to the first function in the function pipeline. By default this is an EdgeX dtos.Event since typical usage is receiving Events from the EdgeX MessageBus. There are scenarios where the incoming data is not an EdgeX Event . One example scenario is two application services are chained via the EdgeX MessageBus. The output of the first service is inference data from analyzing the original Event data, and published back to the EdgeX MessageBus. The second service needs to be able to let the SDK know the target type of the input data it is expecting. For usages where the incoming data is not events , the TargetType of the expected incoming data can be set when the ApplicationService instance is created using the NewAppServiceWithTargetType() factory function. Example - Set and use custom Target Type type Person struct { FirstName string `json:\"first_name\"` LastName string `json:\"last_name\"` } service := pkg . NewAppServiceWithTargetType ( serviceKey , & Person {}) TargetType must be set to a pointer to an instance of your target type such as &Person{} . The first function in your function pipeline will be passed an instance of your target type, not a pointer to it. In the example above, the first function in the pipeline would start something like: func MyPersonFunction ( ctx interfaces . AppFunctionContext , data interface {}) ( bool , interface {}) { ctx . LoggingClient (). Debug ( \"MyPersonFunction executing\" ) if data == nil { return false , errors . New ( \"no data received to MyPersonFunction\" ) } person , ok := data .( Person ) if ! ok { return false , errors . New ( \"MyPersonFunction type received is not a Person\" ) } // .... The SDK supports un-marshaling JSON or CBOR encoded data into an instance of the target type. If your incoming data is not JSON or CBOR encoded, you then need to set the TargetType to &[]byte . If the target type is set to &[]byte the incoming data will not be un-marshaled. The content type, if set, will be set on the interfaces.AppFunctionContext and can be access via the InputContentType() API. Your first function will be responsible for decoding the data or not. Command Line Options See the Common Command Line Options for the set of command line options common to all EdgeX services. The following command line options are specific to Application Services. Skip Version Check -s/--skipVersionCheck Indicates the service should skip the Core Service's version compatibility check. Service Key -sk/--serviceKey Sets the service key that is used with Registry, Configuration Provider and security services. The default service key is set by the application service. If the name provided contains the placeholder text , this text will be replaced with the name of the profile used. If profile is not set, the text is simply removed Can be overridden with EDGEX_SERVICE_KEY environment variable. Environment Variables See the Common Environment Variables section for the list of environment variables common to all EdgeX Services. The remaining in this section are specific to Application Services. EDGEX_SERVICE_KEY This environment variable overrides the -sk/--serviceKey command-line option and the default set by the application service. Note If the name provided contains the text , this text will be replaced with the name of the profile used. Example - Service Key EDGEX_SERVICE_KEY: app--mycloud profile: http-export then service key will be app-http-export-mycloud EdgeX 2.0 The deprecated lowercase `edgex_service environment variable specific have been removed for EdgeX 2.0 Custom Configuration Applications can specify custom configuration in the TOML file in two ways. Application Settings The first simple way is to add items to the ApplicationSetting section. This is a map of string key/value pairs, i.e. map[string]string . Use for simple string values or comma separated list of string values. The ApplicationService API provides the follow access APIs for this configuration section: ApplicationSettings() map[string]string Returns the whole list of application settings GetAppSetting(setting string) (string, error) Returns single entry from the map who's key matches the passed in setting value GetAppSettingStrings(setting string) ([]string, error) Returns list of strings for the entry who's key matches the passed in setting value. The Entry is assumed to be a comma separated list of strings. Structure Custom Configuration EdgeX 2.0 Structure Custom Configuration is new for Edgex 2.0 The second is the more complex Structured Custom Configuration which allows the Application Service to define and watch it's own structured section in the service's TOML configuration file. The ApplicationService API provides the follow APIs to enable structured custom configuration: LoadCustomConfig(config UpdatableConfig, sectionName string) error Loads the service's custom configuration from local file or the Configuration Provider (if enabled). The Configuration Provider will also be seeded with the custom configuration the first time the service is started, if service is using the Configuration Provider. The UpdateFromRaw interface will be called on the custom configuration when the configuration is loaded from the Configuration Provider. ListenForCustomConfigChanges(configToWatch interface{}, sectionName string, changedCallback func(interface{})) error Starts a listener on the Configuration Provider for changes to the specified section of the custom configuration. When changes are received from the Configuration Provider the UpdateWritableFromRaw interface will be called on the custom configuration to apply the updates and then signal that the changes occurred via changedCallback. See the Application Service Template for an example of using the new Structured Custom Configuration capability. See here for defining the structured custom configuration See here for loading, validating and watching the configuration Store and Forward The Store and Forward capability allows for export functions to persist data on failure and for the export of the data to be retried at a later time. Note The order the data exported via this retry mechanism is not guaranteed to be the same order in which the data was initial received from Core Data Configuration Writable.StoreAndForward allows enabling, setting the interval between retries and the max number of retries. If running with Configuration Provider, these setting can be changed on the fly via Consul without having to restart the service. Example - Store and Forward configuration [Writable.StoreAndForward] Enabled = false RetryInterval = \"5m\" MaxRetryCount = 10 Note RetryInterval should be at least 1 second (eg. '1s') or greater. If a value less than 1 second is specified, 1 second will be used. Endless retries will occur when MaxRetryCount is set to 0. If MaxRetryCount is set to less than 0, a default of 1 retry will be used. Database configuration section describes which database type to use and the information required to connect to the database. This section is required if Store and Forward is enabled. It is optional if not using Redis for the EdgeX MessageBus which is now the default. Example - Database configuration [Database] Type = \"redisdb\" Host = \"localhost\" Port = 6379 Timeout = \"30s\" EdgeX 2.0 Support for Mongo DB has been removed in EdgeX 2.0 How it works When an export function encounters an error sending data it can call SetRetryData(payload []byte) on the AppFunctionContext . This will store the data for later retry. If the Application Service is stopped and then restarted while stored data hasn't been successfully exported, the export retry will resume once the service is up and running again. Note It is important that export functions return an error and stop pipeline execution after the call to SetRetryData . See HTTPPost function in SDK as an example When the RetryInterval expires, the function pipeline will be re-executed starting with the export function that saved the data. The saved data will be passed to the export function which can then attempt to resend the data. Note The export function will receive the data as it was stored, so it is important that any transformation of the data occur in functions prior to the export function. The export function should only export the data that it receives. One of three out comes can occur after the export retried has completed. Export retry was successful In this case, the stored data is removed from the database and the execution of the pipeline functions after the export function, if any, continues. Export retry fails and retry count has not been exceeded In this case, the stored data is updated in the database with the incremented retry count Export retry fails and retry count has been exceeded In this case, the stored data is removed from the database and never retried again. Note Changing Writable.Pipeline.ExecutionOrder will invalidate all currently stored data and result in it all being removed from the database on the next retry. This is because the position of the export function can no longer be guaranteed and no way to ensure it is properly executed on the retry. Custom Storage The default backing store is redis. Custom implementations of the StoreClient interface can be provided if redis does not meet your requirements. type StoreClient interface { // Store persists a stored object to the data store and returns the assigned UUID. Store ( o StoredObject ) ( id string , err error ) // RetrieveFromStore gets an object from the data store. RetrieveFromStore ( appServiceKey string ) ( objects [] StoredObject , err error ) // Update replaces the data currently in the store with the provided data. Update ( o StoredObject ) error // RemoveFromStore removes an object from the data store. RemoveFromStore ( o StoredObject ) error // Disconnect ends the connection. Disconnect () error } A factory function to create these clients can then be registered with your service by calling RegisterCustomStoreFactory service . RegisterCustomStoreFactory ( \"jetstream\" , func ( cfg interfaces . DatabaseInfo , cred config . Credentials ) ( interfaces . StoreClient , error ) { conn , err := nats . Connect ( fmt . Sprintf ( \"nats://%s:%d\" , cfg . Host , cfg . Port )) if err != nil { return nil , err } js , err := conn . JetStream () if err != nil { return nil , err } kv , err := js . KeyValue ( serviceKey ) if err != nil { kv , err = js . CreateKeyValue ( & nats . KeyValueConfig { Bucket : serviceKey }) } return & JetstreamStore { conn : conn , serviceKey : serviceKey , kv : kv , }, err }) and configured using the registered name in the Database section: [Database] Type = \"jetstream\" Host = \"broker\" Port = 4222 Timeout = \"5s\" Secrets Configuration All instances of App Services running in secure mode require a SecretStore to be configured. With the use of Redis Pub/Sub as the default EdgeX MessageBus all App Services need the redisdb known secret added to their SecretStore so they can connect to the Secure EdgeX MessageBus. See the Secure MessageBus documentation for more details. Example - SecretStore configuration [SecretStore] Type = \"vault\" Host = \"localhost\" Port = 8200 Path = \"app-sample/\" Protocol = \"http\" RootCaCertPath = \"\" ServerName = \"\" TokenFile = \"/tmp/edgex/secrets/app-sample/secrets-token.json\" [SecretStore.Authentication] AuthType = \"X-Vault-Token\" EdgeX 2.0 For Edgex 2.0 all Application Service Secret Stores are exclusive so the explicit [SecretStoreExclusive] configuration has been removed. Storing Secrets Secure Mode When running an application service in secure mode, secrets can be stored in the SecretStore by making an HTTP POST call to the /api/v2/secret API route in the application service. The secret data POSTed is stored and retrieved from the SecretStore based on values in the [SecretStore] section of the configuration file. Once a secret is stored, only the service that added the secret will be able to retrieve it. For secret retrieval see Getting Secrets section below. Example - JSON message body { \"path\" : \"MyPath\" , \"secretData\" : [ { \"key\" : \"MySecretKey\" , \"value\" : \"MySecretValue\" } ] } Note Path specifies the type or location of the secret in the SecretStore. It is appended to the base path from the [SecretStore] configuration. Insecure Mode When running in insecure mode, the secrets are stored and retrieved from the Writable.InsecureSecrets section of the service's configuration toml file. Insecure secrets and their paths can be configured as below. Example - InsecureSecrets Configuration [ Writable . InsecureSecrets ] [Writable.InsecureSecrets.AWS] Path = \"aws\" [Writable.InsecureSecrets.AWS.Secrets] username = \"aws-user\" password = \"aws-pw\" [Writable.InsecureSecrets.DB] Path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\" Getting Secrets Application Services can retrieve their secrets from their SecretStore using the interfaces.ApplicationService.GetSecret() API or from the interfaces.AppFunctionContext.GetSecret() API When in secure mode, the secrets are retrieved from the SecretStore based on the [SecretStore] configuration values. When running in insecure mode, the secrets are retrieved from the [Writable.InsecureSecrets] configuration. Background Publishing Application Services using the MessageBus trigger can request a background publisher using the AddBackgroundPublisher API in the SDK. This method takes an int representing the background channel's capacity as the only parameter and returns a reference to a BackgroundPublisher. This reference can then be used by background processes to publish to the configured MessageBus output. A custom topic can be provided to use instead of the configured message bus output as well. Edgex 2.0 For EdgeX 2.0 the background publish operation takes a full AppContext instead of just the parameters used to create a message envelope. This allows the background publisher to leverage context-based topic formatting functionality as the trigger output. Example - Background Publisher func runJob ( service interfaces . ApplicationService , done chan struct {}){ ticker := time . NewTicker ( 1 * time . Minute ) //initialize background publisher with a channel capacity of 10 and a custom topic publisher , err := service . AddBackgroundPublisherWithTopic ( 10 , \"custom-topic\" ) if err != nil { // do something } go func ( pub interfaces . BackgroundPublisher ) { for { select { case <- ticker . C : msg := myDataService . GetMessage () payload , err := json . Marshal ( message ) if err != nil { //do something } ctx := svc . BuildContext ( uuid . NewString (), common . ContentTypeJSON ) // modify context as needed err = pub . Publish ( payload , ctx ) if err != nil { //do something } case <- j . done : ticker . Stop () return } } }( publisher ) } func main () { service := pkg . NewAppService ( serviceKey ) done := make ( chan struct {}) defer close ( done ) //pass publisher to your background job runJob ( service , done ) service . SetFunctionsPipeline ( All , My , Functions , ) service . MakeItRun () os . Exit ( 0 ) } Stopping the Service Application Services will listen for SIGTERM / SIGINT signals from the OS and stop the function pipeline in response. The pipeline can also be exited programmatically by calling sdk.MakeItStop() on the running ApplicationService instance. This can be useful for cases where you want to stop a service in response to a runtime condition, e.g. receiving a \"poison pill\" message through its trigger. Received Topic EdgeX 2.0 Received Topic is new for Edgex 2.0 When messages are received via the EdgeX MessageBus or External MQTT triggers, the topic that the data was received on is seeded into the new Context Storage on the AppFunctionContext with the key receivedtopic . This make the Received Topic available to all functions in the pipeline. The SDK provides the interfaces.RECEIVEDTOPIC constant for this key. See the Context Storage section for more details on extracting values. Pipeline Per Topics EdgeX 2.1 Pipeline Per Topics is new for EdgeX 2.1 The Pipeline Per Topics feature allows for multiple function pipelines to be defined. Each will execute only when one of the specified pipeline topics matches the received topic. The pipeline topics can have wildcards ( # ) allowing the topic to match a variety of received topics. Each pipeline has its own set of functions (transforms) that are executed on the received message. If the # wildcard is used by itself for a pipeline topic, it will match all received topics and the specified functions pipeline will execute on every message received. Note The Pipeline Per Topics feature is targeted for EdgeX MessageBus and External MQTT triggers, but can be used with Custom or HTTP triggers. When used with the HTTP trigger the incoming topic will always be blank , so the pipeline's topics must contain a single topic set to the # wildcard so that all messages received are processed by the pipeline. Example pipeline topics with wildcards \"#\" - Matches all messages received \"edegex/events/#\" - Matches all messages received with the based topic `edegex/events/` \"edegex/events/core/#\" - Matches all messages received just from Core Data \"edegex/events/device/#\" - Matches all messages received just from Device services \"edegex/events/#/my-profile/#\" - Matches all messages received from Core Data or Device services for `my-profile` \"edegex/events/#/#/my-device/#\" - Matches all messages received from Core Data or Device services for `my-device` \"edegex/events/#/#/#/my-source\" - Matches all messages received from Core Data or Device services for `my-source` Refer to the Filter By Topics section for details on the structure of the received topic. All pipeline function capabilities such as Store and Forward, Batching, etc. can be used with one or more of the multiple function pipelines. Store and Forward uses the Pipeline's ID to find and restart the pipeline on retries. Example - Adding multiple function pipelines This example adds two pipelines. One to process data from the Random-Float-Device device and one to process data from the Int32 and Int64 sources. sample := functions . NewSample () err = service . AddFunctionsPipelineForTopics ( \"Floats-Pipeline\" , [] string { \"edgex/events/#/#/Random-Float-Device/#\" }, transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { ... return - 1 } err = app . service . AddFunctionsPipelineForTopics ( \"Int32-Pipleine\" , [] string { \"edgex/events/#/#/#/Int32\" , \"edgex/events/#/#/#/Int64\" }, transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { ... return - 1 } Built-in Application Service Metrics EdgeX 2.2 Built-in Application Service Metrics are new for EdgeX 2.2 All application services now have a limited set of built in metrics. More may be added in future releases. The current built-in metrics are: MessagesReceived - This is a counter metric that counts the number of messages received by the application service. PipelineMessagesProcessed - This is a counter metric that counts the number of messages processed by the individual function pipelines defined by the application service. The metric data is tagged with the specific function pipeline ID the count is for. PipelineMessageProcessingTime - This is a timer metric that tracks the amount of time taken to process messages by the individual function pipelines defined by the application service. The metric data is tagged with the specific function pipeline ID the timer is for. Note The time tracked for this metric is only for the function pipeline processing time. The overhead of receiving the messages and handing them to the appropriate function pipelines is not included. Accounting for this overhead may be added as another timer metric in a future release. Reporting of these built-in metrics is disabled by default in the Writable.Telemetry configuration section. See Writable.Telemetry configuration details in the Application Service Configuration section for complete detail on this section. If the configuration for these built-in metrics are missing, then the reporting of the metrics will be disabled. Example - Service Telemetry Configuration with built-in metrics enabled for reporting [Writable.Telemetry] Interval = \"30s\" PublishTopicPrefix = \"edgex/telemetry\" # // will be added to this Publish Topic prefix [ Writable . Telemetry . Metrics ] # All service's metric names must be present in this list. MessagesReceived = true PipelineMessagesProcessed = true PipelineMessageProcessingTime = true [ Writable . Telemetry . Tags ] # Contains the service level tags to be attached to all the service's metrics # Gateway=\"my-iot-gateway\" # Tag must be added here or via Consul Env Override can only change existing value, not added new ones. Custom Application Service Metrics EdgeX 2.2 Custom Application Service Metrics are new for EdgeX 2.2 The Custom Application Service Metrics capability allows for custom application services to define, collect and report their own custom service metrics. Note The SDK will soon implement a limited set of common Application Service Metrics. The following are the steps to collect and report custom service metrics: Determine the metric type that needs to be collected counter - Track the integer count of something gauge - Track the integer value of something gaugeFloat64 - Track the float64 value of something timer - Track the time it takes to accomplish a task Create instance of the metric type from github.com/rcrowley/go-metrics myCounter = gometrics.NewCounter() myGauge = gometrics.NewGauge() myGaugeFloat64 = gometrics.NewGaugeFloat64() myTimer = gometrics.NewTime() Determine if there are any tags to report along with your metric. Not common so nil is typically passed for the tags map[strings]string parameter in the next step. Register your metric(s) with the MetricsManager from the service or pipeline function context reference. See Application Service API and App Function Context API for more details: service.MetricsManager().Register(\"MyCounterName\", myCounter, nil) ctx.MetricsManager().Register(\"MyCounterName\", myCounter, nil) Collect the metric myCounter.Inc(someIntvalue) myCounter.Dec(someIntvalue) myGauge.Update(someIntvalue) myGaugeFloat64.Update(someFloatvalue) myTimer.Update(someDuration) myTimer.Time(func { do sometime}) myTimer.UpdateSince(someTimeValue ) Configure reporting of the service's metrics. See Writable.Telemetry configuration details in the Application Service Configuration section for more detail. Example - Service Telemetry Configuration [Writable.Telemetry] Interval = \"30s\" PublishTopicPrefix = \"edgex/telemetry\" # // will be added to this Publish Topic prefix [ Writable . Telemetry . Metrics ] # All service's metric names must be present in this list. MyCounterName = true MyGaugeName = true MyGaugeFloat64Name = true MyTimerName = true [ Writable . Telemetry . Tags ] # Contains the service level tags to be attached to all the service's metrics # Gateway=\"my-iot-gateway\" # Tag must be added here or via Consul Env Override can only change existing value, not added new ones. Note The metric names used in the above configuration (to enable or disable reporting of a metric) must match the metric name used when the metric is registered. A partial match of starts with is acceptable, i.e. the metric name registered starts with the above configured name.","title":"Advanced Topics"},{"location":"microservices/application/AdvancedTopics/#advanced-topics","text":"The following items discuss topics that are a bit beyond the basic use cases of the Application Functions SDK when interacting with EdgeX.","title":"Advanced Topics"},{"location":"microservices/application/AdvancedTopics/#configurable-functions-pipeline","text":"This SDK provides the capability to define the functions pipeline via configuration rather than code by using the app-service-configurable application service. See the App Service Configurable section for more details.","title":"Configurable Functions Pipeline"},{"location":"microservices/application/AdvancedTopics/#custom-rest-endpoints","text":"It is not uncommon to require your own custom REST endpoints when building an Application Service. Rather than spin up your own webserver inside of your app (alongside the already existing running webserver), we've exposed a method that allows you add your own routes to the existing webserver. A few routes are reserved and cannot be used: /api/v2/version /api/v2/ping /api/v2/metrics /api/v2/config /api/v2/trigger /api/v2/secret To add your own route, use the AddRoute() API provided on the ApplicationService interface. Example - Add Custom REST route myhandler := func ( writer http . ResponseWriter , req * http . Request ) { service := req . Context (). Value ( interfaces . AppServiceContextKey ).( interfaces . ApplicationService ) service . LoggingClient (). Info ( \"TEST\" ) writer . Header (). Set ( \"Content-Type\" , \"text/plain\" ) writer . Write ([] byte ( \"hello\" )) writer . WriteHeader ( 200 ) } service := pkg . NewAppService ( serviceKey ) service . AddRoute ( \"/myroute\" , myHandler , \"GET\" ) Under the hood, this simply adds the provided route, handler, and method to the gorilla mux.Router used in the SDK. For more information on gorilla mux you can check out the github repo here . You can access the interfaces.ApplicationService API for resources such as the logging client by pulling it from the context as shown above -- this is useful for when your routes might not be defined in your main.go where you have access to the interfaces.ApplicationService instance.","title":"Custom REST Endpoints"},{"location":"microservices/application/AdvancedTopics/#target-type","text":"The target type is the object type of the incoming data that is sent to the first function in the function pipeline. By default this is an EdgeX dtos.Event since typical usage is receiving Events from the EdgeX MessageBus. There are scenarios where the incoming data is not an EdgeX Event . One example scenario is two application services are chained via the EdgeX MessageBus. The output of the first service is inference data from analyzing the original Event data, and published back to the EdgeX MessageBus. The second service needs to be able to let the SDK know the target type of the input data it is expecting. For usages where the incoming data is not events , the TargetType of the expected incoming data can be set when the ApplicationService instance is created using the NewAppServiceWithTargetType() factory function. Example - Set and use custom Target Type type Person struct { FirstName string `json:\"first_name\"` LastName string `json:\"last_name\"` } service := pkg . NewAppServiceWithTargetType ( serviceKey , & Person {}) TargetType must be set to a pointer to an instance of your target type such as &Person{} . The first function in your function pipeline will be passed an instance of your target type, not a pointer to it. In the example above, the first function in the pipeline would start something like: func MyPersonFunction ( ctx interfaces . AppFunctionContext , data interface {}) ( bool , interface {}) { ctx . LoggingClient (). Debug ( \"MyPersonFunction executing\" ) if data == nil { return false , errors . New ( \"no data received to MyPersonFunction\" ) } person , ok := data .( Person ) if ! ok { return false , errors . New ( \"MyPersonFunction type received is not a Person\" ) } // .... The SDK supports un-marshaling JSON or CBOR encoded data into an instance of the target type. If your incoming data is not JSON or CBOR encoded, you then need to set the TargetType to &[]byte . If the target type is set to &[]byte the incoming data will not be un-marshaled. The content type, if set, will be set on the interfaces.AppFunctionContext and can be access via the InputContentType() API. Your first function will be responsible for decoding the data or not.","title":"Target Type"},{"location":"microservices/application/AdvancedTopics/#command-line-options","text":"See the Common Command Line Options for the set of command line options common to all EdgeX services. The following command line options are specific to Application Services.","title":"Command Line Options"},{"location":"microservices/application/AdvancedTopics/#skip-version-check","text":"-s/--skipVersionCheck Indicates the service should skip the Core Service's version compatibility check.","title":"Skip Version Check"},{"location":"microservices/application/AdvancedTopics/#service-key","text":"-sk/--serviceKey Sets the service key that is used with Registry, Configuration Provider and security services. The default service key is set by the application service. If the name provided contains the placeholder text , this text will be replaced with the name of the profile used. If profile is not set, the text is simply removed Can be overridden with EDGEX_SERVICE_KEY environment variable.","title":"Service Key"},{"location":"microservices/application/AdvancedTopics/#environment-variables","text":"See the Common Environment Variables section for the list of environment variables common to all EdgeX Services. The remaining in this section are specific to Application Services.","title":"Environment Variables"},{"location":"microservices/application/AdvancedTopics/#edgex_service_key","text":"This environment variable overrides the -sk/--serviceKey command-line option and the default set by the application service. Note If the name provided contains the text , this text will be replaced with the name of the profile used. Example - Service Key EDGEX_SERVICE_KEY: app--mycloud profile: http-export then service key will be app-http-export-mycloud EdgeX 2.0 The deprecated lowercase `edgex_service environment variable specific have been removed for EdgeX 2.0","title":"EDGEX_SERVICE_KEY"},{"location":"microservices/application/AdvancedTopics/#custom-configuration","text":"Applications can specify custom configuration in the TOML file in two ways.","title":"Custom Configuration"},{"location":"microservices/application/AdvancedTopics/#application-settings","text":"The first simple way is to add items to the ApplicationSetting section. This is a map of string key/value pairs, i.e. map[string]string . Use for simple string values or comma separated list of string values. The ApplicationService API provides the follow access APIs for this configuration section: ApplicationSettings() map[string]string Returns the whole list of application settings GetAppSetting(setting string) (string, error) Returns single entry from the map who's key matches the passed in setting value GetAppSettingStrings(setting string) ([]string, error) Returns list of strings for the entry who's key matches the passed in setting value. The Entry is assumed to be a comma separated list of strings.","title":"Application Settings"},{"location":"microservices/application/AdvancedTopics/#structure-custom-configuration","text":"EdgeX 2.0 Structure Custom Configuration is new for Edgex 2.0 The second is the more complex Structured Custom Configuration which allows the Application Service to define and watch it's own structured section in the service's TOML configuration file. The ApplicationService API provides the follow APIs to enable structured custom configuration: LoadCustomConfig(config UpdatableConfig, sectionName string) error Loads the service's custom configuration from local file or the Configuration Provider (if enabled). The Configuration Provider will also be seeded with the custom configuration the first time the service is started, if service is using the Configuration Provider. The UpdateFromRaw interface will be called on the custom configuration when the configuration is loaded from the Configuration Provider. ListenForCustomConfigChanges(configToWatch interface{}, sectionName string, changedCallback func(interface{})) error Starts a listener on the Configuration Provider for changes to the specified section of the custom configuration. When changes are received from the Configuration Provider the UpdateWritableFromRaw interface will be called on the custom configuration to apply the updates and then signal that the changes occurred via changedCallback. See the Application Service Template for an example of using the new Structured Custom Configuration capability. See here for defining the structured custom configuration See here for loading, validating and watching the configuration","title":"Structure Custom Configuration"},{"location":"microservices/application/AdvancedTopics/#store-and-forward","text":"The Store and Forward capability allows for export functions to persist data on failure and for the export of the data to be retried at a later time. Note The order the data exported via this retry mechanism is not guaranteed to be the same order in which the data was initial received from Core Data","title":"Store and Forward"},{"location":"microservices/application/AdvancedTopics/#configuration","text":"Writable.StoreAndForward allows enabling, setting the interval between retries and the max number of retries. If running with Configuration Provider, these setting can be changed on the fly via Consul without having to restart the service. Example - Store and Forward configuration [Writable.StoreAndForward] Enabled = false RetryInterval = \"5m\" MaxRetryCount = 10 Note RetryInterval should be at least 1 second (eg. '1s') or greater. If a value less than 1 second is specified, 1 second will be used. Endless retries will occur when MaxRetryCount is set to 0. If MaxRetryCount is set to less than 0, a default of 1 retry will be used. Database configuration section describes which database type to use and the information required to connect to the database. This section is required if Store and Forward is enabled. It is optional if not using Redis for the EdgeX MessageBus which is now the default. Example - Database configuration [Database] Type = \"redisdb\" Host = \"localhost\" Port = 6379 Timeout = \"30s\" EdgeX 2.0 Support for Mongo DB has been removed in EdgeX 2.0","title":"Configuration"},{"location":"microservices/application/AdvancedTopics/#how-it-works","text":"When an export function encounters an error sending data it can call SetRetryData(payload []byte) on the AppFunctionContext . This will store the data for later retry. If the Application Service is stopped and then restarted while stored data hasn't been successfully exported, the export retry will resume once the service is up and running again. Note It is important that export functions return an error and stop pipeline execution after the call to SetRetryData . See HTTPPost function in SDK as an example When the RetryInterval expires, the function pipeline will be re-executed starting with the export function that saved the data. The saved data will be passed to the export function which can then attempt to resend the data. Note The export function will receive the data as it was stored, so it is important that any transformation of the data occur in functions prior to the export function. The export function should only export the data that it receives. One of three out comes can occur after the export retried has completed. Export retry was successful In this case, the stored data is removed from the database and the execution of the pipeline functions after the export function, if any, continues. Export retry fails and retry count has not been exceeded In this case, the stored data is updated in the database with the incremented retry count Export retry fails and retry count has been exceeded In this case, the stored data is removed from the database and never retried again. Note Changing Writable.Pipeline.ExecutionOrder will invalidate all currently stored data and result in it all being removed from the database on the next retry. This is because the position of the export function can no longer be guaranteed and no way to ensure it is properly executed on the retry.","title":"How it works"},{"location":"microservices/application/AdvancedTopics/#custom-storage","text":"The default backing store is redis. Custom implementations of the StoreClient interface can be provided if redis does not meet your requirements. type StoreClient interface { // Store persists a stored object to the data store and returns the assigned UUID. Store ( o StoredObject ) ( id string , err error ) // RetrieveFromStore gets an object from the data store. RetrieveFromStore ( appServiceKey string ) ( objects [] StoredObject , err error ) // Update replaces the data currently in the store with the provided data. Update ( o StoredObject ) error // RemoveFromStore removes an object from the data store. RemoveFromStore ( o StoredObject ) error // Disconnect ends the connection. Disconnect () error } A factory function to create these clients can then be registered with your service by calling RegisterCustomStoreFactory service . RegisterCustomStoreFactory ( \"jetstream\" , func ( cfg interfaces . DatabaseInfo , cred config . Credentials ) ( interfaces . StoreClient , error ) { conn , err := nats . Connect ( fmt . Sprintf ( \"nats://%s:%d\" , cfg . Host , cfg . Port )) if err != nil { return nil , err } js , err := conn . JetStream () if err != nil { return nil , err } kv , err := js . KeyValue ( serviceKey ) if err != nil { kv , err = js . CreateKeyValue ( & nats . KeyValueConfig { Bucket : serviceKey }) } return & JetstreamStore { conn : conn , serviceKey : serviceKey , kv : kv , }, err }) and configured using the registered name in the Database section: [Database] Type = \"jetstream\" Host = \"broker\" Port = 4222 Timeout = \"5s\"","title":"Custom Storage"},{"location":"microservices/application/AdvancedTopics/#secrets","text":"","title":"Secrets"},{"location":"microservices/application/AdvancedTopics/#configuration_1","text":"All instances of App Services running in secure mode require a SecretStore to be configured. With the use of Redis Pub/Sub as the default EdgeX MessageBus all App Services need the redisdb known secret added to their SecretStore so they can connect to the Secure EdgeX MessageBus. See the Secure MessageBus documentation for more details. Example - SecretStore configuration [SecretStore] Type = \"vault\" Host = \"localhost\" Port = 8200 Path = \"app-sample/\" Protocol = \"http\" RootCaCertPath = \"\" ServerName = \"\" TokenFile = \"/tmp/edgex/secrets/app-sample/secrets-token.json\" [SecretStore.Authentication] AuthType = \"X-Vault-Token\" EdgeX 2.0 For Edgex 2.0 all Application Service Secret Stores are exclusive so the explicit [SecretStoreExclusive] configuration has been removed.","title":"Configuration"},{"location":"microservices/application/AdvancedTopics/#storing-secrets","text":"","title":"Storing Secrets"},{"location":"microservices/application/AdvancedTopics/#secure-mode","text":"When running an application service in secure mode, secrets can be stored in the SecretStore by making an HTTP POST call to the /api/v2/secret API route in the application service. The secret data POSTed is stored and retrieved from the SecretStore based on values in the [SecretStore] section of the configuration file. Once a secret is stored, only the service that added the secret will be able to retrieve it. For secret retrieval see Getting Secrets section below. Example - JSON message body { \"path\" : \"MyPath\" , \"secretData\" : [ { \"key\" : \"MySecretKey\" , \"value\" : \"MySecretValue\" } ] } Note Path specifies the type or location of the secret in the SecretStore. It is appended to the base path from the [SecretStore] configuration.","title":"Secure Mode"},{"location":"microservices/application/AdvancedTopics/#insecure-mode","text":"When running in insecure mode, the secrets are stored and retrieved from the Writable.InsecureSecrets section of the service's configuration toml file. Insecure secrets and their paths can be configured as below. Example - InsecureSecrets Configuration [ Writable . InsecureSecrets ] [Writable.InsecureSecrets.AWS] Path = \"aws\" [Writable.InsecureSecrets.AWS.Secrets] username = \"aws-user\" password = \"aws-pw\" [Writable.InsecureSecrets.DB] Path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\"","title":"Insecure Mode"},{"location":"microservices/application/AdvancedTopics/#getting-secrets","text":"Application Services can retrieve their secrets from their SecretStore using the interfaces.ApplicationService.GetSecret() API or from the interfaces.AppFunctionContext.GetSecret() API When in secure mode, the secrets are retrieved from the SecretStore based on the [SecretStore] configuration values. When running in insecure mode, the secrets are retrieved from the [Writable.InsecureSecrets] configuration.","title":"Getting Secrets"},{"location":"microservices/application/AdvancedTopics/#background-publishing","text":"Application Services using the MessageBus trigger can request a background publisher using the AddBackgroundPublisher API in the SDK. This method takes an int representing the background channel's capacity as the only parameter and returns a reference to a BackgroundPublisher. This reference can then be used by background processes to publish to the configured MessageBus output. A custom topic can be provided to use instead of the configured message bus output as well. Edgex 2.0 For EdgeX 2.0 the background publish operation takes a full AppContext instead of just the parameters used to create a message envelope. This allows the background publisher to leverage context-based topic formatting functionality as the trigger output. Example - Background Publisher func runJob ( service interfaces . ApplicationService , done chan struct {}){ ticker := time . NewTicker ( 1 * time . Minute ) //initialize background publisher with a channel capacity of 10 and a custom topic publisher , err := service . AddBackgroundPublisherWithTopic ( 10 , \"custom-topic\" ) if err != nil { // do something } go func ( pub interfaces . BackgroundPublisher ) { for { select { case <- ticker . C : msg := myDataService . GetMessage () payload , err := json . Marshal ( message ) if err != nil { //do something } ctx := svc . BuildContext ( uuid . NewString (), common . ContentTypeJSON ) // modify context as needed err = pub . Publish ( payload , ctx ) if err != nil { //do something } case <- j . done : ticker . Stop () return } } }( publisher ) } func main () { service := pkg . NewAppService ( serviceKey ) done := make ( chan struct {}) defer close ( done ) //pass publisher to your background job runJob ( service , done ) service . SetFunctionsPipeline ( All , My , Functions , ) service . MakeItRun () os . Exit ( 0 ) }","title":"Background Publishing"},{"location":"microservices/application/AdvancedTopics/#stopping-the-service","text":"Application Services will listen for SIGTERM / SIGINT signals from the OS and stop the function pipeline in response. The pipeline can also be exited programmatically by calling sdk.MakeItStop() on the running ApplicationService instance. This can be useful for cases where you want to stop a service in response to a runtime condition, e.g. receiving a \"poison pill\" message through its trigger.","title":"Stopping the Service"},{"location":"microservices/application/AdvancedTopics/#received-topic","text":"EdgeX 2.0 Received Topic is new for Edgex 2.0 When messages are received via the EdgeX MessageBus or External MQTT triggers, the topic that the data was received on is seeded into the new Context Storage on the AppFunctionContext with the key receivedtopic . This make the Received Topic available to all functions in the pipeline. The SDK provides the interfaces.RECEIVEDTOPIC constant for this key. See the Context Storage section for more details on extracting values.","title":"Received Topic"},{"location":"microservices/application/AdvancedTopics/#pipeline-per-topics","text":"EdgeX 2.1 Pipeline Per Topics is new for EdgeX 2.1 The Pipeline Per Topics feature allows for multiple function pipelines to be defined. Each will execute only when one of the specified pipeline topics matches the received topic. The pipeline topics can have wildcards ( # ) allowing the topic to match a variety of received topics. Each pipeline has its own set of functions (transforms) that are executed on the received message. If the # wildcard is used by itself for a pipeline topic, it will match all received topics and the specified functions pipeline will execute on every message received. Note The Pipeline Per Topics feature is targeted for EdgeX MessageBus and External MQTT triggers, but can be used with Custom or HTTP triggers. When used with the HTTP trigger the incoming topic will always be blank , so the pipeline's topics must contain a single topic set to the # wildcard so that all messages received are processed by the pipeline. Example pipeline topics with wildcards \"#\" - Matches all messages received \"edegex/events/#\" - Matches all messages received with the based topic `edegex/events/` \"edegex/events/core/#\" - Matches all messages received just from Core Data \"edegex/events/device/#\" - Matches all messages received just from Device services \"edegex/events/#/my-profile/#\" - Matches all messages received from Core Data or Device services for `my-profile` \"edegex/events/#/#/my-device/#\" - Matches all messages received from Core Data or Device services for `my-device` \"edegex/events/#/#/#/my-source\" - Matches all messages received from Core Data or Device services for `my-source` Refer to the Filter By Topics section for details on the structure of the received topic. All pipeline function capabilities such as Store and Forward, Batching, etc. can be used with one or more of the multiple function pipelines. Store and Forward uses the Pipeline's ID to find and restart the pipeline on retries. Example - Adding multiple function pipelines This example adds two pipelines. One to process data from the Random-Float-Device device and one to process data from the Int32 and Int64 sources. sample := functions . NewSample () err = service . AddFunctionsPipelineForTopics ( \"Floats-Pipeline\" , [] string { \"edgex/events/#/#/Random-Float-Device/#\" }, transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { ... return - 1 } err = app . service . AddFunctionsPipelineForTopics ( \"Int32-Pipleine\" , [] string { \"edgex/events/#/#/#/Int32\" , \"edgex/events/#/#/#/Int64\" }, transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { ... return - 1 }","title":"Pipeline Per Topics"},{"location":"microservices/application/AdvancedTopics/#built-in-application-service-metrics","text":"EdgeX 2.2 Built-in Application Service Metrics are new for EdgeX 2.2 All application services now have a limited set of built in metrics. More may be added in future releases. The current built-in metrics are: MessagesReceived - This is a counter metric that counts the number of messages received by the application service. PipelineMessagesProcessed - This is a counter metric that counts the number of messages processed by the individual function pipelines defined by the application service. The metric data is tagged with the specific function pipeline ID the count is for. PipelineMessageProcessingTime - This is a timer metric that tracks the amount of time taken to process messages by the individual function pipelines defined by the application service. The metric data is tagged with the specific function pipeline ID the timer is for. Note The time tracked for this metric is only for the function pipeline processing time. The overhead of receiving the messages and handing them to the appropriate function pipelines is not included. Accounting for this overhead may be added as another timer metric in a future release. Reporting of these built-in metrics is disabled by default in the Writable.Telemetry configuration section. See Writable.Telemetry configuration details in the Application Service Configuration section for complete detail on this section. If the configuration for these built-in metrics are missing, then the reporting of the metrics will be disabled. Example - Service Telemetry Configuration with built-in metrics enabled for reporting [Writable.Telemetry] Interval = \"30s\" PublishTopicPrefix = \"edgex/telemetry\" # // will be added to this Publish Topic prefix [ Writable . Telemetry . Metrics ] # All service's metric names must be present in this list. MessagesReceived = true PipelineMessagesProcessed = true PipelineMessageProcessingTime = true [ Writable . Telemetry . Tags ] # Contains the service level tags to be attached to all the service's metrics # Gateway=\"my-iot-gateway\" # Tag must be added here or via Consul Env Override can only change existing value, not added new ones.","title":"Built-in Application Service Metrics"},{"location":"microservices/application/AdvancedTopics/#custom-application-service-metrics","text":"EdgeX 2.2 Custom Application Service Metrics are new for EdgeX 2.2 The Custom Application Service Metrics capability allows for custom application services to define, collect and report their own custom service metrics. Note The SDK will soon implement a limited set of common Application Service Metrics. The following are the steps to collect and report custom service metrics: Determine the metric type that needs to be collected counter - Track the integer count of something gauge - Track the integer value of something gaugeFloat64 - Track the float64 value of something timer - Track the time it takes to accomplish a task Create instance of the metric type from github.com/rcrowley/go-metrics myCounter = gometrics.NewCounter() myGauge = gometrics.NewGauge() myGaugeFloat64 = gometrics.NewGaugeFloat64() myTimer = gometrics.NewTime() Determine if there are any tags to report along with your metric. Not common so nil is typically passed for the tags map[strings]string parameter in the next step. Register your metric(s) with the MetricsManager from the service or pipeline function context reference. See Application Service API and App Function Context API for more details: service.MetricsManager().Register(\"MyCounterName\", myCounter, nil) ctx.MetricsManager().Register(\"MyCounterName\", myCounter, nil) Collect the metric myCounter.Inc(someIntvalue) myCounter.Dec(someIntvalue) myGauge.Update(someIntvalue) myGaugeFloat64.Update(someFloatvalue) myTimer.Update(someDuration) myTimer.Time(func { do sometime}) myTimer.UpdateSince(someTimeValue ) Configure reporting of the service's metrics. See Writable.Telemetry configuration details in the Application Service Configuration section for more detail. Example - Service Telemetry Configuration [Writable.Telemetry] Interval = \"30s\" PublishTopicPrefix = \"edgex/telemetry\" # // will be added to this Publish Topic prefix [ Writable . Telemetry . Metrics ] # All service's metric names must be present in this list. MyCounterName = true MyGaugeName = true MyGaugeFloat64Name = true MyTimerName = true [ Writable . Telemetry . Tags ] # Contains the service level tags to be attached to all the service's metrics # Gateway=\"my-iot-gateway\" # Tag must be added here or via Consul Env Override can only change existing value, not added new ones. Note The metric names used in the above configuration (to enable or disable reporting of a metric) must match the metric name used when the metric is registered. A partial match of starts with is acceptable, i.e. the metric name registered starts with the above configured name.","title":"Custom Application Service Metrics"},{"location":"microservices/application/AppFunctionContextAPI/","text":"App Function Context API The context parameter passed to each function/transform provides operations and data associated with each execution of the pipeline. EdgeX 2.0 For EdgeX 2.0 the AppFunctionContext API replaces the direct access to the appcontext.Context struct. Let's take a look at its API: type AppFunctionContext interface { CorrelationID () string InputContentType () string SetResponseData ( data [] byte ) ResponseData () [] byte SetResponseContentType ( string ) ResponseContentType () string SetRetryData ( data [] byte ) GetSecret ( path string , keys ... string ) ( map [ string ] string , error ) SecretsLastUpdated () time . Time LoggingClient () logger . LoggingClient EventClient () interfaces . EventClient CommandClient () interfaces . CommandClient NotificationClient () interfaces . NotificationClient SubscriptionClient () interfaces . SubscriptionClient DeviceServiceClient () interfaces . DeviceServiceClient DeviceProfileClient () interfaces . DeviceProfileClient DeviceClient () interfaces . DeviceClient MetricsManager () bootstrapInterfaces . MetricsManager PushToCore ( event dtos . Event ) ( common . BaseWithIdResponse , error ) GetDeviceResource ( profileName string , resourceName string ) ( dtos . DeviceResource , error ) AddValue ( key string , value string ) RemoveValue ( key string ) GetValue ( key string ) ( string , bool ) GetAllValues () map [ string ] string ApplyValues ( format string ) ( string , error ) PipelineId () string } Response Data SetResponseData SetResponseData(data []byte) This API sets the response data that will be returned to the trigger when pipeline execution is complete. ResponseData ResponseData() This API returns the data that will be returned to the trigger when pipeline execution is complete. SetResponseContentType SetResponseContentType(string) This API sets the content type that will be returned to the trigger when pipeline execution is complete. ResponseContentType ResponseContentType() This API returns the content type that will be returned to the trigger when pipeline execution is complete. Clients LoggingClient LoggingClient() logger.LoggingClient Returns a LoggingClient to leverage logging libraries/service utilized throughout the EdgeX framework. The SDK has initialized everything so it can be used to log Trace , Debug , Warn , Info , and Error messages as appropriate. Example - LoggingClient ctx . LoggingClient (). Info ( \"Hello World\" ) c . LoggingClient (). Errorf ( \"Some error occurred: %w\" , err ) EventClient EventClient() interfaces.EventClient Returns an EventClient to leverage Core Data's Event API. See interface definition for more details. This client is useful for querying events and is used by the PushToCore convenience API described below. Note if Core Data is not specified in the Clients configuration, this will return nil. CommandClient CommandClient() interfaces.CommandClient Returns a CommandClient to leverage Core Command's Command API. See interface definition for more details. Useful for sending commands to devices. Note if Core Command is not specified in the Clients configuration, this will return nil. NotificationClient NotificationClient() interfaces.NotificationClient Returns a NotificationClient to leverage Support Notifications' Notifications API. See interface definition for more details. Useful for sending notifications. Note if Support Notifications is not specified in the Clients configuration, this will return nil. SubscriptionClient SubscriptionClient() interfaces.SubscriptionClient Returns a SubscriptionClient to leverage Support Notifications' Subscription API. See interface definition for more details. Useful for creating notification subscriptions. Note if Support Notifications is not specified in the Clients configuration, this will return nil. DeviceServiceClient DeviceServiceClient() interfaces.DeviceServiceClient Returns a DeviceServiceClient to leverage Core Metadata's DeviceService API. See interface definition for more details. Useful for querying information about Device Services. Note if Core Metadata is not specified in the Clients configuration, this will return nil. DeviceProfileClient DeviceProfileClient() interfaces.DeviceProfileClient Returns a DeviceProfileClient to leverage Core Metadata's DeviceProfile API. See interface definition for more details. Useful for querying information about Device Profiles and is used by the GetDeviceResource helper function below. Note if Core Metadata is not specified in the Clients configuration, this will return nil. DeviceClient DeviceClient() interfaces.DeviceClient Returns a DeviceClient to leverage Core Metadata's Device API. See interface definition for more details. Useful for querying information about Devices. Note if Core Metadata is not specified in the Clients configuration, this will return nil. Note about Clients Each of the clients above is only initialized if the Clients section of the configuration contains an entry for the service associated with the Client API. If it isn't in the configuration the client will be nil . Your code must check for nil to avoid panic in case it is missing from the configuration. Only add the clients to your configuration that your Application Service will actually be using. All application services need Core-Data for version compatibility check done on start-up. The following is an example Clients section of a configuration.toml with all supported clients specified: Example - Client Configuration Section [Clients] [Clients.core-data] Protocol = 'http' Host = 'localhost' Port = 59880 [Clients.core-metadata] Protocol = 'http' Host = 'localhost' Port = 59881 [Clients.core-command] Protocol = 'http' Host = 'localhost' Port = 59882 [Clients.support-notifications] Protocol = 'http' Host = 'localhost' Port = 59860 Context Storage The context API exposes a map-like interface that can be used to store custom data specific to a given pipeline execution. This data is persisted for retry if needed. Currently only strings are supported, and keys are treated as case-insensitive. There following values are seeded into the Context Storage when an Event is received: Profile Name (key to retrieve value is interfaces.PROFILENAME ) Device Name (key to retrieve value is interfaces.DEVICENAME ) Source Name (key to retrieve value is interfaces.SOURCENAME ) Received Topic (key to retrieve value is interfaces.RECEIVEDTOPIC ) Note Received Topic only available when the message was received from the Edgex MessageBus or External MQTT triggers. Storage can be accessed using the following methods: AddValue AddValue(key string, value string) This API stores a value for access within a pipeline execution RemoveValue RemoveValue(key string) This API deletes a value stored in the context at the given key GetValue GetValue(key string) (string, bool) This API attempts to retrieve a value stored in the context at the given key GetAllValues GetAllValues() map[string]string This API returns a read-only copy of all data stored in the context ApplyValues ApplyValues(format string) (string, error) This API will replace placeholders of the form {context-key-name} with the value found in the context at context-key-name . Note that key matching is case insensitive. An error will be returned if any placeholders in the provided string do NOT have a corresponding entry in the context storage map. Secrets GetSecret GetSecret(path string, keys ...string) This API is used to retrieve secrets from the secret store. path specifies the type or location of the secrets to retrieve. If specified, it is appended to the base path from the exclusive secret store configuration. keys specifies the list of secrets to be retrieved. If no keys are provided then all the keys associated with the specified path will be returned. SecretsLastUpdated SecretsLastUpdated() This API returns that timestamp for when the secrets in the SecretStore where last updated. Useful when a connection to external source needs to be redone when the credentials have been updated. Miscellaneous CorrelationID CorrelationID() This API returns the ID used to track the EdgeX event through entire EdgeX framework. PipelineId PipelineId() string This API returns the ID of the pipeline currently executing. Useful when logging messages from pipeline functions so the message contain the ID of the pipeline that executed the pipeline function. InputContentType InputContentType() This API returns the content type of the data that initiated the pipeline execution. Only useful when the TargetType for the pipeline is []byte, otherwise the data will be the type specified by TargetType. GetDeviceResource GetDeviceResource(profileName string, resourceName string) (dtos.DeviceResource, error) This API retrieves the DeviceResource for the given profile / resource name. Results are cached to minimize HTTP traffic to core-metadata. PushToCore PushToCore(event dtos.Event) This API is used to push data to EdgeX Core Data so that it can be shared with other applications that are subscribed to the message bus that core-data publishes to. This function will return the new EdgeX Event with the ID populated, along with any error encountered. Note that CorrelationId will not be available. Note If validation is turned on in CoreServices then your deviceName and readingName must exist in the CoreMetadata and be properly registered in EdgeX. Warning Be aware that without a filter in your pipeline, it is possible to create an infinite loop when the Message Bus trigger is used. Choose your device-name and reading name appropriately. SetRetryData SetRetryData(data []byte) This method can be used to store data for later retry. This is useful when creating a custom export function that needs to retry on failure. The payload data will be stored for later retry based on Store and Forward configuration. When the retry is triggered, the function pipeline will be re-executed starting with the function that called this API. That function will be passed the stored data, so it is important that all transformations occur in functions prior to the export function. The Context will also be restored to the state when the function called this API. See Store and Forward for more details. Note Store and Forward be must enabled when calling this API, otherwise the data is ignored. MetricsManager MetricsManager() bootstrapInterfaces.MetricsManager This API returns the Metrics Manager used to register counter, gauge, gaugeFloat64 or timer metric types from github.com/rcrowley/go-metrics myCounterMetricName := \"MyCounter\" myCounter := gometrics . NewCounter () myTags := map [ string ] string { \"Tag1\" : \"Value1\" } ctx . MetricsManager (). Register ( myCounterMetricName , myCounter , myTags )","title":"App Function Context API"},{"location":"microservices/application/AppFunctionContextAPI/#app-function-context-api","text":"The context parameter passed to each function/transform provides operations and data associated with each execution of the pipeline. EdgeX 2.0 For EdgeX 2.0 the AppFunctionContext API replaces the direct access to the appcontext.Context struct. Let's take a look at its API: type AppFunctionContext interface { CorrelationID () string InputContentType () string SetResponseData ( data [] byte ) ResponseData () [] byte SetResponseContentType ( string ) ResponseContentType () string SetRetryData ( data [] byte ) GetSecret ( path string , keys ... string ) ( map [ string ] string , error ) SecretsLastUpdated () time . Time LoggingClient () logger . LoggingClient EventClient () interfaces . EventClient CommandClient () interfaces . CommandClient NotificationClient () interfaces . NotificationClient SubscriptionClient () interfaces . SubscriptionClient DeviceServiceClient () interfaces . DeviceServiceClient DeviceProfileClient () interfaces . DeviceProfileClient DeviceClient () interfaces . DeviceClient MetricsManager () bootstrapInterfaces . MetricsManager PushToCore ( event dtos . Event ) ( common . BaseWithIdResponse , error ) GetDeviceResource ( profileName string , resourceName string ) ( dtos . DeviceResource , error ) AddValue ( key string , value string ) RemoveValue ( key string ) GetValue ( key string ) ( string , bool ) GetAllValues () map [ string ] string ApplyValues ( format string ) ( string , error ) PipelineId () string }","title":"App Function Context API"},{"location":"microservices/application/AppFunctionContextAPI/#response-data","text":"","title":"Response Data"},{"location":"microservices/application/AppFunctionContextAPI/#setresponsedata","text":"SetResponseData(data []byte) This API sets the response data that will be returned to the trigger when pipeline execution is complete.","title":"SetResponseData"},{"location":"microservices/application/AppFunctionContextAPI/#responsedata","text":"ResponseData() This API returns the data that will be returned to the trigger when pipeline execution is complete.","title":"ResponseData"},{"location":"microservices/application/AppFunctionContextAPI/#setresponsecontenttype","text":"SetResponseContentType(string) This API sets the content type that will be returned to the trigger when pipeline execution is complete.","title":"SetResponseContentType"},{"location":"microservices/application/AppFunctionContextAPI/#responsecontenttype","text":"ResponseContentType() This API returns the content type that will be returned to the trigger when pipeline execution is complete.","title":"ResponseContentType"},{"location":"microservices/application/AppFunctionContextAPI/#clients","text":"","title":"Clients"},{"location":"microservices/application/AppFunctionContextAPI/#loggingclient","text":"LoggingClient() logger.LoggingClient Returns a LoggingClient to leverage logging libraries/service utilized throughout the EdgeX framework. The SDK has initialized everything so it can be used to log Trace , Debug , Warn , Info , and Error messages as appropriate. Example - LoggingClient ctx . LoggingClient (). Info ( \"Hello World\" ) c . LoggingClient (). Errorf ( \"Some error occurred: %w\" , err )","title":"LoggingClient"},{"location":"microservices/application/AppFunctionContextAPI/#eventclient","text":"EventClient() interfaces.EventClient Returns an EventClient to leverage Core Data's Event API. See interface definition for more details. This client is useful for querying events and is used by the PushToCore convenience API described below. Note if Core Data is not specified in the Clients configuration, this will return nil.","title":"EventClient"},{"location":"microservices/application/AppFunctionContextAPI/#commandclient","text":"CommandClient() interfaces.CommandClient Returns a CommandClient to leverage Core Command's Command API. See interface definition for more details. Useful for sending commands to devices. Note if Core Command is not specified in the Clients configuration, this will return nil.","title":"CommandClient"},{"location":"microservices/application/AppFunctionContextAPI/#notificationclient","text":"NotificationClient() interfaces.NotificationClient Returns a NotificationClient to leverage Support Notifications' Notifications API. See interface definition for more details. Useful for sending notifications. Note if Support Notifications is not specified in the Clients configuration, this will return nil.","title":"NotificationClient"},{"location":"microservices/application/AppFunctionContextAPI/#subscriptionclient","text":"SubscriptionClient() interfaces.SubscriptionClient Returns a SubscriptionClient to leverage Support Notifications' Subscription API. See interface definition for more details. Useful for creating notification subscriptions. Note if Support Notifications is not specified in the Clients configuration, this will return nil.","title":"SubscriptionClient"},{"location":"microservices/application/AppFunctionContextAPI/#deviceserviceclient","text":"DeviceServiceClient() interfaces.DeviceServiceClient Returns a DeviceServiceClient to leverage Core Metadata's DeviceService API. See interface definition for more details. Useful for querying information about Device Services. Note if Core Metadata is not specified in the Clients configuration, this will return nil.","title":"DeviceServiceClient"},{"location":"microservices/application/AppFunctionContextAPI/#deviceprofileclient","text":"DeviceProfileClient() interfaces.DeviceProfileClient Returns a DeviceProfileClient to leverage Core Metadata's DeviceProfile API. See interface definition for more details. Useful for querying information about Device Profiles and is used by the GetDeviceResource helper function below. Note if Core Metadata is not specified in the Clients configuration, this will return nil.","title":"DeviceProfileClient"},{"location":"microservices/application/AppFunctionContextAPI/#deviceclient","text":"DeviceClient() interfaces.DeviceClient Returns a DeviceClient to leverage Core Metadata's Device API. See interface definition for more details. Useful for querying information about Devices. Note if Core Metadata is not specified in the Clients configuration, this will return nil.","title":"DeviceClient"},{"location":"microservices/application/AppFunctionContextAPI/#note-about-clients","text":"Each of the clients above is only initialized if the Clients section of the configuration contains an entry for the service associated with the Client API. If it isn't in the configuration the client will be nil . Your code must check for nil to avoid panic in case it is missing from the configuration. Only add the clients to your configuration that your Application Service will actually be using. All application services need Core-Data for version compatibility check done on start-up. The following is an example Clients section of a configuration.toml with all supported clients specified: Example - Client Configuration Section [Clients] [Clients.core-data] Protocol = 'http' Host = 'localhost' Port = 59880 [Clients.core-metadata] Protocol = 'http' Host = 'localhost' Port = 59881 [Clients.core-command] Protocol = 'http' Host = 'localhost' Port = 59882 [Clients.support-notifications] Protocol = 'http' Host = 'localhost' Port = 59860","title":"Note about Clients"},{"location":"microservices/application/AppFunctionContextAPI/#context-storage","text":"The context API exposes a map-like interface that can be used to store custom data specific to a given pipeline execution. This data is persisted for retry if needed. Currently only strings are supported, and keys are treated as case-insensitive. There following values are seeded into the Context Storage when an Event is received: Profile Name (key to retrieve value is interfaces.PROFILENAME ) Device Name (key to retrieve value is interfaces.DEVICENAME ) Source Name (key to retrieve value is interfaces.SOURCENAME ) Received Topic (key to retrieve value is interfaces.RECEIVEDTOPIC ) Note Received Topic only available when the message was received from the Edgex MessageBus or External MQTT triggers. Storage can be accessed using the following methods:","title":"Context Storage"},{"location":"microservices/application/AppFunctionContextAPI/#addvalue","text":"AddValue(key string, value string) This API stores a value for access within a pipeline execution","title":"AddValue"},{"location":"microservices/application/AppFunctionContextAPI/#removevalue","text":"RemoveValue(key string) This API deletes a value stored in the context at the given key","title":"RemoveValue"},{"location":"microservices/application/AppFunctionContextAPI/#getvalue","text":"GetValue(key string) (string, bool) This API attempts to retrieve a value stored in the context at the given key","title":"GetValue"},{"location":"microservices/application/AppFunctionContextAPI/#getallvalues","text":"GetAllValues() map[string]string This API returns a read-only copy of all data stored in the context","title":"GetAllValues"},{"location":"microservices/application/AppFunctionContextAPI/#applyvalues","text":"ApplyValues(format string) (string, error) This API will replace placeholders of the form {context-key-name} with the value found in the context at context-key-name . Note that key matching is case insensitive. An error will be returned if any placeholders in the provided string do NOT have a corresponding entry in the context storage map.","title":"ApplyValues"},{"location":"microservices/application/AppFunctionContextAPI/#secrets","text":"","title":"Secrets"},{"location":"microservices/application/AppFunctionContextAPI/#getsecret","text":"GetSecret(path string, keys ...string) This API is used to retrieve secrets from the secret store. path specifies the type or location of the secrets to retrieve. If specified, it is appended to the base path from the exclusive secret store configuration. keys specifies the list of secrets to be retrieved. If no keys are provided then all the keys associated with the specified path will be returned.","title":"GetSecret"},{"location":"microservices/application/AppFunctionContextAPI/#secretslastupdated","text":"SecretsLastUpdated() This API returns that timestamp for when the secrets in the SecretStore where last updated. Useful when a connection to external source needs to be redone when the credentials have been updated.","title":"SecretsLastUpdated"},{"location":"microservices/application/AppFunctionContextAPI/#miscellaneous","text":"","title":"Miscellaneous"},{"location":"microservices/application/AppFunctionContextAPI/#correlationid","text":"CorrelationID() This API returns the ID used to track the EdgeX event through entire EdgeX framework.","title":"CorrelationID"},{"location":"microservices/application/AppFunctionContextAPI/#pipelineid","text":"PipelineId() string This API returns the ID of the pipeline currently executing. Useful when logging messages from pipeline functions so the message contain the ID of the pipeline that executed the pipeline function.","title":"PipelineId"},{"location":"microservices/application/AppFunctionContextAPI/#inputcontenttype","text":"InputContentType() This API returns the content type of the data that initiated the pipeline execution. Only useful when the TargetType for the pipeline is []byte, otherwise the data will be the type specified by TargetType.","title":"InputContentType"},{"location":"microservices/application/AppFunctionContextAPI/#getdeviceresource","text":"GetDeviceResource(profileName string, resourceName string) (dtos.DeviceResource, error) This API retrieves the DeviceResource for the given profile / resource name. Results are cached to minimize HTTP traffic to core-metadata.","title":"GetDeviceResource"},{"location":"microservices/application/AppFunctionContextAPI/#pushtocore","text":"PushToCore(event dtos.Event) This API is used to push data to EdgeX Core Data so that it can be shared with other applications that are subscribed to the message bus that core-data publishes to. This function will return the new EdgeX Event with the ID populated, along with any error encountered. Note that CorrelationId will not be available. Note If validation is turned on in CoreServices then your deviceName and readingName must exist in the CoreMetadata and be properly registered in EdgeX. Warning Be aware that without a filter in your pipeline, it is possible to create an infinite loop when the Message Bus trigger is used. Choose your device-name and reading name appropriately.","title":"PushToCore"},{"location":"microservices/application/AppFunctionContextAPI/#setretrydata","text":"SetRetryData(data []byte) This method can be used to store data for later retry. This is useful when creating a custom export function that needs to retry on failure. The payload data will be stored for later retry based on Store and Forward configuration. When the retry is triggered, the function pipeline will be re-executed starting with the function that called this API. That function will be passed the stored data, so it is important that all transformations occur in functions prior to the export function. The Context will also be restored to the state when the function called this API. See Store and Forward for more details. Note Store and Forward be must enabled when calling this API, otherwise the data is ignored.","title":"SetRetryData"},{"location":"microservices/application/AppFunctionContextAPI/#metricsmanager","text":"MetricsManager() bootstrapInterfaces.MetricsManager This API returns the Metrics Manager used to register counter, gauge, gaugeFloat64 or timer metric types from github.com/rcrowley/go-metrics myCounterMetricName := \"MyCounter\" myCounter := gometrics . NewCounter () myTags := map [ string ] string { \"Tag1\" : \"Value1\" } ctx . MetricsManager (). Register ( myCounterMetricName , myCounter , myTags )","title":"MetricsManager"},{"location":"microservices/application/AppServiceConfigurable/","text":"App Service Configurable Getting Started App-Service-Configurable is provided as an easy way to get started with processing data flowing through EdgeX. This service leverages the App Functions SDK and provides a way for developers to use configuration instead of having to compile standalone services to utilize built in functions in the SDK. Please refer to Available Configurable Pipeline Functions section below for full list of built-in functions that can be used in the configurable pipeline. To get started with App Service Configurable, you'll want to start by determining which functions are required in your pipeline. Using a simple example, let's assume you wish to use the following functions from the SDK: FilterByDeviceName - to filter events for a specific device. Transform - to transform the data to XML HTTPExport - to send the data to an HTTP endpoint that takes our XML data Once the functions have been identified, we'll go ahead and build out the configuration in the configuration.toml file under the [Writable.Pipeline] section. Example - Writable.Pipeline [Writable] LogLevel = \"DEBUG\" [Writable.Pipeline] ExecutionOrder = \"FilterByDeviceName, Transform, HTTPExport\" [Writable.Pipeline.Functions] [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] FilterValues = \"Random-Float-Device, Random-Integer-Device\" [Writable.Pipeline.Functions.Transform] [Writable.Pipeline.Functions.Transform.Parameters] Type = \"xml\" [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" The first line of note is ExecutionOrder = \"FilterByDeviceName, Transform, HTTPExport\" . This specifies the order in which to execute your functions. Each function specified here must also be placed in the [Writeable.Pipeline.Functions] section. Next, each function and its required information is listed. Each function typically has associated Parameters that must be configured to properly execute the function as designated by [Writable.Pipeline.Functions.{FunctionName}.Parameters] . Knowing which parameters are required for each function, can be referenced by taking a look at the Available Configurable Pipeline Functions section below. Note By default, the configuration provided is set to use EdgexMessageBus as a trigger. This means you must have EdgeX Running with devices sending data in order to trigger the pipeline. You can also change the trigger to be HTTP. For more details on triggers, view the Triggers documentation located in the Triggers section. That's it! Now we can run/deploy this service and the functions pipeline will process the data with functions we've defined. Pipeline Per Topics EdgeX 2.1 Pipeline Per Topics is new for EdgeX 2.1 The above pipeline configuration in Getting Started section is the preferred way if your use case only requires a single functions pipeline. For use cases that require multiple functions pipelines in order to process the data differently based on the profile , device or source for the Event, there is the Pipeline Per Topics feature. This feature allows multiple pipelines to be configured in the [Writable.Pipeline.PerTopicPipelines] section. This section is a map of pipelines. The map key must be unique , but isn't used so can be any value. Each pipleline is defined by the following configuration settings: Id - This is the unique ID given to each pipeline Topics - Comma separated list of topics that control when the pipeline is executed. See Pipeline Per Topics for details on using wildcards in the topic. ExecutionOrder - This is the list of functions, in order, that the pipeline will execute. Same as ExecutionOrder in the above example in the Getting Started section Example - Writable.Pipeline.PerTopicPipelines In this example Events from the device Random-Float-Device are transformed to JSON and then HTTP exported. At the same time, Events for the source Int8 are transformed to XML and then HTTP exported to same endpoint. Note the custom naming for TransformJson and TransformXml . This is taking advantage of the Multiple Instances of a Function described below. [Writable] LogLevel = \"DEBUG\" [Writable.Pipeline] [Writable.Pipeline.PerTopicPipelines] [Writable.Pipeline.PerTopicPipelines.float] Id = \"float-pipeline\" Topics = \"edgex/events/device/#/Random-Float-Device/#, edgex/events/device/#/Random-Integer-Device/#\" ExecutionOrder = \"TransformJson, HTTPExport\" [Writable.Pipeline.PerTopicPipelines.int8] Id = \"int8-pipeline\" Topic = \"edgex/events/device/#/#/Int8\" ExecutionOrder = \"TransformXml, HTTPExport\" [Writable.Pipeline.Functions] [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] FilterValues = \"Random-Float-Device, Random-Integer-Device\" [Writable.Pipeline.Functions.TransformJson] [Writable.Pipeline.Functions.TransformJson.Parameters] Type = \"json\" [Writable.Pipeline.Functions.TransformXml] [Writable.Pipeline.Functions.TransformXml.Parameters] Type = \"xml\" [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" Note The Pipeline Per Topics feature is targeted for EdgeX MessageBus and External MQTT triggers, but can be used with Custom or HTTP triggers. When used with the HTTP trigger the incoming topic will always be blank , so the pipeline's topics must contain a single topic set to the # wildcard so that all messages received are processed by the pipeline. Environment Variable Overrides For Docker EdgeX services no longer have docker specific profiles. They now rely on environment variable overrides in the docker compose files for the docker specific differences. Example - Environment settings required in the compose files for App Service Configurable EDGEX_PROFILE : [ target profile ] SERVICE_HOST : [ services network host name ] EDGEX_SECURITY_SECRET_STORE : \"false\" # only need to disable as default is true CLIENTS_CORE_COMMAND_HOST : edgex-core-command CLIENTS_CORE_DATA_HOST : edgex-core-data CLIENTS_CORE_METADATA_HOST : edgex-core-metadata CLIENTS_SUPPORT_NOTIFICATIONS_HOST : edgex-support-notifications CLIENTS_SUPPORT_SCHEDULER_HOST : edgex-support-scheduler DATABASES_PRIMARY_HOST : edgex-redis MESSAGEQUEUE_HOST : edgex-redis REGISTRY_HOST : edgex-core-consul TRIGGER_EDGEXMESSAGEBUS_PUBLISHHOST_HOST : edgex-redis TRIGGER_EDGEXMESSAGEBUS_SUBSCRIBEHOST_HOST : edgex-redis Example - Docker compose entry for App Service Configurable in no-secure compose file app-service-rules : container_name : edgex-app-rules-engine depends_on : - consul - data environment : CLIENTS_CORE_COMMAND_HOST : edgex-core-command CLIENTS_CORE_DATA_HOST : edgex-core-data CLIENTS_CORE_METADATA_HOST : edgex-core-metadata CLIENTS_SUPPORT_NOTIFICATIONS_HOST : edgex-support-notifications CLIENTS_SUPPORT_SCHEDULER_HOST : edgex-support-scheduler DATABASES_PRIMARY_HOST : edgex-redis EDGEX_PROFILE : rules-engine EDGEX_SECURITY_SECRET_STORE : \"false\" MESSAGEQUEUE_HOST : edgex-redis REGISTRY_HOST : edgex-core-consul SERVICE_HOST : edgex-app-rules-engine TRIGGER_EDGEXMESSAGEBUS_PUBLISHHOST_HOST : edgex-redis TRIGGER_EDGEXMESSAGEBUS_SUBSCRIBEHOST_HOST : edgex-redis hostname : edgex-app-rules-engine image : edgexfoundry/app-service-configurable:2.0.0 networks : edgex-network : {} ports : - 127.0.0.1:59701:59701/tcp read_only : true security_opt : - no-new-privileges:true user : 2002:2001 Note App Service Configurable is designed to be run multiple times each with different profiles. This is why in the above example the name edgex-app-rules-engine is used for the instance running the rules-engine profile. Deploying Multiple Instances using profiles App Service Configurable was designed to be deployed as multiple instances for different purposes. Since the function pipeline is specified in the configuration.toml file, we can use this as a way to run each instance with a different function pipeline. App Service Configurable does not have the standard default configuration at /res/configuration.toml . This default configuration has been moved to the sample profile. This forces you to specify the profile for the configuration you would like to run. The profile is specified using the -p/--profile=[profilename] command line option or the EDGEX_PROFILE=[profilename] environment variable override. The profile name selected is used in the service key ( app-[profile name] ) to make each instance unique, e.g. AppService-sample when specifying sample as the profile. Edgex 2.0 Default service key for App Service Configurable instances has changed in Edgex 2.0 from AppService-[profile name] to app-[profile name] Note If you need to run multiple instances with the same profile, e.g. http-export , but configured differently, you will need to override the service key with a custom name for one or more of the services. This is done with the -sk/-serviceKey command-line option or the EDGEX_SERVICE_KEY environment variable. See the Command-line Options and Environment Overrides sections for more detail. Note Functions can be declared in a profile but not used in the pipeline ExecutionOrder allowing them to be added to the pipeline ExecutionOrder later at runtime if needed. The following profiles and their purposes are provided with App Service Configurable. rules-engine Profile used to push Event messages to the Rules Engine via the Redis Pub/Sub Message Bus. This is used in the default docker compose files for the app-rules-engine service One can optionally add Filter function via environment overrides WRITABLE_PIPELINE_EXECUTIONORDER: \"FilterByDeviceName, HTTPExport\" WRITABLE_PIPELINE_FUNCTIONS_FILTERBYDEVICENAME_PARAMETERS_DEVICENAMES: \"[comma separated list]\" There are many optional functions and parameters provided in this profile. See the complete profile for more details http-export Starter profile used for exporting data via HTTP. Requires further configuration which can easily be accomplished using environment variable overrides Required: WRITABLE_PIPELINE_FUNCTIONS_HTTPEXPORT_PARAMETERS_URL: [Your URL] There are many more optional functions and parameters provided in this profile. See the complete profile for more details. metrics-influxdb Edgex 2.2 The metrics-influxdb profile is new for Edgex 2.2 Starter profile used for exporting telemetry data from other EdgeX services to InfluxDB via HTTP export. This profile configures the service to receive telemetry data from other services, transform it to Line Protocol syntax, batch the data and then export it to an InfluxDB service via HTTP. Requires further configuration which can easily be accomplished using environment variable overrides. Required: WRITABLE_PIPELINE_FUNCTIONS_HTTPEXPORT_PARAMETERS_URL: [Your InfluxDB URL] Example value: `\"http://localhost:8086/api/v2/write?org=metrics&bucket=edgex&precision=ns\"`` `WRITABLE_INSECURESECRETS_INFLUXDB_SECRETS_TOKEN : [Your InfluxDB Token] Example value: \"Token 29ER8iMgQ5DPD_icTnSwH_77aUhSvD0AATkvMM59kZdIJOTNoJqcP-RHFCppblG3wSOb7LOqjp1xubA80uaWhQ==\" If using secure mode, store the token in the service's secret store via POST to the service's /secret endpoint Example JSON to post to /secret endpoint { \"apiVersion\" : \"v2\" , \"path\" : \"influxdb\" , \"secretData\" :[ { \"key\" : \"Token\" , \"value\" : \"Token 29ER8iMgQ5DPD_icTnSwH_77aUhSvD0AATkvMM59kZdIJOTNoJqcP-RHFCppblG3wSOb7LOqjp1xubA80uaWhQ==\" }] } Optional Additional Tags: WRITABLE_PIPELINE_FUNCTIONS_TOLINEPROTOCOL_PARAMETERS_TAGS: Currently set to empty string Example value: `\"tag1:value1, tag2:value2\" Optional Batching parameters (see Batch function for more details): WRITABLE_PIPELINE_FUNCTIONS_BATCH_PARAMETERS_MODE: Currently set to \"bytimecount\" Valid values are \"bycount\" , \"bytime\" or `\"bytimecount\"`` `WRITABLE_PIPELINE_FUNCTIONS_BATCH_PARAMETERS_BATCHTHRESHOLD: Currently set to 100 WRITABLE_PIPELINE_FUNCTIONS_BATCH_PARAMETERS_TIMEINTERVAL: Currently set to \"60s\" mqtt-export Starter profile used for exporting data via MQTT. Requires further configuration which can easily be accomplished using environment variable overrides Required: WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_BROKERADDRESS: [Your Broker Address] There are many optional functions and parameters provided in this profile. See the complete profile for more details push-to-core Example profile demonstrating how to use the PushToCore function. Provided as an exmaple that can be copied and modified to create new custom profile. See the complete profile for more details Requires further configuration which can easily be accomplished using environment variable overrides Required: WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_PROFILENAME: [Your Event's profile name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_DEVICENAME: [Your Event's device name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_SOURCENAME: [Your Event's source name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_RESOURCENAME: [Your Event reading's resource name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_VALUETYPE: [Your Event reading's value type] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_MEDIATYPE: [Your Event binary reading's media type] Required only when ValueType is Binary sample Sample profile with all available functions declared and a sample pipeline. Provided as a sample that can be copied and modified to create new custom profiles. See the complete profile for more details functional-tests Profile used for the TAF functional testing external-mqtt-trigger Profile used for the TAF functional testing of external MQTT Trigger What if my input data isn't an EdgeX Event ? The default TargetType for data flowing into the functions pipeline is an EdgeX Event DTO. There are cases when this incoming data might not be an EdgeX Event DTO. There are two setting that configure the TargetType to non-Event data. UseTargetTypeOfByteArray In these cases the Pipeline can be configured using UseTargetTypeOfByteArray=true to set the TargetType to be a byte array/slice, i.e. []byte . The first function in the pipeline must then be one that can handle the []byte data. The compression , encryption and export functions are examples of pipeline functions that will take input data that is []byte . Example - Configure the functions pipeline to compress , encrypt and then export the []byte data via HTTP [Writable] LogLevel = \"DEBUG\" [Writable.Pipeline] UseTargetTypeOfByteArray = true ExecutionOrder = \"Compress, Encrypt, HTTPExport\" [Writable.Pipeline.Functions.Compress] [Writable.Pipeline.Functions.Compress.Parameters] Alogrithm = \"gzip\" [Writable.Pipeline.Functions.Encrypt] [Writable.Pipeline.Functions.Encrypt.Parameters] Algorithm = \"aes\" Key = \"aquqweoruqwpeoruqwpoeruqwpoierupqoweiurpoqwiuerpqowieurqpowieurpoqiweuroipwqure\" InitVector = \"123456789012345678901234567890\" [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" Url = \"http://my.api.net/edgexdata\" MimeType = \"application/text\" If along with this pipeline configuration, you also configured the Trigger to be http trigger, you could then send any data to the app-service-configurable' s /api/v2/trigger endpoint and have it compressed, encrypted and sent to your configured URL above. Example - HTTP Trigger configuration [Trigger] Type = \"http\" UseTargetTypeOfMetric Edgex 2.2 New for EdgeX 2.2 is the UseTargetTypeOfMetric setting This setting when set to true will cause the TargeType to be &dtos.Metric{} and is meant to be used in conjunction with the new ToLineProtocol function. See ToLineProtocol section below for more details. In addition the Trigger SubscribeTopics must be set to \"edgex/telemetry/#\" so that the function receives the metric data from the other services. Example - UseTargetTypeOfMetric [Writable.Pipeline] UseTargetTypeOfMetric = true ExecutionOrder = \"ToLineProtocol, ...\" ... [Writable.Pipeline.Functions.ToLineProtocol] [Writable.Pipeline.Functions.ToLineProtocol.Parameters] Tags = \"\" # optional comma separated list of additional tags to add to the metric in to form \"tag:value,...\" ... [Trigger] Type=\"edgex-messagebus\" [Trigger.EdgexMessageBus] ... [Trigger.EdgexMessageBus.SubscribeHost] ... SubscribeTopics=\"edgex/telemetry/#\" Multiple Instances of a Function Edgex 2.0 New for EdgeX 2.0 Now multiple instances of the same configurable pipeline function can be specified, configured differently and used together in the functions pipeline. Previously the function names specified in the [Writable.Pipeline.Functions] section had to match a built-in configurable pipeline function name exactly. Now the names specified only need to start with a built-in configurable pipeline function name. See the HttpExport section below for an example. Available Configurable Pipeline Functions Below are the functions that are available to use in the configurable pipeline function pipeline ( [Writable.Pipeline] ) section of the configuration. The function names below can be added to the Writable.Pipeline.ExecutionOrder setting (comma separated list) and must also be present or added to the [Writable.Pipeline.Functions] section as [Writable.Pipeline.Functions.{FunctionName}] . The functions will also have the [Writable.Pipeline.Functions.{FunctionName}.Parameters] section where the function's parameters are configured. Please refer to the Getting Started section above for an example. Note The Parameters section for each function is a key/value map of string values. So even tough the parameter is referred to as an Integer or Boolean, it has to be specified as a valid string representation, e.g. \"20\" or \"true\". Please refer to the function's detailed documentation by clicking the function name below. AddTags Parameters tags - String containing comma separated list of tag key/value pairs. The tag key/value pairs are colon seperated Example [Writable.Pipeline.Functions.AddTags] [Writable.Pipeline.Functions.AddTags.Parameters] tags = \"GatewayId:HoustonStore000123,Latitude:29.630771,Longitude:-95.377603\" Batch Parameters Mode - The batch mode to use. can be 'bycount', 'bytime' or 'bytimecount' BatchThreshold - Number of items to batch before sending batched items to the next function in the pipeline. Used with 'bycount' and 'bytimecount' modes TimeInterval - Amount of time to batch before sending batched items to the next function in the pipeline. Used with 'bytime' and 'bytimecount' modes IsEventData - If true, specifies that the data being batched is Events and to un-marshal the batched data to []Event prior to returning the batched data. By default the batched data returned is [][]byte MergeOnSend - If true, specifies that the data being batched is to be merged to a single []byte prior to returning the batched data. By default the batched data returned is [][]byte Example [Writable.Pipeline.Functions.Batch] [Writable.Pipeline.Functions.Batch.Parameters] Mode = \"bytimecount\" # can be \"bycount\", \"bytime\" or \"bytimecount\" BatchThreshold = \"30\" TimeInterval = \"60s\" IsEventData = \"false\" MergeOnSend = \"false\" or [Writable.Pipeline.Functions.Batch] [Writable.Pipeline.Functions.Batch.Parameters] Mode = \"bytimecount\" # can be \"bycount\", \"bytime\" or \"bytimecount\" BatchThreshold = \"30\" TimeInterval = \"60s\" IsEventData = \"true\" MergeOnSend = \"false\" or [Writable.Pipeline.Functions.Batch] [Writable.Pipeline.Functions.Batch.Parameters] Mode = \"bytimecount\" # can be \"bycount\", \"bytime\" or \"bytimecount\" BatchThreshold = \"30\" TimeInterval = \"60s\" IsEventData = \"false\" MergeOnSend = \"true\" EdgeX 2.0 For EdgeX 2.0 the BatchByCount , BatchByTime , and BatchByTimeCount configurable pipeline functions have been replaced by single Batch configurable pipeline function with additional Mode parameter. EdgeX 2.1 The IsEventData setting is new for EdgeX 2.1 EdgeX 2.1 The MergeOnSend setting is new for EdgeX 2.2 Compress Parameters Algorithm - Compression algorithm to use. Can be 'gzip' or 'zlib' Example [Writable.Pipeline.Functions.Compress] [Writable.Pipeline.Functions.Compress.Parameters] Algorithm = \"gzip\" EdgeX 2.0 For EdgeX 2.0 the CompressWithGZIP and CompressWithZLIB configurable pipeline functions have been replaced by the single Compress configurable pipeline function with additional Algorithm parameter. Encrypt Parameters Algorithm - AES (deprecated) or AES256 Key - (optional, deprecated) Encryption key used for the encryption. Required if not using Secret Store for the encryption key data InitVector - (deprecated) Initialization vector used for the encryption. SecretPath - (required for AES256) Path in the Secret Store where the encryption key is located. Required if Key not specified. SecretName - (required for AES256) Name of the secret for the encryption key in the Secret Store . Required if Key not specified. Example # Encrypt with key specified in configuration [Writable.Pipeline.Functions.Encrypt] [Writable.Pipeline.Functions.Encrypt.Parameters] Algorithm = \"aes\" Key = \"aquqweoruqwpeoruqwpoeruqwpoierupqoweiurpoqwiuerpqowieurqpowieurpoqiweuroipwqure\" InitVector = \"123456789012345678901234567890\" # Encrypt with key pulled from Secret Store [Writable.Pipeline.Functions.Encrypt] [Writable.Pipeline.Functions.Encrypt.Parameters] Algorithm = \"aes\" InitVector = \"123456789012345678901234567890\" SecretPath = \"aes\" SecretName = \"key\" EdgeX 2.0 For EdgeX 2.0 the EncryptWithAES configurable pipeline function have been replaced by the Encrypt configurable pipeline function with additional Algorithm parameter. In addition the ability to pull the encryption key from the Secret Store has been added. FilterByDeviceName Parameters DeviceNames - Comma separated list of device names for filtering FilterOut - Boolean indicating if the data matching the device names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] DeviceNames = \"Random-Float-Device,Random-Integer-Device\" FilterOut = \"false\" FilterByProfileName Parameters ProfileNames - Comma separated list of profile names for filtering FilterOut - Boolean indicating if the data matching the profile names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterByProfileName] [Writable.Pipeline.Functions.FilterByProfileName.Parameters] ProfileNames = \"Random-Float-Device, Random-Integer-Device\" FilterOut = \"false\" EdgeX 2.0 The FilterByProfileName configurable pipeline function is new for EdgeX 2.0 FilterByResourceName Parameters ResourceName - Comma separated list of reading resource names for filtering FilterOut - Boolean indicating if the readings matching the resource names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterByResourceName] [Writable.Pipeline.Functions.FilterByResourceName.Parameters] ResourceNames = \"Int8, Int64\" FilterOut = \"true\" EdgeX 2.0 For EdgeX 2.0 the FilterByValueDescriptor configurable pipeline function has been renamed to FilterByResourceName and parameter names adjusted. FilterBySourceName Parameters SourceNames - Comma separated list of source names for filtering. Source name is either the device command name or the resource name that created the Event FilterOut - Boolean indicating if the data matching the device names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterBySourceName] [Writable.Pipeline.Functions.FilterBySource.Parameters] SourceNames = \"Bool, BoolArray\" FilterOut = \"false\" EdgeX 2.0 The FilterBySourceName configurable pipeline function is new for EdgeX 2.0 HTTPExport Parameters Method - HTTP Method to use. Can be post or put Url - HTTP endpoint to POST/PUT the data. MimeType - Optional mime type for the data. Defaults to application/json if not set. PersistOnError - Indicates to persist the data if the POST fails. Store and Forward must also be enabled if this is set to \"true\". ContinueOnSendError - For chained multi destination exports, if true continues after send error so next export function executes. ReturnInputData - For chained multi destination exports if true, passes the input data to next export function. HeaderName - (Optional) Name of the header key to add to the HTTP header SecretPath - (Optional) Path of the secret in the Secret Store where the header value is stored. SecretName - (Optional) Name of the secret for the header value in the Secret Store . Example # Simple HTTP Export [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" # HTTP Export with secret header data pull from Secret Store [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" HeaderName = \"MyApiKey\" SecretPath = \"http\" SecretName = \"apikey\" # Http Export to multiple destinations [Writable.Pipeline] ExecutionOrder = \"HTTPExport1, HTTPExport2\" [Writable.Pipeline.Functions.HTTPExport1] [Writable.Pipeline.Functions.HTTPExport1.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api1.net/edgexdata2\" ContinueOnSendError = \"true\" ReturnInputData = \"true\" [Writable.Pipeline.Functions.HTTPExport2] [Writable.Pipeline.Functions.HTTPExport2.Parameters] Method = \"put\" MimeType = \"application/xml\" Url = \"http://my.api2.net/edgexdata2\" EdgeX 2.0 For EdgeX 2.0 the HTTPPost , HTTPPostJSON , HTTPPostXML , HTTPPut , HTTPPutJSON , and HTTPPutXML configurable pipeline functions have been replaced by the single HTTPExport function with additional Method parameter. ContinueOnSendError and ReturnInputData parameter have been added to support multi destination exports. In addition the HeaderName and SecretName parameters have replaced the SecretHeaderName parameter. EdgeX 2.0 The capability to chain Http Export functions to export to multiple destinations is new for Edgex 2.0. EdgeX 2.0 Multiple instances (configured differently) of the same configurable pipeline function is new for EdgeX 2.0. The function names in the Writable.Pipeline.Functions section now only need to start with a built-in configurable pipeline function name, rather than be an exact match. JSONLogic Parameters Rule - The JSON formatted rule that with be executed on the data by JSONLogic Example [Writable.Pipeline.Functions.JSONLogic] [Writable.Pipeline.Functions.JSONLogic.Parameters] Rule = \"{ \\\"and\\\" : [{\\\"<\\\" : [{ \\\"var\\\" : \\\"temp\\\" }, 110 ]}, {\\\"==\\\" : [{ \\\"var\\\" : \\\"sensor.type\\\" }, \\\"temperature\\\" ]} ] }\" MQTTExport Parameters BrokerAddress - URL specify the address of the MQTT Broker Topic - Topic to publish the data ClientId - Id to use when connecting to the MQTT Broker Qos - MQTT Quality of Service (QOS) setting to use (0, 1 or 2). Please refer here for more details on QOS values AutoReconnect - Boolean specifying if reconnect should be automatic if connection to MQTT broker is lost Retain - Boolean specifying if the MQTT Broker should save the last message published as the \u201cLast Good Message\u201d on that topic. SkipVerify - Boolean indicating if the certificate verification should be skipped. PersistOnError - Indicates to persist the data if the POST fails. Store and Forward must also be enabled if this is set to \"true\". AuthMode - Mode of authentication to use when connecting to the MQTT Broker none - No authentication required usernamepassword - Use username and password authentication. The Secret Store (Vault or InsecureSecrets ) must contain the username and password secrets. clientcert - Use Client Certificate authentication. The Secret Store (Vault or InsecureSecrets ) must contain the clientkey and clientcert secrets. cacert - Use CA Certificate authentication. The Secret Store (Vault or InsecureSecrets ) must contain the cacert secret. SecretPath - Path in the secret store where authentication secrets are stored. Note Authmode=cacert is only needed when client authentication (e.g. usernamepassword ) is not required, but a CA Cert is needed to validate the broker's SSL/TLS cert. Example # Simple MQTT Export [Writable.Pipeline.Functions.MQTTExport] [Writable.Pipeline.Functions.MQTTExport.Parameters] BrokerAddress = \"tcps://localhost:8883\" Topic = \"mytopic\" ClientId = \"myclientid\" # MQTT Export with auth credentials pull from the Secret Store [Writable.Pipeline.Functions.MQTTExport] [Writable.Pipeline.Functions.MQTTExport.Parameters] BrokerAddress = \"tcps://my-broker-host.com:8883\" Topic = \"mytopic\" ClientId = \"myclientid\" Qos = \"2\" AutoReconnect = \"true\" Retain = \"true\" SkipVerify = \"false\" PersistOnError = \"true\" AuthMode = \"usernamepassword\" SecretPath = \"mqtt\" EdgeX 2.0 For EdgeX 2.0 the MQTTSecretSend configurable pipeline function has been renamed to MQTTExport and the deprecated MQTTSend configurable pipeline function has been removed PushToCore Parameters ProfileName - Profile name to use for the new Event DeviceName - Device name to use for the new Event ResourceName - Resource name name to use for the new Event's SourceName and Reading's ResourceName ValueType - Value type to use the new Event Reading's value type MediaType - Media type to use the new Event Reading's value type. Required when the value type is Binary Example [Writable.Pipeline.Functions.PushToCore] [Writable.Pipeline.Functions.PushToCore.Parameters] ProfileName = \"MyProfile\" DeviceName = \"MyDevice\" ResourceName = \"SomeResource\" ValueType = \"String\" EdgeX 2.0 For EdgeX 2.0 the ProfileName , ValueType and MediaType parameters are new and the ReadingName parameter has been renamed to ResourceName . SetResponseData Parameters ResponseContentType - Used to specify content-type header for response - optional Example [Writable.Pipeline.Functions.SetResponseData] [Writable.Pipeline.Functions.SetResponseData.Parameters] ResponseContentType = \"application/json\" EdgeX 2.0 For EdgeX 2.0 the SetOutputData configurable pipeline function has been renamed to SetResponseData . Transform Parameters Type - Type of transformation to perform. Can be 'xml' or 'json' Example [Writable.Pipeline.Functions.Transform] [Writable.Pipeline.Functions.Transform.Parameters] Type = \"xml\" EdgeX 2.0 For EdgeX 2.0 the TransformToJSON and TransformToXML configurable pipeline functions have been replaced by the single Transform configurable pipeline function with additional Type parameter. ToLineProtocol EdgeX 2.2 ToLineProtocol is new for Edgex 2.2 Parameters Tags - optional comma separated list of additional tags to add to the metric in to form \"tag:value,...\" Example [Writable.Pipeline.Functions.ToLineProtocol] [Writable.Pipeline.Functions.ToLineProtocol.Parameters] Tags = \"\" # optional comma separated list of additional tags to add to the metric in to form \"tag:value,...\" Note The new UseTargetTypeOfMetric setting must be set to true when using this function. See the UseTargetTypeOfMetric section above for more details.","title":"App Service Configurable"},{"location":"microservices/application/AppServiceConfigurable/#app-service-configurable","text":"","title":"App Service Configurable"},{"location":"microservices/application/AppServiceConfigurable/#getting-started","text":"App-Service-Configurable is provided as an easy way to get started with processing data flowing through EdgeX. This service leverages the App Functions SDK and provides a way for developers to use configuration instead of having to compile standalone services to utilize built in functions in the SDK. Please refer to Available Configurable Pipeline Functions section below for full list of built-in functions that can be used in the configurable pipeline. To get started with App Service Configurable, you'll want to start by determining which functions are required in your pipeline. Using a simple example, let's assume you wish to use the following functions from the SDK: FilterByDeviceName - to filter events for a specific device. Transform - to transform the data to XML HTTPExport - to send the data to an HTTP endpoint that takes our XML data Once the functions have been identified, we'll go ahead and build out the configuration in the configuration.toml file under the [Writable.Pipeline] section. Example - Writable.Pipeline [Writable] LogLevel = \"DEBUG\" [Writable.Pipeline] ExecutionOrder = \"FilterByDeviceName, Transform, HTTPExport\" [Writable.Pipeline.Functions] [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] FilterValues = \"Random-Float-Device, Random-Integer-Device\" [Writable.Pipeline.Functions.Transform] [Writable.Pipeline.Functions.Transform.Parameters] Type = \"xml\" [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" The first line of note is ExecutionOrder = \"FilterByDeviceName, Transform, HTTPExport\" . This specifies the order in which to execute your functions. Each function specified here must also be placed in the [Writeable.Pipeline.Functions] section. Next, each function and its required information is listed. Each function typically has associated Parameters that must be configured to properly execute the function as designated by [Writable.Pipeline.Functions.{FunctionName}.Parameters] . Knowing which parameters are required for each function, can be referenced by taking a look at the Available Configurable Pipeline Functions section below. Note By default, the configuration provided is set to use EdgexMessageBus as a trigger. This means you must have EdgeX Running with devices sending data in order to trigger the pipeline. You can also change the trigger to be HTTP. For more details on triggers, view the Triggers documentation located in the Triggers section. That's it! Now we can run/deploy this service and the functions pipeline will process the data with functions we've defined.","title":"Getting Started"},{"location":"microservices/application/AppServiceConfigurable/#pipeline-per-topics","text":"EdgeX 2.1 Pipeline Per Topics is new for EdgeX 2.1 The above pipeline configuration in Getting Started section is the preferred way if your use case only requires a single functions pipeline. For use cases that require multiple functions pipelines in order to process the data differently based on the profile , device or source for the Event, there is the Pipeline Per Topics feature. This feature allows multiple pipelines to be configured in the [Writable.Pipeline.PerTopicPipelines] section. This section is a map of pipelines. The map key must be unique , but isn't used so can be any value. Each pipleline is defined by the following configuration settings: Id - This is the unique ID given to each pipeline Topics - Comma separated list of topics that control when the pipeline is executed. See Pipeline Per Topics for details on using wildcards in the topic. ExecutionOrder - This is the list of functions, in order, that the pipeline will execute. Same as ExecutionOrder in the above example in the Getting Started section Example - Writable.Pipeline.PerTopicPipelines In this example Events from the device Random-Float-Device are transformed to JSON and then HTTP exported. At the same time, Events for the source Int8 are transformed to XML and then HTTP exported to same endpoint. Note the custom naming for TransformJson and TransformXml . This is taking advantage of the Multiple Instances of a Function described below. [Writable] LogLevel = \"DEBUG\" [Writable.Pipeline] [Writable.Pipeline.PerTopicPipelines] [Writable.Pipeline.PerTopicPipelines.float] Id = \"float-pipeline\" Topics = \"edgex/events/device/#/Random-Float-Device/#, edgex/events/device/#/Random-Integer-Device/#\" ExecutionOrder = \"TransformJson, HTTPExport\" [Writable.Pipeline.PerTopicPipelines.int8] Id = \"int8-pipeline\" Topic = \"edgex/events/device/#/#/Int8\" ExecutionOrder = \"TransformXml, HTTPExport\" [Writable.Pipeline.Functions] [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] FilterValues = \"Random-Float-Device, Random-Integer-Device\" [Writable.Pipeline.Functions.TransformJson] [Writable.Pipeline.Functions.TransformJson.Parameters] Type = \"json\" [Writable.Pipeline.Functions.TransformXml] [Writable.Pipeline.Functions.TransformXml.Parameters] Type = \"xml\" [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" Note The Pipeline Per Topics feature is targeted for EdgeX MessageBus and External MQTT triggers, but can be used with Custom or HTTP triggers. When used with the HTTP trigger the incoming topic will always be blank , so the pipeline's topics must contain a single topic set to the # wildcard so that all messages received are processed by the pipeline.","title":"Pipeline Per Topics"},{"location":"microservices/application/AppServiceConfigurable/#environment-variable-overrides-for-docker","text":"EdgeX services no longer have docker specific profiles. They now rely on environment variable overrides in the docker compose files for the docker specific differences. Example - Environment settings required in the compose files for App Service Configurable EDGEX_PROFILE : [ target profile ] SERVICE_HOST : [ services network host name ] EDGEX_SECURITY_SECRET_STORE : \"false\" # only need to disable as default is true CLIENTS_CORE_COMMAND_HOST : edgex-core-command CLIENTS_CORE_DATA_HOST : edgex-core-data CLIENTS_CORE_METADATA_HOST : edgex-core-metadata CLIENTS_SUPPORT_NOTIFICATIONS_HOST : edgex-support-notifications CLIENTS_SUPPORT_SCHEDULER_HOST : edgex-support-scheduler DATABASES_PRIMARY_HOST : edgex-redis MESSAGEQUEUE_HOST : edgex-redis REGISTRY_HOST : edgex-core-consul TRIGGER_EDGEXMESSAGEBUS_PUBLISHHOST_HOST : edgex-redis TRIGGER_EDGEXMESSAGEBUS_SUBSCRIBEHOST_HOST : edgex-redis Example - Docker compose entry for App Service Configurable in no-secure compose file app-service-rules : container_name : edgex-app-rules-engine depends_on : - consul - data environment : CLIENTS_CORE_COMMAND_HOST : edgex-core-command CLIENTS_CORE_DATA_HOST : edgex-core-data CLIENTS_CORE_METADATA_HOST : edgex-core-metadata CLIENTS_SUPPORT_NOTIFICATIONS_HOST : edgex-support-notifications CLIENTS_SUPPORT_SCHEDULER_HOST : edgex-support-scheduler DATABASES_PRIMARY_HOST : edgex-redis EDGEX_PROFILE : rules-engine EDGEX_SECURITY_SECRET_STORE : \"false\" MESSAGEQUEUE_HOST : edgex-redis REGISTRY_HOST : edgex-core-consul SERVICE_HOST : edgex-app-rules-engine TRIGGER_EDGEXMESSAGEBUS_PUBLISHHOST_HOST : edgex-redis TRIGGER_EDGEXMESSAGEBUS_SUBSCRIBEHOST_HOST : edgex-redis hostname : edgex-app-rules-engine image : edgexfoundry/app-service-configurable:2.0.0 networks : edgex-network : {} ports : - 127.0.0.1:59701:59701/tcp read_only : true security_opt : - no-new-privileges:true user : 2002:2001 Note App Service Configurable is designed to be run multiple times each with different profiles. This is why in the above example the name edgex-app-rules-engine is used for the instance running the rules-engine profile.","title":"Environment Variable Overrides For Docker"},{"location":"microservices/application/AppServiceConfigurable/#deploying-multiple-instances-using-profiles","text":"App Service Configurable was designed to be deployed as multiple instances for different purposes. Since the function pipeline is specified in the configuration.toml file, we can use this as a way to run each instance with a different function pipeline. App Service Configurable does not have the standard default configuration at /res/configuration.toml . This default configuration has been moved to the sample profile. This forces you to specify the profile for the configuration you would like to run. The profile is specified using the -p/--profile=[profilename] command line option or the EDGEX_PROFILE=[profilename] environment variable override. The profile name selected is used in the service key ( app-[profile name] ) to make each instance unique, e.g. AppService-sample when specifying sample as the profile. Edgex 2.0 Default service key for App Service Configurable instances has changed in Edgex 2.0 from AppService-[profile name] to app-[profile name] Note If you need to run multiple instances with the same profile, e.g. http-export , but configured differently, you will need to override the service key with a custom name for one or more of the services. This is done with the -sk/-serviceKey command-line option or the EDGEX_SERVICE_KEY environment variable. See the Command-line Options and Environment Overrides sections for more detail. Note Functions can be declared in a profile but not used in the pipeline ExecutionOrder allowing them to be added to the pipeline ExecutionOrder later at runtime if needed. The following profiles and their purposes are provided with App Service Configurable.","title":"Deploying Multiple Instances using profiles"},{"location":"microservices/application/AppServiceConfigurable/#rules-engine","text":"Profile used to push Event messages to the Rules Engine via the Redis Pub/Sub Message Bus. This is used in the default docker compose files for the app-rules-engine service One can optionally add Filter function via environment overrides WRITABLE_PIPELINE_EXECUTIONORDER: \"FilterByDeviceName, HTTPExport\" WRITABLE_PIPELINE_FUNCTIONS_FILTERBYDEVICENAME_PARAMETERS_DEVICENAMES: \"[comma separated list]\" There are many optional functions and parameters provided in this profile. See the complete profile for more details","title":"rules-engine"},{"location":"microservices/application/AppServiceConfigurable/#http-export","text":"Starter profile used for exporting data via HTTP. Requires further configuration which can easily be accomplished using environment variable overrides Required: WRITABLE_PIPELINE_FUNCTIONS_HTTPEXPORT_PARAMETERS_URL: [Your URL] There are many more optional functions and parameters provided in this profile. See the complete profile for more details.","title":"http-export"},{"location":"microservices/application/AppServiceConfigurable/#metrics-influxdb","text":"Edgex 2.2 The metrics-influxdb profile is new for Edgex 2.2 Starter profile used for exporting telemetry data from other EdgeX services to InfluxDB via HTTP export. This profile configures the service to receive telemetry data from other services, transform it to Line Protocol syntax, batch the data and then export it to an InfluxDB service via HTTP. Requires further configuration which can easily be accomplished using environment variable overrides. Required: WRITABLE_PIPELINE_FUNCTIONS_HTTPEXPORT_PARAMETERS_URL: [Your InfluxDB URL] Example value: `\"http://localhost:8086/api/v2/write?org=metrics&bucket=edgex&precision=ns\"`` `WRITABLE_INSECURESECRETS_INFLUXDB_SECRETS_TOKEN : [Your InfluxDB Token] Example value: \"Token 29ER8iMgQ5DPD_icTnSwH_77aUhSvD0AATkvMM59kZdIJOTNoJqcP-RHFCppblG3wSOb7LOqjp1xubA80uaWhQ==\" If using secure mode, store the token in the service's secret store via POST to the service's /secret endpoint Example JSON to post to /secret endpoint { \"apiVersion\" : \"v2\" , \"path\" : \"influxdb\" , \"secretData\" :[ { \"key\" : \"Token\" , \"value\" : \"Token 29ER8iMgQ5DPD_icTnSwH_77aUhSvD0AATkvMM59kZdIJOTNoJqcP-RHFCppblG3wSOb7LOqjp1xubA80uaWhQ==\" }] } Optional Additional Tags: WRITABLE_PIPELINE_FUNCTIONS_TOLINEPROTOCOL_PARAMETERS_TAGS: Currently set to empty string Example value: `\"tag1:value1, tag2:value2\" Optional Batching parameters (see Batch function for more details): WRITABLE_PIPELINE_FUNCTIONS_BATCH_PARAMETERS_MODE: Currently set to \"bytimecount\" Valid values are \"bycount\" , \"bytime\" or `\"bytimecount\"`` `WRITABLE_PIPELINE_FUNCTIONS_BATCH_PARAMETERS_BATCHTHRESHOLD: Currently set to 100 WRITABLE_PIPELINE_FUNCTIONS_BATCH_PARAMETERS_TIMEINTERVAL: Currently set to \"60s\"","title":"metrics-influxdb"},{"location":"microservices/application/AppServiceConfigurable/#mqtt-export","text":"Starter profile used for exporting data via MQTT. Requires further configuration which can easily be accomplished using environment variable overrides Required: WRITABLE_PIPELINE_FUNCTIONS_MQTTEXPORT_PARAMETERS_BROKERADDRESS: [Your Broker Address] There are many optional functions and parameters provided in this profile. See the complete profile for more details","title":"mqtt-export"},{"location":"microservices/application/AppServiceConfigurable/#push-to-core","text":"Example profile demonstrating how to use the PushToCore function. Provided as an exmaple that can be copied and modified to create new custom profile. See the complete profile for more details Requires further configuration which can easily be accomplished using environment variable overrides Required: WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_PROFILENAME: [Your Event's profile name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_DEVICENAME: [Your Event's device name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_SOURCENAME: [Your Event's source name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_RESOURCENAME: [Your Event reading's resource name] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_VALUETYPE: [Your Event reading's value type] WRITABLE_PIPELINE_FUNCTIONS_PUSHTOCORE_MEDIATYPE: [Your Event binary reading's media type] Required only when ValueType is Binary","title":"push-to-core"},{"location":"microservices/application/AppServiceConfigurable/#sample","text":"Sample profile with all available functions declared and a sample pipeline. Provided as a sample that can be copied and modified to create new custom profiles. See the complete profile for more details","title":"sample"},{"location":"microservices/application/AppServiceConfigurable/#functional-tests","text":"Profile used for the TAF functional testing","title":"functional-tests"},{"location":"microservices/application/AppServiceConfigurable/#external-mqtt-trigger","text":"Profile used for the TAF functional testing of external MQTT Trigger","title":"external-mqtt-trigger"},{"location":"microservices/application/AppServiceConfigurable/#what-if-my-input-data-isnt-an-edgex-event","text":"The default TargetType for data flowing into the functions pipeline is an EdgeX Event DTO. There are cases when this incoming data might not be an EdgeX Event DTO. There are two setting that configure the TargetType to non-Event data.","title":"What if my input data isn't an EdgeX Event ?"},{"location":"microservices/application/AppServiceConfigurable/#usetargettypeofbytearray","text":"In these cases the Pipeline can be configured using UseTargetTypeOfByteArray=true to set the TargetType to be a byte array/slice, i.e. []byte . The first function in the pipeline must then be one that can handle the []byte data. The compression , encryption and export functions are examples of pipeline functions that will take input data that is []byte . Example - Configure the functions pipeline to compress , encrypt and then export the []byte data via HTTP [Writable] LogLevel = \"DEBUG\" [Writable.Pipeline] UseTargetTypeOfByteArray = true ExecutionOrder = \"Compress, Encrypt, HTTPExport\" [Writable.Pipeline.Functions.Compress] [Writable.Pipeline.Functions.Compress.Parameters] Alogrithm = \"gzip\" [Writable.Pipeline.Functions.Encrypt] [Writable.Pipeline.Functions.Encrypt.Parameters] Algorithm = \"aes\" Key = \"aquqweoruqwpeoruqwpoeruqwpoierupqoweiurpoqwiuerpqowieurqpowieurpoqiweuroipwqure\" InitVector = \"123456789012345678901234567890\" [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" Url = \"http://my.api.net/edgexdata\" MimeType = \"application/text\" If along with this pipeline configuration, you also configured the Trigger to be http trigger, you could then send any data to the app-service-configurable' s /api/v2/trigger endpoint and have it compressed, encrypted and sent to your configured URL above. Example - HTTP Trigger configuration [Trigger] Type = \"http\"","title":"UseTargetTypeOfByteArray"},{"location":"microservices/application/AppServiceConfigurable/#usetargettypeofmetric","text":"Edgex 2.2 New for EdgeX 2.2 is the UseTargetTypeOfMetric setting This setting when set to true will cause the TargeType to be &dtos.Metric{} and is meant to be used in conjunction with the new ToLineProtocol function. See ToLineProtocol section below for more details. In addition the Trigger SubscribeTopics must be set to \"edgex/telemetry/#\" so that the function receives the metric data from the other services. Example - UseTargetTypeOfMetric [Writable.Pipeline] UseTargetTypeOfMetric = true ExecutionOrder = \"ToLineProtocol, ...\" ... [Writable.Pipeline.Functions.ToLineProtocol] [Writable.Pipeline.Functions.ToLineProtocol.Parameters] Tags = \"\" # optional comma separated list of additional tags to add to the metric in to form \"tag:value,...\" ... [Trigger] Type=\"edgex-messagebus\" [Trigger.EdgexMessageBus] ... [Trigger.EdgexMessageBus.SubscribeHost] ... SubscribeTopics=\"edgex/telemetry/#\"","title":"UseTargetTypeOfMetric"},{"location":"microservices/application/AppServiceConfigurable/#multiple-instances-of-a-function","text":"Edgex 2.0 New for EdgeX 2.0 Now multiple instances of the same configurable pipeline function can be specified, configured differently and used together in the functions pipeline. Previously the function names specified in the [Writable.Pipeline.Functions] section had to match a built-in configurable pipeline function name exactly. Now the names specified only need to start with a built-in configurable pipeline function name. See the HttpExport section below for an example.","title":"Multiple Instances of a Function"},{"location":"microservices/application/AppServiceConfigurable/#available-configurable-pipeline-functions","text":"Below are the functions that are available to use in the configurable pipeline function pipeline ( [Writable.Pipeline] ) section of the configuration. The function names below can be added to the Writable.Pipeline.ExecutionOrder setting (comma separated list) and must also be present or added to the [Writable.Pipeline.Functions] section as [Writable.Pipeline.Functions.{FunctionName}] . The functions will also have the [Writable.Pipeline.Functions.{FunctionName}.Parameters] section where the function's parameters are configured. Please refer to the Getting Started section above for an example. Note The Parameters section for each function is a key/value map of string values. So even tough the parameter is referred to as an Integer or Boolean, it has to be specified as a valid string representation, e.g. \"20\" or \"true\". Please refer to the function's detailed documentation by clicking the function name below.","title":"Available Configurable Pipeline Functions"},{"location":"microservices/application/AppServiceConfigurable/#addtags","text":"Parameters tags - String containing comma separated list of tag key/value pairs. The tag key/value pairs are colon seperated Example [Writable.Pipeline.Functions.AddTags] [Writable.Pipeline.Functions.AddTags.Parameters] tags = \"GatewayId:HoustonStore000123,Latitude:29.630771,Longitude:-95.377603\"","title":"AddTags"},{"location":"microservices/application/AppServiceConfigurable/#batch","text":"Parameters Mode - The batch mode to use. can be 'bycount', 'bytime' or 'bytimecount' BatchThreshold - Number of items to batch before sending batched items to the next function in the pipeline. Used with 'bycount' and 'bytimecount' modes TimeInterval - Amount of time to batch before sending batched items to the next function in the pipeline. Used with 'bytime' and 'bytimecount' modes IsEventData - If true, specifies that the data being batched is Events and to un-marshal the batched data to []Event prior to returning the batched data. By default the batched data returned is [][]byte MergeOnSend - If true, specifies that the data being batched is to be merged to a single []byte prior to returning the batched data. By default the batched data returned is [][]byte Example [Writable.Pipeline.Functions.Batch] [Writable.Pipeline.Functions.Batch.Parameters] Mode = \"bytimecount\" # can be \"bycount\", \"bytime\" or \"bytimecount\" BatchThreshold = \"30\" TimeInterval = \"60s\" IsEventData = \"false\" MergeOnSend = \"false\" or [Writable.Pipeline.Functions.Batch] [Writable.Pipeline.Functions.Batch.Parameters] Mode = \"bytimecount\" # can be \"bycount\", \"bytime\" or \"bytimecount\" BatchThreshold = \"30\" TimeInterval = \"60s\" IsEventData = \"true\" MergeOnSend = \"false\" or [Writable.Pipeline.Functions.Batch] [Writable.Pipeline.Functions.Batch.Parameters] Mode = \"bytimecount\" # can be \"bycount\", \"bytime\" or \"bytimecount\" BatchThreshold = \"30\" TimeInterval = \"60s\" IsEventData = \"false\" MergeOnSend = \"true\" EdgeX 2.0 For EdgeX 2.0 the BatchByCount , BatchByTime , and BatchByTimeCount configurable pipeline functions have been replaced by single Batch configurable pipeline function with additional Mode parameter. EdgeX 2.1 The IsEventData setting is new for EdgeX 2.1 EdgeX 2.1 The MergeOnSend setting is new for EdgeX 2.2","title":"Batch"},{"location":"microservices/application/AppServiceConfigurable/#compress","text":"Parameters Algorithm - Compression algorithm to use. Can be 'gzip' or 'zlib' Example [Writable.Pipeline.Functions.Compress] [Writable.Pipeline.Functions.Compress.Parameters] Algorithm = \"gzip\" EdgeX 2.0 For EdgeX 2.0 the CompressWithGZIP and CompressWithZLIB configurable pipeline functions have been replaced by the single Compress configurable pipeline function with additional Algorithm parameter.","title":"Compress"},{"location":"microservices/application/AppServiceConfigurable/#encrypt","text":"Parameters Algorithm - AES (deprecated) or AES256 Key - (optional, deprecated) Encryption key used for the encryption. Required if not using Secret Store for the encryption key data InitVector - (deprecated) Initialization vector used for the encryption. SecretPath - (required for AES256) Path in the Secret Store where the encryption key is located. Required if Key not specified. SecretName - (required for AES256) Name of the secret for the encryption key in the Secret Store . Required if Key not specified. Example # Encrypt with key specified in configuration [Writable.Pipeline.Functions.Encrypt] [Writable.Pipeline.Functions.Encrypt.Parameters] Algorithm = \"aes\" Key = \"aquqweoruqwpeoruqwpoeruqwpoierupqoweiurpoqwiuerpqowieurqpowieurpoqiweuroipwqure\" InitVector = \"123456789012345678901234567890\" # Encrypt with key pulled from Secret Store [Writable.Pipeline.Functions.Encrypt] [Writable.Pipeline.Functions.Encrypt.Parameters] Algorithm = \"aes\" InitVector = \"123456789012345678901234567890\" SecretPath = \"aes\" SecretName = \"key\" EdgeX 2.0 For EdgeX 2.0 the EncryptWithAES configurable pipeline function have been replaced by the Encrypt configurable pipeline function with additional Algorithm parameter. In addition the ability to pull the encryption key from the Secret Store has been added.","title":"Encrypt"},{"location":"microservices/application/AppServiceConfigurable/#filterbydevicename","text":"Parameters DeviceNames - Comma separated list of device names for filtering FilterOut - Boolean indicating if the data matching the device names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterByDeviceName] [Writable.Pipeline.Functions.FilterByDeviceName.Parameters] DeviceNames = \"Random-Float-Device,Random-Integer-Device\" FilterOut = \"false\"","title":"FilterByDeviceName"},{"location":"microservices/application/AppServiceConfigurable/#filterbyprofilename","text":"Parameters ProfileNames - Comma separated list of profile names for filtering FilterOut - Boolean indicating if the data matching the profile names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterByProfileName] [Writable.Pipeline.Functions.FilterByProfileName.Parameters] ProfileNames = \"Random-Float-Device, Random-Integer-Device\" FilterOut = \"false\" EdgeX 2.0 The FilterByProfileName configurable pipeline function is new for EdgeX 2.0","title":"FilterByProfileName"},{"location":"microservices/application/AppServiceConfigurable/#filterbyresourcename","text":"Parameters ResourceName - Comma separated list of reading resource names for filtering FilterOut - Boolean indicating if the readings matching the resource names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterByResourceName] [Writable.Pipeline.Functions.FilterByResourceName.Parameters] ResourceNames = \"Int8, Int64\" FilterOut = \"true\" EdgeX 2.0 For EdgeX 2.0 the FilterByValueDescriptor configurable pipeline function has been renamed to FilterByResourceName and parameter names adjusted.","title":"FilterByResourceName"},{"location":"microservices/application/AppServiceConfigurable/#filterbysourcename","text":"Parameters SourceNames - Comma separated list of source names for filtering. Source name is either the device command name or the resource name that created the Event FilterOut - Boolean indicating if the data matching the device names should be filtered out or filtered for. Example [Writable.Pipeline.Functions.FilterBySourceName] [Writable.Pipeline.Functions.FilterBySource.Parameters] SourceNames = \"Bool, BoolArray\" FilterOut = \"false\" EdgeX 2.0 The FilterBySourceName configurable pipeline function is new for EdgeX 2.0","title":"FilterBySourceName"},{"location":"microservices/application/AppServiceConfigurable/#httpexport","text":"Parameters Method - HTTP Method to use. Can be post or put Url - HTTP endpoint to POST/PUT the data. MimeType - Optional mime type for the data. Defaults to application/json if not set. PersistOnError - Indicates to persist the data if the POST fails. Store and Forward must also be enabled if this is set to \"true\". ContinueOnSendError - For chained multi destination exports, if true continues after send error so next export function executes. ReturnInputData - For chained multi destination exports if true, passes the input data to next export function. HeaderName - (Optional) Name of the header key to add to the HTTP header SecretPath - (Optional) Path of the secret in the Secret Store where the header value is stored. SecretName - (Optional) Name of the secret for the header value in the Secret Store . Example # Simple HTTP Export [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" # HTTP Export with secret header data pull from Secret Store [Writable.Pipeline.Functions.HTTPExport] [Writable.Pipeline.Functions.HTTPExport.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api.net/edgexdata\" HeaderName = \"MyApiKey\" SecretPath = \"http\" SecretName = \"apikey\" # Http Export to multiple destinations [Writable.Pipeline] ExecutionOrder = \"HTTPExport1, HTTPExport2\" [Writable.Pipeline.Functions.HTTPExport1] [Writable.Pipeline.Functions.HTTPExport1.Parameters] Method = \"post\" MimeType = \"application/xml\" Url = \"http://my.api1.net/edgexdata2\" ContinueOnSendError = \"true\" ReturnInputData = \"true\" [Writable.Pipeline.Functions.HTTPExport2] [Writable.Pipeline.Functions.HTTPExport2.Parameters] Method = \"put\" MimeType = \"application/xml\" Url = \"http://my.api2.net/edgexdata2\" EdgeX 2.0 For EdgeX 2.0 the HTTPPost , HTTPPostJSON , HTTPPostXML , HTTPPut , HTTPPutJSON , and HTTPPutXML configurable pipeline functions have been replaced by the single HTTPExport function with additional Method parameter. ContinueOnSendError and ReturnInputData parameter have been added to support multi destination exports. In addition the HeaderName and SecretName parameters have replaced the SecretHeaderName parameter. EdgeX 2.0 The capability to chain Http Export functions to export to multiple destinations is new for Edgex 2.0. EdgeX 2.0 Multiple instances (configured differently) of the same configurable pipeline function is new for EdgeX 2.0. The function names in the Writable.Pipeline.Functions section now only need to start with a built-in configurable pipeline function name, rather than be an exact match.","title":"HTTPExport"},{"location":"microservices/application/AppServiceConfigurable/#jsonlogic","text":"Parameters Rule - The JSON formatted rule that with be executed on the data by JSONLogic Example [Writable.Pipeline.Functions.JSONLogic] [Writable.Pipeline.Functions.JSONLogic.Parameters] Rule = \"{ \\\"and\\\" : [{\\\"<\\\" : [{ \\\"var\\\" : \\\"temp\\\" }, 110 ]}, {\\\"==\\\" : [{ \\\"var\\\" : \\\"sensor.type\\\" }, \\\"temperature\\\" ]} ] }\"","title":"JSONLogic"},{"location":"microservices/application/AppServiceConfigurable/#mqttexport","text":"Parameters BrokerAddress - URL specify the address of the MQTT Broker Topic - Topic to publish the data ClientId - Id to use when connecting to the MQTT Broker Qos - MQTT Quality of Service (QOS) setting to use (0, 1 or 2). Please refer here for more details on QOS values AutoReconnect - Boolean specifying if reconnect should be automatic if connection to MQTT broker is lost Retain - Boolean specifying if the MQTT Broker should save the last message published as the \u201cLast Good Message\u201d on that topic. SkipVerify - Boolean indicating if the certificate verification should be skipped. PersistOnError - Indicates to persist the data if the POST fails. Store and Forward must also be enabled if this is set to \"true\". AuthMode - Mode of authentication to use when connecting to the MQTT Broker none - No authentication required usernamepassword - Use username and password authentication. The Secret Store (Vault or InsecureSecrets ) must contain the username and password secrets. clientcert - Use Client Certificate authentication. The Secret Store (Vault or InsecureSecrets ) must contain the clientkey and clientcert secrets. cacert - Use CA Certificate authentication. The Secret Store (Vault or InsecureSecrets ) must contain the cacert secret. SecretPath - Path in the secret store where authentication secrets are stored. Note Authmode=cacert is only needed when client authentication (e.g. usernamepassword ) is not required, but a CA Cert is needed to validate the broker's SSL/TLS cert. Example # Simple MQTT Export [Writable.Pipeline.Functions.MQTTExport] [Writable.Pipeline.Functions.MQTTExport.Parameters] BrokerAddress = \"tcps://localhost:8883\" Topic = \"mytopic\" ClientId = \"myclientid\" # MQTT Export with auth credentials pull from the Secret Store [Writable.Pipeline.Functions.MQTTExport] [Writable.Pipeline.Functions.MQTTExport.Parameters] BrokerAddress = \"tcps://my-broker-host.com:8883\" Topic = \"mytopic\" ClientId = \"myclientid\" Qos = \"2\" AutoReconnect = \"true\" Retain = \"true\" SkipVerify = \"false\" PersistOnError = \"true\" AuthMode = \"usernamepassword\" SecretPath = \"mqtt\" EdgeX 2.0 For EdgeX 2.0 the MQTTSecretSend configurable pipeline function has been renamed to MQTTExport and the deprecated MQTTSend configurable pipeline function has been removed","title":"MQTTExport"},{"location":"microservices/application/AppServiceConfigurable/#pushtocore","text":"Parameters ProfileName - Profile name to use for the new Event DeviceName - Device name to use for the new Event ResourceName - Resource name name to use for the new Event's SourceName and Reading's ResourceName ValueType - Value type to use the new Event Reading's value type MediaType - Media type to use the new Event Reading's value type. Required when the value type is Binary Example [Writable.Pipeline.Functions.PushToCore] [Writable.Pipeline.Functions.PushToCore.Parameters] ProfileName = \"MyProfile\" DeviceName = \"MyDevice\" ResourceName = \"SomeResource\" ValueType = \"String\" EdgeX 2.0 For EdgeX 2.0 the ProfileName , ValueType and MediaType parameters are new and the ReadingName parameter has been renamed to ResourceName .","title":"PushToCore"},{"location":"microservices/application/AppServiceConfigurable/#setresponsedata","text":"Parameters ResponseContentType - Used to specify content-type header for response - optional Example [Writable.Pipeline.Functions.SetResponseData] [Writable.Pipeline.Functions.SetResponseData.Parameters] ResponseContentType = \"application/json\" EdgeX 2.0 For EdgeX 2.0 the SetOutputData configurable pipeline function has been renamed to SetResponseData .","title":"SetResponseData"},{"location":"microservices/application/AppServiceConfigurable/#transform","text":"Parameters Type - Type of transformation to perform. Can be 'xml' or 'json' Example [Writable.Pipeline.Functions.Transform] [Writable.Pipeline.Functions.Transform.Parameters] Type = \"xml\" EdgeX 2.0 For EdgeX 2.0 the TransformToJSON and TransformToXML configurable pipeline functions have been replaced by the single Transform configurable pipeline function with additional Type parameter.","title":"Transform"},{"location":"microservices/application/AppServiceConfigurable/#tolineprotocol","text":"EdgeX 2.2 ToLineProtocol is new for Edgex 2.2 Parameters Tags - optional comma separated list of additional tags to add to the metric in to form \"tag:value,...\" Example [Writable.Pipeline.Functions.ToLineProtocol] [Writable.Pipeline.Functions.ToLineProtocol.Parameters] Tags = \"\" # optional comma separated list of additional tags to add to the metric in to form \"tag:value,...\" Note The new UseTargetTypeOfMetric setting must be set to true when using this function. See the UseTargetTypeOfMetric section above for more details.","title":"ToLineProtocol"},{"location":"microservices/application/ApplicationFunctionsSDK/","text":"App Functions SDK Introduction Welcome the App Functions SDK for EdgeX. This SDK is meant to provide all the plumbing necessary for developers to get started in processing/transforming/exporting data out of EdgeX. If you're new to the SDK - checkout the Getting Started guide. If you're already familiar - checkout the various sections about the SDK: Section Description Application Service API Provides a list of all available APIs on the interface use to build Application Services App Function Context API Provides a list of all available APIs on the context interface that is available inside of a pipeline function Pipeline Function Error Handling Describes how to properly handle pipeline execution failures Built-In Pipeline Functions Provides a list of the available pipeline functions/transforms in the SDK Advanced Topics Learn about other ways to leverage the SDK beyond basic use cases The App Functions SDK implements a small REST API which can be seen Here .","title":"App Functions SDK Introduction"},{"location":"microservices/application/ApplicationFunctionsSDK/#app-functions-sdk-introduction","text":"Welcome the App Functions SDK for EdgeX. This SDK is meant to provide all the plumbing necessary for developers to get started in processing/transforming/exporting data out of EdgeX. If you're new to the SDK - checkout the Getting Started guide. If you're already familiar - checkout the various sections about the SDK: Section Description Application Service API Provides a list of all available APIs on the interface use to build Application Services App Function Context API Provides a list of all available APIs on the context interface that is available inside of a pipeline function Pipeline Function Error Handling Describes how to properly handle pipeline execution failures Built-In Pipeline Functions Provides a list of the available pipeline functions/transforms in the SDK Advanced Topics Learn about other ways to leverage the SDK beyond basic use cases The App Functions SDK implements a small REST API which can be seen Here .","title":"App Functions SDK Introduction"},{"location":"microservices/application/ApplicationServiceAPI/","text":"Application Service API The ApplicationService API is the central API for creating an EdgeX Application Service. EdgeX 2.0 For EdgeX 2.0 the ApplicationService API and factory functions replace direct access to the AppFunctionsSDK struct. The new ApplicationService API is as follows: type AppFunction = func ( appCxt AppFunctionContext , data interface {}) ( bool , interface {}) type FunctionPipeline struct { Id string Transforms [] AppFunction Topic string Hash string } type ApplicationService interface { ApplicationSettings () map [ string ] string GetAppSetting ( setting string ) ( string , error ) GetAppSettingStrings ( setting string ) ([] string , error ) LoadCustomConfig ( config UpdatableConfig , sectionName string ) error ListenForCustomConfigChanges ( configToWatch interface {}, sectionName string , changedCallback func ( interface {})) error SetFunctionsPipeline ( transforms ... AppFunction ) error *** DEPRECATED *** SetDefaultFunctionsPipeline ( transforms ... AppFunction ) error AddFunctionsPipelineByTopics ( id string , topics [] string , transforms ... AppFunction ) error LoadConfigurablePipeline () ([] AppFunction , error ) *** DEPRECATED by LoadConfigurableFunctionPipelines *** LoadConfigurableFunctionPipelines () ( map [ string ] FunctionPipeline , error ) MakeItRun () error MakeItStop () GetSecret ( path string , keys ... string ) ( map [ string ] string , error ) StoreSecret ( path string , secretData map [ string ] string ) error LoggingClient () logger . LoggingClient EventClient () interfaces . EventClient CommandClient () interfaces . CommandClient NotificationClient () interfaces . NotificationClient SubscriptionClient () interfaces . SubscriptionClient DeviceServiceClient () interfaces . DeviceServiceClient DeviceProfileClient () interfaces . DeviceProfileClient DeviceClient () interfaces . DeviceClient RegistryClient () registry . Client MetricsManager () bootstrapInterfaces . MetricsManager AddBackgroundPublisher ( capacity int ) ( BackgroundPublisher , error ) AddBackgroundPublisherWithTopic ( capacity int , topic string ) ( BackgroundPublisher , error ) BuildContext ( correlationId string , contentType string ) AppFunctionContext AddRoute ( route string , handler func ( http . ResponseWriter , * http . Request ), methods ... string ) error RequestTimeout () time . Duration RegisterCustomTriggerFactory ( name string , factory func ( TriggerConfig ) ( Trigger , error )) error RegisterCustomStoreFactory ( name string , factory func ( cfg DatabaseInfo , cred config . Credentials ) ( StoreClient , error )) error } Factory Functions The App Functions SDK provides two factory functions for creating an ApplicationService NewAppService NewAppService(serviceKey string) (interfaces.ApplicationService, bool) This factory function returns an interfaces.ApplicationService using the default Target Type of dtos.Event and initializes the service. The second bool return parameter will be true if successfully initialized, otherwise it will be false when error(s) occurred during initialization. All error(s) are logged so the caller just needs to call os.Exit(-1) if false is returned. Example - NewAppService const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppService ( serviceKey ) if ! ok { os . Exit ( - 1 ) } NewAppServiceWithTargetType NewAppServiceWithTargetType(serviceKey string, targetType interface{}) (interfaces.ApplicationService, bool) This factory function returns an interfaces.ApplicationService using the passed in Target Type and initializes the service. The second bool return parameter will be true if successfully initialized, otherwise it will be false when error(s) occurred during initialization. All error(s) are logged so the caller just needs to call os.Exit(-1) if false is returned. See the Target Type advanced topic for more details. Example - NewAppServiceWithTargetType const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppServiceWithTargetType ( serviceKey , & [] byte {}) if ! ok { os . Exit ( - 1 ) } Custom Configuration APIs The following ApplicationService APIs allow your service to access their custom configuration from the TOML file and/or Configuration Provider. See the Custom Configuration advanced topic for more details. ApplicationSettings ApplicationSettings() map[string]string This API returns the complete key/value map of custom settings Example - ApplicationSettings [ApplicationSettings] Greeting = \"Hello World\" appSettings := service . ApplicationSettings () greeting := appSettings [ \"Greeting\" ] service . LoggingClient . Info ( greeting ) GetAppSetting GetAppSetting(setting string) (string, error) This API is a convenience API that returns a single setting from the [ApplicationSetting] section of the service configuration. An error is returned if the specified setting is not found. Example - GetAppSetting [ApplicationSettings] Greeting = \"Hello World\" greeting , err := service . GetAppSetting [ \"Greeting\" ] if err != nil { ... } service . LoggingClient . Info ( greeting ) GetAppSettingStrings GetAppSettingStrings(setting string) ([]string, error) This API is a convenience API that parses the string value for the specified custom application setting as a comma separated list. It returns the list of strings. An error is returned if the specified setting is not found. Example - GetAppSettingStrings [ApplicationSettings] Greetings = \"Hello World, Welcome World, Hi World\" greetings , err := service . GetAppSettingStrings [ \"Greetings\" ] if err != nil { ... } for _ , greeting := range greetings { service . LoggingClient . Info ( greeting ) } LoadCustomConfig LoadCustomConfig(config UpdatableConfig, sectionName string) error This API loads the service's Structured Custom Configuration from local file or the Configuration Provider (if enabled). The Configuration Provider will also be seeded with the custom configuration if service is using the Configuration Provider. The UpdateFromRaw API ( UpdatableConfig interface) will be called on the custom configuration when the configuration is loaded from the Configuration Provider. The custom config must implement the UpdatableConfig interface. Example - LoadCustomConfig [ AppCustom ] # Can be any name you choose ResourceNames = \"Boolean, Int32, Uint32, Float32, Binary\" SomeValue = 123 [AppCustom.SomeService] Host = \"localhost\" Port = 9080 Protocol = \"http\" type ServiceConfig struct { AppCustom AppCustomConfig } type AppCustomConfig struct { ResourceNames string SomeValue int SomeService HostInfo } func ( c * ServiceConfig ) UpdateFromRaw ( rawConfig interface {}) bool { configuration , ok := rawConfig .( * ServiceConfig ) if ! ok { return false //errors.New(\"unable to cast raw config to type 'ServiceConfig'\") } * c = * configuration return true } ... serviceConfig := & ServiceConfig {} err := service . LoadCustomConfig ( serviceConfig , \"AppCustom\" ) if err != nil { ... } See the App Service Template for a complete example of using Structured Custom Configuration ListenForCustomConfigChanges ListenForCustomConfigChanges(configToWatch interface{}, sectionName string, changedCallback func(interface{})) error This API starts a listener on the Configuration Provider for changes to the specified section of the custom configuration. When changes are received from the Configuration Provider the provided changedCallback function is called with the updated section of configuration. The service must then implement the code to copy the updates into it's copy of the configuration and respond to the updates if needed. Example - ListenForCustomConfigChanges [ AppCustom ] # Can be any name you choose ResourceNames = \"Boolean, Int32, Uint32, Float32, Binary\" SomeValue = 123 [AppCustom.SomeService] Host = \"localhost\" Port = 9080 Protocol = \"http\" ... err := service . ListenForCustomConfigChanges ( & serviceConfig . AppCustom , \"AppCustom\" , ProcessConfigUpdates ) if err != nil { logger . Errorf ( \"unable to watch custom writable configuration: %s\" , err . Error ()) } ... func ( app * myApp ) ProcessConfigUpdates ( rawWritableConfig interface {}) { updated , ok := rawWritableConfig .( * config . AppCustomConfig ) if ! ok { ... return } previous := app . serviceConfig . AppCustom app . serviceConfig . AppCustom = * updated if reflect . DeepEqual ( previous , updated ) { logger . Info ( \"No changes detected\" ) return } if previous . SomeValue != updated . SomeValue { logger . Infof ( \"AppCustom.SomeValue changed to: %d\" , updated . SomeValue ) } if previous . ResourceNames != updated . ResourceNames { logger . Infof ( \"AppCustom.ResourceNames changed to: %s\" , updated . ResourceNames ) } if ! reflect . DeepEqual ( previous . SomeService , updated . SomeService ) { logger . Infof ( \"AppCustom.SomeService changed to: %v\" , updated . SomeService ) } } See the App Service Template for a complete example of using Structured Custom Configuration Function Pipeline APIs The following ApplicationService APIs allow your service to set the Functions Pipeline and start and stop the Functions Pipeline. AppFunction type AppFunction = func(appCxt AppFunctionContext, data interface{}) (bool, interface{}) This type defines the signature that all pipeline functions must implement. FunctionPipeline This type defines the struct that contains the metadata for a functions pipeline instance. type FunctionPipeline struct { Id string Transforms [] AppFunction Topic string Hash string } SetFunctionsPipeline SetFunctionsPipeline(transforms ...AppFunction) error This API has been deprecated (Replaced by SetDefaultFunctionsPipeline) and will be removed in a future release. Functions the same as SetDefaultFunctionsPipeline. SetDefaultFunctionsPipeline SetDefaultFunctionsPipeline(transforms ...AppFunction) error This API sets the default functions pipeline with the specified list of Application Functions. This pipeline is executed for all messages received from the configured trigger. Note that the functions are executed in the order provided in the list. An error is returned if the list is empty. Example - SetDefaultFunctionsPipeline sample := functions . NewSample () err = service . SetDefaultFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { app . lc . Errorf ( \"SetDefaultFunctionsPipeline returned error: %s\" , err . Error ()) return - 1 } AddFunctionsPipelineForTopics AddFunctionsPipelineForTopics(id string, topics []string, transforms ...AppFunction) error This API adds a functions pipeline with the specified unique ID and list of functions (transforms) to be executed when the received topic matches one of the specified pipeline topics. See the Pipeline Per Topic section for more details. Example - AddFunctionsPipelineForTopics sample := functions . NewSample () err = service . AddFunctionsPipelineForTopic ( \"Floats-Pipeline\" , [] string { \"edgex/events/#/#/Random-Float-Device/#\" }, transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { ... return - 1 } LoadConfigurablePipeline LoadConfigurablePipeline() ([]AppFunction, error) This API loads the default function pipeline from configuration. An error is returned if the configuration is not valid, i.e. missing required function parameters, invalid function name, etc. Warning This API is Deprecated , has been replaced by LoadConfigurableFunctionPipelines below and will be removed in a future release. LoadConfigurableFunctionPipelines LoadConfigurableFunctionPipelines() (map[string]FunctionPipeline, error) This API loads the function pipelines (default and per topic) from configuration. An error is returned if the configuration is not valid, i.e. missing required function parameters, invalid function name, etc. Note This API is only useful if pipeline is always defined in configuration as is with App Service Configurable. Example - LoadConfigurableFunctionPipelines configuredPipelines , err := service . LoadConfigurableFunctionPipelines () if err != nil { ... os . Exit ( - 1 ) } ... for _ , pipeline := range configuredPipelines { switch pipeline . Id { case interfaces . DefaultPipelineId : if err = service . SetFunctionsPipeline ( pipeline . Transforms ... ); err != nil { ... os . Exit ( - 1 ) } default : if err = service . AddFunctionsPipelineForTopic ( pipeline . Id , pipeline . Topic , pipeline . Transforms ... ); err != nil { ... os . Exit ( - 1 ) } } } MakeItRun MakeItRun() error This API starts the configured trigger to allow the Functions Pipeline to execute when the trigger receives data. The internal webserver is also started. This is a long running API which does not return until the service is stopped or MakeItStop() is called. An error is returned if the trigger can not be create or initialized or if the internal webserver encounters an error. Example - MakeItRun if err := service . MakeItRun (); err != nil { logger . Errorf ( \"MakeItRun returned error: %s\" , err . Error ()) os . exit ( - 1 ) } // Do any required cleanup here, if needed os . exit ( 0 ) MakeItStop MakeItStop() This API stops the configured trigger so that the functions pipeline no longer executes. The internal webserver continues to accept requests. See Stopping the Service advanced topic for more details Example - MakeItStop service . MakeItStop () ... Secrets APIs The following ApplicationService APIs allow your service retrieve and store secrets from/to the service's SecretStore. See the Secrets advanced topic for more details about using secrets. GetSecret GetSecret(path string, keys ...string) (map[string]string, error) This API returns the secret data from the secret store (secure or insecure) for the specified path. An error is returned if the path is not found or any of the keys (if specified) are not found. Omit keys if all secret data for the specified path is required. Example - GetSecret secretData , err := service . GetSecret ( \"mqtt\" ) if err != nil { ... } username := secretData [ \"user\" ] password := secretData [ \"password\" ] ... StoreSecret StoreSecret(path string, secretData map[string]string) error This API stores the specified secret data into the secret store (secure mode only) for the specified path An error is returned if: Specified secret data is empty Not using the secure secret store, i.e. not valid with InsecureSecrets configuration Secure secret provider is not properly initialized Connection issues with Secret Store service. Note Typically Application Services only needs to retrieve secrets via the code. The /secret REST API is used to seed secrets into the service's SecretStore. Example - StoreSecret secretData := generateMqttCredentials () err := service . StoreSecret ( \"mqtt\" , secretData ) if err != nil { ... } ... Client APIs The following ApplicationService APIs allow your service access the various EdgeX clients and their APIs. LoggingClient LoggingClient() logger.LoggingClient This API returns the LoggingClient instance which the service uses to log messages. See the LoggingClient interface for more details. Example - LoggingClient service . LoggingClient (). Info ( \"Hello World\" ) service . LoggingClient (). Errorf ( \"Some error occurred: %w\" , err ) RegistryClient RegistryClient() registry.Client This API returns the Registry Client. Note the registry must been enabled, otherwise this will return nil. See the Registry Client interface for more details. Useful if service needs to add additional health checks or needs to get endpoint of another registered service. EventClient EventClient() interfaces.EventClient This API returns the Event Client. Note if Core Data is not specified in the Clients configuration, this will return nil. See the Event Client interface for more details. Useful for adding, deleting or querying Events. CommandClient CommandClient() interfaces.CommandClient This API returns the Command Client. Note if Support Command is not specified in the Clients configuration, this will return nil. See the Command Client interface for more details. Useful for issuing commands to devices. NotificationClient NotificationClient() interfaces.NotificationClient This API returns the Notification Client. Note if Support Notifications is not specified in the Clients configuration, this will return nil. See the Notification Client interface for more details. Useful for sending notifications. SubscriptionClient SubscriptionClient() interfaces.SubscriptionClient This API returns the Subscription client. Note if Support Notifications is not specified in the Clients configuration, this will return nil. See the Subscription Client interface for more details. Useful for creating notification subscriptions. DeviceServiceClient DeviceServiceClient() interfaces.DeviceServiceClient This API returns the Device Service Client. Note if Core Metadata is not specified in the Clients configuration, this will return nil. See the Device Service Client interface for more details. Useful for querying information about a Device Service. DeviceProfileClient DeviceProfileClient() interfaces.DeviceProfileClient This API returns the Device Profile Client. Note if Core Metadata is not specified in the Clients configuration, this will return nil. See the Device Profile Client interface for more details. Useful for querying information about a Device Profile such as Device Resource details. DeviceClient DeviceClient() interfaces.DeviceClient This API returns the Device Client. Note if Core Metadata is not specified in the Clients configuration, this will return nil. See the Device Client interface for more details. Useful for querying list of devices for a specific Device Service or Device Profile. Background Publisher APIs The following ApplicationService APIs allow Application Services to have background publishers. See the Background Publishing advanced topic for more details and example. AddBackgroundPublisher AddBackgroundPublisher(capacity int) (BackgroundPublisher, error) This API adds and returns a BackgroundPublisher which is used to publish asynchronously to the Edgex MessageBus. Not valid for use with the HTTP or External MQTT triggers AddBackgroundPublisherWithTopic AddBackgroundPublisherWithTopic(capacity int, topic string) (BackgroundPublisher, error) This API adds and returns a BackgroundPublisher which is used to publish asynchronously to the Edgex MessageBus on the specified topic. Not valid for use with the HTTP or External MQTT triggers. BuildContext BuildContext(correlationId string, contentType string) AppFunctionContext This API allows external callers that may need a context (eg background publishers) to easily create one. Other APIs AddRoute AddRoute(route string, handler func(http.ResponseWriter, *http.Request), methods ...string) error This API adds a custom REST route to the application service's internal webserver. A reference to the ApplicationService is add the the context that is passed to the handler, which can be retrieved using the AppService key. See Custom REST Endpoints advanced topic for more details and example. RequestTimeout RequestTimeout() time.Duration This API returns the parsed value for the Service.RequestTimeout configuration setting. The setting is parsed on start-up so that any error is caught then. Example - RequestTimeout [Service] : RequestTimeout = \"60s\" : timeout := service . RequestTimeout () RegisterCustomTriggerFactory RegisterCustomTriggerFactory(name string, factory func(TriggerConfig) (Trigger, error)) error This API registers a trigger factory for a custom trigger to be used. See the Custom Triggers section for more details and example. RegisterCustomStoreFactory RegisterCustomStoreFactory(name string, factory func(cfg DatabaseInfo, cred config.Credentials) (StoreClient, error)) error This API registers a factory to construct a custom store client for the store & forward loop. MetricsManager MetricsManager() bootstrapInterfaces.MetricsManager This API returns the Metrics Manager used to register counter, gauge, gaugeFloat64 or timer metric types from github.com/rcrowley/go-metrics myCounterMetricName := \"MyCounter\" myCounter := gometrics . NewCounter () myTags := map [ string ] string { \"Tag1\" : \"Value1\" } app . service . MetricsManager (). Register ( myCounterMetricName , myCounter , myTags )","title":"Application Service API"},{"location":"microservices/application/ApplicationServiceAPI/#application-service-api","text":"The ApplicationService API is the central API for creating an EdgeX Application Service. EdgeX 2.0 For EdgeX 2.0 the ApplicationService API and factory functions replace direct access to the AppFunctionsSDK struct. The new ApplicationService API is as follows: type AppFunction = func ( appCxt AppFunctionContext , data interface {}) ( bool , interface {}) type FunctionPipeline struct { Id string Transforms [] AppFunction Topic string Hash string } type ApplicationService interface { ApplicationSettings () map [ string ] string GetAppSetting ( setting string ) ( string , error ) GetAppSettingStrings ( setting string ) ([] string , error ) LoadCustomConfig ( config UpdatableConfig , sectionName string ) error ListenForCustomConfigChanges ( configToWatch interface {}, sectionName string , changedCallback func ( interface {})) error SetFunctionsPipeline ( transforms ... AppFunction ) error *** DEPRECATED *** SetDefaultFunctionsPipeline ( transforms ... AppFunction ) error AddFunctionsPipelineByTopics ( id string , topics [] string , transforms ... AppFunction ) error LoadConfigurablePipeline () ([] AppFunction , error ) *** DEPRECATED by LoadConfigurableFunctionPipelines *** LoadConfigurableFunctionPipelines () ( map [ string ] FunctionPipeline , error ) MakeItRun () error MakeItStop () GetSecret ( path string , keys ... string ) ( map [ string ] string , error ) StoreSecret ( path string , secretData map [ string ] string ) error LoggingClient () logger . LoggingClient EventClient () interfaces . EventClient CommandClient () interfaces . CommandClient NotificationClient () interfaces . NotificationClient SubscriptionClient () interfaces . SubscriptionClient DeviceServiceClient () interfaces . DeviceServiceClient DeviceProfileClient () interfaces . DeviceProfileClient DeviceClient () interfaces . DeviceClient RegistryClient () registry . Client MetricsManager () bootstrapInterfaces . MetricsManager AddBackgroundPublisher ( capacity int ) ( BackgroundPublisher , error ) AddBackgroundPublisherWithTopic ( capacity int , topic string ) ( BackgroundPublisher , error ) BuildContext ( correlationId string , contentType string ) AppFunctionContext AddRoute ( route string , handler func ( http . ResponseWriter , * http . Request ), methods ... string ) error RequestTimeout () time . Duration RegisterCustomTriggerFactory ( name string , factory func ( TriggerConfig ) ( Trigger , error )) error RegisterCustomStoreFactory ( name string , factory func ( cfg DatabaseInfo , cred config . Credentials ) ( StoreClient , error )) error }","title":"Application Service API"},{"location":"microservices/application/ApplicationServiceAPI/#factory-functions","text":"The App Functions SDK provides two factory functions for creating an ApplicationService","title":"Factory Functions"},{"location":"microservices/application/ApplicationServiceAPI/#newappservice","text":"NewAppService(serviceKey string) (interfaces.ApplicationService, bool) This factory function returns an interfaces.ApplicationService using the default Target Type of dtos.Event and initializes the service. The second bool return parameter will be true if successfully initialized, otherwise it will be false when error(s) occurred during initialization. All error(s) are logged so the caller just needs to call os.Exit(-1) if false is returned. Example - NewAppService const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppService ( serviceKey ) if ! ok { os . Exit ( - 1 ) }","title":"NewAppService"},{"location":"microservices/application/ApplicationServiceAPI/#newappservicewithtargettype","text":"NewAppServiceWithTargetType(serviceKey string, targetType interface{}) (interfaces.ApplicationService, bool) This factory function returns an interfaces.ApplicationService using the passed in Target Type and initializes the service. The second bool return parameter will be true if successfully initialized, otherwise it will be false when error(s) occurred during initialization. All error(s) are logged so the caller just needs to call os.Exit(-1) if false is returned. See the Target Type advanced topic for more details. Example - NewAppServiceWithTargetType const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppServiceWithTargetType ( serviceKey , & [] byte {}) if ! ok { os . Exit ( - 1 ) }","title":"NewAppServiceWithTargetType"},{"location":"microservices/application/ApplicationServiceAPI/#custom-configuration-apis","text":"The following ApplicationService APIs allow your service to access their custom configuration from the TOML file and/or Configuration Provider. See the Custom Configuration advanced topic for more details.","title":"Custom Configuration APIs"},{"location":"microservices/application/ApplicationServiceAPI/#applicationsettings","text":"ApplicationSettings() map[string]string This API returns the complete key/value map of custom settings Example - ApplicationSettings [ApplicationSettings] Greeting = \"Hello World\" appSettings := service . ApplicationSettings () greeting := appSettings [ \"Greeting\" ] service . LoggingClient . Info ( greeting )","title":"ApplicationSettings"},{"location":"microservices/application/ApplicationServiceAPI/#getappsetting","text":"GetAppSetting(setting string) (string, error) This API is a convenience API that returns a single setting from the [ApplicationSetting] section of the service configuration. An error is returned if the specified setting is not found. Example - GetAppSetting [ApplicationSettings] Greeting = \"Hello World\" greeting , err := service . GetAppSetting [ \"Greeting\" ] if err != nil { ... } service . LoggingClient . Info ( greeting )","title":"GetAppSetting"},{"location":"microservices/application/ApplicationServiceAPI/#getappsettingstrings","text":"GetAppSettingStrings(setting string) ([]string, error) This API is a convenience API that parses the string value for the specified custom application setting as a comma separated list. It returns the list of strings. An error is returned if the specified setting is not found. Example - GetAppSettingStrings [ApplicationSettings] Greetings = \"Hello World, Welcome World, Hi World\" greetings , err := service . GetAppSettingStrings [ \"Greetings\" ] if err != nil { ... } for _ , greeting := range greetings { service . LoggingClient . Info ( greeting ) }","title":"GetAppSettingStrings"},{"location":"microservices/application/ApplicationServiceAPI/#loadcustomconfig","text":"LoadCustomConfig(config UpdatableConfig, sectionName string) error This API loads the service's Structured Custom Configuration from local file or the Configuration Provider (if enabled). The Configuration Provider will also be seeded with the custom configuration if service is using the Configuration Provider. The UpdateFromRaw API ( UpdatableConfig interface) will be called on the custom configuration when the configuration is loaded from the Configuration Provider. The custom config must implement the UpdatableConfig interface. Example - LoadCustomConfig [ AppCustom ] # Can be any name you choose ResourceNames = \"Boolean, Int32, Uint32, Float32, Binary\" SomeValue = 123 [AppCustom.SomeService] Host = \"localhost\" Port = 9080 Protocol = \"http\" type ServiceConfig struct { AppCustom AppCustomConfig } type AppCustomConfig struct { ResourceNames string SomeValue int SomeService HostInfo } func ( c * ServiceConfig ) UpdateFromRaw ( rawConfig interface {}) bool { configuration , ok := rawConfig .( * ServiceConfig ) if ! ok { return false //errors.New(\"unable to cast raw config to type 'ServiceConfig'\") } * c = * configuration return true } ... serviceConfig := & ServiceConfig {} err := service . LoadCustomConfig ( serviceConfig , \"AppCustom\" ) if err != nil { ... } See the App Service Template for a complete example of using Structured Custom Configuration","title":"LoadCustomConfig"},{"location":"microservices/application/ApplicationServiceAPI/#listenforcustomconfigchanges","text":"ListenForCustomConfigChanges(configToWatch interface{}, sectionName string, changedCallback func(interface{})) error This API starts a listener on the Configuration Provider for changes to the specified section of the custom configuration. When changes are received from the Configuration Provider the provided changedCallback function is called with the updated section of configuration. The service must then implement the code to copy the updates into it's copy of the configuration and respond to the updates if needed. Example - ListenForCustomConfigChanges [ AppCustom ] # Can be any name you choose ResourceNames = \"Boolean, Int32, Uint32, Float32, Binary\" SomeValue = 123 [AppCustom.SomeService] Host = \"localhost\" Port = 9080 Protocol = \"http\" ... err := service . ListenForCustomConfigChanges ( & serviceConfig . AppCustom , \"AppCustom\" , ProcessConfigUpdates ) if err != nil { logger . Errorf ( \"unable to watch custom writable configuration: %s\" , err . Error ()) } ... func ( app * myApp ) ProcessConfigUpdates ( rawWritableConfig interface {}) { updated , ok := rawWritableConfig .( * config . AppCustomConfig ) if ! ok { ... return } previous := app . serviceConfig . AppCustom app . serviceConfig . AppCustom = * updated if reflect . DeepEqual ( previous , updated ) { logger . Info ( \"No changes detected\" ) return } if previous . SomeValue != updated . SomeValue { logger . Infof ( \"AppCustom.SomeValue changed to: %d\" , updated . SomeValue ) } if previous . ResourceNames != updated . ResourceNames { logger . Infof ( \"AppCustom.ResourceNames changed to: %s\" , updated . ResourceNames ) } if ! reflect . DeepEqual ( previous . SomeService , updated . SomeService ) { logger . Infof ( \"AppCustom.SomeService changed to: %v\" , updated . SomeService ) } } See the App Service Template for a complete example of using Structured Custom Configuration","title":"ListenForCustomConfigChanges"},{"location":"microservices/application/ApplicationServiceAPI/#function-pipeline-apis","text":"The following ApplicationService APIs allow your service to set the Functions Pipeline and start and stop the Functions Pipeline.","title":"Function Pipeline APIs"},{"location":"microservices/application/ApplicationServiceAPI/#appfunction","text":"type AppFunction = func(appCxt AppFunctionContext, data interface{}) (bool, interface{}) This type defines the signature that all pipeline functions must implement.","title":"AppFunction"},{"location":"microservices/application/ApplicationServiceAPI/#functionpipeline","text":"This type defines the struct that contains the metadata for a functions pipeline instance. type FunctionPipeline struct { Id string Transforms [] AppFunction Topic string Hash string }","title":"FunctionPipeline"},{"location":"microservices/application/ApplicationServiceAPI/#setfunctionspipeline","text":"SetFunctionsPipeline(transforms ...AppFunction) error This API has been deprecated (Replaced by SetDefaultFunctionsPipeline) and will be removed in a future release. Functions the same as SetDefaultFunctionsPipeline.","title":"SetFunctionsPipeline"},{"location":"microservices/application/ApplicationServiceAPI/#setdefaultfunctionspipeline","text":"SetDefaultFunctionsPipeline(transforms ...AppFunction) error This API sets the default functions pipeline with the specified list of Application Functions. This pipeline is executed for all messages received from the configured trigger. Note that the functions are executed in the order provided in the list. An error is returned if the list is empty. Example - SetDefaultFunctionsPipeline sample := functions . NewSample () err = service . SetDefaultFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { app . lc . Errorf ( \"SetDefaultFunctionsPipeline returned error: %s\" , err . Error ()) return - 1 }","title":"SetDefaultFunctionsPipeline"},{"location":"microservices/application/ApplicationServiceAPI/#addfunctionspipelinefortopics","text":"AddFunctionsPipelineForTopics(id string, topics []string, transforms ...AppFunction) error This API adds a functions pipeline with the specified unique ID and list of functions (transforms) to be executed when the received topic matches one of the specified pipeline topics. See the Pipeline Per Topic section for more details. Example - AddFunctionsPipelineForTopics sample := functions . NewSample () err = service . AddFunctionsPipelineForTopic ( \"Floats-Pipeline\" , [] string { \"edgex/events/#/#/Random-Float-Device/#\" }, transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , sample . LogEventDetails , sample . ConvertEventToXML , sample . OutputXML ) if err != nil { ... return - 1 }","title":"AddFunctionsPipelineForTopics"},{"location":"microservices/application/ApplicationServiceAPI/#loadconfigurablepipeline","text":"LoadConfigurablePipeline() ([]AppFunction, error) This API loads the default function pipeline from configuration. An error is returned if the configuration is not valid, i.e. missing required function parameters, invalid function name, etc. Warning This API is Deprecated , has been replaced by LoadConfigurableFunctionPipelines below and will be removed in a future release.","title":"LoadConfigurablePipeline"},{"location":"microservices/application/ApplicationServiceAPI/#loadconfigurablefunctionpipelines","text":"LoadConfigurableFunctionPipelines() (map[string]FunctionPipeline, error) This API loads the function pipelines (default and per topic) from configuration. An error is returned if the configuration is not valid, i.e. missing required function parameters, invalid function name, etc. Note This API is only useful if pipeline is always defined in configuration as is with App Service Configurable. Example - LoadConfigurableFunctionPipelines configuredPipelines , err := service . LoadConfigurableFunctionPipelines () if err != nil { ... os . Exit ( - 1 ) } ... for _ , pipeline := range configuredPipelines { switch pipeline . Id { case interfaces . DefaultPipelineId : if err = service . SetFunctionsPipeline ( pipeline . Transforms ... ); err != nil { ... os . Exit ( - 1 ) } default : if err = service . AddFunctionsPipelineForTopic ( pipeline . Id , pipeline . Topic , pipeline . Transforms ... ); err != nil { ... os . Exit ( - 1 ) } } }","title":"LoadConfigurableFunctionPipelines"},{"location":"microservices/application/ApplicationServiceAPI/#makeitrun","text":"MakeItRun() error This API starts the configured trigger to allow the Functions Pipeline to execute when the trigger receives data. The internal webserver is also started. This is a long running API which does not return until the service is stopped or MakeItStop() is called. An error is returned if the trigger can not be create or initialized or if the internal webserver encounters an error. Example - MakeItRun if err := service . MakeItRun (); err != nil { logger . Errorf ( \"MakeItRun returned error: %s\" , err . Error ()) os . exit ( - 1 ) } // Do any required cleanup here, if needed os . exit ( 0 )","title":"MakeItRun"},{"location":"microservices/application/ApplicationServiceAPI/#makeitstop","text":"MakeItStop() This API stops the configured trigger so that the functions pipeline no longer executes. The internal webserver continues to accept requests. See Stopping the Service advanced topic for more details Example - MakeItStop service . MakeItStop () ...","title":"MakeItStop"},{"location":"microservices/application/ApplicationServiceAPI/#secrets-apis","text":"The following ApplicationService APIs allow your service retrieve and store secrets from/to the service's SecretStore. See the Secrets advanced topic for more details about using secrets.","title":"Secrets APIs"},{"location":"microservices/application/ApplicationServiceAPI/#getsecret","text":"GetSecret(path string, keys ...string) (map[string]string, error) This API returns the secret data from the secret store (secure or insecure) for the specified path. An error is returned if the path is not found or any of the keys (if specified) are not found. Omit keys if all secret data for the specified path is required. Example - GetSecret secretData , err := service . GetSecret ( \"mqtt\" ) if err != nil { ... } username := secretData [ \"user\" ] password := secretData [ \"password\" ] ...","title":"GetSecret"},{"location":"microservices/application/ApplicationServiceAPI/#storesecret","text":"StoreSecret(path string, secretData map[string]string) error This API stores the specified secret data into the secret store (secure mode only) for the specified path An error is returned if: Specified secret data is empty Not using the secure secret store, i.e. not valid with InsecureSecrets configuration Secure secret provider is not properly initialized Connection issues with Secret Store service. Note Typically Application Services only needs to retrieve secrets via the code. The /secret REST API is used to seed secrets into the service's SecretStore. Example - StoreSecret secretData := generateMqttCredentials () err := service . StoreSecret ( \"mqtt\" , secretData ) if err != nil { ... } ...","title":"StoreSecret"},{"location":"microservices/application/ApplicationServiceAPI/#client-apis","text":"The following ApplicationService APIs allow your service access the various EdgeX clients and their APIs.","title":"Client APIs"},{"location":"microservices/application/ApplicationServiceAPI/#loggingclient","text":"LoggingClient() logger.LoggingClient This API returns the LoggingClient instance which the service uses to log messages. See the LoggingClient interface for more details. Example - LoggingClient service . LoggingClient (). Info ( \"Hello World\" ) service . LoggingClient (). Errorf ( \"Some error occurred: %w\" , err )","title":"LoggingClient"},{"location":"microservices/application/ApplicationServiceAPI/#registryclient","text":"RegistryClient() registry.Client This API returns the Registry Client. Note the registry must been enabled, otherwise this will return nil. See the Registry Client interface for more details. Useful if service needs to add additional health checks or needs to get endpoint of another registered service.","title":"RegistryClient"},{"location":"microservices/application/ApplicationServiceAPI/#eventclient","text":"EventClient() interfaces.EventClient This API returns the Event Client. Note if Core Data is not specified in the Clients configuration, this will return nil. See the Event Client interface for more details. Useful for adding, deleting or querying Events.","title":"EventClient"},{"location":"microservices/application/ApplicationServiceAPI/#commandclient","text":"CommandClient() interfaces.CommandClient This API returns the Command Client. Note if Support Command is not specified in the Clients configuration, this will return nil. See the Command Client interface for more details. Useful for issuing commands to devices.","title":"CommandClient"},{"location":"microservices/application/ApplicationServiceAPI/#notificationclient","text":"NotificationClient() interfaces.NotificationClient This API returns the Notification Client. Note if Support Notifications is not specified in the Clients configuration, this will return nil. See the Notification Client interface for more details. Useful for sending notifications.","title":"NotificationClient"},{"location":"microservices/application/ApplicationServiceAPI/#subscriptionclient","text":"SubscriptionClient() interfaces.SubscriptionClient This API returns the Subscription client. Note if Support Notifications is not specified in the Clients configuration, this will return nil. See the Subscription Client interface for more details. Useful for creating notification subscriptions.","title":"SubscriptionClient"},{"location":"microservices/application/ApplicationServiceAPI/#deviceserviceclient","text":"DeviceServiceClient() interfaces.DeviceServiceClient This API returns the Device Service Client. Note if Core Metadata is not specified in the Clients configuration, this will return nil. See the Device Service Client interface for more details. Useful for querying information about a Device Service.","title":"DeviceServiceClient"},{"location":"microservices/application/ApplicationServiceAPI/#deviceprofileclient","text":"DeviceProfileClient() interfaces.DeviceProfileClient This API returns the Device Profile Client. Note if Core Metadata is not specified in the Clients configuration, this will return nil. See the Device Profile Client interface for more details. Useful for querying information about a Device Profile such as Device Resource details.","title":"DeviceProfileClient"},{"location":"microservices/application/ApplicationServiceAPI/#deviceclient","text":"DeviceClient() interfaces.DeviceClient This API returns the Device Client. Note if Core Metadata is not specified in the Clients configuration, this will return nil. See the Device Client interface for more details. Useful for querying list of devices for a specific Device Service or Device Profile.","title":"DeviceClient"},{"location":"microservices/application/ApplicationServiceAPI/#background-publisher-apis","text":"The following ApplicationService APIs allow Application Services to have background publishers. See the Background Publishing advanced topic for more details and example.","title":"Background Publisher APIs"},{"location":"microservices/application/ApplicationServiceAPI/#addbackgroundpublisher","text":"AddBackgroundPublisher(capacity int) (BackgroundPublisher, error) This API adds and returns a BackgroundPublisher which is used to publish asynchronously to the Edgex MessageBus. Not valid for use with the HTTP or External MQTT triggers","title":"AddBackgroundPublisher"},{"location":"microservices/application/ApplicationServiceAPI/#addbackgroundpublisherwithtopic","text":"AddBackgroundPublisherWithTopic(capacity int, topic string) (BackgroundPublisher, error) This API adds and returns a BackgroundPublisher which is used to publish asynchronously to the Edgex MessageBus on the specified topic. Not valid for use with the HTTP or External MQTT triggers.","title":"AddBackgroundPublisherWithTopic"},{"location":"microservices/application/ApplicationServiceAPI/#buildcontext","text":"BuildContext(correlationId string, contentType string) AppFunctionContext This API allows external callers that may need a context (eg background publishers) to easily create one.","title":"BuildContext"},{"location":"microservices/application/ApplicationServiceAPI/#other-apis","text":"","title":"Other APIs"},{"location":"microservices/application/ApplicationServiceAPI/#addroute","text":"AddRoute(route string, handler func(http.ResponseWriter, *http.Request), methods ...string) error This API adds a custom REST route to the application service's internal webserver. A reference to the ApplicationService is add the the context that is passed to the handler, which can be retrieved using the AppService key. See Custom REST Endpoints advanced topic for more details and example.","title":"AddRoute"},{"location":"microservices/application/ApplicationServiceAPI/#requesttimeout","text":"RequestTimeout() time.Duration This API returns the parsed value for the Service.RequestTimeout configuration setting. The setting is parsed on start-up so that any error is caught then. Example - RequestTimeout [Service] : RequestTimeout = \"60s\" : timeout := service . RequestTimeout ()","title":"RequestTimeout"},{"location":"microservices/application/ApplicationServiceAPI/#registercustomtriggerfactory","text":"RegisterCustomTriggerFactory(name string, factory func(TriggerConfig) (Trigger, error)) error This API registers a trigger factory for a custom trigger to be used. See the Custom Triggers section for more details and example.","title":"RegisterCustomTriggerFactory"},{"location":"microservices/application/ApplicationServiceAPI/#registercustomstorefactory","text":"RegisterCustomStoreFactory(name string, factory func(cfg DatabaseInfo, cred config.Credentials) (StoreClient, error)) error This API registers a factory to construct a custom store client for the store & forward loop.","title":"RegisterCustomStoreFactory"},{"location":"microservices/application/ApplicationServiceAPI/#metricsmanager","text":"MetricsManager() bootstrapInterfaces.MetricsManager This API returns the Metrics Manager used to register counter, gauge, gaugeFloat64 or timer metric types from github.com/rcrowley/go-metrics myCounterMetricName := \"MyCounter\" myCounter := gometrics . NewCounter () myTags := map [ string ] string { \"Tag1\" : \"Value1\" } app . service . MetricsManager (). Register ( myCounterMetricName , myCounter , myTags )","title":"MetricsManager"},{"location":"microservices/application/ApplicationServices/","text":"Application Services Application Services are a means to get data from EdgeX Foundry to be processed at the edge and/or sent to external systems (be it analytics package, enterprise or on-prem application, cloud systems like Azure IoT, AWS IoT, or Google IoT Core, etc.). Application Services provide the means for data to be prepared (transformed, enriched, filtered, etc.) and groomed (formatted, compressed, encrypted, etc.) before being sent to an endpoint of choice or published back to other Application Service to consume. The export endpoints supported out of the box today include HTTP and MQTT endpoints, but custom endpoints can be implemented along side the existing functionality. Application Services are based on the idea of a \"Functions Pipeline\". A functions pipeline is a collection of functions that process messages (in this case EdgeX event/reading messages) in the order that you've specified. Triggers seed the first function in the pipeline with the data received by the Application Service. A trigger is something like a message landing in a watched message queue. The most commonly used Trigger is the MessageBus Trigger. See the Triggers section for more details An Applications Functions Software Development Kit (or App Functions SDK ) is available to help create Application Services. Currently the only SDK supported language is Golang, with the intention that community developed and supported SDKs may come in the future for other languages. The SDK is available as a Golang module to remain operating system (OS) agnostic and to comply with the latest EdgeX guidelines on dependency management. Any application built on top of the Application Functions SDK is considered an App Service. This SDK is provided to help build Application Services by assembling triggers, pre-existing functions and custom functions of your making into a pipeline. Standard Functions As mentioned, an Application Service is a function pipeline. The SDK provides some standard functions that can be used in a functions pipeline. In the future, additional functions will be provided \"standard\" or in other words provided with the SDK. Additionally, developers can implement their own custom functions and add those to their Application Service functions pipeline. One of the most common use cases for working with data that comes from the MessageBus is to filter data down to what is relevant for a given application and to format it. To help facilitate this, six primary functions are included in the SDK. The first is the FilterByProfileName function which will remove events that do or do not match the configured ProfileNames and execution of the pipeline will cease if no event remains after filtering. The second is the FilterByDeviceName function which will remove events that do or do not match the configured DeviceNames and execution of the pipeline will cease if no event remains after filtering. The third is the FilterBySourceName function which will remove events that do or do not match the configured SourceNames and execution of the pipeline will cease if no event remains after filtering. A SourceName is the name of the source (command or resource) that the Event was created from. The fourth is the FilterByResourceName which exhibits the same behavior as DeviceNameFilter except filtering the event's Readings on ResourceName instead of DeviceName . Execution of the pipeline will cease if no readings remain after filtering. The fifth and sixth provided functions in the SDK transform the data received to either XML or JSON by calling XMLTransform or JSONTransform . EdgeX 2.0 The FilterByProfileName and FilterBySourceName pipeline functions are new in EdgeX 2.0 with the addition of the ProfileName and SourceName on the V2 Event DTO. FilterByResourceName replaces the FileterByValueDescriptor pipeline function in EdgeX 2.0 with the change of Name to ResourceName on the V2 Reading DTO. This function serves the same purpose of filtering Event Readings. Typically, after filtering and transforming the data as needed, exporting is the last step in a pipeline to ship the data where it needs to go. There are three primary functions included in the SDK to help facilitate this. The first are the HTTPPost/HTTPPut functions that will POST/PUT the provided data to a specified endpoint, and the third is an MQTTSecretSend() function that will publish the provided data to an MQTT Broker as specified in the configuration. See Built-in Functions section for full list of SDK supplied functions Note The App SDK provides much more functionality than just filtering, formatting and exporting. The above simple example is provided to demonstrate how the functions pipeline works. With the ability to write your custom pipeline functions, your custom application services can do what ever your use case demands. There are three primary triggers that have been included in the SDK that initiate the start of the function pipeline. First is the HTTP Trigger via a POST to the endpoint /api/v2/trigger with the EdgeX Event data as the body. Second is the EdgeX MessageBus Trigger with connection details as specified in the configuration and the third it the External MQTT Trigger with connection details as specified in the configuration. See the Triggers section for full list of available Triggers Finally, data may be sent back to the Trigger response by calling .SetResponseData() on the context. If the trigger is HTTP, then it will be an HTTP Response. If the trigger is EdgeX MessageBus, then it will be published to the configured host and publish topic. If the trigger is External MQTT, then it will be published to the configured publish topic.","title":"Introduction"},{"location":"microservices/application/ApplicationServices/#application-services","text":"Application Services are a means to get data from EdgeX Foundry to be processed at the edge and/or sent to external systems (be it analytics package, enterprise or on-prem application, cloud systems like Azure IoT, AWS IoT, or Google IoT Core, etc.). Application Services provide the means for data to be prepared (transformed, enriched, filtered, etc.) and groomed (formatted, compressed, encrypted, etc.) before being sent to an endpoint of choice or published back to other Application Service to consume. The export endpoints supported out of the box today include HTTP and MQTT endpoints, but custom endpoints can be implemented along side the existing functionality. Application Services are based on the idea of a \"Functions Pipeline\". A functions pipeline is a collection of functions that process messages (in this case EdgeX event/reading messages) in the order that you've specified. Triggers seed the first function in the pipeline with the data received by the Application Service. A trigger is something like a message landing in a watched message queue. The most commonly used Trigger is the MessageBus Trigger. See the Triggers section for more details An Applications Functions Software Development Kit (or App Functions SDK ) is available to help create Application Services. Currently the only SDK supported language is Golang, with the intention that community developed and supported SDKs may come in the future for other languages. The SDK is available as a Golang module to remain operating system (OS) agnostic and to comply with the latest EdgeX guidelines on dependency management. Any application built on top of the Application Functions SDK is considered an App Service. This SDK is provided to help build Application Services by assembling triggers, pre-existing functions and custom functions of your making into a pipeline.","title":"Application Services"},{"location":"microservices/application/ApplicationServices/#standard-functions","text":"As mentioned, an Application Service is a function pipeline. The SDK provides some standard functions that can be used in a functions pipeline. In the future, additional functions will be provided \"standard\" or in other words provided with the SDK. Additionally, developers can implement their own custom functions and add those to their Application Service functions pipeline. One of the most common use cases for working with data that comes from the MessageBus is to filter data down to what is relevant for a given application and to format it. To help facilitate this, six primary functions are included in the SDK. The first is the FilterByProfileName function which will remove events that do or do not match the configured ProfileNames and execution of the pipeline will cease if no event remains after filtering. The second is the FilterByDeviceName function which will remove events that do or do not match the configured DeviceNames and execution of the pipeline will cease if no event remains after filtering. The third is the FilterBySourceName function which will remove events that do or do not match the configured SourceNames and execution of the pipeline will cease if no event remains after filtering. A SourceName is the name of the source (command or resource) that the Event was created from. The fourth is the FilterByResourceName which exhibits the same behavior as DeviceNameFilter except filtering the event's Readings on ResourceName instead of DeviceName . Execution of the pipeline will cease if no readings remain after filtering. The fifth and sixth provided functions in the SDK transform the data received to either XML or JSON by calling XMLTransform or JSONTransform . EdgeX 2.0 The FilterByProfileName and FilterBySourceName pipeline functions are new in EdgeX 2.0 with the addition of the ProfileName and SourceName on the V2 Event DTO. FilterByResourceName replaces the FileterByValueDescriptor pipeline function in EdgeX 2.0 with the change of Name to ResourceName on the V2 Reading DTO. This function serves the same purpose of filtering Event Readings. Typically, after filtering and transforming the data as needed, exporting is the last step in a pipeline to ship the data where it needs to go. There are three primary functions included in the SDK to help facilitate this. The first are the HTTPPost/HTTPPut functions that will POST/PUT the provided data to a specified endpoint, and the third is an MQTTSecretSend() function that will publish the provided data to an MQTT Broker as specified in the configuration. See Built-in Functions section for full list of SDK supplied functions Note The App SDK provides much more functionality than just filtering, formatting and exporting. The above simple example is provided to demonstrate how the functions pipeline works. With the ability to write your custom pipeline functions, your custom application services can do what ever your use case demands. There are three primary triggers that have been included in the SDK that initiate the start of the function pipeline. First is the HTTP Trigger via a POST to the endpoint /api/v2/trigger with the EdgeX Event data as the body. Second is the EdgeX MessageBus Trigger with connection details as specified in the configuration and the third it the External MQTT Trigger with connection details as specified in the configuration. See the Triggers section for full list of available Triggers Finally, data may be sent back to the Trigger response by calling .SetResponseData() on the context. If the trigger is HTTP, then it will be an HTTP Response. If the trigger is EdgeX MessageBus, then it will be published to the configured host and publish topic. If the trigger is External MQTT, then it will be published to the configured publish topic.","title":"Standard Functions"},{"location":"microservices/application/BuiltIn/","text":"Built-In Pipeline Functions All pipeline functions define a type and a factory function which is used to initialize an instance of the type with the required options. The instances returned by these factory functions give access to their appropriate pipeline function pointers when setting up the function pipeline. Example NewFilterFor ([] { \"Device1\" , \"Device2\" }). FilterByDeviceName Batching Included in the SDK is an in-memory batch function that will hold on to your data before continuing the pipeline. There are three functions provided for batching each with their own strategy. Factory Method Description NewBatchByTime(timeInterval string) This function returns a BatchConfig instance with time being the strategy that is used for determining when to release the batched data and continue the pipeline. timeInterval is the duration to wait (i.e. 10s ). The time begins after the first piece of data is received. If no data has been received no data will be sent forward. NewBatchByCount(batchThreshold int) This function returns a BatchConfig instance with count being the strategy that is used for determining when to release the batched data and continue the pipeline. batchThreshold is how many events to hold on to (i.e. 25 ). The count begins after the first piece of data is received and once the threshold is met, the batched data will continue forward and the counter will be reset. NewBatchByTimeAndCount(timeInterval string, batchThreshold int) This function returns a BatchConfig instance with a combination of both time and count being the strategy that is used for determining when to release the batched data and continue the pipeline. Whichever occurs first will trigger the data to continue and be reset. Examples NewBatchByTime ( \"10s\" ). Batch NewBatchByCount ( 10 ). Batch NewBatchByTimeAndCount ( \"30s\" , 10 ). Batch Property Description IsEventData The IsEventData flag, when true, lets this function know that the data being batched is Events and to un-marshal the data a []Event prior to returning the batched data. MergeOnSend The MergeOnSend flag, when true, will merge the [][]byte data to a single []byte prior to sending the data to the next function in the pipeline. Edgex 2.1 New for EdgeX 2.1 is the IsEventData flag on the BatchConfig instance. Batch with IsEventData flag set to true. batch := NewBatchByTimeAndCount(\"30s\", 10) batch.IsEventData = true ... batch.Batch Edgex 2.2 New for EdgeX 2.2 is the MergeOnSend flag on the BatchConfig instance. Batch with MergeOnSend flag set to true. batch := NewBatchByTimeAndCount(\"30s\", 10) batch.MergeOnSend = true ... batch.Batch Batch Batch - This pipeline function will apply the selected strategy in your pipeline. By default the batched data returned by this function is [][]byte . This is because this function doesn't need to know the type of the individual items batched. It simply marshals the items to JSON if the data isn't already a []byte . Warning Keep memory usage in mind as you determine the thresholds for both time and count. The larger they are the more memory is required and could lead to performance issue. Compression There are two compression types included in the SDK that can be added to your pipeline. These transforms return a []byte . Factory Method Description NewCompression() This factory function returns a Compression instance that is used to access the compression functions. GZIP CompressWithGZIP - This pipeline function receives either a string , []byte , or json.Marshaler type, GZIP compresses the data, converts result to base64 encoded string, which is returned as a []byte to the pipeline. Example NewCompression (). CompressWithGZIP ZLIB CompressWithZLIB - This pipeline function receives either a string , []byte , or json.Marshaler type, ZLIB compresses the data, converts result to base64 encoded string, which is returned as a []byte to the pipeline. Example NewCompression (). CompressWithZLIB Conversion There are two conversions included in the SDK that can be added to your pipeline. These transforms return a string . Factory Method Description NewConversion() This factory function returns a Conversion instance that is used to access the conversion functions. JSON TransformToJSON - This pipeline function receives an dtos.Event type and converts it to JSON format and returns the JSON string to the pipeline. Example NewConversion (). TransformToJSON XML TransformToXML - This pipeline function receives an dtos.Event type, converts it to XML format and returns the XML string to the pipeline. Example NewConversion (). TransformToXML Core Data There is one Core Data function that enables interactions with the Core Data REST API Factory Method Description NewCoreDataSimpleReading(profileName string, deviceName string, resourceName string, valueType string) This factory function returns a CoreData instance configured to push a Simple reading. The CoreData instance returned is used to access core data functions. NewCoreDataBinaryReading(profileName string, deviceName string, resourceName string, mediaType string) This factory function returns a CoreData instance configured to push a Binary reading. The CoreData instance returned is used to access core data functions. NewCoreDataObejctReading(profileName string, deviceName string, resourceName string) This factory function returns a CoreData instance configured to push an Object reading. The CoreData instance returned is used to access core data functions. EdgeX 2.0 For EdgeX 2.0 the NewCoreData factory function has been replaced with the NewCoreDataSimpleReading and NewCoreDataBinaryReading functions EdgeX 2.1 The NewCoreDataObejctReading factory method is new for EdgeX 2.1 Push to Core Data PushToCoreData - This pipeline function provides the capability to push a new Event/Reading to Core Data. The data passed into this function from the pipeline is wrapped in an EdgeX Event with the Event and Reading metadata specified from the factory function options. The function returns the new EdgeX Event with ID populated. Example NewCoreDataSimpleReading ( \"my-profile\" , \"my-device\" , \"my-resource\" , \"string\" ). PushToCoreData Data Protection There are two transforms included in the SDK that can be added to your pipeline for data protection. Encryption (Deprecated) EdgeX 2.1 This is deprecated in EdgeX 2.1 - it is recommended to use the new AESProtection transform. Please see this security advisory for more detail. Factory Method Description NewEncryption(key string, initializationVector string) This function returns a Encryption instance initialized with the passed in key and initialization vector . This Encryption instance is used to access the following encryption function that will use the specified key and initialization vector . NewEncryptionWithSecrets(secretPath string, secretName string, initializationVector string) This function returns a Encryption instance initialized with the passed in secret path , Secret name and initialization vector . This Encryption instance is used to access the following encryption function that will use the encryption key from the Secret Store and the passed in initialization vector . It uses the passed in secret path and secret name to pull the encryption key from the Secret Store EdgeX 2.0 New for EdgeX 2.0 is the ability to pull the encryption key from the Secret Store. The encryption key must be seeded into the Secret Store using the /api/v2/secret endpoint on the running instance of the Application Service prior to the Encryption function executing. See App Functions SDK swagger for more details on this endpoint. EncryptWithAES - This pipeline function receives either a string , []byte , or json.Marshaller type and encrypts it using AES encryption and returns a []byte to the pipeline. Example NewEncryption ( \"key\" , \"initializationVector\" ). EncryptWithAES or NewEncryptionWithSecrets ( \"aes\" , \"aes-key\" , \"initializationVector\" ). EncryptWithAES ) Note The algorithm used used with app-service-configurable configuration to access this transform is AES AESProtection Edgex 2.1 This transform provides AES 256 encryption with a random initialization vector and authentication using a SHA 512 hash. It can only be configured using secrets. Factory Method Description NewAESProtection(secretPath string, secretName string) This function returns a Encryption instance initialized with the passed in secretPath and secretName It requires a 64-byte key from secrets which is split in half, the first half used for encryption, the second for generating the signature. Encrypt : This pipeline function receives either a string , []byte , or json.Marshaller type and encrypts it using AES256 encryption, signs it with a SHA512 hash and returns a []byte to the pipeline of the following form: initialization vector ciphertext signing hash 16 bytes variable bytes 32 bytes Example transforms . NewAESProtection ( secretPath , secretName ). Encrypt ( ctx , data ) Note The Algorithm used with app-service-configurable configuration to access this transform is AES256 Export There are two export functions included in the SDK that can be added to your pipeline. HTTP Export EdgeX 2.0 For EdgeX 2.0 the signature of the NewHTTPSenderWithSecretHeader factory function has changed. See below for details. Factory Method Description NewHTTPSender(url string, mimeType string, persistOnError bool) This factory function returns a HTTPSender instance initialized with the passed in url, mime type and persistOnError values. NewHTTPSenderWithSecretHeader(url string, mimeType string, persistOnError bool, headerName string, secretPath string, secretName string) This factory function returns a HTTPSender instance similar to the above function however will set up the HTTPSender to add a header to the HTTP request using the headerName for the field name and the secretPath and secretName to pull the header field value from the Secret Store. NewHTTPSenderWithOptions(options HTTPSenderOptions) This factory function returns a HTTPSender using the passed in options to configure it. EdgeX 2.0 New in EdgeX 2.0 is the ability to chain multiple instances of the HTTP exports to accomplish exporting to multiple destinations. The new NewHTTPSenderWithOptions factory function was added to allow for configuring all the options, including the new ContinueOnSendError and ReturnInputData options that enable this chaining. // HTTPSenderOptions contains all options available to the sender type HTTPSenderOptions struct { // URL of destination URL string // MimeType to send to destination MimeType string // PersistOnError enables use of store & forward loop if true PersistOnError bool // HTTPHeaderName to use for passing configured secret HTTPHeaderName string // SecretPath to search for configured secret SecretPath string // SecretName for configured secret SecretName string // URLFormatter specifies custom formatting behavior to be applied to configured URL. // If nothing specified, default behavior is to attempt to replace placeholders in the // form '{some-context-key}' with the values found in the context storage. URLFormatter StringValuesFormatter // ContinueOnSendError allows execution of subsequent chained senders after errors if true ContinueOnSendError bool // ReturnInputData enables chaining multiple HTTP senders if true ReturnInputData bool } HTTP POST HTTPPost - This pipeline function receives either a string , []byte , or json.Marshaler type from the previous function in the pipeline and posts it to the configured endpoint and returns the HTTP response. If no previous function exists, then the event that triggered the pipeline, marshaled to json, will be used. If the post fails and persistOnError=true and Store and Forward is enabled, the data will be stored for later retry. See Store and Forward for more details. If ReturnInputData=true the function will return the data that it received instead of the HTTP response. This allows the following function in the pipeline to be another HTTP Export which receives the same data but is configured to send to a different endpoint. When chaining for multiple HTTP Exports you need to decide how to handle errors. Do you want to stop execution of the pipeline or continue so that the next HTTP Export function can attempt to export to its endpoint. This is where ContinueOnSendError comes in. If set to true the error is logged and the function returns the received data for the next function to use. ContinueOnSendError=true can only be used when ReturnInputData=true and cannot be use when PersistOnError=true . Example POST NewHTTPSender(\"https://myendpoint.com\",\"application/json\",false).HTTPPost PUT NewHTTPSender(\"https://myendpoint.com\",\"application/json\",false).HTTPPut POST with secure header NewHTTPSenderWithSecretHeader(\"https://myendpoint.com\",\"application/json\",false,\"Authentication\",\"/jwt\",\"AuthToken\").HTTPPost PUT with secure header NewHTTPSenderWithSecretHeader(\"https://myendpoint.com\",\"application/json\",false,\"Authentication\",\"/jwt\",\"AuthToken\").HTTPPPut HTTP PUT HTTPPut - This pipeline function operates the same as HTTPPost but uses the PUT method rather than POST . URL Formatting EdgeX 2.0 URL Formatting is new in EdgeX 2.0 The configured URL is dynamically formatted prior to the POST/PUT request. The default formatter (used if URLFormatter is nil) simply replaces any placeholder text, {key-name} , in the configured URL with matching values from the new Context Storage . An error will occur if a specified placeholder does not exist in the Context Storage . See the Context Storage documentation for more details on seeded values and storing your own values. The URLFormatter option allows you to override the default formatter with your own custom URL formatting scheme. Example Export the Events to different endpoints base on their device name Url=\"http://myhost.com/edgex-events/{devicename}\" MQTT Export EdgeX 2.0 New for EdgeX 2.0 is the the new NewMQTTSecretSenderWithTopicFormatter factory function. The deprecated NewMQTTSender factory function has been removed. Factory Method Description NewMQTTSecretSender(mqttConfig MQTTSecretConfig, persistOnError bool) This factory function returns a MQTTSecretSender instance initialized with the options specified in the MQTTSecretConfig and persistOnError . NewMQTTSecretSenderWithTopicFormatter(mqttConfig MQTTSecretConfig, persistOnError bool, topicFormatter StringValuesFormatter) This factory function returns a MQTTSecretSender instance initialized with the options specified in the MQTTSecretConfig , persistOnError and topicFormatter . See Topic Formatting below for more details. EdgeX 2.0 New in EdgeX 2.0 the KeepAlive and ConnectTimeout MQTTSecretConfig settings have been added. type MQTTSecretConfig struct { // BrokerAddress should be set to the complete broker address i.e. mqtts://mosquitto:8883/mybroker BrokerAddress string // ClientId to connect with the broker with. ClientId string // The name of the path in secret provider to retrieve your secrets SecretPath string // AutoReconnect indicated whether or not to retry connection if disconnected AutoReconnect bool // KeepAlive is the interval duration between client sending keepalive ping to broker KeepAlive string // ConnectTimeout is the duration for timing out on connecting to the broker ConnectTimeout string // Topic that you wish to publish to Topic string // QoS for MQTT Connection QoS byte // Retain setting for MQTT Connection Retain bool // SkipCertVerify SkipCertVerify bool // AuthMode indicates what to use when connecting to the broker. // Options are \"none\", \"cacert\" , \"usernamepassword\", \"clientcert\". // If a CA Cert exists in the SecretPath then it will be used for // all modes except \"none\". AuthMode string } Secrets in the Secret Store may be located at any path however they must have some or all the follow keys at the specified SecretPath . username - username to connect to the broker password - password used to connect to the broker clientkey - client private key in PEM format clientcert - client cert in PEM format cacert - ca cert in PEM format The AuthMode setting you choose depends on what secret values above are used. For example, if \"none\" is specified as auth mode all keys will be ignored. Similarly, if AuthMode is set to \"clientcert\" username and password will be ignored. Topic Formatting EdgeX 2.0 Topic Formatting is new in EdgeX 2.0 The configured Topic is dynamically formatted prior to publishing . The default formatter (used if topicFormatter is nil) simply replaces any placeholder text, {key-name} , in the configured Topic with matching values from the new Context Storage . An error will occur if a specified placeholder does not exist in the Context Storage . See the Context Storage documentation for more details on seeded values and storing your own values. The topicFormatter option allows you to override the default formatter with your own custom topic formatting scheme. Filtering There are four basic types of filtering included in the SDK to add to your pipeline. There is also an option to Filter Out specific items. These provided filter functions return a type of dtos.Event . If filtering results in no remaining data, the pipeline execution for that pass is terminated. If no values are provided for filtering, then data flows through unfiltered. Factory Method Description NewFilterFor([]string filterValues) This factory function returns a Filter instance initialized with the passed in filter values with FilterOut set to false . This Filter instance is used to access the following filter functions that will operate using the specified filter values. NewFilterOut([]string filterValues) This factory function returns a Filter instance initialized with the passed in filter values with FilterOut set to true . This Filter instance is used to access the following filter functions that will operate using the specified filter values. EdgeX 2.0 For EdgeX 2.0 the NewFilter factory function has been renamed to NewFilterFor and the new NewFilterOut factory function has been added. type Filter struct { // Holds the values to be filtered FilterValues [] string // Determines if items in FilterValues should be filtered out. If set to true all items found in the filter will be removed. If set to false all items found in the filter will be returned. If FilterValues is empty then all items will be returned. FilterOut bool } EdgeX 2.0 New for EdgeX 2.0 are the FilterByProfileName and FilterBySourceName pipeline functions. The FilterByValueDescriptor pipeline function has been renamed to FilterByResourceName By Profile Name FilterByProfileName - This pipeline function will filter the event data down to Events that either have (For) or don't have (Out) the specified profiles names. Example NewFilterFor ([] { \"Profile1\" , \"Profile2\" }). FilterByProfileName By Device Name FilterByDeviceName - This pipeline function will filter the event data down to Events that either have (For) or don't have (Out) the specified device names. Example NewFilterFor ([] { \"Device1\" , \"Device2\" }). FilterByDeviceName By Source Name FilterBySourceName - This pipeline function will filter the event data down to Events that either have (For) or don't have (Out) the specified source names. Source name is either the resource name or command name responsible for the Event creation. Example NewFilterFor ([] { \"Source1\" , \"Source2\" }). FilterBySourceName By Resource Name FilterByResourceName - This pipeline function will filter the Event's reading data down to Readings that either have (For) or don't have (Out) the specified resource names. If the result of filtering is zero Readings remaining, the function terminates pipeline execution. Example NewFilterFor ([] { \"Resource1\" , \"Resource2\" }). FilterByResourceName JSON Logic Factory Method Description NewJSONLogic(rule string) This factory function returns a JSONLogic instance initialized with the passed in JSON rule. The rule passed in should be a JSON string conforming to the specification here: http://jsonlogic.com/operations.html. Evaluate Evaluate - This is the pipeline function that will be used in the pipeline to apply the JSON rule to data coming in on the pipeline. If the condition of your rule is met, then the pipeline will continue and the data will continue to flow to the next function in the pipeline. If the condition of your rule is NOT met, then pipeline execution stops. Example NewJSONLogic ( \"{ \\\"in\\\" : [{ \\\"var\\\" : \\\"device\\\" }, [\\\"Random-Integer-Device\\\",\\\"Random-Float-Device\\\"] ] }\" ). Evaluate Note Only operations that return true or false are supported. See http://jsonlogic.com/operations.html# for the complete list of operations paying attention to return values. Any operator that returns manipulated data is currently not supported. For more advanced scenarios checkout LF Edge eKuiper . Tip Leverage http://jsonlogic.com/play.html to get your rule right before implementing in code. JSON can be a bit tricky to get right in code with all the escaped double quotes. Response Data There is one response data function included in the SDK that can be added to your pipeline. Factory Method Description NewResponseData() This factory function returns a ResponseData instance that is used to access the following pipeline function below. Content Type ResponseContentType - This property is used to set the content-type of the response. Example responseData := NewResponseData () responseData . ResponseContentType = \"application/json\" Set Response Data SetResponseData - This pipeline function receives either a string , []byte , or json.Marshaler type from the previous function in the pipeline and sets it as the response data that the pipeline returns to the configured trigger. If configured to use the EdgeXMessageBus trigger, the data will be published back to the EdgeX MessageBus as determined by the configuration. Similar, if configured to use the ExternalMQTT trigger, the data will be published back to the external MQTT Broker as determined by the configuration. If configured to use HTTP trigger the data is returned as the HTTP response. Note Calling SetResponseData() and SetResponseContentType() from the Context API in a custom function can be used in place of adding this function to your pipeline. Tags There is one Tags transform included in the SDK that can be added to your pipeline. Factory Method Description NewGenericTags(tags map[string]interface{} ) Tags This factory function returns a Tags instance initialized with the passed in collection of generic tag key/value pairs. This Tags instance is used to access the following Tags function that will use the specified collection of tag key/value pairs. This allows for generic complex types for the Tag values. NewTags(tags map[string]string ) Tags This factory function returns a Tags instance initialized with the passed in collection of tag key/value pairs. This Tags instance is used to access the following Tags function that will use the specified collection of tag key/value pairs. This factor function has been Deprecated. Use NewGenericTags instead . EdgeX 2.1 The Tags property on Events in Edgex 2.1 has changed from map[string]string to map[string]interface{} . The new NewGenericTags() factory function takes this new definition and replaces the deprecated NewTags() factory function. Add Tags AddTags - This pipeline function receives an Edgex Event type and adds the collection of specified tags to the Event's Tags collection. Example var myTags = map [ string ] interface {}{ \"MyValue\" : 123 , \"GatewayId\" : \"HoustonStore000123\" , \"Coordinates\" : map [ string ] float32 { \"Latitude\" : 29.630771 , \"Longitude\" : \"-95.377603\" , }, } NewGenericTags ( myTags ). AddTags MetricsProcessor EdgeX 2.2 The MetricsProcessor is new in EdgeX 2.2 MetricsProcessor contains configuration and functions for processing the new dtos.Metrics type. Factory Method Description NewMetricsProcessor(additionalTags map[string]interface{}) (*MetricsProcessor, error) This factory function returns a `MetricsProcessor instance initialized with the passed in collection of additionalTags (name/value pairs). This MetricsProcessor instance is used to access the following functions that will process a dtos.Metric instance. The additionalTags are added as metric tags to the processed data. An error will be returned if any of the additionalTags have an invalid name. Currently must be non-blank. ToLineProtocol ToLineProtocol - This pipeline function will transform the received dtos.Metric to a Line Protocol formatted string. See https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/ for details on the Line Protocol syntax. Note When ToLineProtocol is the first function in the functions pipeline, the TargetType for the service must be set to &dtos.Metric{} . See Target Type section for details on setting the service's TargetType . The Trigger configuration must also be set so SubscribeTopics=\"edgex/telemetry/#\" in order to receive the dtos.Metric data from other services. See the new App Service Configurable metrics-influxdb profile for an example. Example mp , err := NewMetricsProcessor ( map [ string ] string { \"MyTag\" : \"MyTagValue\" }) if err != nil { ... handle error } ... mp . ToLineProtocol Warning Any service using the MetricsProcessor needs to disable its own Telemetry reporting to avoid circular data generation from processing. To do this set the services Writeable.Telemetry configuration to: [Writable.Telemetry] Interval = \"0s\" # Don't report any metrics as that would be cyclic processing.","title":"Built-In Pipeline Functions"},{"location":"microservices/application/BuiltIn/#built-in-pipeline-functions","text":"All pipeline functions define a type and a factory function which is used to initialize an instance of the type with the required options. The instances returned by these factory functions give access to their appropriate pipeline function pointers when setting up the function pipeline. Example NewFilterFor ([] { \"Device1\" , \"Device2\" }). FilterByDeviceName","title":"Built-In Pipeline Functions"},{"location":"microservices/application/BuiltIn/#batching","text":"Included in the SDK is an in-memory batch function that will hold on to your data before continuing the pipeline. There are three functions provided for batching each with their own strategy. Factory Method Description NewBatchByTime(timeInterval string) This function returns a BatchConfig instance with time being the strategy that is used for determining when to release the batched data and continue the pipeline. timeInterval is the duration to wait (i.e. 10s ). The time begins after the first piece of data is received. If no data has been received no data will be sent forward. NewBatchByCount(batchThreshold int) This function returns a BatchConfig instance with count being the strategy that is used for determining when to release the batched data and continue the pipeline. batchThreshold is how many events to hold on to (i.e. 25 ). The count begins after the first piece of data is received and once the threshold is met, the batched data will continue forward and the counter will be reset. NewBatchByTimeAndCount(timeInterval string, batchThreshold int) This function returns a BatchConfig instance with a combination of both time and count being the strategy that is used for determining when to release the batched data and continue the pipeline. Whichever occurs first will trigger the data to continue and be reset. Examples NewBatchByTime ( \"10s\" ). Batch NewBatchByCount ( 10 ). Batch NewBatchByTimeAndCount ( \"30s\" , 10 ). Batch Property Description IsEventData The IsEventData flag, when true, lets this function know that the data being batched is Events and to un-marshal the data a []Event prior to returning the batched data. MergeOnSend The MergeOnSend flag, when true, will merge the [][]byte data to a single []byte prior to sending the data to the next function in the pipeline. Edgex 2.1 New for EdgeX 2.1 is the IsEventData flag on the BatchConfig instance. Batch with IsEventData flag set to true. batch := NewBatchByTimeAndCount(\"30s\", 10) batch.IsEventData = true ... batch.Batch Edgex 2.2 New for EdgeX 2.2 is the MergeOnSend flag on the BatchConfig instance. Batch with MergeOnSend flag set to true. batch := NewBatchByTimeAndCount(\"30s\", 10) batch.MergeOnSend = true ... batch.Batch","title":"Batching"},{"location":"microservices/application/BuiltIn/#batch","text":"Batch - This pipeline function will apply the selected strategy in your pipeline. By default the batched data returned by this function is [][]byte . This is because this function doesn't need to know the type of the individual items batched. It simply marshals the items to JSON if the data isn't already a []byte . Warning Keep memory usage in mind as you determine the thresholds for both time and count. The larger they are the more memory is required and could lead to performance issue.","title":"Batch"},{"location":"microservices/application/BuiltIn/#compression","text":"There are two compression types included in the SDK that can be added to your pipeline. These transforms return a []byte . Factory Method Description NewCompression() This factory function returns a Compression instance that is used to access the compression functions.","title":"Compression"},{"location":"microservices/application/BuiltIn/#gzip","text":"CompressWithGZIP - This pipeline function receives either a string , []byte , or json.Marshaler type, GZIP compresses the data, converts result to base64 encoded string, which is returned as a []byte to the pipeline. Example NewCompression (). CompressWithGZIP","title":"GZIP"},{"location":"microservices/application/BuiltIn/#zlib","text":"CompressWithZLIB - This pipeline function receives either a string , []byte , or json.Marshaler type, ZLIB compresses the data, converts result to base64 encoded string, which is returned as a []byte to the pipeline. Example NewCompression (). CompressWithZLIB","title":"ZLIB"},{"location":"microservices/application/BuiltIn/#conversion","text":"There are two conversions included in the SDK that can be added to your pipeline. These transforms return a string . Factory Method Description NewConversion() This factory function returns a Conversion instance that is used to access the conversion functions.","title":"Conversion"},{"location":"microservices/application/BuiltIn/#json","text":"TransformToJSON - This pipeline function receives an dtos.Event type and converts it to JSON format and returns the JSON string to the pipeline. Example NewConversion (). TransformToJSON","title":"JSON"},{"location":"microservices/application/BuiltIn/#xml","text":"TransformToXML - This pipeline function receives an dtos.Event type, converts it to XML format and returns the XML string to the pipeline. Example NewConversion (). TransformToXML","title":"XML"},{"location":"microservices/application/BuiltIn/#core-data","text":"There is one Core Data function that enables interactions with the Core Data REST API Factory Method Description NewCoreDataSimpleReading(profileName string, deviceName string, resourceName string, valueType string) This factory function returns a CoreData instance configured to push a Simple reading. The CoreData instance returned is used to access core data functions. NewCoreDataBinaryReading(profileName string, deviceName string, resourceName string, mediaType string) This factory function returns a CoreData instance configured to push a Binary reading. The CoreData instance returned is used to access core data functions. NewCoreDataObejctReading(profileName string, deviceName string, resourceName string) This factory function returns a CoreData instance configured to push an Object reading. The CoreData instance returned is used to access core data functions. EdgeX 2.0 For EdgeX 2.0 the NewCoreData factory function has been replaced with the NewCoreDataSimpleReading and NewCoreDataBinaryReading functions EdgeX 2.1 The NewCoreDataObejctReading factory method is new for EdgeX 2.1","title":"Core Data"},{"location":"microservices/application/BuiltIn/#push-to-core-data","text":"PushToCoreData - This pipeline function provides the capability to push a new Event/Reading to Core Data. The data passed into this function from the pipeline is wrapped in an EdgeX Event with the Event and Reading metadata specified from the factory function options. The function returns the new EdgeX Event with ID populated. Example NewCoreDataSimpleReading ( \"my-profile\" , \"my-device\" , \"my-resource\" , \"string\" ). PushToCoreData","title":"Push to Core Data"},{"location":"microservices/application/BuiltIn/#data-protection","text":"There are two transforms included in the SDK that can be added to your pipeline for data protection.","title":"Data Protection"},{"location":"microservices/application/BuiltIn/#encryption-deprecated","text":"EdgeX 2.1 This is deprecated in EdgeX 2.1 - it is recommended to use the new AESProtection transform. Please see this security advisory for more detail. Factory Method Description NewEncryption(key string, initializationVector string) This function returns a Encryption instance initialized with the passed in key and initialization vector . This Encryption instance is used to access the following encryption function that will use the specified key and initialization vector . NewEncryptionWithSecrets(secretPath string, secretName string, initializationVector string) This function returns a Encryption instance initialized with the passed in secret path , Secret name and initialization vector . This Encryption instance is used to access the following encryption function that will use the encryption key from the Secret Store and the passed in initialization vector . It uses the passed in secret path and secret name to pull the encryption key from the Secret Store EdgeX 2.0 New for EdgeX 2.0 is the ability to pull the encryption key from the Secret Store. The encryption key must be seeded into the Secret Store using the /api/v2/secret endpoint on the running instance of the Application Service prior to the Encryption function executing. See App Functions SDK swagger for more details on this endpoint. EncryptWithAES - This pipeline function receives either a string , []byte , or json.Marshaller type and encrypts it using AES encryption and returns a []byte to the pipeline. Example NewEncryption ( \"key\" , \"initializationVector\" ). EncryptWithAES or NewEncryptionWithSecrets ( \"aes\" , \"aes-key\" , \"initializationVector\" ). EncryptWithAES ) Note The algorithm used used with app-service-configurable configuration to access this transform is AES","title":"Encryption (Deprecated)"},{"location":"microservices/application/BuiltIn/#aesprotection","text":"Edgex 2.1 This transform provides AES 256 encryption with a random initialization vector and authentication using a SHA 512 hash. It can only be configured using secrets. Factory Method Description NewAESProtection(secretPath string, secretName string) This function returns a Encryption instance initialized with the passed in secretPath and secretName It requires a 64-byte key from secrets which is split in half, the first half used for encryption, the second for generating the signature. Encrypt : This pipeline function receives either a string , []byte , or json.Marshaller type and encrypts it using AES256 encryption, signs it with a SHA512 hash and returns a []byte to the pipeline of the following form: initialization vector ciphertext signing hash 16 bytes variable bytes 32 bytes Example transforms . NewAESProtection ( secretPath , secretName ). Encrypt ( ctx , data ) Note The Algorithm used with app-service-configurable configuration to access this transform is AES256","title":"AESProtection"},{"location":"microservices/application/BuiltIn/#export","text":"There are two export functions included in the SDK that can be added to your pipeline.","title":"Export"},{"location":"microservices/application/BuiltIn/#http-export","text":"EdgeX 2.0 For EdgeX 2.0 the signature of the NewHTTPSenderWithSecretHeader factory function has changed. See below for details. Factory Method Description NewHTTPSender(url string, mimeType string, persistOnError bool) This factory function returns a HTTPSender instance initialized with the passed in url, mime type and persistOnError values. NewHTTPSenderWithSecretHeader(url string, mimeType string, persistOnError bool, headerName string, secretPath string, secretName string) This factory function returns a HTTPSender instance similar to the above function however will set up the HTTPSender to add a header to the HTTP request using the headerName for the field name and the secretPath and secretName to pull the header field value from the Secret Store. NewHTTPSenderWithOptions(options HTTPSenderOptions) This factory function returns a HTTPSender using the passed in options to configure it. EdgeX 2.0 New in EdgeX 2.0 is the ability to chain multiple instances of the HTTP exports to accomplish exporting to multiple destinations. The new NewHTTPSenderWithOptions factory function was added to allow for configuring all the options, including the new ContinueOnSendError and ReturnInputData options that enable this chaining. // HTTPSenderOptions contains all options available to the sender type HTTPSenderOptions struct { // URL of destination URL string // MimeType to send to destination MimeType string // PersistOnError enables use of store & forward loop if true PersistOnError bool // HTTPHeaderName to use for passing configured secret HTTPHeaderName string // SecretPath to search for configured secret SecretPath string // SecretName for configured secret SecretName string // URLFormatter specifies custom formatting behavior to be applied to configured URL. // If nothing specified, default behavior is to attempt to replace placeholders in the // form '{some-context-key}' with the values found in the context storage. URLFormatter StringValuesFormatter // ContinueOnSendError allows execution of subsequent chained senders after errors if true ContinueOnSendError bool // ReturnInputData enables chaining multiple HTTP senders if true ReturnInputData bool }","title":"HTTP Export"},{"location":"microservices/application/BuiltIn/#http-post","text":"HTTPPost - This pipeline function receives either a string , []byte , or json.Marshaler type from the previous function in the pipeline and posts it to the configured endpoint and returns the HTTP response. If no previous function exists, then the event that triggered the pipeline, marshaled to json, will be used. If the post fails and persistOnError=true and Store and Forward is enabled, the data will be stored for later retry. See Store and Forward for more details. If ReturnInputData=true the function will return the data that it received instead of the HTTP response. This allows the following function in the pipeline to be another HTTP Export which receives the same data but is configured to send to a different endpoint. When chaining for multiple HTTP Exports you need to decide how to handle errors. Do you want to stop execution of the pipeline or continue so that the next HTTP Export function can attempt to export to its endpoint. This is where ContinueOnSendError comes in. If set to true the error is logged and the function returns the received data for the next function to use. ContinueOnSendError=true can only be used when ReturnInputData=true and cannot be use when PersistOnError=true . Example POST NewHTTPSender(\"https://myendpoint.com\",\"application/json\",false).HTTPPost PUT NewHTTPSender(\"https://myendpoint.com\",\"application/json\",false).HTTPPut POST with secure header NewHTTPSenderWithSecretHeader(\"https://myendpoint.com\",\"application/json\",false,\"Authentication\",\"/jwt\",\"AuthToken\").HTTPPost PUT with secure header NewHTTPSenderWithSecretHeader(\"https://myendpoint.com\",\"application/json\",false,\"Authentication\",\"/jwt\",\"AuthToken\").HTTPPPut","title":"HTTP POST"},{"location":"microservices/application/BuiltIn/#http-put","text":"HTTPPut - This pipeline function operates the same as HTTPPost but uses the PUT method rather than POST .","title":"HTTP PUT"},{"location":"microservices/application/BuiltIn/#url-formatting","text":"EdgeX 2.0 URL Formatting is new in EdgeX 2.0 The configured URL is dynamically formatted prior to the POST/PUT request. The default formatter (used if URLFormatter is nil) simply replaces any placeholder text, {key-name} , in the configured URL with matching values from the new Context Storage . An error will occur if a specified placeholder does not exist in the Context Storage . See the Context Storage documentation for more details on seeded values and storing your own values. The URLFormatter option allows you to override the default formatter with your own custom URL formatting scheme. Example Export the Events to different endpoints base on their device name Url=\"http://myhost.com/edgex-events/{devicename}\"","title":"URL Formatting"},{"location":"microservices/application/BuiltIn/#mqtt-export","text":"EdgeX 2.0 New for EdgeX 2.0 is the the new NewMQTTSecretSenderWithTopicFormatter factory function. The deprecated NewMQTTSender factory function has been removed. Factory Method Description NewMQTTSecretSender(mqttConfig MQTTSecretConfig, persistOnError bool) This factory function returns a MQTTSecretSender instance initialized with the options specified in the MQTTSecretConfig and persistOnError . NewMQTTSecretSenderWithTopicFormatter(mqttConfig MQTTSecretConfig, persistOnError bool, topicFormatter StringValuesFormatter) This factory function returns a MQTTSecretSender instance initialized with the options specified in the MQTTSecretConfig , persistOnError and topicFormatter . See Topic Formatting below for more details. EdgeX 2.0 New in EdgeX 2.0 the KeepAlive and ConnectTimeout MQTTSecretConfig settings have been added. type MQTTSecretConfig struct { // BrokerAddress should be set to the complete broker address i.e. mqtts://mosquitto:8883/mybroker BrokerAddress string // ClientId to connect with the broker with. ClientId string // The name of the path in secret provider to retrieve your secrets SecretPath string // AutoReconnect indicated whether or not to retry connection if disconnected AutoReconnect bool // KeepAlive is the interval duration between client sending keepalive ping to broker KeepAlive string // ConnectTimeout is the duration for timing out on connecting to the broker ConnectTimeout string // Topic that you wish to publish to Topic string // QoS for MQTT Connection QoS byte // Retain setting for MQTT Connection Retain bool // SkipCertVerify SkipCertVerify bool // AuthMode indicates what to use when connecting to the broker. // Options are \"none\", \"cacert\" , \"usernamepassword\", \"clientcert\". // If a CA Cert exists in the SecretPath then it will be used for // all modes except \"none\". AuthMode string } Secrets in the Secret Store may be located at any path however they must have some or all the follow keys at the specified SecretPath . username - username to connect to the broker password - password used to connect to the broker clientkey - client private key in PEM format clientcert - client cert in PEM format cacert - ca cert in PEM format The AuthMode setting you choose depends on what secret values above are used. For example, if \"none\" is specified as auth mode all keys will be ignored. Similarly, if AuthMode is set to \"clientcert\" username and password will be ignored.","title":"MQTT Export"},{"location":"microservices/application/BuiltIn/#topic-formatting","text":"EdgeX 2.0 Topic Formatting is new in EdgeX 2.0 The configured Topic is dynamically formatted prior to publishing . The default formatter (used if topicFormatter is nil) simply replaces any placeholder text, {key-name} , in the configured Topic with matching values from the new Context Storage . An error will occur if a specified placeholder does not exist in the Context Storage . See the Context Storage documentation for more details on seeded values and storing your own values. The topicFormatter option allows you to override the default formatter with your own custom topic formatting scheme.","title":"Topic Formatting"},{"location":"microservices/application/BuiltIn/#filtering","text":"There are four basic types of filtering included in the SDK to add to your pipeline. There is also an option to Filter Out specific items. These provided filter functions return a type of dtos.Event . If filtering results in no remaining data, the pipeline execution for that pass is terminated. If no values are provided for filtering, then data flows through unfiltered. Factory Method Description NewFilterFor([]string filterValues) This factory function returns a Filter instance initialized with the passed in filter values with FilterOut set to false . This Filter instance is used to access the following filter functions that will operate using the specified filter values. NewFilterOut([]string filterValues) This factory function returns a Filter instance initialized with the passed in filter values with FilterOut set to true . This Filter instance is used to access the following filter functions that will operate using the specified filter values. EdgeX 2.0 For EdgeX 2.0 the NewFilter factory function has been renamed to NewFilterFor and the new NewFilterOut factory function has been added. type Filter struct { // Holds the values to be filtered FilterValues [] string // Determines if items in FilterValues should be filtered out. If set to true all items found in the filter will be removed. If set to false all items found in the filter will be returned. If FilterValues is empty then all items will be returned. FilterOut bool } EdgeX 2.0 New for EdgeX 2.0 are the FilterByProfileName and FilterBySourceName pipeline functions. The FilterByValueDescriptor pipeline function has been renamed to FilterByResourceName","title":"Filtering"},{"location":"microservices/application/BuiltIn/#by-profile-name","text":"FilterByProfileName - This pipeline function will filter the event data down to Events that either have (For) or don't have (Out) the specified profiles names. Example NewFilterFor ([] { \"Profile1\" , \"Profile2\" }). FilterByProfileName","title":"By Profile Name"},{"location":"microservices/application/BuiltIn/#by-device-name","text":"FilterByDeviceName - This pipeline function will filter the event data down to Events that either have (For) or don't have (Out) the specified device names. Example NewFilterFor ([] { \"Device1\" , \"Device2\" }). FilterByDeviceName","title":"By Device Name"},{"location":"microservices/application/BuiltIn/#by-source-name","text":"FilterBySourceName - This pipeline function will filter the event data down to Events that either have (For) or don't have (Out) the specified source names. Source name is either the resource name or command name responsible for the Event creation. Example NewFilterFor ([] { \"Source1\" , \"Source2\" }). FilterBySourceName","title":"By Source Name"},{"location":"microservices/application/BuiltIn/#by-resource-name","text":"FilterByResourceName - This pipeline function will filter the Event's reading data down to Readings that either have (For) or don't have (Out) the specified resource names. If the result of filtering is zero Readings remaining, the function terminates pipeline execution. Example NewFilterFor ([] { \"Resource1\" , \"Resource2\" }). FilterByResourceName","title":"By Resource Name"},{"location":"microservices/application/BuiltIn/#json-logic","text":"Factory Method Description NewJSONLogic(rule string) This factory function returns a JSONLogic instance initialized with the passed in JSON rule. The rule passed in should be a JSON string conforming to the specification here: http://jsonlogic.com/operations.html.","title":"JSON Logic"},{"location":"microservices/application/BuiltIn/#evaluate","text":"Evaluate - This is the pipeline function that will be used in the pipeline to apply the JSON rule to data coming in on the pipeline. If the condition of your rule is met, then the pipeline will continue and the data will continue to flow to the next function in the pipeline. If the condition of your rule is NOT met, then pipeline execution stops. Example NewJSONLogic ( \"{ \\\"in\\\" : [{ \\\"var\\\" : \\\"device\\\" }, [\\\"Random-Integer-Device\\\",\\\"Random-Float-Device\\\"] ] }\" ). Evaluate Note Only operations that return true or false are supported. See http://jsonlogic.com/operations.html# for the complete list of operations paying attention to return values. Any operator that returns manipulated data is currently not supported. For more advanced scenarios checkout LF Edge eKuiper . Tip Leverage http://jsonlogic.com/play.html to get your rule right before implementing in code. JSON can be a bit tricky to get right in code with all the escaped double quotes.","title":"Evaluate"},{"location":"microservices/application/BuiltIn/#response-data","text":"There is one response data function included in the SDK that can be added to your pipeline. Factory Method Description NewResponseData() This factory function returns a ResponseData instance that is used to access the following pipeline function below.","title":"Response Data"},{"location":"microservices/application/BuiltIn/#content-type","text":"ResponseContentType - This property is used to set the content-type of the response. Example responseData := NewResponseData () responseData . ResponseContentType = \"application/json\"","title":"Content Type"},{"location":"microservices/application/BuiltIn/#set-response-data","text":"SetResponseData - This pipeline function receives either a string , []byte , or json.Marshaler type from the previous function in the pipeline and sets it as the response data that the pipeline returns to the configured trigger. If configured to use the EdgeXMessageBus trigger, the data will be published back to the EdgeX MessageBus as determined by the configuration. Similar, if configured to use the ExternalMQTT trigger, the data will be published back to the external MQTT Broker as determined by the configuration. If configured to use HTTP trigger the data is returned as the HTTP response. Note Calling SetResponseData() and SetResponseContentType() from the Context API in a custom function can be used in place of adding this function to your pipeline.","title":"Set Response Data"},{"location":"microservices/application/BuiltIn/#tags","text":"There is one Tags transform included in the SDK that can be added to your pipeline. Factory Method Description NewGenericTags(tags map[string]interface{} ) Tags This factory function returns a Tags instance initialized with the passed in collection of generic tag key/value pairs. This Tags instance is used to access the following Tags function that will use the specified collection of tag key/value pairs. This allows for generic complex types for the Tag values. NewTags(tags map[string]string ) Tags This factory function returns a Tags instance initialized with the passed in collection of tag key/value pairs. This Tags instance is used to access the following Tags function that will use the specified collection of tag key/value pairs. This factor function has been Deprecated. Use NewGenericTags instead . EdgeX 2.1 The Tags property on Events in Edgex 2.1 has changed from map[string]string to map[string]interface{} . The new NewGenericTags() factory function takes this new definition and replaces the deprecated NewTags() factory function.","title":"Tags"},{"location":"microservices/application/BuiltIn/#add-tags","text":"AddTags - This pipeline function receives an Edgex Event type and adds the collection of specified tags to the Event's Tags collection. Example var myTags = map [ string ] interface {}{ \"MyValue\" : 123 , \"GatewayId\" : \"HoustonStore000123\" , \"Coordinates\" : map [ string ] float32 { \"Latitude\" : 29.630771 , \"Longitude\" : \"-95.377603\" , }, } NewGenericTags ( myTags ). AddTags","title":"Add Tags"},{"location":"microservices/application/BuiltIn/#metricsprocessor","text":"EdgeX 2.2 The MetricsProcessor is new in EdgeX 2.2 MetricsProcessor contains configuration and functions for processing the new dtos.Metrics type. Factory Method Description NewMetricsProcessor(additionalTags map[string]interface{}) (*MetricsProcessor, error) This factory function returns a `MetricsProcessor instance initialized with the passed in collection of additionalTags (name/value pairs). This MetricsProcessor instance is used to access the following functions that will process a dtos.Metric instance. The additionalTags are added as metric tags to the processed data. An error will be returned if any of the additionalTags have an invalid name. Currently must be non-blank.","title":"MetricsProcessor"},{"location":"microservices/application/BuiltIn/#tolineprotocol","text":"ToLineProtocol - This pipeline function will transform the received dtos.Metric to a Line Protocol formatted string. See https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/ for details on the Line Protocol syntax. Note When ToLineProtocol is the first function in the functions pipeline, the TargetType for the service must be set to &dtos.Metric{} . See Target Type section for details on setting the service's TargetType . The Trigger configuration must also be set so SubscribeTopics=\"edgex/telemetry/#\" in order to receive the dtos.Metric data from other services. See the new App Service Configurable metrics-influxdb profile for an example. Example mp , err := NewMetricsProcessor ( map [ string ] string { \"MyTag\" : \"MyTagValue\" }) if err != nil { ... handle error } ... mp . ToLineProtocol Warning Any service using the MetricsProcessor needs to disable its own Telemetry reporting to avoid circular data generation from processing. To do this set the services Writeable.Telemetry configuration to: [Writable.Telemetry] Interval = \"0s\" # Don't report any metrics as that would be cyclic processing.","title":"ToLineProtocol"},{"location":"microservices/application/ErrorHandling/","text":"Pipeline Function Error Handling Each transform returns a true or false as part of the return signature. This is called the continuePipeline flag and indicates whether the SDK should continue calling successive transforms in the pipeline. return false, nil will stop the pipeline and stop processing the event. This is useful, for example, when filtering on values and nothing matches the criteria you've filtered on. return false, error , will stop the pipeline as well and the SDK will log the error you have returned. return true, nil tells the SDK to continue, and will call the next function in the pipeline with your result. The SDK will return control back to main when receiving a SIGTERM/SIGINT event to allow for custom clean up.","title":"Pipeline Function Error Handling"},{"location":"microservices/application/ErrorHandling/#pipeline-function-error-handling","text":"Each transform returns a true or false as part of the return signature. This is called the continuePipeline flag and indicates whether the SDK should continue calling successive transforms in the pipeline. return false, nil will stop the pipeline and stop processing the event. This is useful, for example, when filtering on values and nothing matches the criteria you've filtered on. return false, error , will stop the pipeline as well and the SDK will log the error you have returned. return true, nil tells the SDK to continue, and will call the next function in the pipeline with your result. The SDK will return control back to main when receiving a SIGTERM/SIGINT event to allow for custom clean up.","title":"Pipeline Function Error Handling"},{"location":"microservices/application/GeneralAppServiceConfig/","text":"Application Service Configuration Similar to other EdgeX services, configuration is first determined by the configuration.toml file in the /res folder. Once loaded any environment overrides are applied. If -cp is passed to the application on startup, the SDK will leverage the specific configuration provider (i.e Consul) to push the configuration into the provider and monitor Writeable configuration from there. You will find the configuration under the edgex/appservices/2.0/ key in the provider (i.e Consul). On re-restart the service will pull the configuration from the provider and apply any environment overrides. This section describes the configuration elements that are unique to Application Services Please first refer to the general Configuration documentation for configuration properties common across all EdgeX services. Note * indicates the configuration value can be changed on the fly if using a configuration provider (like Consul). ** indicates the configuration value can be changed but the service must be restarted. Writable The tabs below provide additional entries in the Writable section which are applicable to Application Services. Writable.StoreAndForward Writable.Pipeline Writable.InsecureSecrets Writable.Telemetry The section configures the Store and Forward capability. Please refer to Store and Forward documentation for more details. Configuration Default Value Enabled false* Indicates whether the Store and Forward capability enabled or disabled RetryInterval \"5m\"* Indicates the duration of time to wait before retries, aka Forward MaxRetryCount 10* Indicates whether maximum number of retries of failed data. The failed data is removed after the maximum retries has been exceeded. A value of 0 indicates endless retries. The section configures the Configurable Function Pipeline which is used only by App Service Configurable. Please refer to App Service Configurable - Getting Started section for more details This section defines Insecure Secrets that are used when running in non-secure mode, i.e. when Vault isn't available. This is a dynamic map of configuration, so can empty if no secrets are used or can have as many or few user define secrets. It simulates a Secret Store in non-secure mode. Below are a few examples that are need if using the indicated capabilities. Configuration Default Value Description DB --- This section defines a block of insecure secrets for database credentials when Redis is used for the MessageBus and/or when Store and Forward is enabled and running is non-secure mode. This section is not required if Store and Forward is not enabled and not using Redis for the MessageBus . path redisdb* Indicates the location in the simulated Secret Store where the DB secret resides. DB Secrets --- This section is the collection of DB secret data username blank* Indicates the value for the username when connecting to the database. When running in non-secure mode it is blank . password blank* Indicates the value for the password when connecting to the database. When running in non-secure mode it is blank . http --- This section defines a block of insecure secrets for HTTP Export, i.e HTTPPost function path http* Indicates the location in the simulated Secret Store where the HTTP secret resides. http Secrets --- This section is the collection of HTTP secret data. See Http Export documentation for more details on use of secret data. headervalue undefined* This indicates the name of the secret value to use as the value in the HTTP header. mqtt --- This section defines a block of insecure secrets for MQTT export, i.e. MQTTSecretSend function. path mqtt* Indicates the location in the simulated Secret Store where the MQTT secret reside. mqtt Secrets --- This section is the collection of MQTT secret data. See Mqtt Export documentation for more details on use of secret data. username blank* Indicates the value for the username when connecting to the MQTT broker using usernamepassword authentication mode. Must be configured to the value the MQTT broker is expecting. password blank* Indicates the value for the password when connecting to the MQTT broker using usernamepassword authentication mode. Must be configured to the value the MQTT broker is expecting. cacert blank* Indicates the value (contents) for the CA Certificate when connecting to the MQTT broker using cacert authentication mode. Must be configured to the value the MQTT broker is expecting. clientcert blank* Indicates the value (contents) for the Client Certificate when connecting to the MQTT broker using clientcert authentication mode. Must be configured to the value the MQTT broker is expecting. clientkey blank* Indicates the value (contents) for the Client Key when connecting to the MQTT broker using clientcert authentication mode. Must be configured to the value the MQTT broker is expecting. Property Default Value Description See Writable.Telemetry at Common Configuration for the Telemetry configuration common to all services Metrics Service metrics that the application service collects. Boolean value indicates if reporting of the metric is enabled. Custom metrics are also included here for custom application services that define custom metrics MessagesReceived = false Enable/disable reporting of the built-in MessagesReceived metric PipelineMessagesProcessed = false Enable/disable reporting of the built-in PipelineMessagesProcessed metrics PipelineMessageProcessingTime = false Enable/disable reporting of the built-in PipelineMessageProcessingTime metrics = false Enable/disable reporting of custome application service's custom metric Tags List of arbitrary service level tags to included with every metric that is reported. i.e. Gateway=\"my-iot-gateway\" Edgex 2.2 New for EdgeX 2.2 All application services have a limited set of built-in service metrics and custom application services can define, collect and report their own custome service metrics. See Built-in Application Service Metrics and Custom Application Service section for more detials Not Writable The tabs below provide additional configuration which are applicable to Application Services that require the service to be restarted after value(s) are changed. HttpServer Database Clients Trigger Trigger EdgeXMessageBus Trigger ExternalMqtt EdgeX 2.0 New for EdgeX 2.0. These setting previously were in the Service configuration section specific to Application Services. Now the Service configuration is the same for all EdgeX services. See the general Configuration documentation for more details on the common Service configuration. This section contains the configuration for the internal Webserver. Only need if configuring the Webserver for HTTPS Configuration Default Value Description Protocol http** Indicates the protocol for the webserver to use SecretName blank** Indicates the name of the secret in the Secret Store where the HTTPS secret data resides HTTPSCertName blank** Indicates the key name in the HTTPS secret data that contains the certificate data to use for HTTPS HTTPSKeyName blank** Indicates the key name in the HTTPS secret data that contains the key data to use for HTTPS This section contains the connection information. It is required when using redis for the MessageBus (which is the default) and/or when the Store and Forward capability is enabled. Note that it has a slightly different format than the database section used in the core services configuration. Configuration Default Value Description Type redisdb** Indicates the type of database used. redisdb is the only valid type. Host localhost** Indicates the hostname for the database Port 6379** Indicates the port number for the database Timeout \"30s\"** Indicates the connection timeout for the database This section defines the connect information for the EdgeX Clients and is the same as that used by all EdgeX services, just which clients are needed differs. Please refer to the Note about Clients section for more details. This section defines the Trigger for incoming data. See the Triggers documentation for more details on the inner working of triggers. EdgeX 2.0 For EdgeX 2.0 the Binding section has been renamed to Trigger . Configuration Default Value Description Type edgex-messagebus** Indicates the Trigger binding type. valid values are edgex-messagebus , external-mqtt , http , or This section defines the message bus connect information. Only used for edgex-messagebus binding type EdgeX 2.0 For EdgeX 2.0 the MessageBus section has been renamed to EdgexMessageBus and moved under the Trigger section. The SubscribeTopic setting has changed to SubscribeTopics and moved under the SubscribeHost section of EdgexMessageBus . The PublishTopic has been moved under the PublishHost section of EdgexMessageBus . Configuration Default Value Description Type redis** Indicates the type of MessageBus being used. Valid type are redis , mqtt , or zero SubscribeHost ... This section defines the connection information for subscribing/publishing to the MessageBus Host localhost** Indicates the hostname for subscribing to the MessageBus Port 6379** Indicates the port number for subscribing to the MessageBus Protocol redis** Indicates the protocol number for subscribing to the MessageBus SubscribeTopics edgex/events/#** MessageBus topic(s) to subscribe to. This is a comma separated list of topics. Supports filtering by subscribe topics. See EdgeXMessageBus Trigger for more details. PublishHost ... This section defines the connection information for publishing to the MessageBus Host localhost** Indicates the hostname for publishing to the Message Bus Port 6379** Indicates the port number for publishing to the Message Bus Protocol redis** Indicates the protocol number for publishing to the Message Bus PublishTopic blank** Indicates the topic in which to publish the function pipeline response data, if any. Supports dynamic topic places holders. See EdgeXMessageBus Trigger for more details. Optional ... This section is used for optional configuration specific to the MessageBus type used. Please refer to go-mod-messaging for more details This section defines the external MQTT Broker connect information. Only used for external-mqtt trigger binding type EdgeX 2.0 For EdgeX 2.0 the MqttBroker section has been renamed to ExternalMqtt and moved under the Trigger section. The ExternalMqtt section now has it's own SubscribeTopics and PublishTopic settings. Note external-mqtt is not the default Trigger type, so there are no default values for ExternalMqtt settings beyond those that the Go compiler gives to the empty struct. Some of those default values are not valid and must be specified, i.e. Authmode Configuration Default Value Description Url blank** Fully qualified URL to connect to the MQTT broker, i.e. tcp://localhost:1883 SubscribeTopics blank** MQTT topic(s) to subscribe to. This is a comma separated list of topics PublishTopic blank** MQTT topic to publish the function pipeline response data, if any. Supports dynamic topic places holders. See ExternalMqtt Trigger for more details. ClientId blank** ClientId to connect to the broker with ConnectTimeout blank** Time duration indicating how long to wait before timing out broker connection, i.e \"30s\" AutoReconnect false** Indicates whether or not to retry connection if disconnected KeepAlive 0** Seconds between client ping when no active data flowing to avoid client being disconnected. Must be greater then 2 QOS 0** Quality of Service 0 (At most once), 1 (At least once) or 2 (Exactly once) Retain false** Retain setting for MQTT Connection SkipCertVerify false** Indicates if the certificate verification should be skipped SecretPath blank** Name of the path in secret provider to retrieve your secrets. Must be non-blank. AuthMode blank** Indicates what to use when connecting to the broker. Must be one of \"none\", \"cacert\" , \"usernamepassword\", \"clientcert\". If a CA Cert exists in the SecretPath then it will be used for all modes except \"none\". RetryDuration 600 Indicates how long (in seconds) to wait timing out on the MQTT client creation RetryInterval 5 Indicates the time (in seconds) that will be waited between attempts to create MQTT client Note Authmode=cacert is only needed when client authentication (e.g. usernamepassword ) is not required, but a CA Cert is needed to validate the broker's SSL/TLS cert. Application Settings Custom Structured Configuration [ApplicationSettings] - Is used for custom application settings and is accessed via the ApplicationSettings() API. The ApplicationSettings API returns a map[string] string containing the contents on the ApplicationSetting section of the configuration.toml file. [ApplicationSettings] ApplicationName = \"My Application Service\" EdgeX 2.0 New for EdgeX 2.0 Custom Application Services can now define their own custom structured configuration section in the configuration.toml file. Any additional sections in the TOML are ignore by the SDK when it parses the file for the SDK defined sections. See the Custom Configuration section of the SDK documentation for more details.","title":"Application Service Configuration"},{"location":"microservices/application/GeneralAppServiceConfig/#application-service-configuration","text":"Similar to other EdgeX services, configuration is first determined by the configuration.toml file in the /res folder. Once loaded any environment overrides are applied. If -cp is passed to the application on startup, the SDK will leverage the specific configuration provider (i.e Consul) to push the configuration into the provider and monitor Writeable configuration from there. You will find the configuration under the edgex/appservices/2.0/ key in the provider (i.e Consul). On re-restart the service will pull the configuration from the provider and apply any environment overrides. This section describes the configuration elements that are unique to Application Services Please first refer to the general Configuration documentation for configuration properties common across all EdgeX services. Note * indicates the configuration value can be changed on the fly if using a configuration provider (like Consul). ** indicates the configuration value can be changed but the service must be restarted.","title":"Application Service Configuration"},{"location":"microservices/application/GeneralAppServiceConfig/#writable","text":"The tabs below provide additional entries in the Writable section which are applicable to Application Services. Writable.StoreAndForward Writable.Pipeline Writable.InsecureSecrets Writable.Telemetry The section configures the Store and Forward capability. Please refer to Store and Forward documentation for more details. Configuration Default Value Enabled false* Indicates whether the Store and Forward capability enabled or disabled RetryInterval \"5m\"* Indicates the duration of time to wait before retries, aka Forward MaxRetryCount 10* Indicates whether maximum number of retries of failed data. The failed data is removed after the maximum retries has been exceeded. A value of 0 indicates endless retries. The section configures the Configurable Function Pipeline which is used only by App Service Configurable. Please refer to App Service Configurable - Getting Started section for more details This section defines Insecure Secrets that are used when running in non-secure mode, i.e. when Vault isn't available. This is a dynamic map of configuration, so can empty if no secrets are used or can have as many or few user define secrets. It simulates a Secret Store in non-secure mode. Below are a few examples that are need if using the indicated capabilities. Configuration Default Value Description DB --- This section defines a block of insecure secrets for database credentials when Redis is used for the MessageBus and/or when Store and Forward is enabled and running is non-secure mode. This section is not required if Store and Forward is not enabled and not using Redis for the MessageBus . path redisdb* Indicates the location in the simulated Secret Store where the DB secret resides. DB Secrets --- This section is the collection of DB secret data username blank* Indicates the value for the username when connecting to the database. When running in non-secure mode it is blank . password blank* Indicates the value for the password when connecting to the database. When running in non-secure mode it is blank . http --- This section defines a block of insecure secrets for HTTP Export, i.e HTTPPost function path http* Indicates the location in the simulated Secret Store where the HTTP secret resides. http Secrets --- This section is the collection of HTTP secret data. See Http Export documentation for more details on use of secret data. headervalue undefined* This indicates the name of the secret value to use as the value in the HTTP header. mqtt --- This section defines a block of insecure secrets for MQTT export, i.e. MQTTSecretSend function. path mqtt* Indicates the location in the simulated Secret Store where the MQTT secret reside. mqtt Secrets --- This section is the collection of MQTT secret data. See Mqtt Export documentation for more details on use of secret data. username blank* Indicates the value for the username when connecting to the MQTT broker using usernamepassword authentication mode. Must be configured to the value the MQTT broker is expecting. password blank* Indicates the value for the password when connecting to the MQTT broker using usernamepassword authentication mode. Must be configured to the value the MQTT broker is expecting. cacert blank* Indicates the value (contents) for the CA Certificate when connecting to the MQTT broker using cacert authentication mode. Must be configured to the value the MQTT broker is expecting. clientcert blank* Indicates the value (contents) for the Client Certificate when connecting to the MQTT broker using clientcert authentication mode. Must be configured to the value the MQTT broker is expecting. clientkey blank* Indicates the value (contents) for the Client Key when connecting to the MQTT broker using clientcert authentication mode. Must be configured to the value the MQTT broker is expecting. Property Default Value Description See Writable.Telemetry at Common Configuration for the Telemetry configuration common to all services Metrics Service metrics that the application service collects. Boolean value indicates if reporting of the metric is enabled. Custom metrics are also included here for custom application services that define custom metrics MessagesReceived = false Enable/disable reporting of the built-in MessagesReceived metric PipelineMessagesProcessed = false Enable/disable reporting of the built-in PipelineMessagesProcessed metrics PipelineMessageProcessingTime = false Enable/disable reporting of the built-in PipelineMessageProcessingTime metrics = false Enable/disable reporting of custome application service's custom metric Tags List of arbitrary service level tags to included with every metric that is reported. i.e. Gateway=\"my-iot-gateway\" Edgex 2.2 New for EdgeX 2.2 All application services have a limited set of built-in service metrics and custom application services can define, collect and report their own custome service metrics. See Built-in Application Service Metrics and Custom Application Service section for more detials","title":"Writable"},{"location":"microservices/application/GeneralAppServiceConfig/#not-writable","text":"The tabs below provide additional configuration which are applicable to Application Services that require the service to be restarted after value(s) are changed. HttpServer Database Clients Trigger Trigger EdgeXMessageBus Trigger ExternalMqtt EdgeX 2.0 New for EdgeX 2.0. These setting previously were in the Service configuration section specific to Application Services. Now the Service configuration is the same for all EdgeX services. See the general Configuration documentation for more details on the common Service configuration. This section contains the configuration for the internal Webserver. Only need if configuring the Webserver for HTTPS Configuration Default Value Description Protocol http** Indicates the protocol for the webserver to use SecretName blank** Indicates the name of the secret in the Secret Store where the HTTPS secret data resides HTTPSCertName blank** Indicates the key name in the HTTPS secret data that contains the certificate data to use for HTTPS HTTPSKeyName blank** Indicates the key name in the HTTPS secret data that contains the key data to use for HTTPS This section contains the connection information. It is required when using redis for the MessageBus (which is the default) and/or when the Store and Forward capability is enabled. Note that it has a slightly different format than the database section used in the core services configuration. Configuration Default Value Description Type redisdb** Indicates the type of database used. redisdb is the only valid type. Host localhost** Indicates the hostname for the database Port 6379** Indicates the port number for the database Timeout \"30s\"** Indicates the connection timeout for the database This section defines the connect information for the EdgeX Clients and is the same as that used by all EdgeX services, just which clients are needed differs. Please refer to the Note about Clients section for more details. This section defines the Trigger for incoming data. See the Triggers documentation for more details on the inner working of triggers. EdgeX 2.0 For EdgeX 2.0 the Binding section has been renamed to Trigger . Configuration Default Value Description Type edgex-messagebus** Indicates the Trigger binding type. valid values are edgex-messagebus , external-mqtt , http , or This section defines the message bus connect information. Only used for edgex-messagebus binding type EdgeX 2.0 For EdgeX 2.0 the MessageBus section has been renamed to EdgexMessageBus and moved under the Trigger section. The SubscribeTopic setting has changed to SubscribeTopics and moved under the SubscribeHost section of EdgexMessageBus . The PublishTopic has been moved under the PublishHost section of EdgexMessageBus . Configuration Default Value Description Type redis** Indicates the type of MessageBus being used. Valid type are redis , mqtt , or zero SubscribeHost ... This section defines the connection information for subscribing/publishing to the MessageBus Host localhost** Indicates the hostname for subscribing to the MessageBus Port 6379** Indicates the port number for subscribing to the MessageBus Protocol redis** Indicates the protocol number for subscribing to the MessageBus SubscribeTopics edgex/events/#** MessageBus topic(s) to subscribe to. This is a comma separated list of topics. Supports filtering by subscribe topics. See EdgeXMessageBus Trigger for more details. PublishHost ... This section defines the connection information for publishing to the MessageBus Host localhost** Indicates the hostname for publishing to the Message Bus Port 6379** Indicates the port number for publishing to the Message Bus Protocol redis** Indicates the protocol number for publishing to the Message Bus PublishTopic blank** Indicates the topic in which to publish the function pipeline response data, if any. Supports dynamic topic places holders. See EdgeXMessageBus Trigger for more details. Optional ... This section is used for optional configuration specific to the MessageBus type used. Please refer to go-mod-messaging for more details This section defines the external MQTT Broker connect information. Only used for external-mqtt trigger binding type EdgeX 2.0 For EdgeX 2.0 the MqttBroker section has been renamed to ExternalMqtt and moved under the Trigger section. The ExternalMqtt section now has it's own SubscribeTopics and PublishTopic settings. Note external-mqtt is not the default Trigger type, so there are no default values for ExternalMqtt settings beyond those that the Go compiler gives to the empty struct. Some of those default values are not valid and must be specified, i.e. Authmode Configuration Default Value Description Url blank** Fully qualified URL to connect to the MQTT broker, i.e. tcp://localhost:1883 SubscribeTopics blank** MQTT topic(s) to subscribe to. This is a comma separated list of topics PublishTopic blank** MQTT topic to publish the function pipeline response data, if any. Supports dynamic topic places holders. See ExternalMqtt Trigger for more details. ClientId blank** ClientId to connect to the broker with ConnectTimeout blank** Time duration indicating how long to wait before timing out broker connection, i.e \"30s\" AutoReconnect false** Indicates whether or not to retry connection if disconnected KeepAlive 0** Seconds between client ping when no active data flowing to avoid client being disconnected. Must be greater then 2 QOS 0** Quality of Service 0 (At most once), 1 (At least once) or 2 (Exactly once) Retain false** Retain setting for MQTT Connection SkipCertVerify false** Indicates if the certificate verification should be skipped SecretPath blank** Name of the path in secret provider to retrieve your secrets. Must be non-blank. AuthMode blank** Indicates what to use when connecting to the broker. Must be one of \"none\", \"cacert\" , \"usernamepassword\", \"clientcert\". If a CA Cert exists in the SecretPath then it will be used for all modes except \"none\". RetryDuration 600 Indicates how long (in seconds) to wait timing out on the MQTT client creation RetryInterval 5 Indicates the time (in seconds) that will be waited between attempts to create MQTT client Note Authmode=cacert is only needed when client authentication (e.g. usernamepassword ) is not required, but a CA Cert is needed to validate the broker's SSL/TLS cert. Application Settings Custom Structured Configuration [ApplicationSettings] - Is used for custom application settings and is accessed via the ApplicationSettings() API. The ApplicationSettings API returns a map[string] string containing the contents on the ApplicationSetting section of the configuration.toml file. [ApplicationSettings] ApplicationName = \"My Application Service\" EdgeX 2.0 New for EdgeX 2.0 Custom Application Services can now define their own custom structured configuration section in the configuration.toml file. Any additional sections in the TOML are ignore by the SDK when it parses the file for the SDK defined sections. See the Custom Configuration section of the SDK documentation for more details.","title":"Not Writable"},{"location":"microservices/application/GettingStarted/","text":"Getting Started with Application Services Types of Application Services There are two flavors of Applications Service which are configurable and custom . This section will describe how and when each flavor should be used. Configurable The App Functions SDK has a full suite of built-in features that are accessible via configuration when using the App Service Configurable service. This service is built using the App Functions SDK and uses configuration profiles to define separate distinct instances of the service. The service comes with a few built in profiles for common use cases, but custom profiles can also be used. If your use case needs can be meet with the built-in functionality then the App Service Configurable service is right for you. See the App Service Configurable section for more details. Custom Custom Application Services are needed when use case needs can not be meet with just the built-in functionality. This is when you must develop you own custom Application Service use the App Functions SDK . Typically this is triggered by the use case needing an custom Pipeline Function . See the App Functions SDK section for all the details on the features you custom Application Service can take advantage of. Template To help accelerate the creation of your custom Application Service the App Functions SDK contains a template for new custom Application Services. This template has TODO's in the code and a README that walk you through the creation of your new custom Application Service. See the template README for more details. Triggers Triggers are common to both Configurable and Custom Application Services. The are the next logical area to get familiar with. See the Triggers section for more details. Configuration Finally service configuration is very important to understand for both Configurable and Custom Application Services. The service configuration documentation is broken into two parts. First is the configuration that is common to all EdgeX services and the second is the configuration that is specific to Application Services. See the Common Configuration and Application Service Configuration sections for more details.","title":"Getting Started"},{"location":"microservices/application/GettingStarted/#getting-started-with-application-services","text":"","title":"Getting Started with Application Services"},{"location":"microservices/application/GettingStarted/#types-of-application-services","text":"There are two flavors of Applications Service which are configurable and custom . This section will describe how and when each flavor should be used.","title":"Types of Application Services"},{"location":"microservices/application/GettingStarted/#configurable","text":"The App Functions SDK has a full suite of built-in features that are accessible via configuration when using the App Service Configurable service. This service is built using the App Functions SDK and uses configuration profiles to define separate distinct instances of the service. The service comes with a few built in profiles for common use cases, but custom profiles can also be used. If your use case needs can be meet with the built-in functionality then the App Service Configurable service is right for you. See the App Service Configurable section for more details.","title":"Configurable"},{"location":"microservices/application/GettingStarted/#custom","text":"Custom Application Services are needed when use case needs can not be meet with just the built-in functionality. This is when you must develop you own custom Application Service use the App Functions SDK . Typically this is triggered by the use case needing an custom Pipeline Function . See the App Functions SDK section for all the details on the features you custom Application Service can take advantage of.","title":"Custom"},{"location":"microservices/application/GettingStarted/#template","text":"To help accelerate the creation of your custom Application Service the App Functions SDK contains a template for new custom Application Services. This template has TODO's in the code and a README that walk you through the creation of your new custom Application Service. See the template README for more details.","title":"Template"},{"location":"microservices/application/GettingStarted/#triggers","text":"Triggers are common to both Configurable and Custom Application Services. The are the next logical area to get familiar with. See the Triggers section for more details.","title":"Triggers"},{"location":"microservices/application/GettingStarted/#configuration","text":"Finally service configuration is very important to understand for both Configurable and Custom Application Services. The service configuration documentation is broken into two parts. First is the configuration that is common to all EdgeX services and the second is the configuration that is specific to Application Services. See the Common Configuration and Application Service Configuration sections for more details.","title":"Configuration"},{"location":"microservices/application/Triggers/","text":"Application Service Triggers Introduction Triggers determine how the App Functions Pipeline begins execution. The trigger is determined by the [Trigger] configuration section in the configuration.toml file. Edgex 2.0 For Edgex 2.0 the [Binding] configuration section has been renamed to [Trigger] . The [MessageBus] section has been renamed to EdgexMessageBus and moved under the [Trigger] section. The [MqttBroker] section has been renamed to ExternalMqtt and moved under the [Trigger] section. There are 4 types of Triggers supported in the App Functions SDK which are discussed in this document EdgeX Message Bus - Default Trigger for most use cases as this is how the App Services receive Events from EdgeX Core Data and/or Devices Services External MQTT - Useful when receiving commands from an external/Cloud MQTT broker. HTTP - Useful during development and testing of custom functions. Custom - Allows custom Application Services to implement their own Custom Trigger EdgeX MessageBus Trigger An EdgeX MessageBus trigger will execute the pipeline every time data is received from the configured Edgex MessageBus SubscribeTopics . The EdgeX MessageBus is the central message bus internal to EdgeX and has a specific message envelope that wraps all data published to this message bus. There currently are three implementations of the EdgeX MessageBus available to be used. These are Redis Pub/Sub (default), MQTT and ZeroMQ (ZMQ). The implementation type is selected via the [Trigger.EdgexMessageBus] configuration described below. Type Configuration Edgex 2.0 For EdgeX 2.0 the SubscribeTopic has been renamed to SubscribeTopics and moved under the EdgexMessageBus SubscribeHost section. The PublishTopic has also been moved under the EdgexMessageBus PublishHost section. Also the legacy type of messagebus has been removed. Here's an example: [Trigger] Type = \"edgex-messagebus\" The Type= is set to edgex-messagebus trigger type. The Context function ctx.SetResponseData([]byte outputData) stores the data to send back to the EdgeX MessageBus on the topic specified by the PublishHost PublishTopic= setting. MessageBus Connection Configuration The other piece of configuration required are the connection settings: [Trigger.EdgexMessageBus] Type = \"redis\" # message bus type (i.e \"redis`, `mqtt` or `zero` for ZeroMQ) [Trigger.EdgexMessageBus.SubscribeHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" SubscribeTopics = \"edgex/events/#\" [Trigger.EdgexMessageBus.PublishHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" PublishTopic = \"\" # optional if publishing response back to the MessageBus Edgex 2.0 For Edgex 2.0 the PublishTopic can now have placeholders. See Publish Topic Placeholders section below for more details As stated above there are three EdgeX MessageBus implementations you can choose from. These type values are as follows: redis - for Redis Pub/Sub (Requires Redis running and Core Data and/or Device Services configure to use Redis Pub/Sub) mqtt - for MQTT (Requires a MQTT Broker running and Core Data and/or Device Services configure to use MQTT) zero - for ZeroMQ (No Broker/Service required. Core Data must be configured to use Zero and Device service configure to use REST to Core Data) Edgex 2.0 For Edgex 2.0 Redis is now the default EdgeX MessageBus implementation used. Also, the Redis implementation changed from Redis streams to Redis Pub/Sub , thus the type value changed from redisstreams to redis Important When using ZMQ for the message bus, the Publish Host MUST be different for each publisher to since the they will bind to the specific port. 5563 for example cannot be used to publish since EdgeX Core Data has bound to that port. Similarly, you cannot have two separate instances of the app functions SDK running and publishing to the same port. This is why once Device services started publishing the the EdgeX MessageBus the default was changed to Redis Pub/Sub Note When using MQTT for the message bus, there is additional configuration required for specifying the MQTT specific options. Example Using MQTT Here is example EdgexMessageBus configuration when using MQTT as the message bus: [Trigger.EdgexMessageBus] Type = \"mqtt\" [Trigger.EdgexMessageBus.SubscribeHost] Host = \"localhost\" Port = 1883 Protocol = \"tcp\" SubscribeTopics = \"edgex/events/#\" [Trigger.EdgexMessageBus.PublishHost] Host = \"localhost\" Port = 1883 Protocol = \"tcp\" PublishTopic = \"\" # optional if publishing response back to the MessageBus [Trigger.EdgexMessageBus.Optional] # MQTT Specific options ClientId = \"new-app-service\" Qos = \"0\" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"30\" # Seconds SkipCertVerify = \"false\" authmode = \"none\" # change to \"usernamepassword\", \"clientcert\", or \"cacert\" for secure MQTT messagebus. secretname = \"mqtt-bus\" EdgeX 2.0 New for EdgeX 2.0 is the Secure MessageBus when use the Redis Pub/Sub implementation. See the Secure MessageBus documentation for more details. EdgeX 2.0 Also new for EdgeX 2.0 is the MQTT MessageBus implementation now supports retrieving secrets from the Secret Store for secure MQTT connection, but there is not any facility yet to generate the credentials on first startup and distribute them to all services, as is done with Redis Pub/sub . This MQTT credentials generation and distribution is a future enhancement for EdgeX security services. Filter By Topics EdgeX 2.0 New for EdgeX 2.0 App services now have the capability to filter by EdgeX MessageBus topics rather then using Filter functions in the functions pipeline. Filtering by topic is more efficient since the App Service never receives the data off the MessageBus. Core Data and/or Device Services now publish to multi-level topics that include the profilename , devicename and sourcename . Sources are the commandname or resourcename that generated the Event. The publish topics now look like this: # From Core Data edgex/events/core/// # From Device Services edgex/events/device/// This with App Services capability to have multiple subscriptions allows for multiple filters by subscriptions. The SubscribeTopics setting takes a comma separated list of subscribe topics. Here are a few examples of how to configure the SubscribeTopics setting under the Trigger.EdgexMessageBus.SubscribeHost section to filter by subscriptions using the profile , device and source names from the SNMP Device Service file here : Filter for all Events SubscribeTopics = \"edgex/events/#\" Filter for Events only from a single class of devices (device profile defines a class of device) SubscribeTopics = \"edgex/events/#/trendnet/#\" Filter for Events only from a single actual device SubscribeTopics = \"edgex/events/#/#/trendnet01/#\" Filter for Events from two specific actual devices SubscribeTopics = \"edgex/events/#/#/trendnet01/#, edgex/events/#/#/trendnet02/#\" Filter for Events from two specific sources. SubscribeTopics = \"edgex/events/#/#/#/Uptime, edgex/events/#/#/#/MacAddress\" Note The above examples are for when Redis is used as the EdgeX MessageBus implementation, which is now the default. The Redis implementation uses the # wildcard character for multi-level and single level. The implementation actually converts all # 's to the * 's. The * is the actual wildcard character used by Redis Pub/Sub. In the first example (multi-level) the # is used at the end in the location for where Core Data's and Device Service's publish topics differ. This location will be core when coming from Core Data or device when coming from a Device Service. The additional use of # within the topic, not at the end, (single-level) allows for any Profile , Device or Source when specifying one of the others. Note For the MQTT implementation of the EdgeX MessageBus, the # is also used for the multi-level wildcard, but the single-level wildcard is the + character. So the first and last examples above would be as follows for when using the MQTT implementation SubscribeTopics = \"edgex/events/#\" SubscribeTopics = \"edgex/events/+/trendnet/#\" SubscribeTopics = \"edgex/events/+/+/trendnet01/#\" SubscribeTopics = \"edgex/events/+/+/trendnet01/#, edgex/events/+/+/trendnet02/#\" SubscribeTopics = \"edgex/events/+/+/+/Uptime, edgex/events/+/+/+/MacAddress\" External MQTT Trigger An External MQTT trigger will execute the pipeline every time data is received from an external MQTT broker on the configured SubscribeTopics . Note The data received from the external MQTT broker is not wrapped with any metadata known to EdgeX. The data is handled as JSON or CBOR. The data is assumed to be JSON unless the first byte in the data is not a { or a [ , in which case it is then assumed to be CBOR. Note The data received, encoded as JSON or CBOR, must match the TargetType defined by your application service. The default TargetType is an Edgex Event . See TargetType for more details. Type Configuration Here's an example: [Trigger] Type = \"external-mqtt\" [Trigger.externalmqtt] Url = \"tls://test.mosquitto.org:8884\" SubscribeTopics = \"edgex/#\" ClientId = \"app-external-mqtt-trigger\" Qos = 0 KeepAlive = 10 Retained = false AutoReconnect = true ConnectTimeout = \"30s\" SkipCertVerify = true AuthMode = \"clientcert\" SecretPath = \"external-mqtt\" RetryDuration = 600 RetryInterval = 5 Edgex 2.0 For EdgeX 2.0 the SubscribeTopic has been renamed to SubscribeTopics and moved under the ExternalMqtt section. The PublishTopic has also been moved under the ExternalMqtt section. The Type= is set to external-mqtt . To receive data from the external MQTT Broker you must set your SubscribeTopics= to the appropriate topic(s) that the external publisher is using. You may also designate a PublishTopic= if you wish to publish data back to the external MQTT Broker. The Context function ctx.SetResponseData([]byte outputData) stores the data to send back to the external MQTT Broker on the topic specified by the PublishTopic= setting. Edgex 2.2 Prior to EdgeX 2.2 if AuthMode is set to usernamepassword , clientcert , or cacert and App Service will be run in secure mode, the required credentials must be stored to Secret Store via Vault CLI, REST API, or WEB UI before starting App Service. Otherwise App Service will fail to initialize the External MQTT Trigger and then shutdown because the required credentials do not exist in the Secret Store at the time service starts. Today, you can start App Service and store the required credentials using the App Service API afterwards. If the credentials found in Secret Store cannot satisfy App Service, it will retry for a certain duration and interval. See Application Service Configuration for more information on the configuration of this retry duration and interval. External MQTT Broker Configuration The other piece of configuration required are the MQTT Broker connection settings: [Trigger.ExternalMqtt] Url = \"tcp://localhost:1883\" # fully qualified URL to connect to the MQTT broker SubscribeTopics = \"SomeTopics\" PublishTopic = \"\" # optional if publishing response back to the the External MQTT Broker ClientId = \"AppService\" ConnectTimeout = \"5s\" # 5 seconds AutoReconnect = true KeepAlive = 10 # Seconds (must be 2 or greater) QoS = 0 # Quality of Service 0 (At most once), 1 (At least once) or 2 (Exactly once) Retain = true SkipCertVerify = false SecretPath = \"mqtt-trigger\" AuthMode = \"none\" # Options are \"none\", \"cacert\" , \"usernamepassword\", \"clientcert\". Edgex 2.0 For Edgex 2.0 the PublishTopic can have placeholders. See Publish Topic Placeholders section below for more details HTTP Trigger Designating an HTTP trigger will allow the pipeline to be triggered by a RESTful POST call to http://[host]:[port]/api/v2/trigger/ . Type Configuration Here's an example: [Trigger] Type = \"http\" The Type= is set to http . This will will enable listening to the api/v2/trigger/ endpoint. No other configuration is required. The Context function ctx.SetResponseData([]byte outputData) stores the data to send back as the response to the requestor that originally triggered the HTTP Request. Note The HTTP trigger uses the content-type from the HTTP Header to determine if the data is JSON or CBOR encoded and the optional X-Correlation-ID to set the correlation ID for the request. Note The data received, encoded as JSON or CBOR, must match the TargetType defined by your application service. The default TargetType is an Edgex Event . See TargetType for more details. Custom Triggers Edgex 2.0 New for EdgeX 2.0 It is also possible to define your own trigger and register a factory function for it with the SDK. You can then configure the trigger by registering a factory function to build it along with a name to use in the config file. These triggers can be registered with: service . RegisterCustomTriggerFactory ( \"my-trigger-name\" , myFactoryFunc ) Note You can NOT override trigger names built into the SDK ( \"edgex-messagebus\", \"external-mqtt\", or \"http\") for a custom trigger. The trigger factory function is bound to an instance of a trigger configuration struct that is provided by the SDK: type TriggerConfig struct { Logger logger . LoggingClient ContextBuilder TriggerContextBuilder // Deprecated: use MessageReceived MessageProcessor TriggerMessageProcessor MessageReceived TriggerMessageHandler ConfigLoader TriggerConfigLoader } This type carries a pointer to the internal edgex logger, along with three functions: ContextBuilder builds an interfaces.AppFunctionContext from a message envelope you construct. MessageProcessor (DEPRECATED) exposes a function that sends your message envelope and context built above into the default function pipeline. MessageReceived exposes a function that sends your message envelope and context to any pipelines configured in the EdgeX service. It also takes a function that will be run to process the response for each successful pipeline. Note The context passed in to Received will be cloned for each pipeline configured to run. If a nil context is passed a new one will be initialized from the message. ConfigLoader exposes a function that loads your custom config struct. By default this is done from the primary EdgeX configuration pipeline, and only loads root-level elements. If you need to override these functions it can be done in the factory function registered with the service. The custom trigger constructed here will then need to implement the trigger interface so that the SDK can invoke it: type Trigger interface { Initialize ( wg * sync . WaitGroup , ctx context . Context , background <- chan BackgroundMessage ) ( bootstrap . Deferred , error ) } type BackgroundMessage interface { Message () types . MessageEnvelope Topic () string } This leaves a lot of flexibility for how you want the trigger to behave (for example you could write a trigger to watch for file changes, or run on a timer). Below is a sample implementation of a trigger that reads lines from os.Stdin and pass the captured string through the edgex function pipeline. In this case the target type for the service is set to &[]byte{} . type stdinTrigger struct { tc appsdk . TriggerConfig } func ( t * stdinTrigger ) Initialize ( wg * sync . WaitGroup , ctx context . Context , _ <- chan interfaces . BackgroundMessage ) ( bootstrap . Deferred , error ) { msgs := make ( chan [] byte ) receiveMessage := true responseHandler := func ( ctx AppFunctionContext , pipeline * FunctionPipeline ) { // do stuff } go func () { fmt . Print ( \"> \" ) rdr := bufio . NewReader ( os . Stdin ) for receiveMessage { s , err := rdr . ReadString ( '\\n' ) s = strings . TrimRight ( s , \"\\n\" ) if err != nil { t . tc . Logger . Error ( err . Error ()) continue } msgs <- [] byte ( s ) } }() go func () { for receiveMessage { select { case <- ctx . Done (): receiveMessage = false case m := <- msgs : go func () { env := types . MessageEnvelope { Payload : m , } ctx := t . tc . ContextBuilder ( env ) err := t . tc . MessageReceived ( ctx , env , responseHandler ) if err != nil { t . tc . Logger . Error ( err . Error ()) } }() } } }() return cancel , nil } This trigger can then be registered by calling: appService . RegisterCustomTriggerFactory ( \"custom-stdin\" , func ( config appsdk . TriggerConfig ) ( appsdk . Trigger , error ) { return & stdinTrigger { tc : config , }, nil }) Type Configuration Here's an example: [Trigger] Type = \"custom-stdin\" Now the custom trigger is configured to be used rather than one of the built-in triggers. A complete working example can be found here Publish Topic Placeholders Edgex 2.0 New for EdgeX 2.0 Both the EdgeX MessageBus and the External MQTT triggers support the new Publish Topic Placeholders capability. The configured PublishTopic for either of these triggers can contain placeholders for runtime replacements. The placeholders are replaced with values from the new Context Storage whose key match the placeholder name. Function pipelines can add values to the Context Storage which can then be used as replacement values in the publish topic. If an EdgeX Event is received by the configured trigger the Event's profilename , devicename and sourcename as well as the will be seeded into the Context Storage . See the Context Storage documentation for more details. The Publish Topic Placeholders format is a simple {} that can appear anywhere in the topic multiple times. An error will occur if a specified placeholder does not exist in the Context Storage . Example PublishTopic = \"data/{profilename}/{devicename}/{custom}\" Received Topic Edgex 2.0 New for EdgeX 2.0 The topic the data was received on for EdgeX MessageBus and the External MQTT triggers is now stored in the new Context Storage with the key receivedtopic . This makes it available to pipeline functions via the Context Storage .","title":"Triggers"},{"location":"microservices/application/Triggers/#application-service-triggers","text":"","title":"Application Service Triggers"},{"location":"microservices/application/Triggers/#introduction","text":"Triggers determine how the App Functions Pipeline begins execution. The trigger is determined by the [Trigger] configuration section in the configuration.toml file. Edgex 2.0 For Edgex 2.0 the [Binding] configuration section has been renamed to [Trigger] . The [MessageBus] section has been renamed to EdgexMessageBus and moved under the [Trigger] section. The [MqttBroker] section has been renamed to ExternalMqtt and moved under the [Trigger] section. There are 4 types of Triggers supported in the App Functions SDK which are discussed in this document EdgeX Message Bus - Default Trigger for most use cases as this is how the App Services receive Events from EdgeX Core Data and/or Devices Services External MQTT - Useful when receiving commands from an external/Cloud MQTT broker. HTTP - Useful during development and testing of custom functions. Custom - Allows custom Application Services to implement their own Custom Trigger","title":"Introduction"},{"location":"microservices/application/Triggers/#edgex-messagebus-trigger","text":"An EdgeX MessageBus trigger will execute the pipeline every time data is received from the configured Edgex MessageBus SubscribeTopics . The EdgeX MessageBus is the central message bus internal to EdgeX and has a specific message envelope that wraps all data published to this message bus. There currently are three implementations of the EdgeX MessageBus available to be used. These are Redis Pub/Sub (default), MQTT and ZeroMQ (ZMQ). The implementation type is selected via the [Trigger.EdgexMessageBus] configuration described below.","title":"EdgeX MessageBus Trigger"},{"location":"microservices/application/Triggers/#type-configuration","text":"Edgex 2.0 For EdgeX 2.0 the SubscribeTopic has been renamed to SubscribeTopics and moved under the EdgexMessageBus SubscribeHost section. The PublishTopic has also been moved under the EdgexMessageBus PublishHost section. Also the legacy type of messagebus has been removed. Here's an example: [Trigger] Type = \"edgex-messagebus\" The Type= is set to edgex-messagebus trigger type. The Context function ctx.SetResponseData([]byte outputData) stores the data to send back to the EdgeX MessageBus on the topic specified by the PublishHost PublishTopic= setting.","title":"Type Configuration"},{"location":"microservices/application/Triggers/#messagebus-connection-configuration","text":"The other piece of configuration required are the connection settings: [Trigger.EdgexMessageBus] Type = \"redis\" # message bus type (i.e \"redis`, `mqtt` or `zero` for ZeroMQ) [Trigger.EdgexMessageBus.SubscribeHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" SubscribeTopics = \"edgex/events/#\" [Trigger.EdgexMessageBus.PublishHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" PublishTopic = \"\" # optional if publishing response back to the MessageBus Edgex 2.0 For Edgex 2.0 the PublishTopic can now have placeholders. See Publish Topic Placeholders section below for more details As stated above there are three EdgeX MessageBus implementations you can choose from. These type values are as follows: redis - for Redis Pub/Sub (Requires Redis running and Core Data and/or Device Services configure to use Redis Pub/Sub) mqtt - for MQTT (Requires a MQTT Broker running and Core Data and/or Device Services configure to use MQTT) zero - for ZeroMQ (No Broker/Service required. Core Data must be configured to use Zero and Device service configure to use REST to Core Data) Edgex 2.0 For Edgex 2.0 Redis is now the default EdgeX MessageBus implementation used. Also, the Redis implementation changed from Redis streams to Redis Pub/Sub , thus the type value changed from redisstreams to redis Important When using ZMQ for the message bus, the Publish Host MUST be different for each publisher to since the they will bind to the specific port. 5563 for example cannot be used to publish since EdgeX Core Data has bound to that port. Similarly, you cannot have two separate instances of the app functions SDK running and publishing to the same port. This is why once Device services started publishing the the EdgeX MessageBus the default was changed to Redis Pub/Sub Note When using MQTT for the message bus, there is additional configuration required for specifying the MQTT specific options.","title":"MessageBus Connection Configuration"},{"location":"microservices/application/Triggers/#example-using-mqtt","text":"Here is example EdgexMessageBus configuration when using MQTT as the message bus: [Trigger.EdgexMessageBus] Type = \"mqtt\" [Trigger.EdgexMessageBus.SubscribeHost] Host = \"localhost\" Port = 1883 Protocol = \"tcp\" SubscribeTopics = \"edgex/events/#\" [Trigger.EdgexMessageBus.PublishHost] Host = \"localhost\" Port = 1883 Protocol = \"tcp\" PublishTopic = \"\" # optional if publishing response back to the MessageBus [Trigger.EdgexMessageBus.Optional] # MQTT Specific options ClientId = \"new-app-service\" Qos = \"0\" # Quality of Service values are 0 (At most once), 1 (At least once) or 2 (Exactly once) KeepAlive = \"10\" # Seconds (must be 2 or greater) Retained = \"false\" AutoReconnect = \"true\" ConnectTimeout = \"30\" # Seconds SkipCertVerify = \"false\" authmode = \"none\" # change to \"usernamepassword\", \"clientcert\", or \"cacert\" for secure MQTT messagebus. secretname = \"mqtt-bus\" EdgeX 2.0 New for EdgeX 2.0 is the Secure MessageBus when use the Redis Pub/Sub implementation. See the Secure MessageBus documentation for more details. EdgeX 2.0 Also new for EdgeX 2.0 is the MQTT MessageBus implementation now supports retrieving secrets from the Secret Store for secure MQTT connection, but there is not any facility yet to generate the credentials on first startup and distribute them to all services, as is done with Redis Pub/sub . This MQTT credentials generation and distribution is a future enhancement for EdgeX security services.","title":"Example Using MQTT"},{"location":"microservices/application/Triggers/#filter-by-topics","text":"EdgeX 2.0 New for EdgeX 2.0 App services now have the capability to filter by EdgeX MessageBus topics rather then using Filter functions in the functions pipeline. Filtering by topic is more efficient since the App Service never receives the data off the MessageBus. Core Data and/or Device Services now publish to multi-level topics that include the profilename , devicename and sourcename . Sources are the commandname or resourcename that generated the Event. The publish topics now look like this: # From Core Data edgex/events/core/// # From Device Services edgex/events/device/// This with App Services capability to have multiple subscriptions allows for multiple filters by subscriptions. The SubscribeTopics setting takes a comma separated list of subscribe topics. Here are a few examples of how to configure the SubscribeTopics setting under the Trigger.EdgexMessageBus.SubscribeHost section to filter by subscriptions using the profile , device and source names from the SNMP Device Service file here : Filter for all Events SubscribeTopics = \"edgex/events/#\" Filter for Events only from a single class of devices (device profile defines a class of device) SubscribeTopics = \"edgex/events/#/trendnet/#\" Filter for Events only from a single actual device SubscribeTopics = \"edgex/events/#/#/trendnet01/#\" Filter for Events from two specific actual devices SubscribeTopics = \"edgex/events/#/#/trendnet01/#, edgex/events/#/#/trendnet02/#\" Filter for Events from two specific sources. SubscribeTopics = \"edgex/events/#/#/#/Uptime, edgex/events/#/#/#/MacAddress\" Note The above examples are for when Redis is used as the EdgeX MessageBus implementation, which is now the default. The Redis implementation uses the # wildcard character for multi-level and single level. The implementation actually converts all # 's to the * 's. The * is the actual wildcard character used by Redis Pub/Sub. In the first example (multi-level) the # is used at the end in the location for where Core Data's and Device Service's publish topics differ. This location will be core when coming from Core Data or device when coming from a Device Service. The additional use of # within the topic, not at the end, (single-level) allows for any Profile , Device or Source when specifying one of the others. Note For the MQTT implementation of the EdgeX MessageBus, the # is also used for the multi-level wildcard, but the single-level wildcard is the + character. So the first and last examples above would be as follows for when using the MQTT implementation SubscribeTopics = \"edgex/events/#\" SubscribeTopics = \"edgex/events/+/trendnet/#\" SubscribeTopics = \"edgex/events/+/+/trendnet01/#\" SubscribeTopics = \"edgex/events/+/+/trendnet01/#, edgex/events/+/+/trendnet02/#\" SubscribeTopics = \"edgex/events/+/+/+/Uptime, edgex/events/+/+/+/MacAddress\"","title":"Filter By Topics"},{"location":"microservices/application/Triggers/#external-mqtt-trigger","text":"An External MQTT trigger will execute the pipeline every time data is received from an external MQTT broker on the configured SubscribeTopics . Note The data received from the external MQTT broker is not wrapped with any metadata known to EdgeX. The data is handled as JSON or CBOR. The data is assumed to be JSON unless the first byte in the data is not a { or a [ , in which case it is then assumed to be CBOR. Note The data received, encoded as JSON or CBOR, must match the TargetType defined by your application service. The default TargetType is an Edgex Event . See TargetType for more details.","title":"External MQTT Trigger"},{"location":"microservices/application/Triggers/#type-configuration_1","text":"Here's an example: [Trigger] Type = \"external-mqtt\" [Trigger.externalmqtt] Url = \"tls://test.mosquitto.org:8884\" SubscribeTopics = \"edgex/#\" ClientId = \"app-external-mqtt-trigger\" Qos = 0 KeepAlive = 10 Retained = false AutoReconnect = true ConnectTimeout = \"30s\" SkipCertVerify = true AuthMode = \"clientcert\" SecretPath = \"external-mqtt\" RetryDuration = 600 RetryInterval = 5 Edgex 2.0 For EdgeX 2.0 the SubscribeTopic has been renamed to SubscribeTopics and moved under the ExternalMqtt section. The PublishTopic has also been moved under the ExternalMqtt section. The Type= is set to external-mqtt . To receive data from the external MQTT Broker you must set your SubscribeTopics= to the appropriate topic(s) that the external publisher is using. You may also designate a PublishTopic= if you wish to publish data back to the external MQTT Broker. The Context function ctx.SetResponseData([]byte outputData) stores the data to send back to the external MQTT Broker on the topic specified by the PublishTopic= setting. Edgex 2.2 Prior to EdgeX 2.2 if AuthMode is set to usernamepassword , clientcert , or cacert and App Service will be run in secure mode, the required credentials must be stored to Secret Store via Vault CLI, REST API, or WEB UI before starting App Service. Otherwise App Service will fail to initialize the External MQTT Trigger and then shutdown because the required credentials do not exist in the Secret Store at the time service starts. Today, you can start App Service and store the required credentials using the App Service API afterwards. If the credentials found in Secret Store cannot satisfy App Service, it will retry for a certain duration and interval. See Application Service Configuration for more information on the configuration of this retry duration and interval.","title":"Type Configuration"},{"location":"microservices/application/Triggers/#external-mqtt-broker-configuration","text":"The other piece of configuration required are the MQTT Broker connection settings: [Trigger.ExternalMqtt] Url = \"tcp://localhost:1883\" # fully qualified URL to connect to the MQTT broker SubscribeTopics = \"SomeTopics\" PublishTopic = \"\" # optional if publishing response back to the the External MQTT Broker ClientId = \"AppService\" ConnectTimeout = \"5s\" # 5 seconds AutoReconnect = true KeepAlive = 10 # Seconds (must be 2 or greater) QoS = 0 # Quality of Service 0 (At most once), 1 (At least once) or 2 (Exactly once) Retain = true SkipCertVerify = false SecretPath = \"mqtt-trigger\" AuthMode = \"none\" # Options are \"none\", \"cacert\" , \"usernamepassword\", \"clientcert\". Edgex 2.0 For Edgex 2.0 the PublishTopic can have placeholders. See Publish Topic Placeholders section below for more details","title":"External MQTT Broker Configuration"},{"location":"microservices/application/Triggers/#http-trigger","text":"Designating an HTTP trigger will allow the pipeline to be triggered by a RESTful POST call to http://[host]:[port]/api/v2/trigger/ .","title":"HTTP Trigger"},{"location":"microservices/application/Triggers/#type-configuration_2","text":"Here's an example: [Trigger] Type = \"http\" The Type= is set to http . This will will enable listening to the api/v2/trigger/ endpoint. No other configuration is required. The Context function ctx.SetResponseData([]byte outputData) stores the data to send back as the response to the requestor that originally triggered the HTTP Request. Note The HTTP trigger uses the content-type from the HTTP Header to determine if the data is JSON or CBOR encoded and the optional X-Correlation-ID to set the correlation ID for the request. Note The data received, encoded as JSON or CBOR, must match the TargetType defined by your application service. The default TargetType is an Edgex Event . See TargetType for more details.","title":"Type Configuration"},{"location":"microservices/application/Triggers/#custom-triggers","text":"Edgex 2.0 New for EdgeX 2.0 It is also possible to define your own trigger and register a factory function for it with the SDK. You can then configure the trigger by registering a factory function to build it along with a name to use in the config file. These triggers can be registered with: service . RegisterCustomTriggerFactory ( \"my-trigger-name\" , myFactoryFunc ) Note You can NOT override trigger names built into the SDK ( \"edgex-messagebus\", \"external-mqtt\", or \"http\") for a custom trigger. The trigger factory function is bound to an instance of a trigger configuration struct that is provided by the SDK: type TriggerConfig struct { Logger logger . LoggingClient ContextBuilder TriggerContextBuilder // Deprecated: use MessageReceived MessageProcessor TriggerMessageProcessor MessageReceived TriggerMessageHandler ConfigLoader TriggerConfigLoader } This type carries a pointer to the internal edgex logger, along with three functions: ContextBuilder builds an interfaces.AppFunctionContext from a message envelope you construct. MessageProcessor (DEPRECATED) exposes a function that sends your message envelope and context built above into the default function pipeline. MessageReceived exposes a function that sends your message envelope and context to any pipelines configured in the EdgeX service. It also takes a function that will be run to process the response for each successful pipeline. Note The context passed in to Received will be cloned for each pipeline configured to run. If a nil context is passed a new one will be initialized from the message. ConfigLoader exposes a function that loads your custom config struct. By default this is done from the primary EdgeX configuration pipeline, and only loads root-level elements. If you need to override these functions it can be done in the factory function registered with the service. The custom trigger constructed here will then need to implement the trigger interface so that the SDK can invoke it: type Trigger interface { Initialize ( wg * sync . WaitGroup , ctx context . Context , background <- chan BackgroundMessage ) ( bootstrap . Deferred , error ) } type BackgroundMessage interface { Message () types . MessageEnvelope Topic () string } This leaves a lot of flexibility for how you want the trigger to behave (for example you could write a trigger to watch for file changes, or run on a timer). Below is a sample implementation of a trigger that reads lines from os.Stdin and pass the captured string through the edgex function pipeline. In this case the target type for the service is set to &[]byte{} . type stdinTrigger struct { tc appsdk . TriggerConfig } func ( t * stdinTrigger ) Initialize ( wg * sync . WaitGroup , ctx context . Context , _ <- chan interfaces . BackgroundMessage ) ( bootstrap . Deferred , error ) { msgs := make ( chan [] byte ) receiveMessage := true responseHandler := func ( ctx AppFunctionContext , pipeline * FunctionPipeline ) { // do stuff } go func () { fmt . Print ( \"> \" ) rdr := bufio . NewReader ( os . Stdin ) for receiveMessage { s , err := rdr . ReadString ( '\\n' ) s = strings . TrimRight ( s , \"\\n\" ) if err != nil { t . tc . Logger . Error ( err . Error ()) continue } msgs <- [] byte ( s ) } }() go func () { for receiveMessage { select { case <- ctx . Done (): receiveMessage = false case m := <- msgs : go func () { env := types . MessageEnvelope { Payload : m , } ctx := t . tc . ContextBuilder ( env ) err := t . tc . MessageReceived ( ctx , env , responseHandler ) if err != nil { t . tc . Logger . Error ( err . Error ()) } }() } } }() return cancel , nil } This trigger can then be registered by calling: appService . RegisterCustomTriggerFactory ( \"custom-stdin\" , func ( config appsdk . TriggerConfig ) ( appsdk . Trigger , error ) { return & stdinTrigger { tc : config , }, nil })","title":"Custom Triggers"},{"location":"microservices/application/Triggers/#type-configuration_3","text":"Here's an example: [Trigger] Type = \"custom-stdin\" Now the custom trigger is configured to be used rather than one of the built-in triggers. A complete working example can be found here","title":"Type Configuration"},{"location":"microservices/application/Triggers/#publish-topic-placeholders","text":"Edgex 2.0 New for EdgeX 2.0 Both the EdgeX MessageBus and the External MQTT triggers support the new Publish Topic Placeholders capability. The configured PublishTopic for either of these triggers can contain placeholders for runtime replacements. The placeholders are replaced with values from the new Context Storage whose key match the placeholder name. Function pipelines can add values to the Context Storage which can then be used as replacement values in the publish topic. If an EdgeX Event is received by the configured trigger the Event's profilename , devicename and sourcename as well as the will be seeded into the Context Storage . See the Context Storage documentation for more details. The Publish Topic Placeholders format is a simple {} that can appear anywhere in the topic multiple times. An error will occur if a specified placeholder does not exist in the Context Storage .","title":"Publish Topic Placeholders"},{"location":"microservices/application/Triggers/#example","text":"PublishTopic = \"data/{profilename}/{devicename}/{custom}\"","title":"Example"},{"location":"microservices/application/Triggers/#received-topic","text":"Edgex 2.0 New for EdgeX 2.0 The topic the data was received on for EdgeX MessageBus and the External MQTT triggers is now stored in the new Context Storage with the key receivedtopic . This makes it available to pipeline functions via the Context Storage .","title":"Received Topic"},{"location":"microservices/application/V2Migration/","text":"V2 Migration Guide EdgeX 2.0 For the EdgeX 2.0 (Ireland) release there are many backward breaking changes. These changes require custom Application Services and custom profiles (app-service-configurable) to be migrated. This section outlines the necessary steps for this migration. Custom Application Services Configuration The migration of any Application Service's configuration starts with migrating configuration common to all EdgeX services. See the V2 Migration of Common Configuration section for details. The remainder of this section focuses on configuration specific to Application Services. SecretStoreExclusive The SecretStoreExclusive section has been removed in EdgeX 2.0. With EdgeX 2.0 all SecretStores are exclusive, so the existing SecretStore section is all that is required. Services requiring known secrets such as redisdb must inform the Security SecretStore Setup service (via environment variables) that the application service requires the secret added to its SecretStore. See the Configuring Add-on Services section for more details. Clients The client used for the version validation check has changed to being from Core Metadata, rather than Core Data. This is because Core Data is now optional when persistence isn't required since all Device Services publish directly to the EdgeX MessageBus. The configuration for Core Metadata is the only Clients entry required, all other (see below) are optional based on use case needs. Note The port numbers for all EdgeX services have changed which must be reflected in the Clients configuration. Please see the Default Service Ports section for complete list of the new port assignments. Example - Core Metadata client configuration [Clients] [Clients.core-metadata] Protocol = \"http\" Host = \"localhost\" Port = 59881 Example - All available clients configured with new port numbers [Clients] # Used for version check on start-up # Also used for DeviceService, DeviceProfile and Device clients [Clients.core-metadata] Protocol = \"http\" Host = \"localhost\" Port = 59881 # Used for Event client which is used by PushToCoreData function [Clients.core-data] Protocol = \"http\" Host = \"localhost\" Port = 59880 # Used for Command client [Clients.core-command] Protocol = \"http\" Host = \"localhost\" Port = 59882 # Used for Notification and Subscription clients [Clients.support-notifications] Protocol = \"http\" Host = \"localhost\" Port = 59860 Trigger The Trigger section (previously named Binding ) has been restructured with EdgexMessageBus (previously named MessageBus ) and ExternalMqtt (previously named MqttBroker ) moved under it. The SubscribeTopics (previously named SubscribeTopic ) has been moved under the EdgexMessageBus.SubscribeHost and ExternalMqtt sections. The PublishTopic has been moved under the EdgexMessageBus.PublishHost and ExternalMqtt sections. EdgeX MessageBus If your Application Service is using the EdgeX MessageBus trigger, you can then simply copy the complete Trigger configuration from the example below and tweak it as needed. Example - EdgeX MessageBus trigger configuration [Trigger] Type = \"edgex-messagebus\" [Trigger.EdgexMessageBus] Type = \"redis\" [Trigger.EdgexMessageBus.SubscribeHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" SubscribeTopics = \"edgex/events/#\" [Trigger.EdgexMessageBus.PublishHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" PublishTopic = \"example\" [Trigger.EdgexMessageBus.Optional] AuthMode = \"usernamepassword\" # required for redis messagebus (secure or insecure). SecretName = \"redisdb\" From the above example you can see the improved structure and the following changes: Default EdgexMessageBus type has changed from ZeroMQ to Redis . Type value for Redis has changed from redistreams to redis . This is because the implementation no longer uses Redis Streams. It now uses Redis Pub/Sub. SubscribeTopics is now plural since it now accepts a comma separated list of topics. The default value uses a multi-level topic with a wild card. This is because Core Data and Device Services now publish to a multi-level topics which have edgex/events as their base. This allows Application Services to filter by topic rather then receive the data and then filter it out via a pipeline filter function. See the Filter By Topics section for more details. The EdgeX MessageBus using Redis is a Secure MessageBus, thus the addition of the AuthMode and SecretName settings which allow the credentials to be pulled from the service's SecretStore. See the Secure MessageBus secure for more details. External MQTT If your Application service is using the External MQTT trigger do the following: Move your existing MqttBroker configuration under the Trigger section (renaming it to ExternalMqtt ) Move your SubscribeTopic (renaming it to SubscribeTopics ) under the ExternalMqtt section. Move your PublishTopic under the ExternalMqtt section. Example - External MQTT trigger configuration [Trigger] Type = \"external-mqtt\" [Trigger.ExternalMqtt] Url = \"tcp://broker.hivemq.com:1883\" SubscribeTopics = \"edgex-trigger\" PublishTopic = \"edgex-trigger-response\" ClientId = \"app-my-service\" ConnectTimeout = \"30s\" AutoReconnect = false KeepAlive = 60 QoS = 0 Retain = false SkipCertVerify = false SecretPath = \"\" AuthMode = \"none\" HTTP The HTTP trigger configuration has not changed beyond the renaming of Binding to Trigger . Example - HTTP trigger configuration [Trigger] Type = \"http\" Code Dependencies You first need to update the go.mod file to specify go 1.16 and the V2 versions of the App Functions SDK and any EdgeX go-mods directly used by your service. Note the extra /v2 for the modules. Example go.mod for V2 module < your service > go 1.16 require ( github . com / edgexfoundry / app - functions - sdk - go / v2 v2 .0.0 github . com / edgexfoundry / go - mod - core - contracts / v2 v2 .0.0 ) Once that is complete then the import statements for these dependencies must be updated to include the /v2 in the path. Example import statements for V2 import ( ... \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg/interfaces\" \"github.com/edgexfoundry/go-mod-core-contracts/v2/dtos\" ) New APIs Next changes you will encounter in your code are that the AppFunctionsSDK and Context structs have been abstracted into the new ApplicationService and AppFunctionContext APIs. See the Application Service API and App Function Context API sections for complete details on these new APIs. The following sections cover migrating your code for these new APIs. main() The following changes to your main() function will be necessary. Create and Initialize Your main() will change to use a factory function to create and initialize the Application Service instance, rather than create instance of AppFunctionsSDK and call Initialize() Example - Create Application Service instance const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppService ( serviceKey ) if ! ok { os . Exit ( - 1 ) } Example - Create Application Service instance with Target Type specified const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppServiceWithTargetType ( serviceKey , & [] byte {}) if ! ok { os . Exit ( - 1 ) } Since the factory function logs all errors, all you need to do is exit if it returns false . Logging Client The Logging client is now accessible from the service.LoggingClient() API. New extended Logging Client API The Logging Client API now has formatted versions of all the logging APIs, which are Infof , Debugf , Tracef , Warnf and Errorf . If your code uses fmt.Sprintf to format your log messages then it can now be simplified by using these new APIs. Application Settings The access functions for retrieving the service's custom Application Settings ( ApplicationSettings , GetAppSettingStrings , and GetAppSetting ) have not changed. An improved capability to have structured custom configuration has been added. See the Structure Custom Configuration section for more details. Functions Pipeline Setting the Functions Pipeline has not changed, but the name of some built in functions have changed and new ones have been added. See the Built-In Pipeline Functions section for more details. Example - Setting Functions Pipeline if err := service . SetFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , transforms . NewConversion (). TransformToXML , transforms . NewHTTPSender ( exportUrl , \"application/xml\" , false ). HTTPPost , ); err != nil { lc . Errorf ( \"SetFunctionsPipeline returned error: %s\" , err . Error ()) os . Exit ( - 1 ) } MakeItRun The MakeItRun API has not changed. Example - Call to MakeItRun err = service . MakeItRun () if err != nil { lc . Errorf ( \"MakeItRun returned error: %s\" , err . Error ()) os . Exit ( - 1 ) } Custom Pipeline Functions Pipeline Function signature The major change to custom Pipeline Functions for EdgeX 2.0 is the new function signature which drives all the other changes. Example - New Pipeline Function signature type AppFunction = func ( ctx AppFunctionContext , data interface {}) ( bool , interface {}) This function signature passes in an instance of the new AppFunctionContext API for the context and now has only a single data instance for the function to operate on. Return Values The definitions for the Pipeline Function return values have not changed. Data The data passed in either set to a single instance for the function to process or nil. Now you no longer need to check the length of the incoming data. Example if data == nil { return false , errors . New ( \"No Data Received\" ) } Logging Client The Logging client is now accessible from the ctx.LoggingClient() API. Clients The available clients have changed with a few additions and ValueDescriptorClient has been removed. See the Context Clients section for complete list of available clients. ResponseData The SetResponseData and ResponseData APIs replace the previous Complete function and direct access to the OutputData field. ResponseContentType The SetResponseContentType and ResponseContentType APIs replace the previous direct access to the ResponseContentType field. RetryData The SetRetryData API replaces the SetRetryData function and direct access to the RetryData field. MarkAsPushed The MarkAsPushed capability has been removed PushToCore The PushToCore API replaces the PushToCoreData function. The API signature has changed. See the PushToCore section for more details. New Capabilities Some new capabilities have been added to the new AppFunctionContext API. See the App Function Context API section for complete details. App Service Configurable Profiles Custom profiles used with App Service Configurable are configuration files. These follow the same migration above for custom Application Service configuration , except for the Configurable Functions Pipeline items. The following are the changes for the Configurable Functions Pipeline: FilterByValueDescriptor changed to FilterByResourceName . See the FilterByResourceName section for details. TransformToXML and TransformToJSON have been collapsed into Transform with additional parameters. See the Transform section for more details. CompressWithGZIP and CompressWithZLIB have been collapsed into Compress with additional parameters. See the Compress section for more details. EncryptWithAES has been changed to Encrypt with additional parameters. See the Encrypt section for more details. BatchByCount , BatchByTime and BatchByTimeAndCount have been collapsed into Batch with additional parameters. See the Batch section for more details. SetOutputData has been renamed to SetResponseData . See the SetResponseData section for more details. PushToCore parameters have changed. See the PushToCore section for more details. HTTPPost , HTTPPostJSON , HTTPPostXML , HTTPPut , HTTPPutJSON and HTTPPutXML have been collapsed into HTTPExport with additional parameters. See the HTTPExport section for more details. MQTTSecretSend has been renamed to MQTTExport with additional parameters. See the MQTTExport section for more details. MarkAsPushed has been removed. The mark as push capability has been removed from Core Data, which this depended on. MQTTSend has been removed. This has been replaced by MQTTExport . See the MQTTExport section for more details. FilterByProfileName and FilterBySourceName have been added. See the FilterByProfileName and FilterBySourceName sections for more details. Ability to define multiple instances of the same Configurable Pipeline Function has been added. See the Multiple Instances of Function section for more details.","title":"V2 Migration Guide"},{"location":"microservices/application/V2Migration/#v2-migration-guide","text":"EdgeX 2.0 For the EdgeX 2.0 (Ireland) release there are many backward breaking changes. These changes require custom Application Services and custom profiles (app-service-configurable) to be migrated. This section outlines the necessary steps for this migration.","title":"V2 Migration Guide"},{"location":"microservices/application/V2Migration/#custom-application-services","text":"","title":"Custom Application Services"},{"location":"microservices/application/V2Migration/#configuration","text":"The migration of any Application Service's configuration starts with migrating configuration common to all EdgeX services. See the V2 Migration of Common Configuration section for details. The remainder of this section focuses on configuration specific to Application Services.","title":"Configuration"},{"location":"microservices/application/V2Migration/#secretstoreexclusive","text":"The SecretStoreExclusive section has been removed in EdgeX 2.0. With EdgeX 2.0 all SecretStores are exclusive, so the existing SecretStore section is all that is required. Services requiring known secrets such as redisdb must inform the Security SecretStore Setup service (via environment variables) that the application service requires the secret added to its SecretStore. See the Configuring Add-on Services section for more details.","title":"SecretStoreExclusive"},{"location":"microservices/application/V2Migration/#clients","text":"The client used for the version validation check has changed to being from Core Metadata, rather than Core Data. This is because Core Data is now optional when persistence isn't required since all Device Services publish directly to the EdgeX MessageBus. The configuration for Core Metadata is the only Clients entry required, all other (see below) are optional based on use case needs. Note The port numbers for all EdgeX services have changed which must be reflected in the Clients configuration. Please see the Default Service Ports section for complete list of the new port assignments. Example - Core Metadata client configuration [Clients] [Clients.core-metadata] Protocol = \"http\" Host = \"localhost\" Port = 59881 Example - All available clients configured with new port numbers [Clients] # Used for version check on start-up # Also used for DeviceService, DeviceProfile and Device clients [Clients.core-metadata] Protocol = \"http\" Host = \"localhost\" Port = 59881 # Used for Event client which is used by PushToCoreData function [Clients.core-data] Protocol = \"http\" Host = \"localhost\" Port = 59880 # Used for Command client [Clients.core-command] Protocol = \"http\" Host = \"localhost\" Port = 59882 # Used for Notification and Subscription clients [Clients.support-notifications] Protocol = \"http\" Host = \"localhost\" Port = 59860","title":"Clients"},{"location":"microservices/application/V2Migration/#trigger","text":"The Trigger section (previously named Binding ) has been restructured with EdgexMessageBus (previously named MessageBus ) and ExternalMqtt (previously named MqttBroker ) moved under it. The SubscribeTopics (previously named SubscribeTopic ) has been moved under the EdgexMessageBus.SubscribeHost and ExternalMqtt sections. The PublishTopic has been moved under the EdgexMessageBus.PublishHost and ExternalMqtt sections.","title":"Trigger"},{"location":"microservices/application/V2Migration/#edgex-messagebus","text":"If your Application Service is using the EdgeX MessageBus trigger, you can then simply copy the complete Trigger configuration from the example below and tweak it as needed. Example - EdgeX MessageBus trigger configuration [Trigger] Type = \"edgex-messagebus\" [Trigger.EdgexMessageBus] Type = \"redis\" [Trigger.EdgexMessageBus.SubscribeHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" SubscribeTopics = \"edgex/events/#\" [Trigger.EdgexMessageBus.PublishHost] Host = \"localhost\" Port = 6379 Protocol = \"redis\" PublishTopic = \"example\" [Trigger.EdgexMessageBus.Optional] AuthMode = \"usernamepassword\" # required for redis messagebus (secure or insecure). SecretName = \"redisdb\" From the above example you can see the improved structure and the following changes: Default EdgexMessageBus type has changed from ZeroMQ to Redis . Type value for Redis has changed from redistreams to redis . This is because the implementation no longer uses Redis Streams. It now uses Redis Pub/Sub. SubscribeTopics is now plural since it now accepts a comma separated list of topics. The default value uses a multi-level topic with a wild card. This is because Core Data and Device Services now publish to a multi-level topics which have edgex/events as their base. This allows Application Services to filter by topic rather then receive the data and then filter it out via a pipeline filter function. See the Filter By Topics section for more details. The EdgeX MessageBus using Redis is a Secure MessageBus, thus the addition of the AuthMode and SecretName settings which allow the credentials to be pulled from the service's SecretStore. See the Secure MessageBus secure for more details.","title":"EdgeX MessageBus"},{"location":"microservices/application/V2Migration/#external-mqtt","text":"If your Application service is using the External MQTT trigger do the following: Move your existing MqttBroker configuration under the Trigger section (renaming it to ExternalMqtt ) Move your SubscribeTopic (renaming it to SubscribeTopics ) under the ExternalMqtt section. Move your PublishTopic under the ExternalMqtt section. Example - External MQTT trigger configuration [Trigger] Type = \"external-mqtt\" [Trigger.ExternalMqtt] Url = \"tcp://broker.hivemq.com:1883\" SubscribeTopics = \"edgex-trigger\" PublishTopic = \"edgex-trigger-response\" ClientId = \"app-my-service\" ConnectTimeout = \"30s\" AutoReconnect = false KeepAlive = 60 QoS = 0 Retain = false SkipCertVerify = false SecretPath = \"\" AuthMode = \"none\"","title":"External MQTT"},{"location":"microservices/application/V2Migration/#http","text":"The HTTP trigger configuration has not changed beyond the renaming of Binding to Trigger . Example - HTTP trigger configuration [Trigger] Type = \"http\"","title":"HTTP"},{"location":"microservices/application/V2Migration/#code","text":"","title":"Code"},{"location":"microservices/application/V2Migration/#dependencies","text":"You first need to update the go.mod file to specify go 1.16 and the V2 versions of the App Functions SDK and any EdgeX go-mods directly used by your service. Note the extra /v2 for the modules. Example go.mod for V2 module < your service > go 1.16 require ( github . com / edgexfoundry / app - functions - sdk - go / v2 v2 .0.0 github . com / edgexfoundry / go - mod - core - contracts / v2 v2 .0.0 ) Once that is complete then the import statements for these dependencies must be updated to include the /v2 in the path. Example import statements for V2 import ( ... \"github.com/edgexfoundry/app-functions-sdk-go/v2/pkg/interfaces\" \"github.com/edgexfoundry/go-mod-core-contracts/v2/dtos\" )","title":"Dependencies"},{"location":"microservices/application/V2Migration/#new-apis","text":"Next changes you will encounter in your code are that the AppFunctionsSDK and Context structs have been abstracted into the new ApplicationService and AppFunctionContext APIs. See the Application Service API and App Function Context API sections for complete details on these new APIs. The following sections cover migrating your code for these new APIs.","title":"New APIs"},{"location":"microservices/application/V2Migration/#main","text":"The following changes to your main() function will be necessary.","title":"main()"},{"location":"microservices/application/V2Migration/#create-and-initialize","text":"Your main() will change to use a factory function to create and initialize the Application Service instance, rather than create instance of AppFunctionsSDK and call Initialize() Example - Create Application Service instance const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppService ( serviceKey ) if ! ok { os . Exit ( - 1 ) } Example - Create Application Service instance with Target Type specified const serviceKey = \"app-myservice\" ... service , ok := pkg . NewAppServiceWithTargetType ( serviceKey , & [] byte {}) if ! ok { os . Exit ( - 1 ) } Since the factory function logs all errors, all you need to do is exit if it returns false .","title":"Create and Initialize"},{"location":"microservices/application/V2Migration/#logging-client","text":"The Logging client is now accessible from the service.LoggingClient() API. New extended Logging Client API The Logging Client API now has formatted versions of all the logging APIs, which are Infof , Debugf , Tracef , Warnf and Errorf . If your code uses fmt.Sprintf to format your log messages then it can now be simplified by using these new APIs.","title":"Logging Client"},{"location":"microservices/application/V2Migration/#application-settings","text":"The access functions for retrieving the service's custom Application Settings ( ApplicationSettings , GetAppSettingStrings , and GetAppSetting ) have not changed. An improved capability to have structured custom configuration has been added. See the Structure Custom Configuration section for more details.","title":"Application Settings"},{"location":"microservices/application/V2Migration/#functions-pipeline","text":"Setting the Functions Pipeline has not changed, but the name of some built in functions have changed and new ones have been added. See the Built-In Pipeline Functions section for more details. Example - Setting Functions Pipeline if err := service . SetFunctionsPipeline ( transforms . NewFilterFor ( deviceNames ). FilterByDeviceName , transforms . NewConversion (). TransformToXML , transforms . NewHTTPSender ( exportUrl , \"application/xml\" , false ). HTTPPost , ); err != nil { lc . Errorf ( \"SetFunctionsPipeline returned error: %s\" , err . Error ()) os . Exit ( - 1 ) }","title":"Functions Pipeline"},{"location":"microservices/application/V2Migration/#makeitrun","text":"The MakeItRun API has not changed. Example - Call to MakeItRun err = service . MakeItRun () if err != nil { lc . Errorf ( \"MakeItRun returned error: %s\" , err . Error ()) os . Exit ( - 1 ) }","title":"MakeItRun"},{"location":"microservices/application/V2Migration/#custom-pipeline-functions","text":"","title":"Custom Pipeline Functions"},{"location":"microservices/application/V2Migration/#pipeline-function-signature","text":"The major change to custom Pipeline Functions for EdgeX 2.0 is the new function signature which drives all the other changes. Example - New Pipeline Function signature type AppFunction = func ( ctx AppFunctionContext , data interface {}) ( bool , interface {}) This function signature passes in an instance of the new AppFunctionContext API for the context and now has only a single data instance for the function to operate on.","title":"Pipeline Function signature"},{"location":"microservices/application/V2Migration/#return-values","text":"The definitions for the Pipeline Function return values have not changed.","title":"Return Values"},{"location":"microservices/application/V2Migration/#data","text":"The data passed in either set to a single instance for the function to process or nil. Now you no longer need to check the length of the incoming data. Example if data == nil { return false , errors . New ( \"No Data Received\" ) }","title":"Data"},{"location":"microservices/application/V2Migration/#logging-client_1","text":"The Logging client is now accessible from the ctx.LoggingClient() API.","title":"Logging Client"},{"location":"microservices/application/V2Migration/#clients_1","text":"The available clients have changed with a few additions and ValueDescriptorClient has been removed. See the Context Clients section for complete list of available clients.","title":"Clients"},{"location":"microservices/application/V2Migration/#responsedata","text":"The SetResponseData and ResponseData APIs replace the previous Complete function and direct access to the OutputData field.","title":"ResponseData"},{"location":"microservices/application/V2Migration/#responsecontenttype","text":"The SetResponseContentType and ResponseContentType APIs replace the previous direct access to the ResponseContentType field.","title":"ResponseContentType"},{"location":"microservices/application/V2Migration/#retrydata","text":"The SetRetryData API replaces the SetRetryData function and direct access to the RetryData field.","title":"RetryData"},{"location":"microservices/application/V2Migration/#markaspushed","text":"The MarkAsPushed capability has been removed","title":"MarkAsPushed"},{"location":"microservices/application/V2Migration/#pushtocore","text":"The PushToCore API replaces the PushToCoreData function. The API signature has changed. See the PushToCore section for more details.","title":"PushToCore"},{"location":"microservices/application/V2Migration/#new-capabilities","text":"Some new capabilities have been added to the new AppFunctionContext API. See the App Function Context API section for complete details.","title":"New Capabilities"},{"location":"microservices/application/V2Migration/#app-service-configurable-profiles","text":"Custom profiles used with App Service Configurable are configuration files. These follow the same migration above for custom Application Service configuration , except for the Configurable Functions Pipeline items. The following are the changes for the Configurable Functions Pipeline: FilterByValueDescriptor changed to FilterByResourceName . See the FilterByResourceName section for details. TransformToXML and TransformToJSON have been collapsed into Transform with additional parameters. See the Transform section for more details. CompressWithGZIP and CompressWithZLIB have been collapsed into Compress with additional parameters. See the Compress section for more details. EncryptWithAES has been changed to Encrypt with additional parameters. See the Encrypt section for more details. BatchByCount , BatchByTime and BatchByTimeAndCount have been collapsed into Batch with additional parameters. See the Batch section for more details. SetOutputData has been renamed to SetResponseData . See the SetResponseData section for more details. PushToCore parameters have changed. See the PushToCore section for more details. HTTPPost , HTTPPostJSON , HTTPPostXML , HTTPPut , HTTPPutJSON and HTTPPutXML have been collapsed into HTTPExport with additional parameters. See the HTTPExport section for more details. MQTTSecretSend has been renamed to MQTTExport with additional parameters. See the MQTTExport section for more details. MarkAsPushed has been removed. The mark as push capability has been removed from Core Data, which this depended on. MQTTSend has been removed. This has been replaced by MQTTExport . See the MQTTExport section for more details. FilterByProfileName and FilterBySourceName have been added. See the FilterByProfileName and FilterBySourceName sections for more details. Ability to define multiple instances of the same Configurable Pipeline Function has been added. See the Multiple Instances of Function section for more details.","title":"App Service Configurable Profiles"},{"location":"microservices/configuration/CommonCommandLineOptions/","text":"Common Command Line Options This section describes the command line options that are common to all EdgeX services. Some services have addition command line options which are documented in the specific sections for those services. ConfDir -c/--confdir Specify local configuration directory. Default is ./res Can be overridden with EDGEX_CONF_DIR environment variable. File -f/--file Indicates the name of the local configuration file. Default is configuration.toml Can be overridden with EDGEX_CONFIG_FILE environment variable. Config Provider -cp/ --configProvider Indicates to use Configuration Provider service at specified URL. URL Format: {type}.{protocol}://{host}:{port} ex: consul.http://localhost:8500 Can be overridden with EDGEX_CONFIGURATION_PROVIDER environment variable. Profile -p/--profile Indicates configuration profile other than default. Default is no profile name resulting in using ./res/configuration.toml if -f and -c are not used. Can be overridden with EDGEX_PROFILE environment variable. Registry -r/ --registry Indicates service should use the Registry. Connection information is pulled from the [Registry] configuration section. Can be overridden with EDGEX_USE_REGISTRY environment variable. Overwrite -o/--overwrite Overwrite configuration in provider with local configuration. Use with cation This will clobber existing settings in provider, problematic if those settings were edited by hand intentionally. Typically only used during development. Help -h/--help Show the help message","title":"Common Command Line Options"},{"location":"microservices/configuration/CommonCommandLineOptions/#common-command-line-options","text":"This section describes the command line options that are common to all EdgeX services. Some services have addition command line options which are documented in the specific sections for those services.","title":"Common Command Line Options"},{"location":"microservices/configuration/CommonCommandLineOptions/#confdir","text":"-c/--confdir Specify local configuration directory. Default is ./res Can be overridden with EDGEX_CONF_DIR environment variable.","title":"ConfDir"},{"location":"microservices/configuration/CommonCommandLineOptions/#file","text":"-f/--file Indicates the name of the local configuration file. Default is configuration.toml Can be overridden with EDGEX_CONFIG_FILE environment variable.","title":"File"},{"location":"microservices/configuration/CommonCommandLineOptions/#config-provider","text":"-cp/ --configProvider Indicates to use Configuration Provider service at specified URL. URL Format: {type}.{protocol}://{host}:{port} ex: consul.http://localhost:8500 Can be overridden with EDGEX_CONFIGURATION_PROVIDER environment variable.","title":"Config Provider"},{"location":"microservices/configuration/CommonCommandLineOptions/#profile","text":"-p/--profile Indicates configuration profile other than default. Default is no profile name resulting in using ./res/configuration.toml if -f and -c are not used. Can be overridden with EDGEX_PROFILE environment variable.","title":"Profile"},{"location":"microservices/configuration/CommonCommandLineOptions/#registry","text":"-r/ --registry Indicates service should use the Registry. Connection information is pulled from the [Registry] configuration section. Can be overridden with EDGEX_USE_REGISTRY environment variable.","title":"Registry"},{"location":"microservices/configuration/CommonCommandLineOptions/#overwrite","text":"-o/--overwrite Overwrite configuration in provider with local configuration. Use with cation This will clobber existing settings in provider, problematic if those settings were edited by hand intentionally. Typically only used during development.","title":"Overwrite"},{"location":"microservices/configuration/CommonCommandLineOptions/#help","text":"-h/--help Show the help message","title":"Help"},{"location":"microservices/configuration/CommonConfiguration/","text":"Common Configuration The tables in each of the tabs below document configuration properties that are common to all services in the EdgeX Foundry platform. Service-specific properties can be found on the respective documentation page for each service. Configuration Properties Edgex 2.0 For EdgeX 2.0 the Logging and Startup sections have been removed. Startup has been replaced with the EDGEX_STARTUP_DURATION (default is 60 secs) and EDGEX_STARTUP_INTERVAL (default is 1 sec) environment variables. Writable Writable.Telemetry Service Service.CORSConfiguration Databases.Primary Registry Clients.[service-key] SecretStore Property Default Value Description entries in the Writable section of the configuration can be changed on the fly while the service is running if the service is running with the -cp/--configProvider flag LogLevel INFO log entry severity level . Log entries not of the default level or higher are ignored. InsecureSecrets --- This section a map of secrets which simulates the SecretStore for accessing secrets when running in non-secure mode. All services have a default entry for Redis DB credentials called redisdb Edgex 2.0 For EdgeX 2.0 the Writable.InsecureSecrets configuration section is new. Property Default Value Description Interval 30s The interval in seconds at which to report the metrics currently being collected and enabled. Value of 0s disables reporting . PublishTopicPrefix \"edgex/telemetry\" The base topic in which to publish (report) metrics currently being collected and enabled. // will be added to this base topic prefix. Metrics Boolean map of service metrics that are being collected. The boolean flag for each indicates if the metric is enabled for reporting. i.e. EventsPersisted = true . The metric name must match one defined by the service. Tags String map of arbitrary tags to be added to every metric that is reported for the service. i.e. Gateway=\"my-iot-gateway\" . The tag names are arbitrary. Edgex 2.2 For EdgeX 2.2 Service Metrics have been added. Currently only Core Data and Application Services are collecting service metrics. Property Default Value Description HealthCheckInterval 10s The interval in seconds at which the service registry(Consul) will conduct a health check of this service. Host localhost Micro service host name Port --- Micro service port number (specific for each service) ServerBindAddr '' (empty string) The interface on which the service's REST server should listen. By default the server is to listen on the interface to which the Host option resolves (leaving it blank). A value of 0.0.0.0 means listen on all available interfaces. App & Device service do not implement this setting StartupMsg --- Message logged when service completes bootstrap start-up MaxResultCount 1024* Read data limit per invocation. *Default value is for core/support services. Application and Device services do not implement this setting. MaxRequestSize 0 Defines the maximum size of http request body in kilbytes. 0 represents default to system max. RequestTimeout 5s Specifies a timeout duration for handling requests Edgex 2.0 For EdgeX 2.0 Protocol and BootTimeout have been removed. CheckInterval and Timeout have been renamed to HealthCheckInterval and RequestTimeout respectively. MaxRequestSize was added for all services. Edgex 2.2 For EdgeX 2.2 Service MaxRequestSize has been implemented to all services, and the unit is kilobyte. Property Default Value Description The settings of controling CORS http headers EnableCORS false Enable or disable CORS support. CORSAllowCredentials false The value of Access-Control-Allow-Credentials http header. It appears only if the value is true . CORSAllowedOrigin \"https://localhost\" The value of Access-Control-Allow-Origin http header. CORSAllowedMethods \"GET, POST, PUT, PATCH, DELETE\" The value of Access-Control-Allow-Methods http header. CORSAllowedHeaders \"Authorization, Accept, Accept-Language, Content-Language, Content-Type, X-Correlation-ID\" The value of Access-Control-Allow-Headers http header. CORSExposeHeaders \"Cache-Control, Content-Language, Content-Length, Content-Type, Expires, Last-Modified, Pragma, X-Correlation-ID\" The value of Access-Control-Expose-Headers http header. CORSMaxAge 3600 The value of Access-Control-Max-Age http header. Edgex 2.1 New for EdgeX 2.1 is the ability to enable CORS access to EdgeX microservices through configuration. To understand more details about these HTTP headers, please refer to MDN Web Docs , and refer to CORS enabling to learn more. Property Default Value Description configuration that govern database connectivity and the type of database to use. While not all services require DB connectivity, most do and so this has been included in the common configuration docs. Host localhost DB host name Port 6379 DB port number Name ---- Database or document store name (Specific to the service) Timeout 5000 DB connection timeout Type redisdb DB type. Redis is the only supported DB Edgex 2.0 For EdgeX 2.0 mongodb has been remove as a supported DB. The credentials username and password have been removed and are now in the Writable.InsecureSecrets.DB section. Property Default Value Description this configuration only takes effect when connecting to the registry for configuration info Host localhost Registry host name Port 8500 Registry port number Type consul Registry implementation type Property Default Value Description Each service has it own collect of Clients that it uses Protocol http The protocol to use when building a URI to local the service endpoint Host localhost The host name or IP address where the service is hosted Port 598xx The port exposed by the target service Edgex 2.0 For EdgeX 2.0 the map keys have changed to be the service's service-key, i.e. Metadata changed to core-metadata Property Default Value Description these config values are used when security is enabled and SecretStore service access is required for obtaining secrets, such as database credentials Type vault The type of the SecretStore service to use. Currenly only vault is supported. Host localhost The host name or IP address associated with the SecretStore service Port 8200 The configured port on which the SecretStore service is listening Path / The service-specific path where the secrets are kept. This path will differ according to the given service. Protocol http The protocol to be used when communicating with the SecretStore service RootCaCertPath blank Default is to not use HTTPS ServerName blank Not needed for HTTP TokenFile /tmp/edgex/secrets/ /secrets-token.json Fully-qualified path to the location of the service's SecretStore access token. This path will differ according to the given service. SecretsFile blank Fully-qualified path to the location of the service's JSON secrets file contains secrets to seed at start-up. See Seeding Service Secrets section for more details on seed a service's secrets. DisableScrubSecretsFile false Controls if the secrets file is scrubbed (secret data remove) and rewritten after importing the secrets. Authentication AuthType X-Vault-Token A header used to indicate how the given service will authenticate with the SecretStore service Edgex 2.0 For EdgeX 2.0 the Protocol default has changed to HTTP which no longer requires RootCaCertPath and ServerName to be set. Path has been reduce to the sub-path for the service since the based path is fixed. TokenFile default value has changed and requires the service-key be used in the path. Writable vs Readable Settings Within a given service's configuration, there are keys whose values can be edited and change the behavior of the service while it is running versus those that are effectively read-only. These writable settings are grouped under a given service key. For example, the top-level groupings for edgex-core-data are: /edgex/core/2.0/edgex-core-data/Writable /edgex/core/2.0/edgex-core-data/Service /edgex/core/2.0/edgex-core-data/Clients /edgex/core/2.0/edgex-core-data/Databases /edgex/core/2.0/edgex-core-data/MessageQueue /edgex/core/2.0/edgex-core-data/Registry /edgex/core/2.0/edgex-core-data/SecretStore Any configuration settings found in a service's Writable section may be changed and affect a service's behavior without a restart. Any modifications to the other settings (read-only configuration) would require a restart.","title":"Common Configuration"},{"location":"microservices/configuration/CommonConfiguration/#common-configuration","text":"The tables in each of the tabs below document configuration properties that are common to all services in the EdgeX Foundry platform. Service-specific properties can be found on the respective documentation page for each service.","title":"Common Configuration"},{"location":"microservices/configuration/CommonConfiguration/#configuration-properties","text":"Edgex 2.0 For EdgeX 2.0 the Logging and Startup sections have been removed. Startup has been replaced with the EDGEX_STARTUP_DURATION (default is 60 secs) and EDGEX_STARTUP_INTERVAL (default is 1 sec) environment variables. Writable Writable.Telemetry Service Service.CORSConfiguration Databases.Primary Registry Clients.[service-key] SecretStore Property Default Value Description entries in the Writable section of the configuration can be changed on the fly while the service is running if the service is running with the -cp/--configProvider flag LogLevel INFO log entry severity level . Log entries not of the default level or higher are ignored. InsecureSecrets --- This section a map of secrets which simulates the SecretStore for accessing secrets when running in non-secure mode. All services have a default entry for Redis DB credentials called redisdb Edgex 2.0 For EdgeX 2.0 the Writable.InsecureSecrets configuration section is new. Property Default Value Description Interval 30s The interval in seconds at which to report the metrics currently being collected and enabled. Value of 0s disables reporting . PublishTopicPrefix \"edgex/telemetry\" The base topic in which to publish (report) metrics currently being collected and enabled. // will be added to this base topic prefix. Metrics Boolean map of service metrics that are being collected. The boolean flag for each indicates if the metric is enabled for reporting. i.e. EventsPersisted = true . The metric name must match one defined by the service. Tags String map of arbitrary tags to be added to every metric that is reported for the service. i.e. Gateway=\"my-iot-gateway\" . The tag names are arbitrary. Edgex 2.2 For EdgeX 2.2 Service Metrics have been added. Currently only Core Data and Application Services are collecting service metrics. Property Default Value Description HealthCheckInterval 10s The interval in seconds at which the service registry(Consul) will conduct a health check of this service. Host localhost Micro service host name Port --- Micro service port number (specific for each service) ServerBindAddr '' (empty string) The interface on which the service's REST server should listen. By default the server is to listen on the interface to which the Host option resolves (leaving it blank). A value of 0.0.0.0 means listen on all available interfaces. App & Device service do not implement this setting StartupMsg --- Message logged when service completes bootstrap start-up MaxResultCount 1024* Read data limit per invocation. *Default value is for core/support services. Application and Device services do not implement this setting. MaxRequestSize 0 Defines the maximum size of http request body in kilbytes. 0 represents default to system max. RequestTimeout 5s Specifies a timeout duration for handling requests Edgex 2.0 For EdgeX 2.0 Protocol and BootTimeout have been removed. CheckInterval and Timeout have been renamed to HealthCheckInterval and RequestTimeout respectively. MaxRequestSize was added for all services. Edgex 2.2 For EdgeX 2.2 Service MaxRequestSize has been implemented to all services, and the unit is kilobyte. Property Default Value Description The settings of controling CORS http headers EnableCORS false Enable or disable CORS support. CORSAllowCredentials false The value of Access-Control-Allow-Credentials http header. It appears only if the value is true . CORSAllowedOrigin \"https://localhost\" The value of Access-Control-Allow-Origin http header. CORSAllowedMethods \"GET, POST, PUT, PATCH, DELETE\" The value of Access-Control-Allow-Methods http header. CORSAllowedHeaders \"Authorization, Accept, Accept-Language, Content-Language, Content-Type, X-Correlation-ID\" The value of Access-Control-Allow-Headers http header. CORSExposeHeaders \"Cache-Control, Content-Language, Content-Length, Content-Type, Expires, Last-Modified, Pragma, X-Correlation-ID\" The value of Access-Control-Expose-Headers http header. CORSMaxAge 3600 The value of Access-Control-Max-Age http header. Edgex 2.1 New for EdgeX 2.1 is the ability to enable CORS access to EdgeX microservices through configuration. To understand more details about these HTTP headers, please refer to MDN Web Docs , and refer to CORS enabling to learn more. Property Default Value Description configuration that govern database connectivity and the type of database to use. While not all services require DB connectivity, most do and so this has been included in the common configuration docs. Host localhost DB host name Port 6379 DB port number Name ---- Database or document store name (Specific to the service) Timeout 5000 DB connection timeout Type redisdb DB type. Redis is the only supported DB Edgex 2.0 For EdgeX 2.0 mongodb has been remove as a supported DB. The credentials username and password have been removed and are now in the Writable.InsecureSecrets.DB section. Property Default Value Description this configuration only takes effect when connecting to the registry for configuration info Host localhost Registry host name Port 8500 Registry port number Type consul Registry implementation type Property Default Value Description Each service has it own collect of Clients that it uses Protocol http The protocol to use when building a URI to local the service endpoint Host localhost The host name or IP address where the service is hosted Port 598xx The port exposed by the target service Edgex 2.0 For EdgeX 2.0 the map keys have changed to be the service's service-key, i.e. Metadata changed to core-metadata Property Default Value Description these config values are used when security is enabled and SecretStore service access is required for obtaining secrets, such as database credentials Type vault The type of the SecretStore service to use. Currenly only vault is supported. Host localhost The host name or IP address associated with the SecretStore service Port 8200 The configured port on which the SecretStore service is listening Path / The service-specific path where the secrets are kept. This path will differ according to the given service. Protocol http The protocol to be used when communicating with the SecretStore service RootCaCertPath blank Default is to not use HTTPS ServerName blank Not needed for HTTP TokenFile /tmp/edgex/secrets/ /secrets-token.json Fully-qualified path to the location of the service's SecretStore access token. This path will differ according to the given service. SecretsFile blank Fully-qualified path to the location of the service's JSON secrets file contains secrets to seed at start-up. See Seeding Service Secrets section for more details on seed a service's secrets. DisableScrubSecretsFile false Controls if the secrets file is scrubbed (secret data remove) and rewritten after importing the secrets. Authentication AuthType X-Vault-Token A header used to indicate how the given service will authenticate with the SecretStore service Edgex 2.0 For EdgeX 2.0 the Protocol default has changed to HTTP which no longer requires RootCaCertPath and ServerName to be set. Path has been reduce to the sub-path for the service since the based path is fixed. TokenFile default value has changed and requires the service-key be used in the path.","title":"Configuration Properties"},{"location":"microservices/configuration/CommonConfiguration/#writable-vs-readable-settings","text":"Within a given service's configuration, there are keys whose values can be edited and change the behavior of the service while it is running versus those that are effectively read-only. These writable settings are grouped under a given service key. For example, the top-level groupings for edgex-core-data are: /edgex/core/2.0/edgex-core-data/Writable /edgex/core/2.0/edgex-core-data/Service /edgex/core/2.0/edgex-core-data/Clients /edgex/core/2.0/edgex-core-data/Databases /edgex/core/2.0/edgex-core-data/MessageQueue /edgex/core/2.0/edgex-core-data/Registry /edgex/core/2.0/edgex-core-data/SecretStore Any configuration settings found in a service's Writable section may be changed and affect a service's behavior without a restart. Any modifications to the other settings (read-only configuration) would require a restart.","title":"Writable vs Readable Settings"},{"location":"microservices/configuration/CommonEnvironmentVariables/","text":"Common Environment Variables There are two types of environment variables used by all EdgeX services. They are standard and overrides . The only difference is that the overrides apply to command-line options and service configuration settings where as standard do not have any corresponding command-line option or configuration setting. Standard Environment Variables This section describes the standard environment variables common to all EdgeX services. Some service may have additional standard environment variables which are documented in those service specific sections. EDGEX_SECURITY_SECRET_STORE This environment variables indicates whether the service is expected to initialize the secure SecretStore which allows the service to access secrets from Vault. Defaults to true if not set or not set to false . When set to true the EdgeX security services must be running. If running EdgeX in non-secure mode you then want this explicitly set to false . Example - Using docker-compose to disable secure SecretStore environment : EDGEX_SECURITY_SECRET_STORE : \"false\" EdgeX 2.0 For EdgeX 2.0 when running in secure mode Consul is secured, which requires all services to have this environment variable be true so they can request their Consul access token from Vault. See the Secure Consul section for more details. EDGEX_STARTUP_DURATION This environment variable sets the total duration in seconds allowed for the services to complete the bootstrap start-up. Default is 60 seconds. Example - Using docker-compose to set start-up duration to 120 seconds environment : EDGEX_STARTUP_DURATION : \"120\" EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version startup_duration has been removed EDGEX_STARTUP_INTERVAL This environment variable sets the retry interval in seconds for the services retrying a failed action during the bootstrap start-up. Default is 1 second. Example - Using docker-compose to set start-up interval to 3 seconds environment : EDGEX_STARTUP_INTERVAL : \"3\" EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version startup_interval has been removed Environment Overrides There are two types of environment overrides which are command-line and configuration . Important Environment variable overrides have precedence over all command-line, local configuration and remote configuration. i.e. configuration setting changed in Consul will be overridden after the service loads the configuration from Consul if that setting has an environment override. Command-line Overrides EDGEX_CONF_DIR This environment variable overrides the -c/--confdir command-line option . Note All EdgeX service Docker images have this option set to /res . Example - Using docker-compose to override the configuration folder name environment : EDGEX_CONF_DIR : \"/my-config\" EDGEX_CONFIG_FILE This environment variable overrides the -f/--file command-line option . Example - Using docker-compose to override the configuration file name used environment : EDGEX_CONFIG_FILE : \"my-config.toml\" EDGEX_CONFIGURATION_PROVIDER This environment variable overrides the -cp/--configProvider command-line option . Note All EdgeX service Docker images have this option set to -cp=consul.http://edgex-core-consul:8500 . Example - Using docker-compose to override with different port number environment : EDGEX_CONFIGURATION_PROVIDER : \"consul.http://edgex-consul:9500\" EDGEX_PROFILE This environment variable overrides the -p/--profile command-line option . When non-empty, the value is used in the path to the configuration file. i.e. /res/my-profile/configuation.toml. This is useful when running multiple instances of a service such as App Service Configurable. Example - Using docker-compose to override the profile to use app-service-rules : image : edgexfoundry/docker-app-service-configurable:2.0.0 environment : EDGEX_PROFILE : \"rules-engine\" ... This sets the profile so that the App Service Configurable uses the rules-engine configuration profile which resides at /res/rules-engine/configuration.toml EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version edgex_profile has been removed EDGEX_USE_REGISTRY This environment variable overrides the -r/--registry command-line option . Note All EdgeX service Docker images have this option set to --registry . Example - Using docker-compose to override use of the Registry environment : EDGEX_USE_REGISTRY : \"false\" EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version edgex_registry has been removed Configuration Overrides Any configuration setting from a service's configuration.toml file can be overridden by environment variables. The environment variable names have the following format: < TOM-SECTION-NAME > _ < TOML-KEY-NAME > < TOML-SECTION-NAME > _ < TOML-SUB-SECTION-NAME > _ < TOML-KEY-NAME > EdgeX 2.0 With EdgeX 2.0 the use of CamelCase environment variable names is no longer supported. Instead the variable names must be all uppercase as in the example below. Also the using of dash - in the TOML-NAME is converted to an underscore _ in the environment variable name. Example - Environment Overrides of Configuration ``` toml TOML : [ Writable ] LogLevel = \"INFO\" ENVVAR : WRITABLE_LOGLEVEL = DEBUG TOML : [ Clients ] [Clients.core-data] Host = \"localhost\" ENVVAR : CLIENTS_CORE_DATA_HOST = edgex-core-data ``` Notable Configuration Overrides This section describes environment variable overrides that have special utility, such as enabling a debug capability or facilitating code development. KONG_SSL_CIPHER_SUITE (edgex-kong service) This variable controls the TLS cipher suite and protocols supported by the EdgeX API Gateway as implemented by Kong. This variable, if unspecified, selects the \"intermediate\" cipher suite which supports TLSv1.2, TLSv1.3, and relatively modern TLS ciphers. The EdgeX framework by default overrides this value to \"modern\" , which currently enables only TLSv1.3 and a fixed cipher suite. The \"modern\" cipher suite is known to be incompatible with older web browsers, but since the target use of the API gateway is to support API clients, not browsers, this behavior was deemed acceptable by the EdgeX Security Working Group on September 8, 2021. TOKENFILEPROVIDER_DEFAULTTOKENTTL (security-secretstore-setup service) This variable controls the TTL of the default secretstore tokens that are created for EdgeX microservices. This variable defaults to 1h (one hour) if unspecified. It is often useful when developing a new microservice to set this value to a higher value, such as 12h . This higher value will allow the secret store token to remain valid long enough for a developer to get a new microservice working and into a state where it can renew its own token. (All secret store tokens in EdgeX expire if not renewed periodically.)","title":"Common Environment Variables"},{"location":"microservices/configuration/CommonEnvironmentVariables/#common-environment-variables","text":"There are two types of environment variables used by all EdgeX services. They are standard and overrides . The only difference is that the overrides apply to command-line options and service configuration settings where as standard do not have any corresponding command-line option or configuration setting.","title":"Common Environment Variables"},{"location":"microservices/configuration/CommonEnvironmentVariables/#standard-environment-variables","text":"This section describes the standard environment variables common to all EdgeX services. Some service may have additional standard environment variables which are documented in those service specific sections.","title":"Standard Environment Variables"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_security_secret_store","text":"This environment variables indicates whether the service is expected to initialize the secure SecretStore which allows the service to access secrets from Vault. Defaults to true if not set or not set to false . When set to true the EdgeX security services must be running. If running EdgeX in non-secure mode you then want this explicitly set to false . Example - Using docker-compose to disable secure SecretStore environment : EDGEX_SECURITY_SECRET_STORE : \"false\" EdgeX 2.0 For EdgeX 2.0 when running in secure mode Consul is secured, which requires all services to have this environment variable be true so they can request their Consul access token from Vault. See the Secure Consul section for more details.","title":"EDGEX_SECURITY_SECRET_STORE"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_startup_duration","text":"This environment variable sets the total duration in seconds allowed for the services to complete the bootstrap start-up. Default is 60 seconds. Example - Using docker-compose to set start-up duration to 120 seconds environment : EDGEX_STARTUP_DURATION : \"120\" EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version startup_duration has been removed","title":"EDGEX_STARTUP_DURATION"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_startup_interval","text":"This environment variable sets the retry interval in seconds for the services retrying a failed action during the bootstrap start-up. Default is 1 second. Example - Using docker-compose to set start-up interval to 3 seconds environment : EDGEX_STARTUP_INTERVAL : \"3\" EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version startup_interval has been removed","title":"EDGEX_STARTUP_INTERVAL"},{"location":"microservices/configuration/CommonEnvironmentVariables/#environment-overrides","text":"There are two types of environment overrides which are command-line and configuration . Important Environment variable overrides have precedence over all command-line, local configuration and remote configuration. i.e. configuration setting changed in Consul will be overridden after the service loads the configuration from Consul if that setting has an environment override.","title":"Environment Overrides"},{"location":"microservices/configuration/CommonEnvironmentVariables/#command-line-overrides","text":"","title":"Command-line Overrides"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_conf_dir","text":"This environment variable overrides the -c/--confdir command-line option . Note All EdgeX service Docker images have this option set to /res . Example - Using docker-compose to override the configuration folder name environment : EDGEX_CONF_DIR : \"/my-config\"","title":"EDGEX_CONF_DIR"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_config_file","text":"This environment variable overrides the -f/--file command-line option . Example - Using docker-compose to override the configuration file name used environment : EDGEX_CONFIG_FILE : \"my-config.toml\"","title":"EDGEX_CONFIG_FILE"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_configuration_provider","text":"This environment variable overrides the -cp/--configProvider command-line option . Note All EdgeX service Docker images have this option set to -cp=consul.http://edgex-core-consul:8500 . Example - Using docker-compose to override with different port number environment : EDGEX_CONFIGURATION_PROVIDER : \"consul.http://edgex-consul:9500\"","title":"EDGEX_CONFIGURATION_PROVIDER"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_profile","text":"This environment variable overrides the -p/--profile command-line option . When non-empty, the value is used in the path to the configuration file. i.e. /res/my-profile/configuation.toml. This is useful when running multiple instances of a service such as App Service Configurable. Example - Using docker-compose to override the profile to use app-service-rules : image : edgexfoundry/docker-app-service-configurable:2.0.0 environment : EDGEX_PROFILE : \"rules-engine\" ... This sets the profile so that the App Service Configurable uses the rules-engine configuration profile which resides at /res/rules-engine/configuration.toml EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version edgex_profile has been removed","title":"EDGEX_PROFILE"},{"location":"microservices/configuration/CommonEnvironmentVariables/#edgex_use_registry","text":"This environment variable overrides the -r/--registry command-line option . Note All EdgeX service Docker images have this option set to --registry . Example - Using docker-compose to override use of the Registry environment : EDGEX_USE_REGISTRY : \"false\" EdgeX 2.0 For EdgeX 2.0 the deprecated lower case version edgex_registry has been removed","title":"EDGEX_USE_REGISTRY"},{"location":"microservices/configuration/CommonEnvironmentVariables/#configuration-overrides","text":"Any configuration setting from a service's configuration.toml file can be overridden by environment variables. The environment variable names have the following format: < TOM-SECTION-NAME > _ < TOML-KEY-NAME > < TOML-SECTION-NAME > _ < TOML-SUB-SECTION-NAME > _ < TOML-KEY-NAME > EdgeX 2.0 With EdgeX 2.0 the use of CamelCase environment variable names is no longer supported. Instead the variable names must be all uppercase as in the example below. Also the using of dash - in the TOML-NAME is converted to an underscore _ in the environment variable name. Example - Environment Overrides of Configuration ``` toml TOML : [ Writable ] LogLevel = \"INFO\" ENVVAR : WRITABLE_LOGLEVEL = DEBUG TOML : [ Clients ] [Clients.core-data] Host = \"localhost\" ENVVAR : CLIENTS_CORE_DATA_HOST = edgex-core-data ```","title":"Configuration Overrides"},{"location":"microservices/configuration/CommonEnvironmentVariables/#notable-configuration-overrides","text":"This section describes environment variable overrides that have special utility, such as enabling a debug capability or facilitating code development.","title":"Notable Configuration Overrides"},{"location":"microservices/configuration/CommonEnvironmentVariables/#kong_ssl_cipher_suite-edgex-kong-service","text":"This variable controls the TLS cipher suite and protocols supported by the EdgeX API Gateway as implemented by Kong. This variable, if unspecified, selects the \"intermediate\" cipher suite which supports TLSv1.2, TLSv1.3, and relatively modern TLS ciphers. The EdgeX framework by default overrides this value to \"modern\" , which currently enables only TLSv1.3 and a fixed cipher suite. The \"modern\" cipher suite is known to be incompatible with older web browsers, but since the target use of the API gateway is to support API clients, not browsers, this behavior was deemed acceptable by the EdgeX Security Working Group on September 8, 2021.","title":"KONG_SSL_CIPHER_SUITE (edgex-kong service)"},{"location":"microservices/configuration/CommonEnvironmentVariables/#tokenfileprovider_defaulttokenttl-security-secretstore-setup-service","text":"This variable controls the TTL of the default secretstore tokens that are created for EdgeX microservices. This variable defaults to 1h (one hour) if unspecified. It is often useful when developing a new microservice to set this value to a higher value, such as 12h . This higher value will allow the secret store token to remain valid long enough for a developer to get a new microservice working and into a state where it can renew its own token. (All secret store tokens in EdgeX expire if not renewed periodically.)","title":"TOKENFILEPROVIDER_DEFAULTTOKENTTL (security-secretstore-setup service)"},{"location":"microservices/configuration/ConfigurationAndRegistry/","text":"Configuration and Registry Providers Introduction The EdgeX registry and configuration service provides other EdgeX Foundry micro services with information about associated services within EdgeX Foundry (such as location and status) and configuration properties (i.e. - a repository of initialization and operating values). Today, EdgeX Foundry uses Consul by Hashicorp as its reference implementation configuration and registry providers. However, abstractions are in place so that these functions could be provided by an alternate implementation. In fact, registration and configuration could be provided by different services under the covers. For more, see the Configuration Provider and Registry Provider sections in this page. Configuration Please refer to the EdgeX Foundry architectural decision record for details (and design decisions) behind the configuration in EdgeX. Local Configuration Because EdgeX Foundry may be deployed and run in several different ways, it is important to understand how configuration is loaded and from where it is sourced. Referring to the cmd directory within the edgex-go repository , each service has its own folder. Inside each service folder there is a res directory (short for \"resource\"). There you will find the configuration files in TOML format that defines each service's configuration. A service may support several different configuration profiles, such as a App Service Configurable does. In this case, the configuration file located directly in the res directory should be considered the default configuration profile. Sub-directories will contain configurations appropriate to the respective profile. As of the Geneva release, EdgeX recommends using environment variable overrides instead of creating profiles to override some subset of config values. App Service Configurable is an exception to this as this is how it defined unique instances using the same executable. If you choose to use profiles as described above, the config profile can be indicated using one of the following command line flags: --profile / -p Taking the Core Data and App Service Configurable services as an examples: ./core-data starts the service using the default profile found locally ./app-service-configurable --profile=rules-engine starts the service using the rules-engine profile found locally Note Again, utilizing environment variables for configuration overrides is the recommended path. Config profiles, for the most part, are not used. Seeding Configuration When utilizing the centralized configuration management for the EdgeX Foundry micro services, it is necessary to seed the required configuration before starting the services. Each service has the built-in capability to perform this seeding operation. A service will use its local configuration file to initialize the structure and relevant values, and then overlay any environment variable override values as specified. The end result will be seeded into the configuration provider if such is being used. In order for a service to seed/load the configuration to/from the configuration provider, use one of the following flags: --configProvider / -cp Again, taking the core-data service as an example: ./core-data -cp=consul.http://localhost:8500 will start the service using configuration values found in the provider or seed them if they do not exist. Note Environment overrides are also applied after the configuration is loaded from the configuration provider. Configuration Structure Configuration information is organized into a hierarchical structure allowing for a logical grouping of services, as well as versioning, beneath an \"edgex\" namespace at root level of the configuration tree. The root namespace separates EdgeX Foundry-related configuration information from other applications that may be using the same configuration provider. Below the root, sub-nodes facilitate grouping of device services, core/support/security services, app services, etc. As an example, the top-level nodes shown when one views the configuration registry might be as follows: edgex (root namespace) core (core/support/security services) devices (device services) appservices ( application services ) Versioning Incorporating versioning into the configuration hierarchy looks like this. edgex (root namespace) core (core/support/security services) 2.0 core-command core-data core-metadata support-notifications support-scheduler sys-mgmt-agent 3.0 devices (device services) 2.0 device-mqtt device-virtual device-modbus 3.0 appservices (application services) 2.0 app-rules-engine 3.0 EdgeX 2.0 For EdgeX 2.0 the version number in the path is now 2.0 and the service keys are now used for the service names. The versions shown correspond to major versions of the given services. For all minor/patch versions associated with a major version, the respective service keys live under the major version in configuration (such as 2.0). Changes to the configuration structure that may be required during the associated minor version development cycles can only be additive. That is, key names will not be removed or changed once set in a major version. Furthermore, sections of the configuration tree cannot be moved from one place to another. In this way, backward compatibility for the lifetime of the major version is maintained. An advantage of grouping all minor/patch versions under a major version involves end-user configuration changes that need to be persisted during an upgrade. A service on startup will not overwrite existing configuration when it runs unless explicitly told to do so via the --overwrite / -o command line flag. Therefore if a user leaves their configuration provider running during an EdgeX Foundry upgrade any customization will be left in place. Environment variable overrides such as those supplied in the docker-compose for a given release will always override existing content in the configuration provider. Configuration Provider You can supply and manage configuration in a centralized manner by utilizing the -cp/--configProvider flag when starting a service. If the flag is provided and points to an application such as HashiCorp's Consul , the service will bootstrap its configuration into the provider, if it doesn't exist. If configuration does already exist, it will load the content from the given location applying any environment variables overrides of which the service is aware. Integration with the configuration provider is handled through the go-mod-configuration module referenced by all services. Registry Provider The registry refers to any platform you may use for service discovery. For the EdgeX Foundry reference implementation, the default provider for this responsibility is Consul. Integration with the registry is handled through the go-mod-registry module referenced by all services. Introduction to Registry The objective of the registry is to enable micro services to find and to communicate with each other. When each micro service starts up, it registers itself with the registry, and the registry continues checking its availability periodically via a specified health check endpoint. When one micro service needs to connect to another one, it connects to the registry to retrieve the available host name and port number of the target micro service and then invokes the target micro service. The following figure shows the basic flow. Consul is the default registry implementation and provides native features for service registration, service discovery, and health checking. Please refer to the Consul official web site for more information: https://www.consul.io Physically, the \"registry\" and \"configuration\" management services are combined and running on the same Consul server node. Web User Interface A web user interface is also provided by Consul. Users can view the available service list and their health status through the web user interface. The web user interface is available at the /ui path on the same port as the HTTP API. By default this is http://localhost:8500/ui . For more detail, please see: https://www.consul.io/intro/getting-started/ui.html Running on Docker For ease of use to install and update, the microservices of EdgeX Foundry are published as Docker images onto Docker Hub and compose files that allow you to run EdgeX and dependent service such as Consul. These compose files can be found here in the edgex-compose repository . See the Getting Started using Docker for more details. Once the EdgeX stack is running in docker verify Consul is running by going to http://localhost:8500/ui in your browser. Running on Local Machine To run Consul on the local machine, following these steps: Download the binary from Consul official website: https://www.consul.io/downloads.html . Please choose the correct binary file according to the operation system. Set up the environment variable. Please refer to https://www.consul.io/intro/getting-started/install.html . Execute the following command: consul agent -data-dir \\$ { DATA_FOLDER } -ui -advertise 127 .0.0.1 -server -bootstrap-expect 1 # ${DATA_FOLDER} could be any folder to put the data files of Consul and it needs the read/write permission. Verify the result: http://localhost:8500/ui","title":"Configuration and Registry Providers"},{"location":"microservices/configuration/ConfigurationAndRegistry/#configuration-and-registry-providers","text":"","title":"Configuration and Registry Providers"},{"location":"microservices/configuration/ConfigurationAndRegistry/#introduction","text":"The EdgeX registry and configuration service provides other EdgeX Foundry micro services with information about associated services within EdgeX Foundry (such as location and status) and configuration properties (i.e. - a repository of initialization and operating values). Today, EdgeX Foundry uses Consul by Hashicorp as its reference implementation configuration and registry providers. However, abstractions are in place so that these functions could be provided by an alternate implementation. In fact, registration and configuration could be provided by different services under the covers. For more, see the Configuration Provider and Registry Provider sections in this page.","title":"Introduction"},{"location":"microservices/configuration/ConfigurationAndRegistry/#configuration","text":"Please refer to the EdgeX Foundry architectural decision record for details (and design decisions) behind the configuration in EdgeX.","title":"Configuration"},{"location":"microservices/configuration/ConfigurationAndRegistry/#local-configuration","text":"Because EdgeX Foundry may be deployed and run in several different ways, it is important to understand how configuration is loaded and from where it is sourced. Referring to the cmd directory within the edgex-go repository , each service has its own folder. Inside each service folder there is a res directory (short for \"resource\"). There you will find the configuration files in TOML format that defines each service's configuration. A service may support several different configuration profiles, such as a App Service Configurable does. In this case, the configuration file located directly in the res directory should be considered the default configuration profile. Sub-directories will contain configurations appropriate to the respective profile. As of the Geneva release, EdgeX recommends using environment variable overrides instead of creating profiles to override some subset of config values. App Service Configurable is an exception to this as this is how it defined unique instances using the same executable. If you choose to use profiles as described above, the config profile can be indicated using one of the following command line flags: --profile / -p Taking the Core Data and App Service Configurable services as an examples: ./core-data starts the service using the default profile found locally ./app-service-configurable --profile=rules-engine starts the service using the rules-engine profile found locally Note Again, utilizing environment variables for configuration overrides is the recommended path. Config profiles, for the most part, are not used.","title":"Local Configuration"},{"location":"microservices/configuration/ConfigurationAndRegistry/#seeding-configuration","text":"When utilizing the centralized configuration management for the EdgeX Foundry micro services, it is necessary to seed the required configuration before starting the services. Each service has the built-in capability to perform this seeding operation. A service will use its local configuration file to initialize the structure and relevant values, and then overlay any environment variable override values as specified. The end result will be seeded into the configuration provider if such is being used. In order for a service to seed/load the configuration to/from the configuration provider, use one of the following flags: --configProvider / -cp Again, taking the core-data service as an example: ./core-data -cp=consul.http://localhost:8500 will start the service using configuration values found in the provider or seed them if they do not exist. Note Environment overrides are also applied after the configuration is loaded from the configuration provider.","title":"Seeding Configuration"},{"location":"microservices/configuration/ConfigurationAndRegistry/#configuration-structure","text":"Configuration information is organized into a hierarchical structure allowing for a logical grouping of services, as well as versioning, beneath an \"edgex\" namespace at root level of the configuration tree. The root namespace separates EdgeX Foundry-related configuration information from other applications that may be using the same configuration provider. Below the root, sub-nodes facilitate grouping of device services, core/support/security services, app services, etc. As an example, the top-level nodes shown when one views the configuration registry might be as follows: edgex (root namespace) core (core/support/security services) devices (device services) appservices ( application services )","title":"Configuration Structure"},{"location":"microservices/configuration/ConfigurationAndRegistry/#versioning","text":"Incorporating versioning into the configuration hierarchy looks like this. edgex (root namespace) core (core/support/security services) 2.0 core-command core-data core-metadata support-notifications support-scheduler sys-mgmt-agent 3.0 devices (device services) 2.0 device-mqtt device-virtual device-modbus 3.0 appservices (application services) 2.0 app-rules-engine 3.0 EdgeX 2.0 For EdgeX 2.0 the version number in the path is now 2.0 and the service keys are now used for the service names. The versions shown correspond to major versions of the given services. For all minor/patch versions associated with a major version, the respective service keys live under the major version in configuration (such as 2.0). Changes to the configuration structure that may be required during the associated minor version development cycles can only be additive. That is, key names will not be removed or changed once set in a major version. Furthermore, sections of the configuration tree cannot be moved from one place to another. In this way, backward compatibility for the lifetime of the major version is maintained. An advantage of grouping all minor/patch versions under a major version involves end-user configuration changes that need to be persisted during an upgrade. A service on startup will not overwrite existing configuration when it runs unless explicitly told to do so via the --overwrite / -o command line flag. Therefore if a user leaves their configuration provider running during an EdgeX Foundry upgrade any customization will be left in place. Environment variable overrides such as those supplied in the docker-compose for a given release will always override existing content in the configuration provider.","title":"Versioning"},{"location":"microservices/configuration/ConfigurationAndRegistry/#configuration-provider","text":"You can supply and manage configuration in a centralized manner by utilizing the -cp/--configProvider flag when starting a service. If the flag is provided and points to an application such as HashiCorp's Consul , the service will bootstrap its configuration into the provider, if it doesn't exist. If configuration does already exist, it will load the content from the given location applying any environment variables overrides of which the service is aware. Integration with the configuration provider is handled through the go-mod-configuration module referenced by all services.","title":"Configuration Provider"},{"location":"microservices/configuration/ConfigurationAndRegistry/#registry-provider","text":"The registry refers to any platform you may use for service discovery. For the EdgeX Foundry reference implementation, the default provider for this responsibility is Consul. Integration with the registry is handled through the go-mod-registry module referenced by all services.","title":"Registry Provider"},{"location":"microservices/configuration/ConfigurationAndRegistry/#introduction-to-registry","text":"The objective of the registry is to enable micro services to find and to communicate with each other. When each micro service starts up, it registers itself with the registry, and the registry continues checking its availability periodically via a specified health check endpoint. When one micro service needs to connect to another one, it connects to the registry to retrieve the available host name and port number of the target micro service and then invokes the target micro service. The following figure shows the basic flow. Consul is the default registry implementation and provides native features for service registration, service discovery, and health checking. Please refer to the Consul official web site for more information: https://www.consul.io Physically, the \"registry\" and \"configuration\" management services are combined and running on the same Consul server node.","title":"Introduction to Registry"},{"location":"microservices/configuration/ConfigurationAndRegistry/#web-user-interface","text":"A web user interface is also provided by Consul. Users can view the available service list and their health status through the web user interface. The web user interface is available at the /ui path on the same port as the HTTP API. By default this is http://localhost:8500/ui . For more detail, please see: https://www.consul.io/intro/getting-started/ui.html","title":"Web User Interface"},{"location":"microservices/configuration/ConfigurationAndRegistry/#running-on-docker","text":"For ease of use to install and update, the microservices of EdgeX Foundry are published as Docker images onto Docker Hub and compose files that allow you to run EdgeX and dependent service such as Consul. These compose files can be found here in the edgex-compose repository . See the Getting Started using Docker for more details. Once the EdgeX stack is running in docker verify Consul is running by going to http://localhost:8500/ui in your browser.","title":"Running on Docker"},{"location":"microservices/configuration/ConfigurationAndRegistry/#running-on-local-machine","text":"To run Consul on the local machine, following these steps: Download the binary from Consul official website: https://www.consul.io/downloads.html . Please choose the correct binary file according to the operation system. Set up the environment variable. Please refer to https://www.consul.io/intro/getting-started/install.html . Execute the following command: consul agent -data-dir \\$ { DATA_FOLDER } -ui -advertise 127 .0.0.1 -server -bootstrap-expect 1 # ${DATA_FOLDER} could be any folder to put the data files of Consul and it needs the read/write permission. Verify the result: http://localhost:8500/ui","title":"Running on Local Machine"},{"location":"microservices/configuration/V2MigrationCommonConfig/","text":"V2 Migration of Common Configuration EdgeX 2.0 For EdgeX 2.0 there have been many breaking changes made to the configuration for all services. This section describes how to migrate the configuration sections that are common to all services. This information only applies if you have existing 1.x configuration that you have modified and need to migrate, rather than use the new V2 version of the configuration and modify it as needed. Writable The Writable section has the new InsecureSecrets sub-section. All services need the following added so they can access the Database and/or MessageBus : [Writable.InsecureSecrets] [Writable.InsecureSecrets.DB] path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\" Logging Remove the [Logging] section. Service The service section is now common to all EdgeX services. The migration to this new version slightly differs for each class of service, i.e. Core/Support, Device or Application Service. The sub-sections below describe the migration for each class. Core/Support For the Core/Support services the following changes are required: Remove BootTimeout Remove Protocol Rename CheckInterval to HealthCheckInterval Rename Timeout to RequestTimeout and change value to be duration string. i.e 5000 changes to 5s Add MaxRequestSize with value of 0 Port value changes to be in proper range for new port assignments. See Port Assignments (TBD) section for more details Device For Device service the changes are the same as Core/Support above plus the following: Remove ConnectRetries Move EnableAsyncReadings to be under the [Device] section Move AsyncBufferSize to be under the [Device] section Move labels to be under the [Device] section Application For Application services the changes are the same as Core/Support above plus the following: Remove ReadMaxLimit Remove ClientMonitor Add ServerBindAddr = \"\" # if blank, uses default Go behavior https://golang.org/pkg/net/#Listen Add MaxResultCount and set value to 0 Databases Remove the Username and Password settings Registry No changes Clients The map key names have changed to uses the service key for each of the target services. Each client entry must be changed to use the appropriate service key as follows: CoreData => core-data Metadata => core-metadata Command => core-command Notifications => support-notifications Scheduler => support-scheduler Remove the [Clients.Logging] section SecretStore All service now require the [SecretStore] section. For those that did not have it previously add the following replacing with the service's actual service key: [SecretStore] Type = 'vault' Protocol = 'http' Host = 'localhost' Port = 8200 Path = '/' TokenFile = '/tmp/edgex/secrets//secrets-token.json' RootCaCertPath = '' ServerName = '' [SecretStore.Authentication] AuthType = 'X-Vault-Token' For those service that previously had the [SecretStore] section, make the following changes replacing with the service's actual service key: Add the Type = 'vault' setting Remove AdditionalRetryAttempts Remove RetryWaitPeriod Change Protocol value to be 'http' Change Path value to be '/' Change TokenFile value to be '/tmp/edgex/secrets//secrets-token.json' Change RootCaCertPath value to be empty, i.e '' Change ServerName value to be empty, i.e ''","title":"V2 Migration of Common Configuration"},{"location":"microservices/configuration/V2MigrationCommonConfig/#v2-migration-of-common-configuration","text":"EdgeX 2.0 For EdgeX 2.0 there have been many breaking changes made to the configuration for all services. This section describes how to migrate the configuration sections that are common to all services. This information only applies if you have existing 1.x configuration that you have modified and need to migrate, rather than use the new V2 version of the configuration and modify it as needed.","title":"V2 Migration of Common Configuration"},{"location":"microservices/configuration/V2MigrationCommonConfig/#writable","text":"The Writable section has the new InsecureSecrets sub-section. All services need the following added so they can access the Database and/or MessageBus : [Writable.InsecureSecrets] [Writable.InsecureSecrets.DB] path = \"redisdb\" [Writable.InsecureSecrets.DB.Secrets] username = \"\" password = \"\"","title":"Writable"},{"location":"microservices/configuration/V2MigrationCommonConfig/#logging","text":"Remove the [Logging] section.","title":"Logging"},{"location":"microservices/configuration/V2MigrationCommonConfig/#service","text":"The service section is now common to all EdgeX services. The migration to this new version slightly differs for each class of service, i.e. Core/Support, Device or Application Service. The sub-sections below describe the migration for each class.","title":"Service"},{"location":"microservices/configuration/V2MigrationCommonConfig/#coresupport","text":"For the Core/Support services the following changes are required: Remove BootTimeout Remove Protocol Rename CheckInterval to HealthCheckInterval Rename Timeout to RequestTimeout and change value to be duration string. i.e 5000 changes to 5s Add MaxRequestSize with value of 0 Port value changes to be in proper range for new port assignments. See Port Assignments (TBD) section for more details","title":"Core/Support"},{"location":"microservices/configuration/V2MigrationCommonConfig/#device","text":"For Device service the changes are the same as Core/Support above plus the following: Remove ConnectRetries Move EnableAsyncReadings to be under the [Device] section Move AsyncBufferSize to be under the [Device] section Move labels to be under the [Device] section","title":"Device"},{"location":"microservices/configuration/V2MigrationCommonConfig/#application","text":"For Application services the changes are the same as Core/Support above plus the following: Remove ReadMaxLimit Remove ClientMonitor Add ServerBindAddr = \"\" # if blank, uses default Go behavior https://golang.org/pkg/net/#Listen Add MaxResultCount and set value to 0","title":"Application"},{"location":"microservices/configuration/V2MigrationCommonConfig/#databases","text":"Remove the Username and Password settings","title":"Databases"},{"location":"microservices/configuration/V2MigrationCommonConfig/#registry","text":"No changes","title":"Registry"},{"location":"microservices/configuration/V2MigrationCommonConfig/#clients","text":"The map key names have changed to uses the service key for each of the target services. Each client entry must be changed to use the appropriate service key as follows: CoreData => core-data Metadata => core-metadata Command => core-command Notifications => support-notifications Scheduler => support-scheduler Remove the [Clients.Logging] section","title":"Clients"},{"location":"microservices/configuration/V2MigrationCommonConfig/#secretstore","text":"All service now require the [SecretStore] section. For those that did not have it previously add the following replacing with the service's actual service key: [SecretStore] Type = 'vault' Protocol = 'http' Host = 'localhost' Port = 8200 Path = '/' TokenFile = '/tmp/edgex/secrets//secrets-token.json' RootCaCertPath = '' ServerName = '' [SecretStore.Authentication] AuthType = 'X-Vault-Token' For those service that previously had the [SecretStore] section, make the following changes replacing