Skip to content

Transformer Developer Guide

Jump to bottom
Kwan edited this page Sep 26, 2019 · 5 revisions

Transformer Developer Guide

Welcome to the Transformer developer guide. The document shows you how to define the translation hints with YANG extensions and write special translation codes using APIs in the Transformer framework. If you are brand new to Translib/Transformer, start with the HLD

Transformer provides an underlying building block for developers to translate data from YANG to ABNF/Redis schema and vice versa, using YANG extensions along the YANG paths. At run time, the YANG extensions are mapped to an in-memory Transformer Spec that provides two-way mapping between YANG and ABNF/Redis schema for Transformer to perform data translation while processing SET/GET operations.

In case that SONiC YANG modules are used by NBI applications, the Transformer performs 1:1 mapping between a YANG object and a SONiC DB object without a need to write special translation codes.

If you use the openconfig YANGs for NBI applications, you may need special handling to translate data between YANG and ABNF schema. In such case, you can annotate YANG extensions and write callbacks to perform translations where required.

In either case, the default application - common-app.go - generically handles both SET and GET requests with data translated by Transformer. Please refer to the appendix - sequence diagram that shows interactions within Translib.  

1. Generate the annotation template file

The goyang package is extended to generate the template annotation file for any input yang files. A new output format type "annotate" can be used to generate the template annotation file.

Add $(SONIC_MGMT_FRAMEWORK)/gopkgs/bin to the PATH to run the goyang binary.

goyang --format=annotate --path=/path/to/yang/models openconfig-acl.yang > openconfig-acl-annot.yang

Sample output: module openconfig-acl-annot {

yang-version "1"

namespace "http://openconfig.net/yang/annotation";
prefix "oc-acl-annot"

import openconfig-packet-match { prefix oc-pkt-match }
import openconfig-interfaces { prefix oc-if }
import openconfig-yang-types { prefix oc-yang }
import openconfig-extensions { prefix oc-ext }

deviation oc-acl:openconfig-acl {
  deviate add {
  }
}

deviation oc-acl:openconfig-acl/oc-acl:acl {
  deviate add {
  }
}

deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state {
  deviate add {
  }
}

deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:state/oc-acl:counter-capability {
  deviate add {
  }
}

deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets {
  deviate add {
  }
}

deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set {
  deviate add {
  }
}

deviation oc-acl:openconfig-acl/oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set/oc-acl:type {
  deviate add {
  }
}

2. Annotate YANG extensions to define translation hints

The translation hints are defined as YANG extensions to support simple table/field name mapping or more complex data translation by overloading the default methods.

  1. sonic-ext:table-name [string]: Map a YANG container/list to TABLE name, processed by the default transformer method. Argument is a table name statically mapped to the given YANG container or list node. The table-name is inherited to all descendant nodes unless another one is defined.

  2. sonic-ext:field-name [string]: Map a YANG leafy - leaf or leaf-list - node to FIELD name, processed by the default transformer method

  3. sonic-ext:key-delimiter [string]: Override the default delimiter, “|”, processed by the default transformer method. Used to concatenate multiple YANG keys of a YANG list into a single DB key. Note that the default delimiter “|” is used when keys are concatenated between current node and the nested nodes, i.e. container/list

  4. sonic-ext:key-name [string]: Fixed key name, used for YANG container mapped to TABLE with a fixed key, processed by the default transformer method. Used to define a fixed key, mainly for container mapped to TABLE key e.g. Redis can have a hash “STP|GLOBAL”

container global
   sonic-ext:table-name “STP”
   sonic-ext:key-name “GLOBAL”

  1. sonic-ext:key-transformer [function]: Overloading default method to generate DB keys(s), used when the key values in a YANG list are different from ones in DB TABLE. A pair of callbacks should be implemented to support 2 way translation - YangToDBfunction, DbToYangfunction

  2. sonic-ext:field-transformer [function]: Overloading default method to generate FIELD value, used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB. A pair of callbacks should be implemented to support 2 way translation - YangToDBfunction, DbToYangfunction

  3. sonic-ext:subtree-transformer [function]: Overloading default method for the current subtree, allows the sub-tree transformer to take full control of translation. Note that, if any other extensions are annotated to the nodes on the subtree, they are not effective. The subtree-transformer is inherited to all descendant nodes unless another one is defined, i.e. the scope of subtree-transformer callback is limited to the current and descendant nodes along the YANG path until a new subtree transformer is annotated. A pair of callbacks should be implemented to support 2 way translation - YangToDBfunction, DbToYangfunction

  4. sonic-ext:db-name [string]: DB name to access data – “APPL_DB”, “ASIC_DB”, “COUNTERS_DB”, “CONFIG_DB”, “FLEX_COUNTER_DB”, “STATE_DB”. The default db-name is CONFIG_DB, Used for GET operation to non CONFIG_DB, applicable only to SONiC YANG. Processed by Transformer core to traverse database. The db-name is inherited to all descendant nodes unless another one. Must be defined with the table-name

  5. sonic-ext:post-transformer [function]: A special hook to update the DB requests right before passing to common-app, analogous to the postponed YangToDB subtree callback that is invoked at the very end by the Transformer. Used to add/update additional data to the maps returned from Transformer before passing to common-app, e.g. add a default acl rule Note that the post-transformer can be annotated only to the top-level container(s) within each module, and called once for the given node during translation

  6. sonic-ext:table-transformer [function]: Dynamically map a YANG container/list to TABLE name(s), allows the table-transformer to map a YANG list/container to table names. Used to dynamically map a YANG list/container to table names based on URI and payload. The table-transformer is inherited to all descendant nodes unless another one is defined

  7. sonic-ext:get-validate [function]: A special hook to validate YANG nodes, to populate data read from database, allows developers to instruct Transformer to choose a YANG node among multiple nodes, while constructing the response payload. Typically used to check the “when” condition to validate YANG node among multiple nodes to choose only valid nodes from sibling nodes.

Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - YangToDBfunction, DbToYangfunction. It is not mandatory to implement both functions. E.g. if you need a translation for GET operation only, you can implement only DbToYangfunction.

The template annotation file can be generated and used by the developers to define extensions to the yang paths as needed to translate data between YANG and ABNF format. Refer to the 3.2.2.7.8 Utilities.

Here is the general guide you can check to find which extensions can be annotated in implementing your model.

1)	If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter 
2)	If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope
3)	If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload 
4)	In Get operation access to non CONFIG_DB, you can use the db-name extension
5)	In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYangxxx function
6)	In case of mapping a container to TABLE/KET, you can use the key-name along with the table-name extension

e.g.

deviation /oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set {
      deviate add {
        sonic-ext:table-name "ACL_TABLE";
        sonic-ext:key-delimiter "_";
      }
    }

deviation /oc-acl:acl/oc-acl:acl-sets/oc-acl:acl-set/oc-acl:acl-entries/oc-acl:acl-entry/oc-acl:actions/oc-acl:config/oc-acl:forwarding-action {
      deviate add {
        sonic-ext:field-name "PACKET_ACTION";
      }
    }

deviation /oc-acl:acl/oc-acl:interfaces {
      deviate add {
        sonic-ext:subtree-transformer "acl_port_bindings_xfmr";
      }
    }

3. Add the list of Openconfig YANG modules and annotation files to the transformer manifest file

Find the YANG models for your feature and add the YANG file names to the manifest file - $(SONIC_MGMT_FRAMEWORK)/config/transformer/models_list. Note that there is no need to add SONiC yangs in this file.

e.g. for openconfig ACL

#List yang models transformer need to load

openconfig-acl-annot.yang
openconfig-acl.yang

4. Implement the overload methods if you need translate data with special handling

There are three types of transformer methods to implement 2-way translation: key-transformer, field-transformer and subtree-transformer. Argument of each transformer is used by the transformer core to dynamically lookup and invoke the function during data translation. A function name is formed by an argument string prefixed by a reserved literal – “YangToDb_” or “DbToYang_”, which distinguishes the direction.

Here is a data structures passed from Transformer to overloaded methods.

type XfmrParams struct {
	d *db.DB
	dbs [db.MaxDB]*db.DB
	curDb db.DBNum
	ygRoot *ygot.GoStruct
	uri string
	oper int
	key string
	dbDataMap *map[db.DBNum]map[string]map[string]db.Value
	param interface{}
}

e.g. for YANG extension defined in the annotation file sonic-ext:subtree-transformer "acl_port_bindings_xfmr";

A pair of functions shall be implemented if you need two way translation: YangToDb_acl_port_bindings_xfmr() and DbToYang_acl_port_bindings_xfmr(). Note that you can implement one if you need to implement 1-way translation only.

e.g. xfmr_acl.go

var YangToDb_acl_port_bindings_xfmr SubTreeXfmrYangToDb = func(inParams XfmrParams) (map[string]map[string]db.Value, error) {
	var err error
	res_map := make(map[string]map[string]db.Value)
	aclTableMap := make(map[string]db.Value)
	log.Info("YangToDb_acl_port_bindings_xfmr: ", inParams.ygRoot, inParams.uri)

	aclObj := getAclRoot(inParams.ygRoot)
	if aclObj.Interfaces == nil {
		return res_map, err
	}
. . .
}
var DbToYang_acl_port_bindings_xfmr SubTreeXfmrDbToYang = func(inParams XfmrParams) error {
	var err error
	data := (*inParams.dbDataMap)[inParams.curDb]
	log.Info("DbToYang_acl_port_bindings_xfmr: ", data, inParams.ygRoot)
. . .
}

Overloaded transformer functions are grouped by YANG module, to have a separate GO file. At init(), all the functions need to be bind for dynamic invocation. e.g. xfmr_acl.go

func init () {
    XlateFuncBind("YangToDb_acl_entry_key_xfmr", YangToDb_acl_entry_key_xfmr)
    XlateFuncBind("DbToYang_acl_entry_key_xfmr", DbToYang_acl_entry_key_xfmr)
    XlateFuncBind("YangToDb_acl_l2_ethertype_xfmr", YangToDb_acl_l2_ethertype_xfmr)
    XlateFuncBind("DbToYang_acl_l2_ethertype_xfmr", DbToYang_acl_l2_ethertype_xfmr)
    XlateFuncBind("YangToDb_acl_ip_protocol_xfmr", 
. . .
}
Clone this wiki locally