Update Doc for Filters

README.md

@@ -4,44 +4,45 @@ FHIR Converter is an open source project that enables conversion of health data
 
 The first version of the FHIR Converter released to open source on Mar 6th, 2020. It used Handlebars template language and Javascript runtime. A new converter engine was released on Nov 13, 2020 that uses Liquid templating language and .Net runtime.
 
-Currently, Microsoft supports two types of converter engines, Handlebars engine and Liquid engine, along with templates and filters. We recommend using Liquid converter for a better alignment with [Azure API for FHIR](https://azure.microsoft.com/en-us/services/azure-api-for-fhir/), [FHIR Server for Azure](https://github.com/microsoft/fhir-server), and [Microsoft Logic Apps](https://azure.microsoft.com/en-us/services/logic-apps/).
+Currently, Microsoft supports two types of converter engines, Handlebars engine and Liquid engine, along with templates and filters. **We strongly recommend using the Liquid converter** for a better alignment with [Azure Healthcare APIs](https://docs.microsoft.com/azure/healthcare-apis/), [Azure API for FHIR](https://azure.microsoft.com/services/azure-api-for-fhir/), [FHIR Server for Azure](https://github.com/microsoft/fhir-server), and [Microsoft Logic Apps](https://azure.microsoft.com/services/logic-apps/).
 
 The following table shows the differences between two converter engines:
 
 |  | Handlebars engine | Liquid engine | 
 | ----- | ----- | ----- |
 | **Template language** | [Handlebars](https://handlebarsjs.com/) | [Liquid](https://shopify.github.io/liquid/) |
 | **Template authoring tool** | Self-hosted web-app | [VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-health-fhir-converter)|
-| **Supported conversions** | 1. HL7v2 to FHIR <br> 2. C-CDA to FHIR | 1. HL7v2 to FHIR <br> 2. C-CDA to FHIR|
-| **Available as** | 1. Self-deployed web service <br> (on-prem or on Azure)| 1. Command line tool <br> 2. $convert-data operation in  [FHIR Server for Azure](https://github.com/microsoft/fhir-server/blob/main/docs/ConvertDataOperation.md) <br> 3. $convert-data operation in [Azure API for FHIR](https://docs.microsoft.com/en-us/azure/healthcare-apis/fhir/convert-data).|
+| **Supported conversions** | 1. HL7v2 to FHIR <br> 2. C-CDA to FHIR | 1. HL7v2 to FHIR <br> 2. C-CDA to FHIR <br> 3. JSON to FHIR **(new!)** |
+| **Available as** | 1. Self-deployed web service <br> (on-prem or on Azure)| 1. Command line tool <br> 2. $convert-data operation in  [FHIR Server for Azure (OSS)](https://github.com/microsoft/fhir-server/blob/main/docs/ConvertDataOperation.md) <br> 3. $convert-data operation in both [Azure API for FHIR](https://docs.microsoft.com/azure/healthcare-apis/azure-api-for-fhir/convert-data) and [Azure Healthcare APIs (preview)](https://docs.microsoft.com/azure/healthcare-apis/fhir/convert-data)|
 
 ⚠ Rest of this document is about the Liquid converter. For the Handlebars converter, please refer to the [Handlebars branch](https://github.com/microsoft/FHIR-Converter/tree/handlebars).
 
-Currently, FHIR Converter supports two types of conversions, **HL7v2 to FHIR** and **C-CDA to FHIR**. The converter uses templates that define mappings between these different data formats. The templates are written in [Liquid](https://shopify.github.io/liquid/) templating language and make use of custom [filters](docs/Filters-and-Tags.md), which make it easy with work with HL7v2 messages and C-CDA documents.
+Currently, FHIR Converter supports three types of conversions, **HL7v2 to FHIR**, **C-CDA to FHIR**, and **JSON to FHIR**. The converter uses templates that define mappings between these different data formats. The templates are written in [Liquid](https://shopify.github.io/liquid/) templating language and make use of custom [filters](docs/Filters-and-Tags.md).
 
 The converter comes with a few ready-to-use templates. If needed, you can create a new template, or modify existing templates to meet your specific conversion requirements.
 
-FHIR Converter with DotLiquid engine transforms the input data into FHIR bundles that are persisted to a FHIR server. The converter is integrated into both [Azure API for FHIR](https://azure.microsoft.com/en-us/services/azure-api-for-fhir/) and [FHIR Server](https://github.com/microsoft/fhir-server) as a [$convert-data](https://docs.microsoft.com/en-us/azure/healthcare-apis/convert-data) operation. It is also available as a command line tool.
+FHIR Converter with DotLiquid engine transforms the input data into FHIR bundles that are persisted to a FHIR server. The converter is integrated into both [Azure API for FHIR](https://docs.microsoft.com/azure/healthcare-apis/azure-api-for-fhir/) and [Azure Healthcare APIs (preview)](https://docs.microsoft.com/azure/healthcare-apis/), as well as in the open-source [FHIR Server](https://github.com/microsoft/fhir-server) as a [`$convert-data` operation](#using-convert-data-in-fhir-server). It is also available as a [command line tool](#using-command-line-tool).
 
 ## Using $convert-data in FHIR server
 
 The `$convert-data` operation is integrated into both Azure API for FHIR and FHIR server to run as part of the service. After enabling `$convert-data` in your server, you can make API calls in the form of ```<<your FHIR server URL>>/$convert-data``` to the server to convert your data into FHIR. In the API call request body, you would include parameters such as `inputData`, `inputDataType`, `templateCollectionReference`, and `rootTemplate`, to specify the message and the type of message you are converting. 
 
 For more information on configuring and using `$convert-data` operation on your server, please refer to these documentation:
 
-- [$convert-data for Azure API for FHIR](https://docs.microsoft.com/en-us/azure/healthcare-apis/fhir/convert-data)
+- [$convert-data for Azure API for FHIR](https://docs.microsoft.com/azure/healthcare-apis/azure-api-for-fhir/convert-data)
+- [$convert-data for Azure Healthcare APIs](https://docs.microsoft.com/azure/healthcare-apis/fhir/convert-data)
 - [$convert-data for FHIR Server](https://github.com/microsoft/fhir-server/blob/main/docs/ConvertDataOperation.md)
 
 ## Using Command line tool
 
 ### Supported parameters
 
-The command line tool is another way of converting data, as well as managing templates. The tool converts a folder containing HL7v2 messages or C-CDA documents into FHIR resources. It accepts the following parameters in the command line:
+The command line tool is another way of converting data, as well as managing templates. The tool converts a folder containing input messages (HL7v2, C-CDA, or JSON) into FHIR resources. It accepts the following parameters in the command line:
 
 | Option | Name | Optionality | Default | Description |
 | ----- | ----- | ----- |----- |----- |
 | -d | TemplateDirectory | Required | | Root directory of templates. |
-| -r | RootTemplate | Required | | Name of root template.<br><br> Valid values for **HL7v2 to FHIR** conversion: "ADT_A01", "OML_O21", "ORU_R01", "VXU_V04". <br><br> Valid values for **C-CDA to FHIR** conversion: "CCD", "ConsultationNote", "DischargeSummary", "HistoryandPhysical", "OperativeNote", "ProcedureNote", "ProgressNote", "ReferralNote", "TransferSummary". |
+| -r | RootTemplate | Required | | Name of root template.<br><br> **HL7v2 to FHIR**: "ADT_A01", "ADT_A02", "ADT_A03", "ADT_A04", "ADT_A05", "ADT_A08", "ADT_A11",  "ADT_A13", "ADT_A14", "ADT_A15", "ADT_A16", "ADT_A25", "ADT_A26", "ADT_A27", "ADT_A28", "ADT_A29", "ADT_A31", "ADT_A47", "ADT_A60", "OML_O21", "ORU_R01", "ORM_O01", "VXU_V04", "SIU_S12", "SIU_S13", "SIU_S14", "SIU_S15", "SIU_S16", "SIU_S17", "SIU_S26", "MDM_T01", "MDM_T02" <br><br> **C-CDA to FHIR**: "CCD", "ConsultationNote", "DischargeSummary", "HistoryandPhysical", "OperativeNote", "ProcedureNote", "ProgressNote", "ReferralNote", "TransferSummary" <br><br> **JSON to FHIR**: "Stu3ChargeItem", "ExamplePatient" <br> (*These JSON templates are sample templates for use, not default templates that adhere to any pre-defined JSON message types. JSON does not have any standardized message types, unlike HL7v2 messages or C-CDA documents. Therefore, instead of "default" templates we provide you with some sample templates that you can use as a starting guide for you to modify and customize.*)  |
 | -c | InputDataContent | Optional| | Input data content. Specify OutputDataFile to get the results. |
 | -n | InputDataFile | Optional| | Input data file. Specify OutputDataFile to get the results. |
 | -f | OutputDataFile | Optional | | Output data file. |
@@ -53,21 +54,35 @@ The command line tool is another way of converting data, as well as managing tem
 
 ### Command line example
 
-Build your executable file, `Microsoft.Health.Fhir.Liquid.Converter.Tool.exe`.
+First, build the executable file, `Microsoft.Health.Fhir.Liquid.Converter.Tool.exe`, in your local directory. Have the input data, as well as the conversion templates, saved in separate folders.
+
+#### Converting data inside folders
+
+You can run the built .exe file in your terminal by running the following command line and specifying the directories:
 
-For converting HL7v2 messages to FHIR resources in a folder:
 ```
 >.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d myTemplateDirectory -r myRootTemplate -i myInputDataFolder -o myOutputDataFolder
 ```
-For converting C-CDA documents to FHIR resources in a folder:
+
+For example, you can run a command like below to convert all C-CDA sample documents inside the `SampleData\Ccda` folder using the default `CCD` template:
+
 ```
->.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d myTemplateDirectory -r myRootTemplate -i myInputDataFolder -o myOutputDataFolder
+>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d {path}\Templates\Ccda -r CCD -i {path}\SampleData\Ccda -o {path to your output folder}
 ```
 
 After running the command line, there will be a series of "Processing..." lines being written in the terminal window. When the conversion is complete, you will see "Conversion completed!" message.
 
+For example, if you were doing the C-CDA to FHIR conversion in a folder as in the command above, you will see something like this:
+
 ![Conversion running on the terminal (screenshot)](docs/conversion-terminal-screenshot.png)
 
+#### Converting individual files
+
+Instead of converting multiples messages and documents inside a folder, you can also convert single files using the parameters in the table above. For example, if you were to convert a C-CDA document into a FHIR resource using the `CCD` template:
+```
+>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -n {path}\SampleData\Ccda\CCD.ccda -d {path}\Templates\Ccda -f {path to your output json file} -r CCD
+```
+
 ## Using Templates
 
 The command line tool supports managing different versions of templates from Azure Container Registry (ACR). You can customize templates and store them in your ACR if default templates are not sufficient for meeting conversion requirements. After [ACR authentication](docs/TemplateManagementCLI.md#authentication), you can pull templates from ACR or push templates to ACR using our command line tool.
@@ -99,6 +114,8 @@ For **HL7v2 to FHIR conversion**, [HL7v2 DotLiquid templates](data/Templates/Hl7
 
 For **C-CDA to FHIR conversion**, [C-CDA DotLiquid templates](data/Templates/Ccda/Utils) generate FHIR resource IDs in two ways: 1) [ID generation template](data/Templates/Ccda/Utils/_GenerateId.liquid) helps generate Patient ID and Practitioner ID; 2) the resource IDs for other resources are generated from the resource object directly.
 
+For **JSON to FHIR conversion**, there is no standardized JSON input message types unlike HL7v2 messages or C-CDA documents. Therefore, instead of default templates we provide you with some sample JSON DotLiquid templates that you can use as a starting guide for your custom JSON conversion templates. You can decide how to generate the resource IDs according to your own inputs, and use our sample templates as a reference.
+
 The Converter introduces a concept of "base resource/base ID". Base resources are independent entities, like Patient, Organization, Device, etc, whose IDs are defined as base ID. Base IDs could be used to generate IDs for other resources that relate to them. It helps enrich the input for hash and thus reduce ID collision.
 For example, a Patient ID is used as part of hash input for an AllergyIntolerance ID, as this resource is closely related with a specific patient.
 

docs/Filters-and-Tags.md

@@ -8,7 +8,7 @@ By default, Liquid provides a set of [standard filters](https://github.com/Shopi
 Besides these filters, FHIR Converter also provides some other filters that are useful in conversion, which are listed below.
 If these filters do not meet your needs, you can also write your own filters.
 
-### Hl7 v2 specific filters
+### Hl7v2 specific filters
 | Filter | Description | Syntax |
 |-|-|-|
 | get_first_segments | Returns first instance of the segments | `{% assign result = hl7v2Data \| get_first_segments: 'segment1\|segment2\|...' -%}` |
@@ -34,6 +34,7 @@ If these filters do not meet your needs, you can also write your own filters.
 | unescape_special_chars | Returns string after removing escaping of special char | `{{ '\\E' \| unescape_special_chars }} #=> '\E'` |
 | match | Returns an array containing matches with a regular expression | `{% assign m = code \| match: '[0123456789.]+' -%}` |
 | to_json_string | Converts to JSON string | `{% assign msgJsonString = msg \| to_json_string -%}` |
+| to_double | Converts string to double | `{{ "100.01" \| to_double }} ` |
 | base64_encode | Returns base64 encoded string | `{{ decodedData \| base64_encode }}` |
 | base64_decode | Returns base64 decoded string | `{{ encodedData \| base64_decode }}` |
 | sha1_hash | Returns SHA1 hash (in hex) of given string | `{{ inputData \| sha1_hash }}` |
@@ -54,9 +55,43 @@ If these filters do not meet your needs, you can also write your own filters.
 ### DateTime filters
 | Filter | Description | Syntax |
 |-|-|-|
-| add_hyphens_date | Adds hyphens to a date without hyphens | `{{ PID.7.Value \| add_hyphens_date }}` |
-| format_as_date_time | Convert an YYYYMMDDHHmmssSSS string, e.g. 20040629175400000 to dateTime format, e.g. 2004-06-29T17:54:00.000z. A parameter could be set to handle time zone with "preserve", "utc" or "local". The default method is "local" | {{ PID.29.Value \| format_as_date_time: 'utc' }} |
-| now | Provides current time in a specific format. The default format is "yyyy-MM-ddTHH:mm:ss.FFFZ" | {{ '' \| now: 'dddd, dd MMMM yyyy HH:mm:ss' }} |
+| add_hyphens_date | Adds hyphens to a date or a partial date that does not have hyphens to make it into a valid FHIR format. The input date format is YYYY, YYYYMM, or YYYYMMDD. The output format is a valid FHIR date or a partial date format: YYYY, YYYY-MM, or YYYY-MM-DD.  | `{{ PID.7.Value \| add_hyphens_date }}` |
+| format_as_date_time | Converts valid HL7v2 and C-CDA datetime to a valid FHIR datetime format. The input datetime format is datetime or partial datetime without hyphens: YYYY[MM[DD[HH[MM[SS[.S[S[S[S]]]]]]]]][+/-ZZZZ]. For example, the input 20040629175400000 will have the output 2004-06-29T17:54:00.000Z. Provides parameters to handle different time zones: preserve, utc, local. The default method is preserve. | `{{ PID.29.Value \| format_as_date_time: 'utc' }}` |
+| now | Provides the current time in a specific format. The default format is yyyy-MM-ddTHH:mm:ss.FFFZ. | `{{ '' \| now: 'dddd, dd MMMM yyyy HH:mm:ss' }}` |
+| add_seconds | Adds double seconds on a valid [FHIR datetime](http://hl7.org/fhir/R4/datatypes.html) (e.g.2021-01-01T00:00:00Z). Provides parameters to handle different time zones: preserve, utc, local. The default method is preserve. | `{{ "2021-01-01T00:00:00Z" \| add_seconds:100.1, 'utc' }}` |
+
+DateTime filters usage examples:
+
+- add_hyphens_date
+```
+    {{ "2001" | add_hyphens_date }} -> 2001
+    {{ "200101" | add_hyphens_date }} -> 2001-01
+    {{ "20010101" | add_hyphens_date }} -> 2001-01-01
+```
+
+- format_as_date_time
+```
+    {{ "20110103143428-0800" | format_as_date_time }} -> 2011-01-03T14:34:28-08:00
+    {{ "20110103143428-0800" | format_as_date_time: 'preserve' }} -> 2011-01-03T14:34:28-08:00
+    {{ "20110103143428-0800" | format_as_date_time: 'utc' }} -> 2011-01-03T22:34:28Z
+```
+>[Note] : `format_as_date_time` and `add_hyphens_date` are used to convert HL7v2 and C-CDA date and datetime format to FHIR. For other date or datetime's reformat, please refer to the standard filter [date](https://shopify.dev/api/liquid/filters/additional-filters#date).
+- now
+```
+    {{ "" | now  }} -> 2022-03-22T06:50:25.071Z // an example time
+    {{ "" | now: 'dddd, dd MMMM yyyy HH:mm:ss' }} -> Tuesday, 22 March 2022 06:52:15
+    {{ "" | now: 'd' }} -> 3/22/2022
+```
+> [Note] : Input string will not affect the result.
+- add_seconds
+```
+    {{ "1970-01-01T00:01:00.000+10:00" | add_seconds: -60, 'utc' }} -> 1969-12-31T14:00:00.000Z
+    {{ "1970-01-01T00:01:00Z" | add_seconds: 60.1230 }} -> 1970-01-01T00:02:00.123Z
+    {{ "1970-01-01T00:01:00+14:00" | add_seconds: 60, 'preserve' }} -> 1970-01-01T00:01:01+14:00
+```
+
+> [Note] : If the input is a partial datetime without time zone, it will be set to the first day of the year and 00:00:00 clock time with local time zone as suffix. e.g. In the location with +08:00 time zone, the input string "201101031434" will be filled to "20110103143400+0800". The template {{ "201101031434" | format_as_date_time: 'utc'}} will output 2011-01-03T06:34:00Z when running at +08:00 location.
+
 
 ### Collection filters
 | Filter | Description | Syntax |