Ship logs to Logz.io using Azure Blob Trigger

Auto-deployment of Azure resources for shipping logs from Azure storage using blob trigger. Each new log in the container path inside the storage account (including sub directories), will trigger the Logz.io function that will ship the file content to Logz.io.

Getting Started

There are 2 options for auto-deployment. Choose the one that is suitable for you and follow the instructions:

Full Auto-Deployment

If you don't have storage account (general purpose v2) with container with logs, or you want to create everything from scratch, this auto-deployment is for you. Will auto-deploy the following resources:

• Storage Account (general purpose v2) + Container
• App Service Plan - Consumption Plan
• Application Insights
• Logz.io Function App + Logz.io Blob Trigger Function
• Storage Account for Logz.io Function App logs

Press the button to start:

Logz.io Function Auto-Deployment

If you already have storage account (general purpose v2) with container with logs, this auto-deployment is for you. Will auto-deploy the following resources:

• App Service Plan - Consumption Plan
• Application Insights
• Logz.io Function App + Logz.io Blob Trigger Function
• Storage Account for Logz.io Function App logs

• When deploying the Logz.io function, logs that were in the container before the deployment will be shipped to Logz.io. If these logs have already been shipped to Logz.io, we recommend that you empty the container before the deployment or use the FilterDate and FilterDateJsonPath parameters.

Instructions

You'll be taken to Azure Custom deployment page. Fill in all the parameters and click Review + create button:

Parameter	Description	Is Required	Value If Empty
Storage Account Name	The storage account (general purpose v2) name.	Required	-
Storage Account Resource Name	The resource name that contains the storage account. (Needed only in Logz.io Function Auto-Deployment)	Required	-
Container Name	The name of the container inside the storage account	Required	-
Format	The format of the log files.	Required	-
Logzio URL	The Logz.io listener URL fot your region. (For more details, see the regions page: https://docs.logz.io/user-guide/accounts/account-region.html)	Required	-
Logzio Token	Your Logz.io logs token. (Can be retrieved from the Manage Token page.)	Required	-
Logs Path	The path from where blob files will trigger the Logz.io function (including subdirectories in that path). Leave empty if you want that every blob file in the container will trigger the Logz.io function.	Not Required	<<ContainerLogsPath>>/{name}
Multiline Regex	The regex that matches the multiline logs in text blob files. Leave empty if you do not use multiline logs in your text blob files.	Not Required	NO_REGEX
Datetime Filter	Every log with datetime greater or equal to this datetime (for example: 2021-11-05T10:10:10) will be shipped to Logz.io (for it to take effect DatetimeFinder and DatetimeFormat must not be empty). Leave empty if you want all logs to be shipped to Logz.io.	Not Required	NO_DATETIME_FILTER
Datetime Finder	If file is csv/json: write the json path of the datetime field inside each log. CSV json path will always be the name of the datetime field. Json json path can be the name of the datetime field if it's in the root, or a path contains fields separated by '.' (for example: metadata.datetime, metadata[:1].datetime). If file is text: write a regex that will get the datetime from each log. If log has many occurrences of datetime, make sure the regex will give the right one (for example: '(?:.*?[0-9]){2}.*?([0-9])' will give the third digit). If this value cannot be found inside a log, the log will be shipped to Logz.io. Leave empty if you are not using DatetimeFilter.	Required if using DatetimeFilter	NO_DATETIME_FINDER
Datetime Format	The datetime format of DatetimeFilter and datetime field in each log (for example: %Y/%m/%dT%H:%M:%S%z is for 2021/11/01T10:10:10+0000 datetime). If the format is wrong, the log will be shipped to Logz.io. Leave empty if you are not using DatetimeFilter.	Required if using DatetimeFilter	NO_DATETIME_FORMAT

On the following screen, press the create button:

If everything went well, you should see the following screen. Press Go to resource group button to go to your resource group with all the created resources:

Resources

Supported Data Types

The Logz.io function supports the following data types:

• Json
• CSV
• Text (supports multiline text - MultilineRegex parameter)

The file name **does not** have to be with these extensions.

Supported File Formats

The Logz.io function supports the following file formats:

• Gzip

The file name **does not** have to be with these extensions.

Searching in Logz.io

All logs that were sent from the function will be under the type azure_blob_trigger

changelog

• v1.0.7:
  • Delete git-secrets workflow
  • Update azure extension bundle [2.*, 3.0.0) -> [4.0.0, 5.0.0)
  • Update Python version 3.8 -> 3.10
• v1.0.6:
  • Function does not send logs and throws exception. Update upload workflow node version to 16 and azure-functions-core-tools to 4 (handles functions with runtime version 4).
• v1.0.5:
  • Support of sending logs of log files with empty lines.
• v1.0.4:
  • Upgrade runtime version from ~3 to ~4
• v1.0.3:
  • Add datetime filter to text format
• v1.0.2:
  • Format parameter tells in which format the logs are. Available formats: JSON, CSV, TEXT
• v1.0.1:
  • filter files logs by date.
• v1.0.0:
  • Initial release