Skip to content

granturing/notebook-monitoring

Folders and files

NameName
Last commit message
Last commit date

Latest commit

7fd77ba · Nov 3, 2022

History

3 Commits
Oct 24, 2022
Nov 3, 2022
Oct 24, 2022

Repository files navigation

Databricks Notebook Command Monitoring

Example of using open source static analysis tools to monitor notebook command logs for security

Background

This uses the Databricks verbose notebook audit logs to process commands and run them through a static analysis tool to detect possible security issues. For this example, it uses the Pyre/Pysa static analysis tool as it has data and control flow analysis with built-in rules for security and the ability to define custom rules. We've included some example configurations for Databricks notebooks under conf/ to enable signatures for dbutils, and the Spark DataFrame API as well as taint configurations.

Instructions

Clone this project into a Databricks repo, attach to a standard or single-user cluster with permissions to read your workspace audit logs. This assumes you've already enabled verbose audit logs and ingested them into a Delta Lake table. Open the src/process-command-logs.py file in Databricks and attach to your cluster. You can change the lookback_days and audit_log_table parameters based on your environment. Then use Run All which will process the audit logs, process them through Pyre/Pysa and generate an embedded report.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages