Prototype M365 Auditing Changes and Enhancements

[schrolla]

💡 Summary

The M365 unified audit log capability tracks actions taken across many of the M365 services. The log types supported depend on services in use, tenant licensing, and licenses applied to individual users. This epic is built around using identified changes to test audit policies from previous work to evaluate feasibility of implementing checks for updated baseline auditing policies.

Motivation and context

Auditing is a critical component for monitoring SaaS usage patterns, potential misuse, and detecting threats. Based on the expanded availability of several log types previously only available to Purview Premium and the publication of the Microsoft Expanded Cloud Logs Implementation Playbook, SCuBA baselines should be reviewed and updated to keep pace with these service updates and latest guidance.

Implementation notes

Prototyping auditing policy and assessment check enhancements will include:

• Review previously identified baseline policy recommendations and test ability to assess through automation/APIs
• Determine if per user checks will cause significant performance issues as the number of users/mailboxes in a tenant scales up

Work Issues

• Conduct hands-on examination of organization and user level audit settings for Exchange Online mailboxes #1433

Acceptance criteria

The following issues are completed

• Purview premium assessment check successfully implemented Update Purview Premium assesment to check status for all users #88
• New or changes to existing audit policies have been identified and agreed upon with TCOs
• Automated assessment checks for new/changed audit policies have been successfully tested
• Baseline audit policies updated to address latest service updates and guidance

[schrolla]

Need to further refine. May split into multiple epics.

[Sloane4]

Script to test timing of querying user mailbox audit settings for all users who have the default audit settings.

Connect-ExchangeOnline -AppId  '' -Organization '' -CertificateThumbprint ''
$Start = Get-Date
# $Mailbox = $(Get-Mailbox -ResultSize unlimited).Count
$Mailbox = $(Get-Mailbox -Filter 'AuditEnabled -eq $true' -ResultSize unlimited | Select-Object -Property Id,DefaultAuditSet | Where-Object {$_.DefaultAuditSet -contains "Admin" -and $_.DefaultAuditSet -contains "Delegate" -and $_.DefaultAuditSet -contains "Owner" }).Count
$TotalTime = NEW-TIMESPAN -Start $Start -End $(Get-Date)
Disconnect-ExchangeOnline -Confirm:$false
Write-Output "$($Mailbox) Mailboxes pulled"
Write-Output "Total Time: $TotalTime

[Sloane4]

Script to generate temporary user mailboxes to use in performance testing per user audit checks against tenants with a given number of mailboxes.

[CmdletBinding()]
param (
    [Parameter(Mandatory=$false)]
    [Alias('id')]
    [string]$AppID = '',

    [Parameter(Mandatory=$false)]
    [Alias('d')]
    [string]$Domain = '',

    [Parameter(Mandatory=$false)]
    [Alias('c')]
    [string]$Certificate = '',

    [Parameter(Mandatory=$false)]
    [Alias('a')]
    [int]$Amount = 5000
)

Import-Module -Name Microsoft.Graph.Users
Import-Module -Name Microsoft.Graph.Groups


# Create mailbox & user
Connect-ExchangeOnline -AppId $AppID -Organization $Domain -CertificateThumbprint $Certificate

for ($i=1; $i -le $Amount; $i++) {
    $RandomWord = 'z' + -join ((65..90) + (97..122) | Get-Random -Count 10 | % {[char]$_})

    $DisplayName = $RandomWord
    $Password = '!AccountPassword!'
    $UserPrincipalName = $RandomWord + '@' + $Domain
    $NewUser = New-Mailbox -Alias $DisplayName -Name $DisplayName -FirstName $DisplayName -LastName "Test" -DisplayName "$DisplayName Test" -MicrosoftOnlineServicesID $UserPrincipalName -Password $(ConvertTo-SecureString -String $Password -AsPlainText -Force)

    $Users += @(@{
        DisplayName = $DisplayName
        Id = $NewUser.ExternalDirectoryObjectId
    })
}

Disconnect-ExchangeOnline -Confirm:$false

# Lock down user
Connect-MgGraph -ClientID $AppID -TenantId $Domain -CertificateThumbprint $Certificate -NoWelcome
$Details = Get-MgContext
$Scopes = $Details | Select -ExpandProperty Scopes
$Scopes = $Scopes -Join ", "
$OrgName = (Get-MgBetaOrganization).DisplayName

Write-Host "Microsoft Graph Connection Information"
Write-Host "--------------------------------------"
Write-Host " "
Write-Host ("Connected to Tenant {0} ({1}) as account {2}" -f $Details.TenantId, $OrgName, $Details.Account)
Write-Host "+-------------------------------------------------------------------------------------------------------------------+"
Write-Host ("Profile set as {0}. The following permission scope is defined: {1}" -f $ProfileName, $Scopes)
Write-Host ""

Start-Sleep -Seconds 15
foreach ($User in $Users) {
    if ($User.Id -ne "") {
        Write-Output "Disabling User: $($User.DisplayName)"
        Update-MgBetaUser -UserId $User.Id -AccountEnabled:$false
        New-MgBetaGroupMember -GroupId "0a1c00d0-1032-4d87-b713-180c05430619" -DirectoryObjectId $User.Id
    }
    else {
        Write-Warning "Failed to make mailbox for user: $($User.Id)"
    }
}
Disconnect-MgGraph

[tkol2022]

New policy proposal

Implement a new policy to check for users that have their mailbox audit logging set to bypass. According to Microsoft, "you can't disable mailbox auditing for specific mailboxes when mailbox auditing on by default is turned on in your organization. However, you can still use the Set-MailboxAuditBypassAssociation cmdlet in Exchange Online PowerShell to prevent all mailbox actions by the specified users from being logged."

This has been logged as a separate issue so we don't need to adjudicate it here but we can certainly examine it if we want to during any current hands-on prototyping.

[buidav]

Might want to consider this policy as well. Disable PowerShell access to inboxes from non-administrative users. Documentation
This was suggested to us during the RFC, but we didn't have the capability to check this in a timely manner. This also has to be done in PowerShell no admin center controls.

This has been logged as a separate issue so we don't need to adjudicate it here.

[tkol2022]

AuditEnabled implications for queries

*My conclusion here needs to be verified.

After reading the mailbox audit bypass article it says "you can't disable mailbox auditing for specific mailboxes when mailbox auditing on by default is turned on in your organization. For example, setting the AuditEnabled mailbox property to False is ignored."

If I understand this correctly, this has implications for queries such as the one in the previous comment that use the AuditEnabled in a filter as shown below:
Get-Mailbox -Filter 'AuditEnabled -eq $true' -ResultSize unlimited

It seems like the article says that if you detect that mailbox audit logging is turned on for the organization, then it wouldn't make sense to execute a query that filters on AuditEnabled because the article says that the property is AuditEnabled ignored in that case. Here is how to check if mailbox audit logging is on for the organization (a value of false means that it is turned on):

Get-OrganizationConfig | Format-List AuditDisabled

[tkol2022]

Get-EXOMailbox yields significant performance improvement over Get-Mailbox

While doing some research of the cmdlets associated with this issue, I learned that Get-EXOMailbox is an enhanced version of the command and via testing it yields some noticeable performance improvements compared to Get-Mailbox both with and without a filter. During some executions Get-Mailbox can take significantly longer. For example during one execution it took 36 seconds versus 2 seconds for Get-EXOMailbox, although that was not the norm.

Preliminary results are below. On a tenant with 517 mailboxes, Get-EXOMailbox was ~ 3 times faster on average. It will be interesting to see what the time difference is on larger tenants.
Get-EXOMailbox | Measure-Object | Select-Object -ExpandProperty Count

Below are average execution times when using a filter clause which gets evaluated in the back-end REST API:

Below are average execution times without a filter:

Code to measure the execution times. Save as a script.

# Set the number of loops
$NumberOfLoops = 10

# Initialize variables to store total execution times
$TotalTimeGetMailbox = [TimeSpan]::Zero
$TotalTimeGetEXOMailbox = [TimeSpan]::Zero

# Loop to measure execution times for both cmdlets
for ($i = 0; $i -lt $NumberOfLoops; $i++) {
    # Measure Get-Mailbox execution time
    $TimeTakenGetMailbox = Measure-Command {
        # Get-Mailbox -ResultSize unlimited -Filter 'AuditEnabled -eq $false'
        Get-Mailbox -ResultSize unlimited
    }
    $TotalTimeGetMailbox += $TimeTakenGetMailbox

    # Measure Get-EXOMailbox execution time
    $TimeTakenGetEXOMailbox = Measure-Command {
        # Get-EXOMailbox -ResultSize unlimited -Filter 'AuditEnabled -eq $false'
        Get-EXOMailbox -ResultSize unlimited
    }
    $TotalTimeGetEXOMailbox += $TimeTakenGetEXOMailbox
    Write-Host "Get-Mailbox: $($TimeTakenGetMailbox.TotalSeconds). Get-EXOMailbox: $($TimeTakenGetEXOMailbox.TotalSeconds)."
}

# Calculate average execution times
$AverageTimeGetMailbox = $TotalTimeGetMailbox.TotalSeconds / $NumberOfLoops
$AverageTimeGetEXOMailbox = $TotalTimeGetEXOMailbox.TotalSeconds / $NumberOfLoops

# Print average execution times
Write-Host "Average execution time for Get-Mailbox: $AverageTimeGetMailbox seconds"
Write-Host "Average execution time for Get-EXOMailbox: $AverageTimeGetEXOMailbox seconds"