How to Sync Two Azure File Shares with an Automation Account

Azure File Shares provide a seamless way to store and access files in the cloud using standard SMB protocols. However, when working with multiple file shares across regions, keeping them in sync can be challenging. In this blog, I will show you how to sync two Azure File Shares - one in West India (source) and another in South India (destination) - using an Azure Automation Account and a PowerShell-based runbook.

Use Case

We have:

Two storage accounts:
- Central India Storage Account: Contains the source file share.
- West India Storage Account: Contains the destination file share.
Azure Automation Account: aac-sync-afs267 is used to run a PowerShell script for syncing. We will make use of the AzCopy tool, which is a command-line utility that you can use to sync or copy files to or from a storage account. We will also use Azure Container Instances to simplify and automate the AzCopy script in Runbook, which will run as part of the container. In this way, we can run the container on a simple schedule to sync the data and only get billed for the time the container was used.
Goal: Automatically sync files from the source file share to the destination file share on a schedule using an Automation Runbook.

Steps to Sync Azure File Shares

A. Set Up Azure Storage Accounts and File Shares

Navigate to the Azure Portal and create the Resource Group where we want place all our resources for the demo:
- Name: rg-afs-sync-wind267
- Region: West India.
Navigate to the Azure Portal and create two Storage Accounts and their respective File Shares:
- Storage Account1 Name: saafswind267
- File Share1 Name: afs-replication-source-wind267
- Region: West India.
- Storage Account2 Name: saafssind267
- File Share2 Name: afs-replication-dest-sind267
- Region: South India.
Upload your demo files and folders to the source file share

afs-replication-source-wind267
Go to the Automation Account and import the required modules:
- Go to Modules Gallery.
- Search for and import the following:
  - Az.Accounts
  - Az.Storage
  - Az.ContainerInstance

B. Set Up Azure Automation Account

Navigate to the Azure Portal and create a system assigned Automation Account if you don’t already have one:
- Name: aac-sync-afs267
- Region: Choose any region (this won’t affect cross-region syncing).
Go to the Automation Account and import the required modules:
- Under Shared Resources Go to Modules Gallery.
- Ensure that the following modules are enabled:
  - Az.Accounts
  - Az.Storage
  - Az.ContainerInstance

C. Grant Permissions to the Automation Account

Ensure that the Automation Account has the necessary permissions to access both storage accounts:

Navigate to your demo Resource Group or each of your Storage Accounts.
Go to Access Control IAM » Role Assignments » Add Role Assignment
Assign the Contributor role to the Automation Account's Managed Identity:
- Go to Access Control (IAM) → Add Role Assignment.
- Assign the role to the Automation Account.

D. Create the PowerShell Runbook

In your Automation Account, go to Runbooks → Create a Runbook.
Enter details:
- Name: SyncAzureFileShares
- Runbook Type: PowerShell
- Description: Sync files between two Azure File Shares.
Paste the following script into the Runbook editor and click Test Pane:

On the Test page, you need to supply the following parameters manually and then click the Start button to test the automation script.
- SOURCEAZURESUBSCRIPTIONID
- SOURCESTORAGEACCOUNTRG
- TARGETSTORAGEACCOUNTRG
- SOURCESTORAGEACCOUNTNAME
- TARGETSTORAGEACCOUNTNAME
- SOURCESTORAGEFILESHARENAME
- TARGETSTORAGEFILESHARENAME

PowerShell Script for Syncing Azure File Shares

<#
.DESCRIPTION
A Runbook example that automatically creates incremental backups of an Azure Files system on a customer-defined schedule and stores the backups in a separate storage account.
It does so by leveraging AzCopy with the sync parameter, which is similar to Robocopy /MIR. Only changes will be copied with every backup, and any deletions on the source will be mirrored on the target.
In this example, AzCopy is running in a Container inside an Azure Container Instance using Service Principal in Azure AD.

.NOTES
Filename : SyncBetweenTwoFileShares
Author1  : Sibonay Koo (Microsoft PM)
Author2  : Charbel Nemnom (Microsoft MVP/MCT)
Version  : 2.1
Date     : 13-February-2021
Updated  : 18-April-2022
Tested   : Az.ContainerInstance PowerShell module version 2.1 and above

.LINK
To provide feedback or for further assistance please email: azurefiles[@]microsoft.com
#>

Param (
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceAzureSubscriptionId,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageAccountRG,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageAccountRG,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageAccountName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageAccountName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $sourceStorageFileShareName,
    [Parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]
    [String] $targetStorageFileShareName
)

# Azure File Share maximum snapshot support limit by the Azure platform is 200
[Int]$maxSnapshots = 200

# Ensures you do not inherit an AzContext in your runbook
Disable-AzContextAutosave -Scope Process

# Connect to Azure with system-assigned managed identity (automation account)
Connect-AzAccount -Identity

# SOURCE Azure Subscription
Select-AzSubscription -SubscriptionId $sourceAzureSubscriptionId

#! Source Storage Account in the primary region
# Get Source Storage Account Key
$sourceStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $sourceStorageAccountRG -Name $sourceStorageAccountName).Value[0]

# Set Azure Storage Context
$sourceContext = New-AzStorageContext -StorageAccountKey $sourceStorageAccountKey -StorageAccountName $sourceStorageAccountName

# List the current snapshots on the source share
$snapshots = Get-AzStorageShare `
    -Context $sourceContext.Context | `
Where-Object { $_.Name -eq $sourceStorageFileShareName -and $_.IsSnapshot -eq $true}

# Delete the oldest (1) manual snapshot in the source share if have 180 or more snapshots (Azure Files snapshot limit is 200)
# This leaves a buffer such that there can always be 6 months of daily snapshots and 10 yearly snapshots taken via Azure Backup
# You may need to adjust this buffer based on your snapshot retention policy
If ((($snapshots.count)+20) -ge $maxSnapshots) {
    $manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Initiator -eq "Manual"}
    Remove-AzStorageShare -Share $manualSnapshots[0].CloudFileShare -Force
}

# Take manual snapshot on the source share
# When taking a snapshot using PowerShell, CLI or from the Portal, the snapshot's metadata is set to a key-value pair with the key being "Manual"
# The value of "AzureBackupProtected" is set to "True" or "False" depending on whether Azure Backup is enabled
$sourceShare = Get-AzStorageShare -Context $sourceContext.Context -Name $sourceStorageFileShareName
$sourceSnapshot = $sourceShare.CloudFileShare.Snapshot()

# Generate source file share SAS URI
$sourceShareSASURI = New-AzStorageShareSASToken -Context $sourceContext `
  -ExpiryTime(get-date).AddDays(1) -FullUri -ShareName $sourceStorageFileShareName -Permission rl
# Set source file share snapshot SAS URI
$sourceSnapSASURI = $sourceSnapshot.SnapshotQualifiedUri.AbsoluteUri + "&" + $sourceShareSASURI.Split('?')[-1]

#! TARGET Storage Account in a different region
# Get Target Storage Account Key
$targetStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $targetStorageAccountRG -Name $targetStorageAccountName).Value[0]

# Set Target Azure Storage Context
$destinationContext = New-AzStorageContext -StorageAccountKey $targetStorageAccountKey -StorageAccountName $targetStorageAccountName

# Generate target SAS URI
$targetShareSASURI = New-AzStorageShareSASToken -Context $destinationContext `
    -ExpiryTime(get-date).AddDays(1) -FullUri -ShareName $targetStorageFileShareName -Permission rwl

# Check if target file share contains data
$targetFileShare = Get-AzStorageFile -Sharename $targetStorageFileShareName -Context $destinationContext.Context

# If target share already contains data, use AzCopy sync to sync data from source to target
# Else if target share is empty, use AzCopy copy as it will be more efficient
# By default, the replicated files that you deleted from the source does not get deleted on the destination.
# The deletion on the destination is optional because some customers want to keep them as backup in the DR location.
# If you want to delete the files, then you want to add the " --delete-destination true" flag to the $command below.
# The "--delete-destination" defines whether to delete extra files from the destination that are not present at the source.
if ($targetFileShare) {
     $command = "azcopy","sync",$sourceSnapSASURI,$targetShareSASURI,"--preserve-smb-info","--preserve-smb-permissions","--recursive"
}
Else {
     $command = "azcopy","copy",$sourceSnapSASURI,$targetShareSASURI,"--preserve-smb-info","--preserve-smb-permissions","--recursive"
}

# The container image (peterdavehello/azcopy:latest) is publicly available on Docker Hub and has the latest AzCopy version installed
# You could also create your own private container image and use it instead
# When you create a new container instance, the default compute resources are set to 1vCPU and 1.5GB RAM
# We recommend starting with 2vCPU and 4GB memory for larger file shares (E.g. 3TB)
# You may need to adjust the CPU and memory based on the size and churn of your file share
# The container will be created in the $location variable based on the source storage account location. Adjust if needed.
$location = (Get-AzResourceGroup -Name $sourceStorageAccountRG).location
$containerGroupName = "syncafsdrjob"

# Set the AZCOPY_BUFFER_GB value at 2 GB which would prevent the container from crashing.
$envVars = New-AzContainerInstanceEnvironmentVariableObject -Name "AZCOPY_BUFFER_GB" -Value "2"

# Create Azure Container Instance Object
$container = New-AzContainerInstanceObject `
-Name $containerGroupName `
-Image "peterdavehello/azcopy:latest" `
-RequestCpu 2 -RequestMemoryInGb 4 `
-Command $command -EnvironmentVariable $envVars

# Create Azure Container Group and run the AzCopy job
$containerGroup = New-AzContainerGroup -ResourceGroupName $sourceStorageAccountRG -Name $containerGroupName `
-Container $container -OsType Linux -Location $location -RestartPolicy never

# List the current snapshots on the target share
$snapshots = Get-AzStorageShare `
    -Context $destinationContext.Context | `
Where-Object { $_.Name -eq $targetStorageFileShareName -and $_.IsSnapshot -eq $true}

# Delete the oldest (1) manual snapshot in the target share if have 190 or more snapshots (Azure Files backup snapshot limit is 200)
If ((($snapshots.count)+10) -ge $maxSnapshots) {
    $manualSnapshots = $snapshots | where-object {$_.ShareProperties.Metadata.Initiator -eq "Manual"}
    Remove-AzStorageShare -Share $manualSnapshots[0].CloudFileShare -Force
}

# Take manual snapshot on the target share
# When taking a snapshot using PowerShell, CLI or from the Portal, the snapshot's metadata is set to a key-value pair with the key being "Manual"
# The value of "AzureBackupProtected" is set to "True" or "False" depending on whether Azure Backup is enabled
$targetShare = Get-AzStorageShare -Context $destinationContext.Context -Name $targetStorageFileShareName
$targetShareSnapshot = $targetShare.CloudFileShare.Snapshot()

Write-Output ("")

E. Publish and Test the Runbook

Publish the Runbook: Click Save and then Publish.
Test the Runbook:
- Go to Runbooks → SyncAzureFileShares → Start.
- Provide the following parameters:
  - Source Storage Account: Central India storage account name
  - Source File Share: Name of the file share in Central India
  - Source Resource Group: Resource group of the Central India storage account
  - Destination Storage Account: West India storage account name
  - Destination File Share: Name of the file share in West India
  - Destination Resource Group: Resource group of the West India storage account

F. Set a Schedule for the Runbook

Navigate to the Runbook and go to Schedules → Add a Schedule.
Create a new schedule:
- Name: SyncAzureFileSharesSchedule.
- Recurrence: Choose daily, hourly, or your preferred interval.
Link the schedule to the Runbook.

Clean up

Go to the Azure Portal » Resource Groups » Select the demo resource group rg-afs-sync-wind267 click Delete resource group, confirm the delete and click Delete.

Conclusion

With this setup, your Azure file shares in West India and South India will stay synchronized automatically based on the schedule. This solution is scalable, efficient, and ensures your file shares remain up to date without manual intervention.

By automating this sync process, you can focus more on your core tasks and less on operational overhead. Happy automating! 😊