Moving files between Azure Storage and RHEL

There are several options when you want to move files in or out of an Azure Storage account to a Red Hat Linux (RHEL) server.  Below is a quick break down of the most commonly used options.

Azure CLI

Azure CLI is design to run on Linux or Windows so is the ideal tool when a Linux machine is involved.  Azure CLI 2.0 (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) is the latest and preferred version.  See https://docs.microsoft.com/en-us/azure/storage/common/storage-azure-cli#create-and-manage-blobs for syntax.

Example:

#Start the copy of the blob

az storage blob copy start –account-name ${storage_account} –account-key ${key} –source-account-name ${source_account} –source-account-key ${source_key} –source-container ${blob_container} –source-blob ${blob_name} –destination-container $target_container –destination-blob $target_blob_name

#Now wait while this copy happening. This could take a while.

while [ $(az storage blob show –account-name ${storage_account} –account-key ${key} –container-name $target_container –name $target_blob_name| jq .properties.copy.status| tr -d ‘”‘) == “pending” ]; do

progress=$(az storage blob show –account-name ${storage_account} –account-key ${key} –container-name ${target_container} –name ${target_blob_name} | jq -r .properties.copy.progress)

done_count=$(echo $progress | sed ‘s,\([0-9]*\)/\([0-9]*\),\1,g’)

total_count=$(echo $progress | sed ‘s,\([0-9]*\)/\([0-9]*\),\2,g’)

progress_percent=$((100 * $done_count / $total_count))

echo “Copying: $progress ($progress_percent %)” && sleep 5

done

echo “Copying done”

AzCopy

AzCopy on Linux is a command-line utility designed for copying data to/from Azure Blob and File storage using simple commands.  See https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-linux?toc=%2fazure%2fstorage%2ffiles%2ftoc.json.  Specific steps for the file copy are here.

To Install on RHEL, you first need to install .NET Core 1.1.1

  1. sudo yum install rh-dotnetcore11
  2. scl enable rh-dotnetcore11 bash
  3. source scl_source enable rh-dotnetcore11

Then you install AzCopy per the article.

  1. wget -O azcopy.tar.gz https://aka.ms/downloadazcopyprlinux
  2. tar -xf azcopy.tar.gz
  3. sudo ./install.sh

Example from RHEL 7.4:

clip_image002

Wget

The final approach is to downloading is to generate a SAS token, append it to the blob URL and simply use wget in Linux.

See https://blogs.msdn.microsoft.com/nicole_welch/2017/05/using-azure-cli-1-0-to-copy-a-file-between-subscriptions/ for details on how to generate the SAS token and append it to the the blob URL (aka, “sourceSAS”).

 

*previously posed on https://blogs.msdn.microsoft.com/nicole_welch/2017/09/moving-files-between-azure-storage-and-rhel/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s