Harness the Power of Azure Batch with Azure CLI: A Step-by-Step Tutorial

Azure Batch is a service provided by Microsoft Azure that helps in running large-scale parallel and high-performance computing applications efficiently in the cloud. Azure CLI, on the other hand, is a command-line interface for interacting with Azure services. In this tutorial, we will use Azure CLI to create an Azure Batch account and perform some basic operations.

Prerequisites

• An Azure subscription
• Azure CLI installed on your machine.

Steps

1. Set the value for the RESOURCE_GROUP variable with the name of the resource group where you want to create the Azure Batch account.

RESOURCE_GROUP=<your resource group>  

2. Generate a random value for the BATCH_ACCOUNT variable to create a unique name for your Batch account.

BATCH_ACCOUNT=batchaccount$RANDOM  

3. Create a new Batch account using the az batch account create command. Replace <choose a location from the list above> with your preferred location.

az batch account create \  
 --name $BATCH_ACCOUNT \  
 --resource-group $RESOURCE_GROUP \  
 --location <choose a location from the list above>  

4. Log in to your new Batch account using the az batch account login command.

az batch account login \  
 --name $BATCH_ACCOUNT \  
 --resource-group $RESOURCE_GROUP \  
 --shared-key-auth  

5. Create a new pool for the Batch account using the az batch pool create command. We are using an Ubuntu image here, but you can replace it with any other image you prefer.

az batch pool create \  
 --id mypool --vm-size Standard_A1_v2 \  
 --target-dedicated-nodes 3 \  
 --image canonical:ubuntuserver:18.04-LTS \  
 --node-agent-sku-id "batch.node.ubuntu 18.04"  

6. Check the state of the pool using the az batch pool show command.

az batch pool show --pool-id mypool \  
 --query "allocationState"  

7. Create a new job using the az batch job create command.

az batch job create \  
 --id myjob \  
 --pool-id mypool  

8. Create a new task for the job using the az batch task create command. This command is used in a loop to create 10 tasks.

for i in {1..10}  
do  
   az batch task create \  
    --task-id mytask$i \  
    --job-id myjob \  
    --command-line "/bin/bash -c 'echo \$(printenv | grep \AZ_BATCH_TASK_ID) processed by; echo \$(printenv | grep \AZ_BATCH_NODE_ID)'"  
done  

9. List the files associated with the task using the az batch task file list command.

az batch task file list \  
 --job-id myjob1 \  
 --task-id mytask5 \  
 --output table  

10. Download the stdout file associated with each task using the az batch task file download command. This command is used in a loop to download the stdout files for all tasks.

mkdir taskoutputs && cd taskoutputs  
  
for i in {1..10}  
do  
az batch task file download \  
    --job-id myjob2 \  
    --task-id mytask$i \  
    --file-path stdout.txt \  
    --destination ./stdout$i.txt  
done  

11. Display the contents of the downloaded stdout files using the cat command.

cat stdout1.txt && cat stdout2.txt  

Considerations

1. Group related resources into a single resource group for easier management and cost control.
2. Use a unique name for your Batch account to avoid conflicts with existing names.
3. When creating a new pool, specify the minimum number of nodes required to run your tasks to save costs.
4. Use a virtual machine (VM) size that is appropriate for the tasks you want to run.
5. Schedule tasks to run at off-peak hours to take advantage of lower compute costs.
6. Configure your tasks to use all available resources to optimize performance.
7. Use auto-pooling to dynamically adjust the size of the pool according to the workload.
8. Keep track of the status and progress of your tasks using the Azure Batch Explorer or other monitoring tools.
9. Use Azure CLI batch commands within scripts for easier automation and repeatable operations.
10. Clean up unused or idle resources to reduce costs and keep your environment organized.

In this tutorial, we learned how to use Azure CLI to create an Azure Batch account, create a pool, job, and task associated with the pool, and download the output of the tasks. With Azure Batch and Azure CLI, you can easily manage and run high-performance computing tasks in the cloud with minimal effort.

Reference Documentation

1. Azure Batch documentation
2. Azure Batch pricing
3. Azure Batch Explorer
4. Azure CLI reference for Azure Batch
5. Azure Batch on GitHub
6. Azure Batch sample code on GitHub