Small Bites of Big Data from AzureCAT
You’ve created your HDInsight Hadoop clusters and now you want to know exactly what you have out there in Azure. Maybe you want to pull the key information into a repository periodically as a reference for future troubleshooting, comparisons, or billing. Maybe you just need to get a quick look at your overall HDInsight usage. This is something you can easily automate with PowerShell.
Environment
First, open Windows Azure PowerShell or powershell_ise.exe.
Set some values for your environment:
$SubName = "YourSubscriptionName"
Select-AzureSubscription -SubscriptionName $SubName
Get-AzureSubscription -Current
$ClusterName = "HDInsightClusterName" #HDInsight cluster name
HDInsight Usage for the Subscription
Take a look at your overall HDInsight usage for this subscription:
Get-AzureHDInsightProperties
Get-AzureHDInsightProperties returns the number of clusters for this subscription, the total HDInsight cores used and available (for head nodes and data nodes), the Azure regions where HDInsight clusters can be created, and the HDInsight versions available for new clusters:
ClusterCount : 2
CoresAvailable : 122
CoresUsed : 48
Locations : {East US, North Europe, Southeast Asia, West Europe...}
MaxCoresAllowed : 170
Versions : {1.6, 2.1, 3.0}
You can also pick out specific pieces of information and write them to a file, store them as variables, or use them elsewhere. This example simply outputs the values to the screen.
write-host '== Max HDInsight Cores for Sub: ' (Get-AzureHDInsightProperties).MaxCoresAllowed
write-host '== Cores Available: ' (Get-AzureHDInsightProperties).CoresAvailable
write-host '== Cores Used: ' (Get-AzureHDInsightProperties).CoresUsed
HDInsight Cluster Information
Get-AzureHDInsightCluster provides information about all existing HDInsight clusters for this subscription:
Get-AzureHDInsightCluster
As you can see this cmdlet tells you the size, connection information, and version.
ClusterSizeInNodes : 4
ConnectionUrl : https://BigCAT.azurehdinsight.net
CreateDate : 4/5/2014 3:37:23 PM
DefaultStorageAccount : sqlcatwomanwestus.blob.core.windows.net
HttpUserName : Admin
Location : West US
Name : BigCAT30
State : Running
StorageAccounts : {}
SubscriptionId : {YourSubID}
UserName : Admin
Version : 3.0.0.0.661685
VersionStatus : Compatible
ClusterSizeInNodes : 4
ConnectionUrl : https://cgrosstest.azurehdinsight.net
CreateDate : 5/5/2014 6:09:58 PM
DefaultStorageAccount : cgrosstest.blob.core.windows.net
HttpUserName : Admin
Location : West US
Name : cgrosstest
State : Running
StorageAccounts : {sqlcatwomanwestus.blob.core.windows.net}
SubscriptionId : {YourSubID}
UserName : Admin
Version : 3.0.2.0.727283
VersionStatus : Compatible
You can also get information about just one HDInsight cluster at a time:
Get-AzureHDInsightCluster -name $ClusterName
Or you can get very granular and look at specific properties, even some that aren’t in the default values:
write-host '== Default Storage Account: ' `
(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageAccountName.split(".")[0]
write-host '== Default Container: ' `
(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageContainerName
This information will be a valuable source of information for tracking past configurations, current usage, and planning. Enjoy your Hadooping!
Sample Script
# Cindy Gross 2014
# Get HDInsight properties
$SubName = "YourSubscriptionName"
Select-AzureSubscription -SubscriptionName $SubName
Get-AzureSubscription -Current
$ClusterName = "YourHDInsightClusterName" #HDInsight cluster name
Get-AzureHDInsightProperties
Get-AzureHDInsightCluster
Get-AzureHDInsightCluster -name $ClusterName
write-host '== Default Storage Account: ' `
(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageAccountName.split(".")[0]
write-host '== Default Container: ' `
(Get-AzureHDInsightCluster -Cluster $ClusterName).DefaultStorageAccount.StorageContainerName
write-host '== Max HDInsight Cores for Sub: ' (Get-AzureHDInsightProperties).MaxCoresAllowed
write-host '== Cores Available: ' (Get-AzureHDInsightProperties).CoresAvailable
write-host '== Cores Used: ' (Get-AzureHDInsightProperties).CoresUsed