Author: Cindy Gross | Befriending Dragons

  • Use Additional Storage Accounts with HDInsight Hive

    When you create an HDInsight Hadoop cluster you pass in one or more storage accounts and their associated keys. This allows you to access the files on all associated storage accounts from the cluster. If you want to use public storage that isn’t passed in at create time that’s easy – simply supply the storage account name each time you run a job. But how do you access data on private storage accounts that need an access key?

    The steps are laid out in this wiki by Eric Hanson: Using an HDInsight Cluster with Alternate Storage Accounts and Metastores

    http://social.technet.microsoft.com/wiki/contents/articles/23256.using-an-hdinsight-cluster-with-alternate-storage-accounts-and-metastores.aspx

    I am providing a variable based variation of the PowerShell sample for Hive. To set up PowerShell for use with Azure see Getting Started with Azure PowerShell Cmdlets–Subscription Management.

    First you will set some values for your environment. If you use your default subscription you don’t need to pass in the subscription name and select it. However, you will always need to specify the HDInsight cluster name. In this example $undefinedStorageAccount is the name of an account that you want to access from a cluster but you didn’t define it when you created the cluster. You always need to specify which container to use for any given reference so you also need to define $undefinedContainer. If the storage account belongs to the current subscription you can simply ask Azure to return the key (#commented out in the example below) or you can paste in the key that someone has given you.

    $subscriptionName = "LocalAzureSubscriptionName"
    
    $clusterName = "HDInsightClusterName"
    
    $undefinedStorageAccount = "AdditionalStorageAccount"
    
    $undefinedContainer = "ContainerOnAdditionalStorageAccount"
    
    #$undefinedStorageKey = Get-AzureStorageKey $undefinedStorageAccount | %{ $_.Primary }
    
    $undefinedStorageKey = "YourActualAccessKeyFromAzurePortal"

    Now choose which of your locally defined subscriptions to use:

    Select-AzureSubscription -SubscriptionName $subscriptionName

    Set the context of the cluster you want to use:

    Use-AzureHDInsightCluster $clusterName

    Now let’s check your HDInsight cluster properties.

    $defaultStorageAccount  = (Get-AzureHDInsightCluster -Name $clusterName).DefaultStorageAccount.StorageAccountName #default/only storage account
    
    $defaultContainerName   = (Get-AzureHDInsightCluster -Subscription $SubID -Cluster $ClusterName).DefaultStorageAccount.StorageContainerName
    
    $definedStorageAccounts = (Get-AzureHDInsightCluster -Name $clusterName).StorageAccounts #no 2nd account is associated, no value is returned

    Let’s check the values and verify that the storage account you want to use is not listed as either the DefaultStorageAccount (every cluster has one) or as one of the additional known storage accounts configured during provisioning (you may have zero, one, or many).

    write-host "===Default storage account"
    
    $defaultStorageAccount
    
    write-host "===Default container name"
    
    $defaultContainerName
    
    write-host "===Other defined storage accounts for this cluster"
    
    $definedStorageAccounts

    Next we’ll get a non-recursive listing of the files in the default location:

    invoke-hive "dfs -ls wasb://$defaultContainerName@$defaultStorageAccount/;" #default storage

    And then try to get a listing for the private storage account that we have not associated with the cluster:

    invoke-hive "dfs -ls wasb://$undefinedContainer@$undefinedStorageAccount/;" #not associated, errors

    Because the storage account access key is not yet known you will see an error similar to this one:

    Logging initialized using configuration in file:/C:/apps/dist/hive-0.12.0.2.0.7.0-1559/conf/hive-log4j.properties
    
    ls: org.apache.hadoop.fs.azure.AzureException: Unable to access container xyz in account abc using anonymous credentials, 
    
    and no credentials found for them  in the configuration.
    
    Command failed with exit code = 1

    But we can fix this! From PowerShell we can pass in “defines” statements to change configuration values, add libraries, etc.

    $defines = @{}
    
    $defines.Add("fs.azure.account.key.$undefinedStorageAccount.blob.core.windows.net", $undefinedStorageKey)
    
    Invoke-Hive -Defines $defines -Query "dfs -ls wasb://$undefinedContainer@$undefinedStorageAccount.blob.core.windows.net/;"

    The access key is only available to this Hive query, but now that I have the variables set I can pass it in to other queries as well. Happy Hiving!

    I hope you enjoyed this small bite of Big Data!

  • Getting Started with Azure PowerShell Cmdlets–Subscription Management

    I’ve started using the Azure PowerShell cmdlets more often to manage virtual machines and HDInsight in Azure. Once you connect to a subscription everything just works. However, the initial steps to get one or more subscriptions configured to be used from your machine or understanding how to change subscription information on your machine can be confusing. Some of the docs are contradictory, outdated, or incomplete. Often they assume you are only a co-admin of one subscription. The below steps should get you going with Azure cmdlets whether you admin one or many subscriptions.

    You need to enable your machine to talk to one or more Azure subscriptions. The first step is creating a certificate. Do NOT do this if you already used the PublishSettings commands unless you first use Remove-AzureSubscription (which removes the locally stored information about the specified subscription). Makecert is more secure than PublishSettings, especially if you (a given email address) have multiple co-administrators per subscription and/or you (a given email address) are a co-administrator of multiple subscriptions.

    The steps to get going are documented in Shep’s blog “Cloud Spelunking, Managing Azure form your Desktop via PowerShell (the Setup)” http://blogs.msdn.com/b/sql_shep/archive/2013/03/29/cloud-spelunking-managing-azure-form-your-desktop-via-powershell.aspx. I’ll go a bit deeper and fill in a few additional details on what Shep calls the “hard” option.

    Create a Certificate

    If you have IIS, Visual Studio, or the Windows SDK you will have some variation of a “Developer Command Prompt” (or VS201x or Visual Studio Command Prompt). Open that command prompt with the “run as administrator” option. Replace YourCertName with a meaningful name and run the below command. The cert always goes to the cert store on your local machine – the last parameter is an optional file based copy of that certificate that we will need for the next step. If you don’t specify the location it goes to %windir%system32. Be very protective of the .cer file – delete it once you have uploaded it. You can always generate another file if you need it.

    makecert -sky exchange -r -n “CN=<YourCertName>” -pe -a sha1 -len 2048 -ss My “c:temp<YourCertName>.cer”

    This certificate is yours – do not share it with others. If you want to reuse the certificate on other machines that you control, you can copy the .cer file to those machines and import them into the local certificate store on each machine. The .cer is just a copy, the actual certificate was loaded into your local certificate store (Manage Computer Certificates) by makecert.

    Upload Certificate to Azure Subscription(s)

    Generally you will not want to share certificates with others. Any certificate you use must be in your local certificate store (Manage Computer Certificates). The same certificate must also be uploaded to the portal and associated with each subscription you wish to manage from your machine.

    From your local machine where you created the certificate in the above step:

    • Log in to the Azure Portal with an email address that is associated with the subscription you want to use from your own machine.
    • Scroll to the bottom of the left pane and choose “SETTINGS”

    settings

    • Choose “MANAGEMENT CERTIFICATES”

    AzurePortalSettingsMgmtCert

    • Click on the “UPLOAD” button in the middle of the bar at the bottom of the screen.

    image

    • In the “Upload a management certificate” dialog navigate to the location specified in the last parameter above or %windir%system32 if you didn’t specify a location. Choose the .cer file you just created with makecert (or export a certificate from the local certificate store – just make sure it has the right properties). If you have multiple subscriptions there is a 2nd drop down box where you need to choose the subscription that the certificate will be associated with.

    image

    • Repeat for any additional subscriptions that you want to manage with the same certificate (or create one certificate per subscription for additional security granularity).

    Install and Configure the Azure PowerShell Cmdlets

    Follow the steps here to install the Azure Cmdlets. Basically you are selecting “Azure PowerShell” from the Web Platform Installer. You can also check in the Web Platform Installer for updated versions of the cmdlets.

    A very common setting that many admins set is the RemoteSigned Execution Policy. This is less secure than AllSigned or Restricted but allows you to use most downloaded scripts.

    Open Windows Azure PowerShell with the “run as admin” option and run:

    Set-ExecutionPolicy RemoteSigned –Force
    Get-ExecutionPolicy –list

    If you see errors when setting the execution policy, search on your specific error or start with this blog: Set-ExecutionPolicy : Windows PowerShell updated your execution policy successfully, but the setting is overridden by a policy defined at a more specific scope!!! You may need to open “Edit Group Policy” (in Windows 8 that opens the Local Group Policy Editor) and make a change.  Sometimes you may need to set each individual scope, but process scope settings go back to the default when the process is closed:

    Set-ExecutionPolicy RemoteSigned -Scope Process -Force

    Then import the Azure cmdlets:

    Import-Module Azure

    You can close the PowerShell window, you no longer need to “run as admin”.

    Enable PowerShell to use a Subscription via a Certificate

    Repeat this section on each machine that will be used to execute PowerShell code. Also repeat for additional subscriptions on each machine.

    Open Windows Azure PowerShell. Optionally type ISE to open the Integrated Scripting Environment where you can edit, save, and run collections of cmdlets.

    First, set some variables. You will need to copy some basic settings from the Azure Management Portal. On the far left side of the portal, scroll all the way to the bottom and choose “SETTINGS” and “MANAGEMENT CERTIFICATES” (see the “Upload Certificate to Azure Subscription(s)” section of this blog for more details – you are copying from the same place where you uploaded the certificate). Choose the certificate you just uploaded. Don’t worry if the numbers are cut off on the screen, if you highlight and copy it will get the whole value, even the part that doesn’t show on the screen. Replace the $subID and $thumbprint below – do not update $myCert as that is done based on your other variables. Execute the code in the PowerShell window.

    #copy SUBSCRIPTION ID from portal 
    #lower left, settings, management certificates
    $subID = "11111111-2222-3333-4444-555555555555"
    #copy THUMBPRINT from portal 
    #lower left, settings, management certificates
    $thumbprint = "1234567891234567891234567891234567891234"
    $myCert = Get-Item cert:\CurrentUserMy$thumbprint  
    

    Now set the subscription name you will use to refer to this subscription from this machine. In most cases you will choose the NAME of the subscription from the portal but that is not required. The matching between your machine’s knowledge of the subscription and the subscription on Azure is done via the SUBSCRIPTION ID. Update $localSubName below and execute the code in the PowerShell window. Note that the local subscription name is case-sensitive.

    #subname to be used locally
    #usually you will choose the actual subscription name
    #stored in %appdata%Windows Azure PowerShellWindowsAzureProfile.xml
    $localSubName = "MyFavSub"
    

    Now that you have set the values for your own environment, run the code to actually update your machine’s knowledge of the subscription. Note that I used the back tick “`” to specify that the command continues on a new line.

    Set-AzureSubscription –SubscriptionName $localSubName `
    –SubscriptionId $subID -Certificate $myCert

    Some operations rely on a default storage account, you may want to set the default storage account you want to use for each subscription.

    #optionally set "current" storage account for this sub
    $defaultStorageAccount = 'MyFavStorageAccount'
    Set-AzureSubscription -SubscriptionName $localSubName `
    -CurrentStorageAccount $defaultStorageAccount
    

    Next you can set the default subscription that you will start with when you open PowerShell on this machine (note that we’ve changed from the Set cmdlet to the Select one):

    Select-AzureSubscription –Default $localSubName

    You can change which of the configured subscriptions is the current one:

    Select-AzureSubscription –Current $localSubName

    Check to see which subscription you are currently using:

    Get-AzureSubscription –Current
    (Get-AzureSubscription -Current).SubscriptionName

    Verify that you can connect and list the services associated with the current subscription:

    Get-AzureService | select ServiceName

    Look at the Local Configuration

    Now let’s look at what got updated on the local machine.

    Open File Explorer and go to %appdata%Windows Azure PowerShell. Open WindowsAzureProfile.xml in Notepad or your favorite editor. Here are a few of the key values for each subscription you have mapped on your machine:

    IsDefault tells you which one is the default subscription for your machine

    <IsDefault>true</IsDefault>

    The thumbprint id is stored as the ManagementCertificate:

    <ManagementCertificate>1234567891234567891234567891234567891234</ManagementCertificate>

    The local name you chose for the subscription is stored in Name (to avoid confusion chose the name used in the portal):

    <Name>MyFavSub</Name>

    The subscription id is stored in SubscriptionId:

    <SubscriptionId>11111111-2222-3333-4444-555555555555</SubscriptionId>

    Remove Subscription

    If you need to remove a subscription from your machine, whether because you no longer have access to it or because you want to change one of the properties such as the name or which certificate you use, you can use Remove-AzureSubscription. This updates your local %appdata%Windows Azure PowerShell.

    #RemoveSub
    #Remove my machine's knowledge of a subscription 
    #Removes info from %appdata%Windows Azure PowerShellWindowsAzureProfile.xml
    Remove-AzureSubscription -SubscriptionName MyFavSub

    Sample Script

    Here is a handy dandy cut/paste version of the above PowerShell code to add a subscription and make it your default and current subscription:

    #copy SUBSCRIPTION ID from portal 
    
    #lower left, settings, management certificates
    
    $subID = "YourOwnSubID"
    
    #copy THUMBPRINT from portal 
    
    #lower left, settings, management certificates
    
    $thumbprint = "YourCertThumbprint"
    
    $myCert = Get-Item cert:\CurrentUserMy$thumbprint  
    
    #subname to be used locally
    
    #usually you will choose the actual subscription name
    
    #stored in %appdata%Windows Azure PowerShellWindowsAzureProfile.xml
    
    $localSubName = "YourSubcriptionName"
    
    #optionally set "current" storage account for this sub
    
    $defaultStorageAccount = 'OptionalDefaultStorage'
    
    Set-AzureSubscription –SubscriptionName $localSubName `
    
        –SubscriptionId $subID -Certificate $myCert
    
    Set-AzureSubscription -SubscriptionName $localSubName `
    
        -CurrentStorageAccount $defaultStorageAccount
    
    Select-AzureSubscription –Default $localSubName
    
    Select-AzureSubscription –Current $localSubName
    
    Get-AzureSubscription –Current
    
    (Get-AzureSubscription -Current).SubscriptionName

    You are Ready for PowerShell Gooey Goodness!

    Woohoo! Now you can access your Azure subscriptions from your machine without entering ids and passwords. You can automate, simplify, and standardize any Azure activity that has an associated cmdlet! Happy PowerShelling!

    
    
  • Would you go to jail for Human Rights?

    AT4WFeb272014JailBus_AshtonPage
    Add the 4 Words supporters being arrested and transported from the Statehouse to the Ada County Jail. Photo by Ashton Page.

    I believe in justice. I believe in human rights. I believe gay rights are human rights. I believe the Equal Protection Clause of the 14th Amendment to the US Constitution guarantees “Equal Justice Under Law”. Idaho is keeping that justice from an entire group of people – those who are gay. Gay teens in Idaho are bullied into suicide. Gay teens and adults are denied jobs or promotions. Gay folks can’t take a picture of their partner to work. Straight employees can’t take pictures of their gay adult child and that child’s partner to work. People of all orientations avoid taking jobs in Idaho or refuse to bring their businesses here because they worry about their families. What do you believe in? What are you willing to do in support of those beliefs?

    I am willing to fight for what is right, to fight for everyone of every sexual orientation and every gender identity. Last week I chose to be arrested, along with 45 other supporters, in support of the “Add the 4 Words” cause. The Republican Leadership in the Idaho legislature won’t even hold a hearing to listen to the stories of those who face discrimination for their real or perceived sexual orientation – and we all have a sexual orientation. We have lobbied legislative leadership, shared poll results showing a vast majority of Idahoans across the state believe firing someone just because they are gay is wrong, and told them our personal stories. Yet still leadership silences our voices, ignores their constituents, and refuses to hold a public hearing where their own constituents could testify. What do you believe in? What are you willing to do in support of those beliefs?

    With the support of former Idaho Governor and former head of the Idaho Republican Party Phil Batt, in the 1960s Idaho added a Human Rights Act. We simply want to Add the Words “sexual orientation” and “gender identity” to that existing act to tell Idahoans that gay folks get the same basic human rights as the rest of us. Former Governor Batt supports this modification to the act he helped create – he wants his gay grandson to feel safe while visiting Idaho. He calls the current legislature’s failure to act “disdain.” What do you believe in? What are you willing to do in support of those beliefs?

    You can spend a few minutes or a few hours or a few days on an action – start as small as you want. Do one action or many. Just do something. Stand up for your beliefs, for the fact that all humans deserve basic dignity and respect and the freedom pursue their happiness. Start with any one of the items below. See how that feels. And come back for more when you’re ready. What do you believe in? What are you willing to do in support of those beliefs?

    Cindy Gross - Arrested in support of Add the Words, Idaho Feb 27, 2014
    Cindy Gross – Arrested in support of Add the Words, Idaho Feb 27, 2014. What will you do? Photo by Ada County Sheriff Deputies.

    1) Tell Governor Otter your thoughts on Idaho’s reputation with regards to human rights, especially gay rights. This is even more important for those of you don’t live in Idaho. Would you visit Idaho? Would you start a business here? Would you bring your family here for a job? Give him a personal story.

    2) Follow @AddTheWords and @AddThe4Words on Twitter. Now tweet why you support gay rights in Idaho and tag one or both of those accounts. Consider tagging Governor Otter, Senator Hill, your own legislators, or national folks who we may be able to engage in the cause.

    3) Donate to Add the 4 Words to help cover the bail and fines of those standing up to be arrested.

    4) Participate in the next non-arrest event sponsored by Add the 4 Words or Add the Words.

    5) Sign up to participate in in the next Add the 4 Words arrest action – you don’t have to be arrested, you can volunteer as an observer or supporter.

    6) Contact your Idaho State Senator and two Idaho State Representatives plus Senator Brent Hill. Note that this is NOT the same as your US Senators (Crapo and Risch) and US Representatives (Labrador and Simpson). There are 35 districts in Idaho – there are many different legislators. I am happy to help you figure out who to contact (cgross1@hotmail.com).

    7) Tell your friends why you believe Add the Words is the right thing to do.

    8) Donate to the Add the Words documentary http://www.indiegogo.com/projects/add-the-words-a-documentary

    9) Follow the Add the Words Blog http://addthewords.blogspot.com/. Offer to contribute a blog post or be interviewed for a blog post.

    What do you believe in? What are you willing to do in support of those beliefs?

  • What Should “Racism” Mean?

    Racism isn’t dead, it just keeps morphing. And spreading. And multiplying. Take all those things you see about racism and add in sexism, homophobia, and other “isms”. Be aware and stop doing it. It isn’t some mysterious other doing it – it’s mainstream Americans.

  • Stupid Cancer – Survivor Spotlight Tonight at 6p

    Stupid cancer. It can be a devastating diagnosis – even if like me you’ve “won the cancer lottery” and ended up with a very treatable breast cancer. In my case I’m cured – a couple of small scars, a bit of a fading sunburn on one side, and memories. I can no longer say I don’t have tattoos; I have three tiny “prison blue” dots across my chest as a reminder of my radiation treatment. Stupid cancer.

    As a professional data geek I see lots of hope for the future of cancer diagnosis, prevention, and treatment. New “Big Data” technologies promise to improve the accuracy of screenings, speed up genetic testing, and improve the lives of those with a cancer diagnosis. Stupid cancer.

    As an amateur community organizer and activist I regularly fight for human rights – gay rights, women’s rights, the rights of those who are being denied the opportunity to live normal lives for no good reason. Cancer denies people the right to live normal lives – I’m working to fit fighting for cancer survivors into my activism. Stupid cancer.

    My mom died from breast cancer in 1997. One of the big surprises of getting my own breast cancer diagnosis was my sense of relief. I hadn’t realized how much I had been waiting for stupid cancer to happen to me. Heredity may only account for 5-10% of breast cancers but I apparently still had it in my subconscious that my mom had it so I would get it. The non-stop awareness campaigns sometimes seem to just make us afraid without giving us something concrete to actually do. Stupid cancer.

    This stupid cancer definitely gave me a new perspective on life. Tune in to hear me talk about being a cancer survivor on tonight’s Stupid Cancer show at 6p MST: http://www.blogtalkradio.com/stupidcancershow/2014/02/18/single-fathers-of-cancer

    “Bereavement from cancer is the one of the most difficult parts of dealing with cancer. Join us for tonight’s bereavement roundtable, where we talk to Justin M. Yopp, Ph.D. (Assistant Professor, UNC School of Medicine, Dep’t of Psychiatry) and Matt Herynk co-founder of Young Cancer Spouses about the coping with the loss of a loved one due to cancer. Survivor Spotlight on blogger Cindy Gross.”

    To listen live, visit http://stupidcancershow.org

    To subscribe to the iTunes podcast, visit http://stupidcancer.org/itunes

    To subscribe via iHeartRadio, visit http://stupidcancer.org/ihearttradio

  • Sample PowerShell Script: HDInsight Custom Create

    This is a working script I use to create various HDInsight clusters. For a really reproducible, automated environment you would want to put this into a .ps1 script that accepts parameters (see here for an example). However, you may find the method below good for learning and experimenting. Replace all the “YOURxyz” sections with your actual information. Beware of oddities introduced by cut/paste such as spaces being replaced by line breaks or quotes being replaced by smart quotes. The # is a comment, some commands that you rarely run are commented out so remove the # to run them if you need them.

    # This PowerShell script is meant to be a cut/paste of specific parts, it is NOT designed to be run as a whole.

    # Do once after you install the cmdlets
    #Get-AzurePublishSettingsFile
    #Import-AzurePublishSettingsFile C:UsersYOURDirectoryDownloadsYOURName-credentials.publishsettings

    # Use if you admin more than one subscription
    #Get-AzureAccount # This may be needed to log in to Azure
    Select-AzureSubscription –SubscriptionName YOURSubscription
    Get-AzureSubscription -Current

    # Many things are easier in the ISE
    ise

    ###############################################
    ### create clusters ###
    ###############################################

    # Add your specific information here
    # Previous failures may make a name unavailable for a while – check to see if previous cluster was partially created
    $ClusterName = “YOURNewHDInsightClusterName” #the name you will give to your cluster
    $Location = “YOURDataCenter” #cluster data center must be East US, West US, or North Europe (as of December 2013)
    $NumOfNodes = 1 #start small
    $StorageAcct1 = “YOURExistingStorageAccountName” #currently must be in same data center as the cluster
    $DefaultContainer = “YOURExistingContainerName” #already exists on the storage account

    # These variables are automatically set for you
    $FullStorage1 = “${StorageAcct1}.blob.core.windows.net”
    $Key1 = Get-AzureStorageKey $StorageAcct1 | %{ $_.Primary }
    $SubID = Get-AzureSubscription -Current | %{ $_.SubscriptionId }
    $SubName = Get-AzureSubscription -Current | %{ $_.SubscriptionName }
    $Cert = Get-AzureSubscription -Current | %{ $_.Certificate }
    $Creds = Get-Credential -Message “New admin account to be created for your HDInsight cluster” #this prompts you

    ###############################################
    # Sample quick create
    ###############################################
    # Equivalent of quick create
    # The ` specifies that the cmd continues on the next line, beware of artifical line breaks added during cut/paste from the blog
    New-AzureHDInsightCluster -Name $ClusterName -ClusterSizeInNodes $NumOfNodes -Subscription $SubID -Location “$Location” `
    -DefaultStorageAccountName $FullStorage1 -DefaultStorageAccountKey $Key1 -DefaultStorageContainerName $DefaultContainer -Credential $Creds

    ###############################################
    # Sample custom create
    ###############################################
    #https://hadoopsdk.codeplex.com/wikipage?title=PowerShell%20Cmdlets%20for%20Cluster%20Management
    # Most params are the same as quick create, use a new cluster name
    # Pass in a 2nd storage account, a SQLAzure db for the metastore (assume same db for Oozie and Hive), add Avro library, some config values
    # Execute all the variable settings from above

    # This value is set for you, don’t change!
    $configvalues = new-object ‘Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.DataObjects.AzureHDInsightHiveConfiguration’

    # Add your specific information here
    $ClusterName = “YOURNewHDInsightClusterName”
    $StorageAcct2 = “YOURExistingStorageAccountName2”
    $MetastoreAzureSQLDBName = “YOURExistingSQLAzureDBName”
    $MetastoreAzureServerName = “YOURExistingSQLAzureServer.database.windows.net” #gives a DNS error if you don’t use the full name
    $configvalues.Configuration = @{ “hive.exec.compress.output”=”true” }  #this is an example of a config value you may pass in

    # These variables are automatically set for you
    $FullStorage2 = “${StorageAcct2}.blob.core.windows.net”
    $Key2 = Get-AzureStorageKey $StorageAcct2 | %{ $_.Primary }
    $MetastoreCreds = Get-Credential -Message “existing id/password for your SQL Azure DB (metastore)” #This prompts for the existing id and password of your existing SQL Azure DB

    # Add a config file value
    # Add AVRO SerDe libraries for Hive (on storage 1)
    $configvalues.AdditionalLibraries = new-object ‘Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.DataObjects.AzureHDInsightDefaultStorageAccount’
    $configvalues.AdditionalLibraries.StorageAccountName = $FullStorage1
    $configvalues.AdditionalLibraries.StorageAccountKey = $Key1
    $configvalues.AdditionalLibraries.StorageContainerName = “hivelibs” #container called hivelibs must exist on specified storage account
    # Create custom cluster
    New-AzureHDInsightClusterConfig -ClusterSizeInNodes $NumOfNodes `
    | Set-AzureHDInsightDefaultStorage -StorageAccountName $FullStorage1 -StorageAccountKey $Key1 -StorageContainerName $DefaultContainer `
    | Add-AzureHDInsightStorage -StorageAccountName $FullStorage2 -StorageAccountKey $Key2 `
    | Add-AzureHDInsightMetastore -SqlAzureServerName $MetastoreAzureServerName -DatabaseName $MetastoreAzureSQLDBName -Credential $MetastoreCreds -MetastoreType OozieMetastore `
    | Add-AzureHDInsightMetastore -SqlAzureServerName $MetastoreAzureServerName -DatabaseName $MetastoreAzureSQLDBName -Credential $MetastoreCreds -MetastoreType HiveMetastore `
    | Add-AzureHDInsightConfigValues -Hive $configvalues `
    | New-AzureHDInsightCluster -Subscription $SubID -Location “$Location” -Name $ClusterName -Credential $Creds

    ###############################################
    # get status, properties, etc.
    ###############################################
    #$SubName = $SubID = Get-AzureSubscription -Current | %{ $_.SubscriptionName }
    Get-AzureHDInsightProperties -Subscription $SubName
    Get-AzureHDInsightCluster -Subscription $SubName
    Get-AzureHDInsightCluster -Subscription $SubName -name YOURClusterName

    ###############################################
    # remove cluster
    ###############################################
    #Remove-AzureHDInsightCluster -Name $ClusterName -Subscription $SubName

  • Your First HDInsight Cluster–Step by Step

    Small Bites of Big Data from AZURECAT
    Big Data Tech Training Series #1
    Cindy Gross | Murshed Zaman

    Sometimes it is just hard to get started. Have you been putting off your first foray into Hadoop? Are you not sure where to begin? Let’s get really basic.

    Prerequisites:

    Log on to the Windows Azure Portal https://manage.windowsazure.com

    Go to storage https://manage.windowsazure.com/#Workspaces/StorageExtension/storage. Create a storage account in a location that is available to HDInsight (as of November 2013 that’s East US, West US, and North Europe). Do NOT choose an affinity group. If you choose to “Enable Geo-Replication” there will be an extra charge – it’s probably not necessary for a demo/test account as you have a limited amount of credit in the trial subscription. In the portal choose the STORAGE icon on the left. Then click on +NEW at the bottom. That opens a QUICK CREATE window. Enter a unique name for your storage, such as sqlcatwomanrules. It only allows lower case letters and numbers.

    StorageNov2013 NewStorageAccountNov2013

    Now click on the HDInsight icon just below the storage icon storage. Choose QUICK CREATE. Enter a unique name for your HDInsight cluster. For a demo choose 4 data nodes. Enter a password that contains upper and lower case letters, a number, and a special character. Choose the storage account you created above. Once you click on “CREATE HDINSIGHT CLUSTER” it will take several minutes for the cluster to be deployed.

    StorageNov2013_HDI QuickCreateNov2013

    Once it completes you are ready to use your cluster!

    ClusterQuickStart

    If you won’t be using the cluster right away, go ahead and delete it (look for the icon at the bottom of the portal) to save compute time and money. You can easily recreate it when you need it.

    image

    Look for more blogs soon on customizing your cluster with CUSTOM CREATE or PowerShell and on automating deployment and jobs with PowerShell. In the meantime see if you can get Invoke-Hive working from PowerShell for some simple Hive commands such as:

    Invoke-Hive “select * from hivesampletable limit 10”

    Big Data Technical Series:

    Your First HDInsight Cluster–Step by Step

    Automating HDInsight cluster creation with PowerShell

  • PowerShell for Azure cmdlets: Subscription was all Wacky

    I was working on some HDInsight scripts in PowerShell and doing lots of experimenting. I’m not sure what exactly I did but all of a sudden everything stopped working. With lots of interruptions from meetings and chats and lunch…. I couldn’t retrace my steps. Everything seemed to fail on the Azure subscription information so I tried to get really basic – what did Get-AzureSubscription|%{$_.SubscriptionName} return? As it turns out, wacky garbage:

    set-azuresubscription ?
    !?
    !
    help
    sqlcatwoman
    ?
    Sqlcatwomanwestus

    What I expected to see was my single subscription:

    sqlcatwoman

    So what happened? The Azure portal only shows one subscription. Obviously those other lines are not valid subscriptions – they look like the output of a help command or an error. Reinstalling the cmdlets, rebooting, and reimporting certificates didn’t help. I turned to my AzureCAT coworkers for help and @elcid98 pointed out this blog post that talks about how subscriptions are used in PowerShell:

    Azure Subscriptions in PowerShell demystified http://yossidahan.wordpress.com/2012/12/12/azure-subscriptions-in-powershell-demystified/

    This caught my attention: “The second file – DefaultSubscriptionData.xml – also lists the available subscriptions and the associated certificates’ thumbprints“. Ok, where is that file? A search finds it in

    C:Users%username%AppDataRoamingWindows Azure Powershell

    I checked and sure enough, where I would expect just one entry I see multiple – and they’re named the same thing as the garbage in my output! I cleared out all but one entry to end up with this:

    <?xml version=”1.0″ encoding=”utf-8″?>
    < Subscriptions xmlns:xsd=”http://www.w3.org/2001/XMLSchema” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” version=”0″ xmlns=”urn:Microsoft.WindowsAzure.Management:WaPSCmdlets”>
    < Subscription name=”sqlcatwoman”>
    < SubscriptionId>You don’t get to see the real info!</SubscriptionId>
    <Thumbprint>Not here either!</Thumbprint>
    < ServiceEndpoint>https://management.core.windows.net</ServiceEndpoint>
    < /Subscription>
    < /Subscriptions>

    Hmmmm…. I still got an error from Get-AzureSubscription. Back to C:Users%username%AppDataRoamingWindows Azure PowerShell. What’s this? WindowsAzureProfile.xml also has all the same junk! I cleared out all the extras to end up with this:

    <?xml version=”1.0″ encoding=”utf-8″?>
    < ProfileData xmlns:i=”http://www.w3.org/2001/XMLSchema-instance” xmlns=”http://schemas.datacontract.org/2004/07/Microsoft.WindowsAzure.Commands.Utilities.Common”>
    < DefaultEnvironmentName>AzureCloud</DefaultEnvironmentName>
    <Environments />
    <Subscriptions>
    < AzureSubscriptionData>
    < ActiveDirectoryEndpoint>https://login.windows.net/</ActiveDirectoryEndpoint>
    < ActiveDirectoryTenantId>More secrets!</ActiveDirectoryTenantId>
    < ActiveDirectoryUserId>cgross@microsoft.com</ActiveDirectoryUserId>
    < CloudStorageAccount i:nil=”true” />
    < IsDefault>true</IsDefault>
    < LoginType i:nil=”true” />
    < ManagementCertificate>Hiding this one too!</ManagementCertificate>
    < ManagementEndpoint>https://management.core.windows.net/</ManagementEndpoint>
    <Name>sqlcatwoman</Name>
    <RegisteredResourceProviders xmlns:d4p1=”http://schemas.microsoft.com/2003/10/Serialization/Arrays” />
    < SubscriptionId>And more secrets</SubscriptionId>
    </AzureSubscriptionData>
    </Subscriptions>
    < /ProfileData>

    Success! Get-AzureSubscription now returns just my single, valid subscription. All my other Azure cmdlets magically started working again. I don’t know how it got that way, but at least now I know where the subscription information is stored. I hope this helps someone else with their Azure subscription PowerShell scripting!

  • Interview with Julie Strauss–Microsoft BI WIT

    clip_image002Julie Strauss is a very accomplished and respected Senior PM at Microsoft. Her current role is technical assistant for Microsoft Data Platform Group (DPG) Corporate Vice President Quentin Clark. She has been the public face of Microsoft BI at conferences and helps deliver great technical content and data stories to the public. Julie loves to help others so she has shared some background on herself and some great business advice that could be helpful to others seeking to improve their success.

    Julie saw a job posting for the support team in Microsoft Norway (at the time Great Plains) looking for an individual willing to learn the ins and outs of the Microsoft BI products. She was excited that the posting indicated a willingness to learn was more important than previous knowledge of the particular Microsoft product. This was how and why Julie came here – she loves the technology and the data driven parts of the business and finds them fascinating.

    Julie has a notable role with a wide range of responsibilities. The majority of her time is spent working on strategic projects to meet the goals of the team at the DPG Vice President level. Projects can vary in nature and cover everything from exploratory and technical projects to organizational projects. She gets to work with many areas of the business and enjoys interactions across the org. In addition to these internal facing responsibilities Julie also manages a set of customer and partner engagements for the business. Overall this role has provided Julie with an amazing learning opportunity. She gets to widen her scope while maintaining her data and BI focus and also use her years of experience from responsibilities ranging through sales, marketing, support, engineering, program management and people management. She merged these experiences into a role as technical assistant that utilizes some aspects of all those areas. Throughout her career she has chosen new jobs that allowed her to stretch and grow with a significant amount of change. But throughout it all she kept one core thing the same – her focus on BI and data. This mix of old and new in each role helps her cultivate new skills while leveraging what she already knows and expanding her influence. Within Microsoft there are many opportunities, something Julie feels is unique in the corporate world, and we can all find a way to shine and grow here.

    imageJulie has an extensive network she finds invaluable in navigating all that opportunity. Her network lets her know about new opportunities and the network members also influence decision makers. She emphasizes that your reputation is everything – your network carries that reputation to others. In a strong network everyone is contributing to each other’s success. She has a large network though at any given point in time she is only actively interacting with a few people.

    In addition to a network of contacts, Julie has closer relationships with a smaller group of people as both a mentor and a mentee. When Julie made the decision to move from marketing to engineering she leveraged her close mentoring relationship with Donald Farmer. Donald knew Julie and her work ethic and was willing to take a chance on Julie’s ability to succeed even though on paper it wasn’t an obvious fit. She stresses the importance of having semi-formal mentoring relationships with people at various levels. She asks various mentors for advice with experiences, projects, and specific interactions. Julie contributes back as a mentor to others – this keeps her coaching skills active. Julie observed that while she doesn’t treat her mentees differently based on their gender they tend to bucket themselves. More often than not women ask how to handle a specific situation or how to become more efficient or appear more confident. On the other hand men are more likely to ask task oriented questions such as how to make a specific change or how to write a better spec. She enjoys helping with both types of questions. Some of her mentees and mentors are people she already knew and some are people she grew to know only after the mentor-mentee relationship started.

    imageI asked Julie what advice she feels is most important to her success that would be helpful to others in the organization. In addition to networking and mentors, she offered these pearls of wisdom:

    • Be willing to take risks and take on new challenges. She has few regrets because she goes after what she wants. She does wonder if having no regrets at all means she didn’t stretch enough. You have to find your own balance.
    • Be true to who you are – how people see you, your brand, should reflect the real you. For Julie it has been very important to never compromise on being true to herself. Julie’s brand is “Give me a challenge and I will work my butt off to get it done, being creative as needed, bringing in people who will make it work.”
    • Never be a victim. Women are strong.
    • Pick something concrete to improve upon and just do it. For example, Julie was ranked as the lowest presenter at a conference. She decided to become a top 10 presenter – she achieved that goal and grew to truly enjoy presenting along the way.
    • Find work you love. Julie finds data fascinating because it is very tangible and with BI you control how it leads to insights, learnings, and possibilities. She loves how data and BI let you use your own imagination and set your own boundaries.
    • State your needs and get buy-in. For example you might tell your manager that you want a promotion and lay out your plan to get there. Then you ask “Is this realistically going to get me to my goal”? Make sure your manager understands your value and gives you feedback, then follow through on the actions with appropriately timed check-ins on whether you are still on track.

    Over the years Julie has lived in Denmark, Norway, the UK, and the US. She is always looking for new challenges whether it’s how to succeed in a new country or job or taking on a demanding project. Whatever she does she is working hard and getting things done. Follow her advice – build your network, find a mentor or two, be clear on expectations, and always be true to who you are.

    I want to thank Julie for sharing herself and her ideas with us – it can be tough to open up but Julie did a stellar job!

  • Transgender…. Are You Uncomfortable Yet?

    I wrote this post for Idaho’s Add the Words blog, I am reposting here to spread the word:

    TDORI started this post to ask you to join us at Boise’s Transgender Day of Remembrance 2013. As I thought about how to write about it, what I could say to get people to attend, I realized most people would just dismiss the day and the event as irrelevant to their lives. But it’s not irrelevant to you – it’s all about the type of world we want to live in. It really is an event you, yes YOU, need to attend.

    I know, you’re busy. Maybe you’re uncomfortable around people who are different than society has taught us is normal. Maybe you aren’t sure what people will think of you if you go. Most people who attend will not be transgender themselves, we are there to show that we don’t think violence is the answer to feeling uncomfortable around others. Are you really ok with using violence against people in a misguided attempt to make them conform or to make the attackers more comfortable with themselves? Because that’s what this is about – remembering all the people who are beaten, raped, and even killed every year just because their gender identity makes people uncomfortable.

    Take a few minutes from your evening on Wednesday, November 20, 2013 to head to the Anne Frank Memorial for Boise’s Transgender Day of Remembrance. The memorial starts at 7p – show that you don’t condone violence by joining us there.
    Cindy Gross – Straight Ally and Equal Rights Supporter
    #TDOR
    GLADD: Transgender Day of Remembrance #TDOR – November 20 http://www.glaad.org/tdor

    10 things every Lutheran should know about Transgender Day of Remembrance http://www.reconcilingworks.org/news/news/611-transgender-day-of-rememberance

    Link or Tweet this blog post via http://tinyurl.com/mw45x5w