Your First HDInsight Cluster–Step by Step

Small Bites of Big Data from AZURECAT
Big Data Tech Training Series #1
Cindy Gross | Murshed Zaman

Sometimes it is just hard to get started. Have you been putting off your first foray into Hadoop? Are you not sure where to begin? Let’s get really basic.

Prerequisites:

Log on to the Windows Azure Portal https://manage.windowsazure.com

Go to storage https://manage.windowsazure.com/#Workspaces/StorageExtension/storage. Create a storage account in a location that is available to HDInsight (as of November 2013 that’s East US, West US, and North Europe). Do NOT choose an affinity group. If you choose to “Enable Geo-Replication” there will be an extra charge – it’s probably not necessary for a demo/test account as you have a limited amount of credit in the trial subscription. In the portal choose the STORAGE icon on the left. Then click on +NEW at the bottom. That opens a QUICK CREATE window. Enter a unique name for your storage, such as sqlcatwomanrules. It only allows lower case letters and numbers.

StorageNov2013 NewStorageAccountNov2013

Now click on the HDInsight icon just below the storage icon storage. Choose QUICK CREATE. Enter a unique name for your HDInsight cluster. For a demo choose 4 data nodes. Enter a password that contains upper and lower case letters, a number, and a special character. Choose the storage account you created above. Once you click on “CREATE HDINSIGHT CLUSTER” it will take several minutes for the cluster to be deployed.

StorageNov2013_HDI QuickCreateNov2013

Once it completes you are ready to use your cluster!

ClusterQuickStart

If you won’t be using the cluster right away, go ahead and delete it (look for the icon at the bottom of the portal) to save compute time and money. You can easily recreate it when you need it.

image

Look for more blogs soon on customizing your cluster with CUSTOM CREATE or PowerShell and on automating deployment and jobs with PowerShell. In the meantime see if you can get Invoke-Hive working from PowerShell for some simple Hive commands such as:

Invoke-Hive “select * from hivesampletable limit 10”

Big Data Technical Series:

Your First HDInsight Cluster–Step by Step

Automating HDInsight cluster creation with PowerShell

PowerShell for Azure cmdlets: Subscription was all Wacky

I was working on some HDInsight scripts in PowerShell and doing lots of experimenting. I’m not sure what exactly I did but all of a sudden everything stopped working. With lots of interruptions from meetings and chats and lunch…. I couldn’t retrace my steps. Everything seemed to fail on the Azure subscription information so I tried to get really basic – what did Get-AzureSubscription|%{$_.SubscriptionName} return? As it turns out, wacky garbage:

set-azuresubscription ?
!?
!
help
sqlcatwoman
?
Sqlcatwomanwestus

What I expected to see was my single subscription:

sqlcatwoman

So what happened? The Azure portal only shows one subscription. Obviously those other lines are not valid subscriptions – they look like the output of a help command or an error. Reinstalling the cmdlets, rebooting, and reimporting certificates didn’t help. I turned to my AzureCAT coworkers for help and @elcid98 pointed out this blog post that talks about how subscriptions are used in PowerShell:

Azure Subscriptions in PowerShell demystified http://yossidahan.wordpress.com/2012/12/12/azure-subscriptions-in-powershell-demystified/

This caught my attention: “The second file – DefaultSubscriptionData.xml – also lists the available subscriptions and the associated certificates’ thumbprints“. Ok, where is that file? A search finds it in

C:Users%username%AppDataRoamingWindows Azure Powershell

I checked and sure enough, where I would expect just one entry I see multiple – and they’re named the same thing as the garbage in my output! I cleared out all but one entry to end up with this:

<?xml version=”1.0″ encoding=”utf-8″?>
< Subscriptions xmlns:xsd=”http://www.w3.org/2001/XMLSchema” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” version=”0″ xmlns=”urn:Microsoft.WindowsAzure.Management:WaPSCmdlets”>
< Subscription name=”sqlcatwoman”>
< SubscriptionId>You don’t get to see the real info!</SubscriptionId>
<Thumbprint>Not here either!</Thumbprint>
< ServiceEndpoint>https://management.core.windows.net</ServiceEndpoint>
< /Subscription>
< /Subscriptions>

Hmmmm…. I still got an error from Get-AzureSubscription. Back to C:Users%username%AppDataRoamingWindows Azure PowerShell. What’s this? WindowsAzureProfile.xml also has all the same junk! I cleared out all the extras to end up with this:

<?xml version=”1.0″ encoding=”utf-8″?>
< ProfileData xmlns:i=”http://www.w3.org/2001/XMLSchema-instance” xmlns=”http://schemas.datacontract.org/2004/07/Microsoft.WindowsAzure.Commands.Utilities.Common”>
< DefaultEnvironmentName>AzureCloud</DefaultEnvironmentName>
<Environments />
<Subscriptions>
< AzureSubscriptionData>
< ActiveDirectoryEndpoint>https://login.windows.net/</ActiveDirectoryEndpoint>
< ActiveDirectoryTenantId>More secrets!</ActiveDirectoryTenantId>
< ActiveDirectoryUserId>cgross@microsoft.com</ActiveDirectoryUserId>
< CloudStorageAccount i:nil=”true” />
< IsDefault>true</IsDefault>
< LoginType i:nil=”true” />
< ManagementCertificate>Hiding this one too!</ManagementCertificate>
< ManagementEndpoint>https://management.core.windows.net/</ManagementEndpoint>
<Name>sqlcatwoman</Name>
<RegisteredResourceProviders xmlns:d4p1=”http://schemas.microsoft.com/2003/10/Serialization/Arrays” />
< SubscriptionId>And more secrets</SubscriptionId>
</AzureSubscriptionData>
</Subscriptions>
< /ProfileData>

Success! Get-AzureSubscription now returns just my single, valid subscription. All my other Azure cmdlets magically started working again. I don’t know how it got that way, but at least now I know where the subscription information is stored. I hope this helps someone else with their Azure subscription PowerShell scripting!

Interview with Julie Strauss–Microsoft BI WIT

clip_image002Julie Strauss is a very accomplished and respected Senior PM at Microsoft. Her current role is technical assistant for Microsoft Data Platform Group (DPG) Corporate Vice President Quentin Clark. She has been the public face of Microsoft BI at conferences and helps deliver great technical content and data stories to the public. Julie loves to help others so she has shared some background on herself and some great business advice that could be helpful to others seeking to improve their success.

Julie saw a job posting for the support team in Microsoft Norway (at the time Great Plains) looking for an individual willing to learn the ins and outs of the Microsoft BI products. She was excited that the posting indicated a willingness to learn was more important than previous knowledge of the particular Microsoft product. This was how and why Julie came here – she loves the technology and the data driven parts of the business and finds them fascinating.

Julie has a notable role with a wide range of responsibilities. The majority of her time is spent working on strategic projects to meet the goals of the team at the DPG Vice President level. Projects can vary in nature and cover everything from exploratory and technical projects to organizational projects. She gets to work with many areas of the business and enjoys interactions across the org. In addition to these internal facing responsibilities Julie also manages a set of customer and partner engagements for the business. Overall this role has provided Julie with an amazing learning opportunity. She gets to widen her scope while maintaining her data and BI focus and also use her years of experience from responsibilities ranging through sales, marketing, support, engineering, program management and people management. She merged these experiences into a role as technical assistant that utilizes some aspects of all those areas. Throughout her career she has chosen new jobs that allowed her to stretch and grow with a significant amount of change. But throughout it all she kept one core thing the same – her focus on BI and data. This mix of old and new in each role helps her cultivate new skills while leveraging what she already knows and expanding her influence. Within Microsoft there are many opportunities, something Julie feels is unique in the corporate world, and we can all find a way to shine and grow here.

imageJulie has an extensive network she finds invaluable in navigating all that opportunity. Her network lets her know about new opportunities and the network members also influence decision makers. She emphasizes that your reputation is everything – your network carries that reputation to others. In a strong network everyone is contributing to each other’s success. She has a large network though at any given point in time she is only actively interacting with a few people.

In addition to a network of contacts, Julie has closer relationships with a smaller group of people as both a mentor and a mentee. When Julie made the decision to move from marketing to engineering she leveraged her close mentoring relationship with Donald Farmer. Donald knew Julie and her work ethic and was willing to take a chance on Julie’s ability to succeed even though on paper it wasn’t an obvious fit. She stresses the importance of having semi-formal mentoring relationships with people at various levels. She asks various mentors for advice with experiences, projects, and specific interactions. Julie contributes back as a mentor to others – this keeps her coaching skills active. Julie observed that while she doesn’t treat her mentees differently based on their gender they tend to bucket themselves. More often than not women ask how to handle a specific situation or how to become more efficient or appear more confident. On the other hand men are more likely to ask task oriented questions such as how to make a specific change or how to write a better spec. She enjoys helping with both types of questions. Some of her mentees and mentors are people she already knew and some are people she grew to know only after the mentor-mentee relationship started.

imageI asked Julie what advice she feels is most important to her success that would be helpful to others in the organization. In addition to networking and mentors, she offered these pearls of wisdom:

  • Be willing to take risks and take on new challenges. She has few regrets because she goes after what she wants. She does wonder if having no regrets at all means she didn’t stretch enough. You have to find your own balance.
  • Be true to who you are – how people see you, your brand, should reflect the real you. For Julie it has been very important to never compromise on being true to herself. Julie’s brand is “Give me a challenge and I will work my butt off to get it done, being creative as needed, bringing in people who will make it work.”
  • Never be a victim. Women are strong.
  • Pick something concrete to improve upon and just do it. For example, Julie was ranked as the lowest presenter at a conference. She decided to become a top 10 presenter – she achieved that goal and grew to truly enjoy presenting along the way.
  • Find work you love. Julie finds data fascinating because it is very tangible and with BI you control how it leads to insights, learnings, and possibilities. She loves how data and BI let you use your own imagination and set your own boundaries.
  • State your needs and get buy-in. For example you might tell your manager that you want a promotion and lay out your plan to get there. Then you ask “Is this realistically going to get me to my goal”? Make sure your manager understands your value and gives you feedback, then follow through on the actions with appropriately timed check-ins on whether you are still on track.

Over the years Julie has lived in Denmark, Norway, the UK, and the US. She is always looking for new challenges whether it’s how to succeed in a new country or job or taking on a demanding project. Whatever she does she is working hard and getting things done. Follow her advice – build your network, find a mentor or two, be clear on expectations, and always be true to who you are.

I want to thank Julie for sharing herself and her ideas with us – it can be tough to open up but Julie did a stellar job!

Transgender…. Are You Uncomfortable Yet?

I wrote this post for Idaho’s Add the Words blog, I am reposting here to spread the word:

TDORI started this post to ask you to join us at Boise’s Transgender Day of Remembrance 2013. As I thought about how to write about it, what I could say to get people to attend, I realized most people would just dismiss the day and the event as irrelevant to their lives. But it’s not irrelevant to you – it’s all about the type of world we want to live in. It really is an event you, yes YOU, need to attend.

I know, you’re busy. Maybe you’re uncomfortable around people who are different than society has taught us is normal. Maybe you aren’t sure what people will think of you if you go. Most people who attend will not be transgender themselves, we are there to show that we don’t think violence is the answer to feeling uncomfortable around others. Are you really ok with using violence against people in a misguided attempt to make them conform or to make the attackers more comfortable with themselves? Because that’s what this is about – remembering all the people who are beaten, raped, and even killed every year just because their gender identity makes people uncomfortable.

Take a few minutes from your evening on Wednesday, November 20, 2013 to head to the Anne Frank Memorial for Boise’s Transgender Day of Remembrance. The memorial starts at 7p – show that you don’t condone violence by joining us there.
Cindy Gross – Straight Ally and Equal Rights Supporter
#TDOR
GLADD: Transgender Day of Remembrance #TDOR – November 20 http://www.glaad.org/tdor

10 things every Lutheran should know about Transgender Day of Remembrance http://www.reconcilingworks.org/news/news/611-transgender-day-of-rememberance

Link or Tweet this blog post via http://tinyurl.com/mw45x5w

Taking it Easy with Radiation

When I was diagnosed with Breast Cancer I quickly found out that I was going to have things relatively easy. In less than three weeks from my diagnosis I had a lumpectomy which removed a very small, low-grade tumor and almost certainly removed every trace of the cancer. This past Monday I started the radiation that will make sure any lingering microscopic cancer cells are gone from my left breast. Two small, neat scars from the surgery are the extent of my side effects so far. I might possibly have a little bit of mild short term fatigue from the radiation towards the end of treatment. I’ve had one week of radiation so that means I have 5 more weeks, 5 days a week, to go. The first five weeks they treat my entire left breast – they created the plan to be very directed and avoid my heart and lungs. The last week has a “boost” that focuses radiation just on the area where the .9cm tumor was. I see the radiation oncologist, Dr. Kuhn, every Tuesday right after the radiation treatment so she can make sure the treatment is going well.

Green Boots
My beautiful green boots!

On the first day of treatment I wore my green cowboy boots. Everyone from radiation check-in to the radiation techs raved about the boots. As a bonus they match the green hospital gowns they have us change into. It’s nice to have something positive to think about because walking past a waiting room full of people waiting for chemo treatments on my way to the elevator down to radiation can be a bit depressing. I feel guilty about having it so much easier than they do.

Radiation Room
The purple section rotates and is where the radiation comes from

The daily treatment is very fast and easy. Before my first treatment they put three freckle sized tattoos, literal pin pricks of blue ink, on my body. Every weekday morning at 9am I go in, lay face-up on a table, and put my hands on the handles behind my head (see the green circle on the picture). They move the sheet under me to get the laser on the ceiling to line up with the three tattoos so the radiation is directed at the correct area. Then the entire machine rotates – the section circled in purple ends up at an angle about a foot above my right side, facing the inside of my left breast. I hear some whirring noises for about a minute – that’s the radiation being dispensed. Then it rotates to be even with my left side, the radiation tech shifts the table a bit, and the machine whirrs again for about a minute. That’s it. I get changed from my hospital gown into my clothes and go home. Fast, easy, painless. It’s amazingly simple and the techs are very friendly and answer all my questions.

The hardest parts are getting there on time each day and not being able to wear deodorant or antiperspirant. I have to use their hospital grade lotion and increase my protein intake but otherwise my life is very normal. Everything is easy and my great circle of friends has been incredibly supportive! I’m very lucky and happy to be getting on with my life!

HDInsight Big Data Talks from #SQLPASS

SQL PASS Summit 2013 was another great data geek week! I chatted with many of you about Big Data, Hadoop, HDInsight, architecting solutions, SQL Server, data, BI, analytics, and general geekiness – great fun! This time around I delivered two talks on Hadoop and HDInsight – the slides from both are attached.

Zero to 60 with HDInsight takes you from an overview of Big Data and why it matters (zero) all the way through an end to end solution (60). We discussed how to create an HDInsight cluster with the Azure portal or PowerShell and talked through the architecture of the data and analysis behind the release of Halo 4. We talked about how you could use the same architectural pattern for many projects and walked through Hive and Pig script examples. We finished up with how to use Power Map (codename GeoFlow) over that data to gain new insights and improve the game experience for the end user.

The next session I co-presented with HDInsight PM Dipti Sangani: CAT: From Question to Insight with HDInsight and BI. We went deeper this time. Not only did we present an end to end story with how our own internal Windows Azure SQL Database team uses telemetry to improve your experience with SQL Server in Azure PaaS but we also went deeper with demos of Hive, Pig, and Oozie. We also gave another archetypical design scenario that will apply to many of your own scenarios and talked about how HDInsight fits with SQL Server and your other existing infrastructure. The deck covers your cloud and on-premises options for Hadoop on Windows including HDInsight Service, Hortonworks HDP for Windows, OneBox, and PDW with Polybase.

Please let me know if you have any questions from the talks or just general HDInsight questions!

PASSSummit2013BigData.zip

How To Really Be Aware This October

Thanks to the “I Had Cancer” site for posting my guest blog this week on How To Really Be Aware This October.

Everywhere we turn we hear about “awareness”. That’s great, but what does it mean? Now it is Breast Cancer Awareness month – but awareness of what? What do you do with all this awareness? Will you be more aware at the end of this October than you were last October? What did you learn? And what will you do differently?

Read the rest here.

Diagnosis – Those First Moments of Breast Cancer

I wrote this Reader’s Opinion for the “Pink Edition” of the Idaho Statesman October 2, 2013 for Breast Cancer Awareness Month:

Do something. Anything. Make the cancer go away. What will I have to endure – will they chop off my breasts, make me ingest toxic drugs, poke me and prod me and stick me with needles and run me through machines? Quick; learn the difference between an MRI and an ultrasound and a mammogram – what does each really do, what does each detect, what kinds of false positives and false negatives might occur with each. This isn’t happening, it’s really a mistake. The test results aren’t really mine. Are they?

Keep Calm and Fight Cancer
Keep Calm and Fight Cancer

Deep breath. Of course it’s my diagnosis. And it’s mine to deal with. I can either panic (been there, done that – just a few seconds ago) and wallow in fear and pain or I can deal with it and move past it. Doesn’t seem like much of a choice, I guess I’ll deal with it.    It.    The breast cancer. I can fight it. I will fight it like a girl – proud and strong. I’ll overcome it. And I did. The cancer is gone. A little lumpectomy – done. A few weeks of radiation just in case there’s a microscopic cancer cell or two left over – coming in a few weeks with few expected side effects. Genetic testing – negative for the common breast and ovarian cancer causing mutations including BRCA. What a relief. I’m cured and I still have breasts that look pretty much like they did before. I’m done. I survived.

I took that fearsome cancer dragon and I turned it into something to live with. I didn’t do it alone. I have a great circle of friends who went with me to the biopsy and doctor appointments and pre-op and post-op…. They sent positive thoughts and showed they cared and they also knew when I just needed to be alone. I have a highly trained surgeon and oncologists and lab techs and nurses and even a nurse navigator assigned by the hospital. I work for a company that provides me with great insurance so I don’t have to worry about paying for all this wonderful medical care and the frequent screenings I will have in the future.

I am so lucky in so many ways. They caught my cancer early, I have wonderful friends, I asked questions and went back for additional screening after the initial false negative, and I have access to medical care many in this country can’t afford – though many more will get the care they need as more and more ObamaCare provisions are enacted.

You too can be lucky – take charge of your own life. Ask questions, take action, and be your own best advocate. Schedule your mammogram. Fight like a girl. Love your friends. Ask questions. Befriend your dragons. Help someone in need. Thank those who help you. Do, be, live, love, grow, change. And never give up.

Jo Ann Morris is Igniting Women with Courage

Ignite: Inspiring Courageous Leaders - A Book of Thought-Provoking Wisdom and a Manual for Action
Ignite: Inspiring Courageous Leaders – A Book of Thought-Provoking Wisdom and a Manual for Action

Go Lead Idaho sponsored a “meet the author” talk by Jo Ann Morris this week at the Boise WaterCooler. Jo Ann is the author of Ignite: Inspiring Courageous Leaders – A Book of Thought-Provoking Wisdom and a Manual for Action and co-founder of White Men as Full Diversity Partners. She describes herself as a proud radical feminist – I wish more people, men and women, had the courage to say that!

Jo Ann’s book, Ignite, helps you to take your own courageous actions. It has a series of “thought exercises” that each start with a powerful quote. She has suggested questions to ask yourself about each quote and there is room to write down the thoughts and feelings evoked by the quote. The exercises make you really think and help you get in the habit of looking beneath the surface and really digging deep. Then you can use your new insights to take action. Thoughts need to be followed by action to be powerful.

Jo Ann talked about taking charge in many ways. We are all responsible for ourselves. And we all need to help those around us.

  • Don’t spend time being nice – nice is overrated. This doesn’t mean to be deliberately mean, but don’t prioritize being nice or being polite above getting things done or getting what you need.
  • To be successful we need to take risks.
  • Don’t wait – step up and offer your ideas and actions.
  • Demand what you’re worth.
  • Be comfortable being uncomfortable.
  • Choose courage.
  • Be vulnerable to be courageous.
Courage!
Courage!

Courage encompasses four things. It can be manifested when you do one or more of these things:

  • See and speak the truth.
  • Champion an unpopular or risky vision.
  • Persevere.
  • Collaborate with AND rely on others. If you don’t rely on those you collaborate with you aren’t truly collaborating or being truly courageous.

In life we need truth, courage, and risk – they can’t really be separated. Women have the power to change the world. Don’t be “honorary men” – lead the way to a world that has a great combination of “feminine” and “masculine” ways of doing things. Have the courage to be the change!

Step up now – in your every-day life, in relationships, at work – and take charge of your own life. Be courageous, be uncomfortable, and be vulnerable. Stand up for yourself, help others, and be a proud radical feminist!

Go Lead Idaho – Get in the Game

This past week I attended another great Go Lead Idaho event – A Legacy of Leading. Go Lead Idaho helps women build leadership skills and helps women engage in politics, public advocacy, and public planning. The speakers this week, Marilyn Monroe Fordham and Rose Bowman, are two veterans of being “first”. Sometimes it’s easy to forget just how recently and severely women’s work and political options were limited to a small subset of opportunities.

Both speakers talked about being strongly discouraged in the 70s and 80s from choosing challenging, non-secretarial type degrees in college and from applying for jobs that were at the time typically reserved for men. Marilyn talked about staying in a banking job for years trying to break through the glass ceiling of “no women can be bank officers”. She eventually left to start her own business as promotion after promotion passed her by. They didn’t even hide why they wouldn’t consider her – they flatly stated it was because she was a woman. The powers that be also talked about the possibility that she might someday get pregnant as a roadblock to many roles – those were the days when women were expected to quit working as soon as they “showed” their pregnancy. While today few people would come out and say so, and many may think they’re being totally fair when evaluating people, there are countless subtle perceptions and reactions that still keep women from being completely successful.

This doesn’t mean we give up or sit around complaining – we need to stand up for ourselves. Don’t get discouraged, keep things positive, and stay focused on the goal. How other people perceive you matters – but don’t let it define you. And don’t try to do it alone. Ask for help and give help to others. Step up to help with projects – you will learn a lot, make new contacts, and show people what you can do. Even if you’re volunteering or doing something outside the scope of your core job you’re still showing people your skills and giving them a reason to remember you the next time an opportunity arises. Always be ready to help others, especially women who may be looking for a female based network. Help others feel confident and build their own circles.

When you are choosing new projects and opportunities challenge yourself. Don’t compare yourself to others and what they could do with the job, project, or role – think of what you can contribute and be creative about it. Others don’t really know more about how to do it than you do – and what you do know how to do could be exactly what is needed whether it’s typical or not. Stretch yourself and don’t focus at first on the practicalities. Figure out what needs to be done then come up with a plan that combines your needs with the needs of the job or project. Many times the schedules and specifics are much more flexible than they seem at first – ask for what you need.

Sometimes the best way to solve a problem is to redefine it. Marilyn recounted how she sat on boards with a mix of men and women and there were often 1-2 people who tried to “win” and dominate discussions. However, as she joined boards that were composed of all women she saw a lot more of a focus on solving the problem and collaborating. Over the years the boards she was on became more efficient as they spent less time “playing golf” and instead focused on getting the work done sooner so they could get back to their responsibilities such as families and full time jobs. This wasn’t because of the inherent gender differences but because the women had different goals in mind and focused on them. They stated their needs, got things done, and made the job better.

When Rose ran for US Senate in Idaho in 1972 she was the first woman to do so. People were less likely to give money to a woman and she was running in the primary against the husbands of friends. She got out, made contacts, networked, but still lost the primary. But she was out there, she showed everyone that a woman could run, and she leveraged the contacts she made into appointments to multiple statewide offices. She made a difference. So what was next – what women have run for US Senate since then in Idaho? None. Women haven’t stepped up. We all have our excuses – we’re too busy, we don’t feel we have the skills, or it just seems like too much work. But really – why hasn’t any woman run again in the last 40 years?

You don’t have to start out with a national political office – but start somewhere. Do something new, extend your comfort zone, grow  your network, and get in the game – any game! Go lead Idaho!

%d bloggers like this: