Archive

Archive for December, 2009

How to Avoid Auto Deployments Behaving Badly

December 18, 2009 4 comments

A couple of posts relating to development/ops gaps, and access control reminded me how very far we have come with our environments over the past few years.

Environment management in general comes with its own set of complications, but what I am interested in knowing is how other Systems/Operations departments are handling auto deployments. Several of the points in the articles above are moot for us and have been for some time. Access to production systems is heavily limited to our Systems Administrators and many of our products are deployed with scripts triggered by team city builds. Code or configuration files are not deployed manually to our end to end test environment, and when they are, generally an auto-deployment script needs to be touched to alter the outcome rather than a human touching a file. Don’t get me wrong, there are still blunders that occur from one environment to another “oh I forgot that we needed to add that file so I didn’t add it to the production release… oops”. However, my main concern is what to do when an auto deployment script behaves badly.

First, let us discuss what auto deployment is to our teams. When code is ready to release to a test or production environment, an auto deployment script will be executed to place the code on the correct server(s). In the case where code is going to production, a Systems Administrator is assigned a ticket to take a few preparation steps, then run the script to deploy the code. Depending on the environment and product, one or all of the following things could happen during an auto-deployment run:

  • Disable/enable monitors
  • Disable/enable nodes in a cluster
  • Run database scripts
  • Svn checkout or switch a directory or file
  • Copy files from one location to another
  • Start/stop services
  • Start/stop web containers

The first time I ran an auto-deployment release, I felt naked. I was so used to knowing every piece of what was actually required for a product to become live that pressing a button felt completely foreign. I worried that something would be missed, and I just wouldn’t know. So what should you do in a case like this? How do you mitigate the risk of the system being compromised? You’re running a script that you know little about other than you just typed several passwords into a config file on a secure server and it’s using your permissions to execute. However, because you have done this manually before, you know what the behavior and outcome of the script should be, so verify it.

The first few times we were vigilant. Check to make sure the node is out, review all the database scripts prior to release, check log files during and after the release, etc. Over time though, we started to become more comfortable with the release runs, and this is when the problems started to arise.

Auto-deployment scripts in recent months have been known to produce any one of the following symptoms:

  • Didn’t run the database scripts, or ran the incorrect database scripts
  • Didn’t remove the node from the cluster or didn’t put it back in
  • Deployed the wrong the version
  • Deployed the wrong application
  • Failed because of a spelling mistake
  • Showed “all green” in our monitoring system, but still caused a serious underlying problem

All these problems are compound, they are the responsibility of both the development and systems teams to realize and mitigate. The real root of the solution is to treat the auto-deployment script as exactly what it is: code. What do you do with code? You test it, you unit test it, and you make sure that it throws exceptions and failures when it doesn’t do what it is supposed to do. Most of the above problems could easily be avoided if the script was intelligent enough to know when it has made a mistake.

So, this is my list of remedies for saving an operations team from feeling naked when learning to run auto deployment scripts.

  • Get involved, have the script demoed to you and understand what it does before you run it. Ensure that every time it changes, you receive a new demo.
  • Don’t be naive, yes, maybe one day this script will run without someone watching over it, but before that happens, sys admins and developers should work out the kinks.
  • Review database scripts with a developer before the script is executed then check to ensure that one of the intended operations occurred.
  • Any time a problem is encountered post-release that is found to be the fault of the script, have the script altered to create a FAILURE state if it ever happens again. Additionally, review the script for other possible failures and have those adjusted too.
  • Check log files, review the script output and review the logs of your application containers.
  • Ask developers to check log files too, you will not always catch an error.
  • Consider having the script pause at critical stages (i.e. between nodes, etc) so that things can be inspected before they go too far.
  • Improve your monitoring systems. Create monitoring situations for the errors that get caused by the releases.

One day, after you have executed a script 10 times and it has not made a mistake, you will trust it. The important thing to remember is that the scripts will continue to evolve, so stay involved and help test every change. After all, as Systems Administrators or Operations staff, it is your job to ensure that the system is operational and take responsibility or alert the appropriate team when it is not.

Melanie Cey
Systems Administration Team Lead

Is Tardiness Contagious?

December 11, 2009 2 comments

Meetings are a regular occurrence in all office environments, and it is no different for an agile software development team. With the majority of of our teams following Scrum, the Daily Scrum, estimation, and retrospectives are just some of those many meetings. Even though my particular team has become quite efficient at running these meetings, there remains one problem that is common amongst almost all of them – people are always showing up late to them. This very problem was the focus of our last root cause analysis meeting (and yes, people showed up late to that one as well :P) as we hoped to identify reasons for meeting tardiness.


After drilling down into the problem, my team identified four categories that played a role in tardiness amongst members of our software development team.

Focus

  • Since we do 100% pair programming on our development team, a developer feels bad if they need to leave for a meeting that their pair is not in.
  • A pair is usually in some sort of flow while working on a problem and they often find it difficult to leave abruptly.

Complacency

  • It is generally okay for meetings to run long, so people start to think it is alright to show up late.
  • Meeting invitation notes (agendas, goals, etc.) are not read until a few minutes prior to the meeting start time, resulting in people being late.
  • It has just become the norm.  We have started accepting people being late as business as usual.

Scheduling

  • Coffee and bathroom detours are made on the route from desk to meeting location.
  • Meetings scheduled on the hour/half-hour are not disruptive enough to garner full attention.

Software

  • Since developers are pairing most of the time, they are rarely at their own desk to get the meeting reminders.
  • The snooze function and meeting reminders in Outlook and Entourage are not reliable.

It was no secret that this was a problem faced by all teams in our department (I can’t speak for other departments) so we came up with some action items that we could start to implement (on our particular team, at least).

  • Simply round people up prior to the meeting. This may only work if you are all located in the same area.
  • The organizer of the meeting should be the first person to show up, and welcome the attendees as they arrive. Many times people show up to a meeting room and no one is there, so they too wander off.
  • Identify the meetings for the day at the Daily Scrum.
  • Just start the meeting on time, regardless of whether everyone is there or not. People who show up mid-meeting will get the point.
  • A more extreme method which will not work all of the time is to simply cancel the meeting if everyone is not there on time.

It’s been a week since we’ve started tackling this problem and I can say that there has already been improvements made. Even though there is still a ways to go, we are confident that tardiness will soon become a thing of the past. The idea that we have become complacent to people showing up late just means that more and more people will follow this trend. If we can just turn that attitude around I think we can successfully make punctuality just as contagious as tardiness.

By Hemant J. Naidu

Movember

December 10, 2009 Comments off

Ah what can I say about Movember, it’s definitely the hairiest time of the year around here at Point2. For those of you who have never heard of it, Movember is the renaming of November to signify the thousands of men all over the world who are growing mustaches for the fight against prostate cancer. Most people at Point2 learned about it last year when our now CTO asked us to join him in growing mustaches to raise money for this cause. Soon after that many of the normally clean shaven men of Point2 were proudly displaying hair where there was no hair before. Again this year we have decided to put the razor away and focus on growing the baddest mustaches our faces will allow.

The development team that I am currently on has four brave men willing to put forth their best mo for a good cause. Mustaches can be very powerful tools, helping to fight against prostate cancer, and even helping our team reach new heights of greatness. During our retrospective last week it was found that our mustaches had a large role in helping us achieve our velocity. We did so well that we decided to hold a Root Cause Analysis to find out how our teams Velocity was so much higher than the previous week. In that meeting our mustaches were cited as an extra member of the team helping complete many tasks, they also helped to bring morale higher than usual. I believe that I was not the only one who came to the conclusion that it was in fact the ‘Mo’ that not only helped us achieve our velocity goal but also to surpass it.

However growing a sweet mustache is not all glory, for most people it takes time filled with ups and downs. Some people may look you at in disgust however you may find that the longer your mustache grows the more attractive you become. Most men usually go through the 5 stages of  ‘Mo’.

The 5 Stages of ‘Mo’:

1.  Denial – Thinking this year I’m not going to look like a porn star from the 70’s
2.  Anger – Grow faster and thicker!!
3.  Bargaining – I’ll grow twice the mustache next year if you just let me shave this thing off
4.  Depression – Oh, my wife won’t kiss me, and everyone looks at me like I’m insane
5.  Acceptance – Well we’ve come this far it’s finally looking like something, and I think that hot blond just gave me the eyes.

Now that the month is over and the mustaches have moved on, things feel different, almost less masculine. However Our Point2 team did raise almost $3000 this year for Prostate Cancer, and I know that everyone who participated feels stronger from the experience. We now know that growing a good friend only takes a month. Well mustaches and Movember go together like bacon and eggs, but all good things must come to an end.

Click here to donate to our team

By: Dean Gaudet

Test Driven Development with a Widescreen Monitor

December 9, 2009 Comments off

My Point2 workstation recently received a monitor upgrade; the old pair had had served me faithfully for many years, but they were worn out. So I am now the proud user of a pair of shiny new widescreen monitors.

The new monitors are much nicer than the previous monitors, and they are also a much higher resolution (1920 x 1080 vs 960 x 1280). Using them is a very enjoyable experience, but at first it seemed like the widescreen aspect ratio was wasting a lot of horizontal space:

Before: my widescreen monitor with a single file open

Before: my widescreen monitor with a single file open

Then a few days ago, in the middle of a pairing session, I discovered IntelliJ’s split screen feature. In my opinion, the combination of a wide screen monitor and split screen is a killer feature for TDD. If you assign your test code in one pane, and your production code to another pane, you can see your test and production code at the same time:

After: my widescreen with two files open simultaneously

To activate split screen choose “Split Vertically” from the Window menu, or right click on a document tab and choose “Split Vertically”.

IntelliJ works very nicely with this split screen configuration; intentions, “Go To Declaration”, and refactoring shortcuts jump nicely to the appropriate screen. Being able to read test and production code at the same time without clicking a button is especially valuable when pairing, as each half of the pair can be reading a different file. One important caveat: this only works correctly if each file is only open in one pane, and IntelliJ doesn’t enforce that for you.

Some of you might wonder what the second monitor is used for, and the answer is pair programming. We pair program all the time at Point2, so I run the two monitors as mirrors. I’ve also added an extra keyboard and mouse, and the combination allows each member of the pair to see exactly what’s going without craning their necks or stretching. It’s a really nice feature for a pairing station.

My "Pairing Station" - a pair friendly workstation

My pair-friendly workstation. My keyboard and trackball are on the left, and the developer pairing with me has their own keyboard, mouse, and monitor on the right. Either of us can drive without any inconvenience.

By: Sean Reilly

Let’s Learn from Each Other: BarCamp Saskatoon 2009

December 4, 2009 Comments off

Last weekend members of Saskatoon’s technical industry congregated at Louis’ Pub on the University of Saskatchewan campus for the annual BarCamp, a technical conference “for the rest of us.” For those of you who are not familiar with BarCamp, here is a little history as explained on the website.

The idea of BarCamp came from Foo Camp where a bunch of the best techie minds in the world got together to talk about new technologies, and to create opportunities for cross-fertilization between people and technologies.

Since only the top minds from O’Reilly, Google, Yahoo, Intel, Apple, etc. were invited to attend, two people in California decided to make a Camp for the rest of us. Now, there are BarCamps somewhere in the world almost every day.

This was the first year that I attended BarCamp so I wasn’t quite sure what to expect. I did understand that it had a laid back atmosphere which is quite apparent since it is held in a bar, but I also knew that the caliber of the talks would be high. The idea behind BarCamp is that everyone who attends has to participate in some way ranging anywhere from helping set up to leading a presentation.

As to be expected there was a high number of Point2ers in attendance, and impressively four of the nine presenters were from the office. Dave Kellow gave a spirited talk on Agile Modeling explaining why it is important, and discussed ideal ways of approaching the subject. Nate Heagy gave a brief, but extremely informative and entertaining presentation on Why You Should Get Good at Javascript. He used this as an opportunity to showcase an iPhone app he wrote using this technology as leverage. Chris Powell teamed up with his buddy Mark Poppen to give an information session on Starting a Web-based Business. Sean Reilly then wowed the crowd with his extensive knowledge of effective Functional Testing techniques, and why they are so important.

The time between all of the great presentations provided an opportunity for the attendees to mingle and find out what each other was up to. Being in a relatively small market, Saskatoon has a limited number of events like this so it is rare for members of the industry to get together and just hang out. Even though many different companies were being represented the atmosphere was of collaboration,  not competition. Everyone was there to learn from each other in hopes of bettering themselves professionally.  It’s events like these that will have a positive effect on Saskatoon’s technical community moving forward.  On behalf of Point2 I would like to thank everyone who made this event possible.

So you may be asking how I contributed at this year’s BarCamp. Well I didn’t help set up or lead any presentations, but I did shoot a lot of photographs to document the event. You can view the shots from the day here. The local media was also there to cover the conference and you can read the resulting article here.

By Hemant J. Naidu

The Agile Game

December 3, 2009 Comments off

Dr. Stuart Brown, founder of the National Institute of Play, believes that the act of play is essential to human development and intelligence. After watching one of his TED talks, Play is more than fun, I was inspired to look into games an agile team can play during a regular sprint. After all, it seems pretty straight forward that human development and intelligence are valuable aspects of a software developer’s career. So I hit up Google for what it could tell me about agile games, and found a bunch of balloon toting, crazy string wielding fun games I could play with my development team. I envisioned my team’s reaction to the introduction of these wild games:

1. Get a bunch of awkward looks.
2. Make a fool of myself to help lead the way
3. Get a few giggles playing the game
4. Try to come up with a different crazy game the following sprint

To me this seemed like a lot of effort for a little flash-in-the-pan fun. I decided I wanted to focus more on games we could play that would infuse themselves into our work stream — Heck, let’s just make our work a game! First let’s discuss what makes a game fun. Games are scenarios where you have a goal to reach while operating within established rules, boundaries and limitations. What makes the game fun is having the freedom to choose different paths toward the goal, being able to use your skills to achieve smaller goals along the way, and to reap the rewards for achieving the goal.

We already have plenty of rules, boundaries and limitations in our workplaces, but let’s talk more specifically to agile software development. I am talking about story breakdown, story estimating, spiking, definition of done for a story, coding standards, pair programming, TDD, stakeholder satisfaction, etc. Once the rulebook is well understood by your team you must give them time to play the game. YOUR ATTENTION PLEASE: the players need uninterrupted blocks of time where they get to play by the rules and innovate a solution. Don’t be afraid, these blocks are measured in hours, not days. Do what you need to do to provide the players with at least 75% of their day as uninterrupted game time (the more the better). If you need meetings with them make sure you consider the following:

1. Do you NEED the meeting?
2. Does everyone need to attend?
3. Can the meeting occur at the edge of a break (start/end of day, right before/after lunch)?

As a Team Lead your job is part referee and part coach. Encourage healthy demonstrations of play and discourage improper play. Help remove problems which affect their ability to play, and celebrate their victories while helping them learn from their defeats.

And finally, a word to the wise – break down your stories into small enough pieces that they can be achieved in no more than 2 work days. Having those frequent moments of accomplishment makes every game more enjoyable.

Logan Peters

Main Course: Hot Ideas

December 2, 2009 1 comment

Many interesting women I know are incredible geeks, and many interesting geeks I know are incredible women! But then, I may be biased, cuz I’m a girl, and yes, I’m a geek too!

The truth shall set you free!

What solidified the fact that I am, indeed, a geek was my first invitation to a Saskatoon Girl Geek Dinner. That was it!   I had no choice but to acknowledge my long-denied nerddom.   I missed SGGD’s first meetup, but I have attended every Saskatoon Girl Geek Dinner since.

Last Tuesday,  Saskatoon’s Girl Geek Dinners held its 4th event since its local inception in early summer, 2008.    The dinner was held at Zu’s stylin’ new digs on Pacific Avenue.   They were awesome hosts!

Organized by Melanie Cey, Brittany Melnyk and Devon McGeary, the GGD agenda included dinner, a presentation,  a workshop,  and an impromptu tour of Zu’s new facility.

The turnout was great!  Not surprisingly, the ratio of women to men was a complete reversal of what we see in the workplace.    Instead of 30:1 men to women, the ratio was 1:30+ men to women.

“I love my team, but…”

After dinner and socializing,  Melanie got the program started by explaining the Girl Geek Dinner concept to attendees.   You can read about GGD here.    Essentially, Melanie acknowledged that working in a still male-dominated field of study offers few or no female role models within the workplace.   Let’s face it: sometimes men and women exist on different planes and as a result life can be frustrating.   But we don’t want to give up, fellas… we love what we do! and we want to keep doing it for as long as it makes us happy.

Cue Ginger!

Ginger Koolick spoke to the group on becoming a consultant.  While outlining logistical processes of becoming self-employed, Ginger’s story had a notably female voice, and included not just business sense, but personal philosophy for leading a fulfilling life.   It was interesting to hear that despite the sudden nature of her decision to go into business for herself,  Ginger actually assembled a comprehensive business plan.   I hope in the future she returns to the group to present retrospectives on her initial plan.

Flexing our Agile muscles…

Devon McGeary ended the presentation layer of our event with a hands-on exercise called “Blitz Planning”,  an exercise which many of us at Point2 (if not all of us) have participated in to some degree within the past few months.   In a nutshell, the session emphasized a quick and collaborative approach to building a project’s stories and tasks, illustrating and revealing sequences, dependencies, moving parts, and necessary expertise. (I know I’m omitting objectives, so if the idea sounds interesting, you should read up on it starting here.)

Refactory!

Just before the evening ended, Zu gals gave us tours of their newly renovated space on Pacific Avenue.   It is a beautiful old building, and the decor is fresh and groovy.  And truly, who wouldn’t love a fireplace in their lunchroom!!!

Next up?

The story has yet to be written… GGDs are cooperatives which focus on science and technology, and of course, women working in those fields.  The Saskatoon chapter is new and growing, and certainly, there are many women (and men) who have much to offer the Saskatoon Girl Geek Dinner community.  Maybe you are one of them??

By: Karen Martens