A large portion of my career has been spent fighting against organizational disarray. Over the years, I’ve become very accustomed to being dropped into an established culture with the intent and promise of doing everything I could to improve it in some way. Sometimes I was alone, and sometimes I was even accompanied by my closest friends and colleagues. While I’ve both succeeded and failed to varying degrees in both situations — and occasionally, even, at the same time — one thing has not changed: this is not an easy task. We were cucumbers, being dropped into jars with the intent to somehow, against the forces of nature, un-pickle pickles.
We’ve all heard the phrase, “What gets measured gets done” but what happens when there’s nothing to measure? Recently, I was involved in a project where there was noticeable delay in getting the numbers behind various performance indicators. As it turns out, the stall was due to the fact that there was so little activity to report, people were concerned that showing “no measurements” equated to showing “no work,” which I suspect was never the intention of the Management By Objectives (MBO) process.
Look, I don’t want you to be offended, but I just don’t want to pair with you. No, I fully understand the benefit of pair programming, and have paired successfully quite a bit with other people. The problem isn’t with pair programming as a concept — the problem is you. I just don’t want to pair with you. You know how people say, “Don’t take it personally?” This isn’t one of those times — you should take this personally. Fundamentally, I am less productive as an engineer and less happy as a person when I pair with you. As a result, I’m not going to.
Ever heard this one: “Come on everyone! If we’re going to make the deadline, we’re going to have to put in some hours!” or “Jimmy is really working hard — look at all the hours he’s putting in!” Yes, Jimmy is putting in a lot of hours. Yes, Jimmy comes in at 9:00 am, and leaves at 11:00 pm. Jimmy is putting in hours. Jimmy is also writing crap that has to be re-written, missing requirements, and pumping out more bugs than functional production ready code. But boy, he sure is putting in hours.
Not a lot of companies do Agile well. I know some very smart people who think Agile is an utter hoax — a gimmick, used to make buckets of money by selling snake oil to gullible organizations that are just struggling to stay relevant. I don’t think it’s because Agile hasn’t been effectively evangelized. I certainly don’t think it’s because there is some fatal flaw with Agile methodologies that makes them abstruse or impractical. I think it’s because switching to Agile is painful, and people are opposed to pain.
Obviously, if you want a high quality product with a clean codebase and low defect counts, you need to be very careful and methodical in your approach to developing software. Ready-Fire-Aim is a sure road to disaster, leading to chaos and confusion. Not only that, but it speaks to a general immaturity and lack of professionalism of the development team and its members. That is, unless you iterate rapidly and continuously improve.
So your organization isn’t responding to reason. What started out as a team that would attempt to exemplify the way your organization wanted to do product development has become a micro-managed monstrosity, full of stakeholders, and absent of accountability. You’ve decided that in the face of this blame-storm you keep hearing referred to as a “team effort” (which you’re starting to believe!), you have to take a stand. Here’s what you’re in for:
A closing interview question I sometimes like to ask is, “Is writing software more like stacking bricks, or playing high speed chess?” Asked my own question, I might answer, “It’s like playing high speed chess in order to figure out which bricks to craft so that I can stack them.” Stacking bricks, with no caveat to explain their origin, couldn’t be a more incorrect metaphor for developing software. It is monotonous, predictable, and not mentally taxing in the slightest. Playing high speed chess, on the other hand, is a grueling intellectual process, and one that cannot be sustained indefinitely.
At a glance, it may not be immediately be clear how ops engineers and software engineer would see eye-to-eye on very many things. For the sake of uptime, ops tend to dislike change, while developers create and enhance, by continuously introducing refinements ( “change” ) — two seemingly opposing ends of the spectrum. However, the part that cause one group to resemble the other is the inconsistency of people. When it comes to being human, one type of engineer may be indistinguishable from the other. The reliance on people to “do the right thing” repeatedly, ironically, is the greatest threat in most organizations when it comes to productivity and efficiency. It is the people that’s most likely to break down. You, the people, are the weakest link.
There is a trend with my posts about software design where I write a lot about how it is a creative process. This is largely because for years, software design was considered to be busy work, and I take extreme issue with that philosophy. I find producing consistently effective software designs to be a very difficult task, and I deeply respect those who are good at it. I also find that those who are good at it have in common that they are all inspired workers.
One of the most common points of extreme contention I find between Agile and Waterfall practitioners involves a heated debate about the appropriate time in the project’s life cycle to be making important decisions about the software’s architecture. In one corner, we have the Waterfall approach of Big Design Up Front (BDUF), and in the other corner we have the Agile approach of Emergent Design. I’d like to outline some of the key differences between the two design philosophies because I think the right choice becomes obvious when they’re contrasted appropriately.
Speed and quality are universally contentious. Engineers always choose quality, but claim to value speed. Business owners always choose speed, but claim to value quality. Each side is skeptical of the other’s intentions when their actions seem to contradict their claims. Why does such an ironic and contentious relationship exist between the funder and the implementer? This post is not a referendum on either side: it’s a bi-directional concession letter, between Competent Business Owner and Competent Engineer.
Held captive in a status meeting, your moment of shame approaches. Vulnerable and exposed, like a naked baby laid before encroaching doom, you await your turn. Your peers’ bold claims, naive honesty, and feeble excuses are recorded for posterity by a stone faced facilitator. They work their way around the room, thinly masking disdain at any answer that is not a crisply delivered “I’m done”. Finally, the time of judgement is upon you. As you fumble to explain your lack of completion, you are interrupted by a question that has struck down even the strongest among us: “When will you be done?” Fighting back a swelling tide of emotion, you try desperately to think of what you can say that you haven’t already. Step aside my child, let me handle this.
I love information technology. I’ve been fortunate, in that I’ve always known what I wanted to do [professionally] and then be able to pursue that with vigor and passion. Over time, as I move up and through my career ladder, I’ve deliberately aligned myself with people who’ve garnered my respect, and conversely, people who recognize my values. As such, I’ve worked for a series of companies fitting a certain profile. Until recently… with some changes in my [personal] life, I suddenly find myself craving something different, something outside of what I’ve known. Making career changes is not the simplest of tasks and required that I exercise some skills I’ve not used in some time.
This week, an incident happened with Knight Capital when their “trading algorithms” allegedly cost the firm hundreds of millions of dollars. Within hours of the story, various camps have been quick to denounce the algorithms and the automation that was supposed to save the day. Of course, the problem is in automation, because automation is supposed to reduce errors and prevent outages and boost security and mitigate risks and improve the bottom line and make ice cream sundaes with a cherry on top. Except when it doesn’t, and now this latest mishap only adds to the argument against such automated practices.
In yesteryear, in the time of our coding forefathers, ancient project managers would gauge a developer’s productivity my measuring the lines of code added per unit time. This was a convenient and intuitive metric, as there was seemingly a correlation between number of lines coded and number of features added. When companies then started rewarding developers according to this metric, it lead to a rash of bad programming habits – most notably copy and paste coding. In modern times, we consider ourselves enlightened, and claim to have realized that not only is there no direct correlation, but that attempting to measure developer productivity in this way is both meaningless and destructive. Yet, if you corner the average development manager and ask them to compare developer productivity, many of them would not be able to resist the urge to pull metrics from source control. This due to a lack of awareness of a complete reversal in the way in which lines of code reflect productivity.
I have a predictable threshold for Macy’s. Not too long ago, I was shopping there with a girl — which is to say that she was shopping, and I was serving dual life-sentences back-to-back for a crime I didn’t commit. The likeliest explanation for my unjust, albeit fashionable, incarceration was that she was a heartless shrew who had no concern for my long-term psychological or emotional well-being. In the unlikely event, however, that she was not pure evil, but merely that the allure of fragrances and cosmetics had acutely undermined her ability to feel specific empathy for the opposite sex — the scents having temporarily transmuted her into a sort of sociopathic shopaholic — I felt it would be in my best interest to point out that not only was Macy’s not where I would like to spend 84 hours of a Saturday afternoon, but also that it was utterly absurd to spend $1.4M — which is only a slight exaggeration — on eye shadow.
In 2005, socioeconomic researchers at Harvard, in conjunction with the local police force, performed a social experiment in Lowell, Massachusetts. After identifying over 30 of the highest crime rate areas in Lowell, the police forces were instructed to clean up litter from the roads and sidewalks, replace broken street lights, and implement a series of other small “cleanup” initiatives for 15 of the 30 areas. In the other 15, the police were instructed to continue operating normally. The result was an almost immediate 20% drop in police calls in the cleaned up areas. This is an interesting case-study for Broken Windows Theory.
It’s a well known fact among software professionals that the moment your friends and family discover your technical competency, you will become their indentured technical support servant for life. Whether it’s re-installing Windows or replacing ink cartridges, no task is too demeaning: if it involves a computer, you’re the first phone call they make. And why not? This is your fault, after all.
As I watch the flight attendant go through the pre-flight safety speech, I cannot help but wonder how many people are paying attention, and more importantly, in a “real” emergency, if people will actually find their nearest exits. That’s not just a problem plaguing airline passengers. I routinely observe managers, developers and engineers ignore smart practices and safety procedures, and head blindly into tasks without proper planning, ill-informed, or worse yet… motivated by fear. It’s not wonder they, along with their code and systems, end up in a prison of their own creation — the kind of legacy scenario we retell like ghost stories, nonetheless, people continue to not heed this information. Knowing where the exits are will help you to avoid getting trapped in your burning jail cell.
Let’s briefly take a step back. Yes, we’re all very concerned with how best to drive software development. We love our methodologies and our philosophies — our little software calendar platitudes and acronyms. We love Martin Fowler, Kent Beck, Eric Evans, Bob Martin, so on and so forth. But really, what is the point? This article is for everyone who has ever been subjected to my ongoing ramblings and critiques of software, processes, and team dynamics. I’d like to try to offer you a moment — if ever so brief — of catharsis. These were my intentions. Here was my point.
Since I’ve already started the fire for more Agile in operations, it makes sense to actually discuss what exactly is involved in doing just that. After all, this isn’t just envy of my fellow software development brethren– then again, who wouldn’t want to be a hip and Agile developer? — these are real methodologies and enlightenment gleamed through blood, sweat and tears and savvy Ops should outright
borrow steal those tough-earned wisdom from the software teams. If nothing else, only to avoid doing any real work so that we may continue to be grumpy and misanthropic stonewalls that system administrators are known for. And play StarCraft.
“Good morning everyone. First, let me say how excited I am on this, the first day of our new project! The goal of the project is to build a single family home, and we have a month to build it. Working closely with our team of engineers and architects we have broken down all the individual requirements for the house, estimated how long they will take, and all we need to do today is prioritize them. Since building houses is intuitive to everyone, this shouldn’t take too long, but I’ve blocked out 2 hours just in case we need extra time. Let’s get started!”
Management loves, and loves to tout about, training. Management distrusts training, because it highlights broken people, systems and processes. How can you love and distrust something at the same time? The dichotomy is a simple reflection that as it is designed today, many if not most training programs are ineffective and are nothing short of last-ditch efforts at salvaging un-qualified employees. This is especially true within the technology sector, a group of professionals that may benefit the most from training yet reaps the least as a result of mandate, motivation and momentum associated with training. There has to be a better way.
Go to a developer-oriented gathering and you’ll hear this: “I have no interest in learning <xyz>” where <xyz> represent some kind of operational tasks or knowledge. Why should they? System administration is not really essential to software engineering, and conversely, ops teams have similar disinterest in writing code. Or they would be doing each other’s job already. That doesn’t mean there aren’t lessons to be learned from one another. In fact, the emergence of devops reflect just that recognition. It’s time for operations to adopt and apply the same discipline and knowledge that their brethren in the software camp have gradually refined over the years. It’s time for agile operations.
It’s not the easiest pill to swallow, and more importantly, imagine trying to convince the management team to not only implement an external security review program, but to provide incentives for the resulting discoveries — that’s right, we’re going to pay others for our mistakes. Yikes! Let’s face it, there will be bugs in software, despite development process, review and QA. Items will get missed, and unintended feature or behavior will creep into the code base. The systems have become complex. The interactions are not always planned or even manageable with 3rd parties. Likely, only non-developers won’t acquiesce to that simple truth. It’s an understandable sentiment by the business stakeholders, as their focus isn’t on design or implementation of complex application logic. However, there should be one common ground within any organization, and that is the need to be diligent stewards of their customers, and by extension, their customers’ information. Security is the bedrock of excellent customer service.
The majority of meetings are a waste of time for the majority of people in them, yet we call them mercilessly whenever we feel there is the slightest communication need. Meetings have no cost that can be directly measured, but a benefit that can: People sat in a meeting for a period of time. Really, as stated, that can only be a good thing — especially if it’s the right people. With this in mind, great care is taken in the selection and culling of meeting attendees. After all, without the right people, how can it have the right outcome? This is where the concept of the meeting breaks down, as “the right outcome” is rarely understood much less articulated to attendees.
I never get why people panic at any point during a deployment. When you’re doing a deployment you’re in only one of two situations: 1) You’ve never deployed the app before 2) You have. If you never deployed the app, there’s no audience to complain if things go wrong, so take your time and figure them out – no need to panic. If you have deployed the app before, you damn sure better have a rollback rip chord in place. Fact is, the vast majority of deployments are done without even the semblance of a rollback plan. The best they have is to restore the database from dump files and build an earlier source control tag. That’s not a plan, that’s inviting disaster.
If, as a developer, you care about security, you need to be constantly running pentests against your own code. Constantly – and I’m not talking about buying an off the shelf tool that will do the scanning for you. Those are important, but they’re something that QA or Operations can use to cross-check your work. What I mean is good, old fashioned, trying to break into the software you just wrote. This shouldn’t be too hard, you wrote it! You know where you usually slack off, so you’re in the best position to find vulnerabilities in your own code.
Terrorism comes in many forms. It’s commonly typified by leveraging hostage situations to undermine national policy and to facilitate foreign influence, or personal ulterior motives. An increasingly common incarnation of what I consider to be corporate terrorism manifests itself as so-called “knowledge anchors”: employees who have been with a company long enough to have acquired intimate knowledge about the business’ problem domain, and have subsequently outlasted any one else who might also have acquired the knowledge.
A lot has been written about execution. Yet it clearly remains a challenge and an elusive goal to both individuals and teams. I could start espousing my own theories of execution, but I have no desire to add to the stream of management mumbo-jumbo regarding roles, responsibilities, metrics and results. Make no mistake, it is precisely the desire for results — the point B, in going from point A to point B — that leads us to examine and question execution. Too often, even capable people [and teams] focus on making history, when the focus should be on making impact.
Here’s a scenario: two women are at Starbucks, talking about how their respective weeks have been. One woman pauses, then exclaims, “I’m just so excited about parenthood! I can’t stop thinking about whether it’s going to be a girl or a boy!” “Oh my gosh! Are you pregnant?!”, the other woman responds, understandably shocked, given that her friend has yet to show any signs of a pregnancy. “Well, not completely. I’ve never really been sold on the whole pregnancy process – you know, the mood swings, the cravings, the added weight. It’s not really right for me; I’m just interested in the ‘having a child’ part. I guess you could say I’m mostly pregnant.”
Agile. Don’t roll your eyes. Yeah yeah yeah you say, and I appreciate that – at least you’re being honest. Agile ain’t panning out. All of these good things were supposed to come out of it, but it all just seems like a disorganized mess. People are complaining, work isn’t getting done, and all the while you’re reading about how good it is. You don’t get it. I understand. You wouldn’t get it, would you? You’re doing it all wrong. You made the common mistake of thinking that somehow Agile was a relief from the rigors of Waterfall methodologies, and that you could be on a perpetual process vacation.
Deploying sophisticated software onto the internet is not easy. It involves lots of fine, intricate steps being executed in exactly the right sequence, and if any one of a thousand steps is not done in precisely the right order, the whole thing will fail. Knowing this, what do we do? We put a bunch of people on standby just in case anything goes wrong, and make sure that everyone is on a conference call and in a too-massive-to-communicate-effectively chat room at 2:00 am so that they can be Johnny-on-the-spot if something goes to wrong. Here’s a tip: If you are so nervous that something will go wrong that you need to have dozens of sleep-drunk people around ready to fix something they might have broken (or harass someone else who may have broken it) then you have too many damn humans involved in a process that needs to be fully automated.
True operation mavens know that downtime is inevitable. It’s going to happen, despite your best efforts. A blip, a stumble, some cable will get cut. Increasing the “nines” carries quite the price tag, and may not be the best way to maximize ROI. The plans for disaster recovery needs to be balanced, so that focus isn’t solely on the prevention of catastrophes. Equally important, is the rapid recovery for business continuance. Because that is the true goal of uptime — to serve pages, apps and data, to provide for the customers, and continue the revenue stream. This is no longer an insurmountable task, given the resources and knowledge at hand.
You need a plan. You’ve just been told by your boss that the highly visible, habitually failing flagship project – yep, the one that’s a year behind schedule – just became your problem. “Don’t worry, you can only come out of this looking good,” you hear, among other things, like “hand-picked” and “Swat Team” – each one making you cringe more than the last. You’ve been here before, and while the stakes may be different, one thing is certain – You. Are. Fucked.
Successful organizations thrive by their talents [people], and it’s mostly a losing battle because so many hiring decisions are simply… bad. Over the years, pundits and experts offer up many theories and philosophies on how to recruit, and then retain, superior personnel. It really comes down to just three things — Proficiency, Passion and Personality. That’s the secret. No more. No less. Interviews, tests and profiles are but the tools to establish how a candidate measure up under each areas.