A large portion of my career has been spent fighting against organizational disarray. Over the years, I’ve become very accustomed to being dropped into an established culture with the intent and promise of doing everything I could to improve it in some way. Sometimes I was alone, and sometimes I was even accompanied by my closest friends and colleagues. While I’ve both succeeded and failed to varying degrees in both situations — and occasionally, even, at the same time — one thing has not changed: this is not an easy task. We were cucumbers, being dropped into jars with the intent to somehow, against the forces of nature, un-pickle pickles.
I have, on many occasions, been a terrible technical lead. For years, I had no idea why people continued to promote me into these positions, even across different jobs. It’s not like I wanted to be a technical lead, they would just walk up and tell me that would be my new job. I was a terrible technical lead, because I could never work out what the job entailed. To become a technical lead, you had to be the most skilled of any of the other engineers — at least that’s the idea. However, to stay skilled you need to code constantly — that’s the “technical” part. The “lead” part, however, never seemed to fit, because to lead, you have to do more than sit at your desk and code all day with your headphones on. How can you stay technical and still lead?
Look, I don’t want you to be offended, but I just don’t want to pair with you. No, I fully understand the benefit of pair programming, and have paired successfully quite a bit with other people. The problem isn’t with pair programming as a concept — the problem is you. I just don’t want to pair with you. You know how people say, “Don’t take it personally?” This isn’t one of those times — you should take this personally. Fundamentally, I am less productive as an engineer and less happy as a person when I pair with you. As a result, I’m not going to.
Ever heard this one: “Come on everyone! If we’re going to make the deadline, we’re going to have to put in some hours!” or “Jimmy is really working hard — look at all the hours he’s putting in!” Yes, Jimmy is putting in a lot of hours. Yes, Jimmy comes in at 9:00 am, and leaves at 11:00 pm. Jimmy is putting in hours. Jimmy is also writing crap that has to be re-written, missing requirements, and pumping out more bugs than functional production ready code. But boy, he sure is putting in hours.
Not a lot of companies do Agile well. I know some very smart people who think Agile is an utter hoax — a gimmick, used to make buckets of money by selling snake oil to gullible organizations that are just struggling to stay relevant. I don’t think it’s because Agile hasn’t been effectively evangelized. I certainly don’t think it’s because there is some fatal flaw with Agile methodologies that makes them abstruse or impractical. I think it’s because switching to Agile is painful, and people are opposed to pain.
Obviously, if you want a high quality product with a clean codebase and low defect counts, you need to be very careful and methodical in your approach to developing software. Ready-Fire-Aim is a sure road to disaster, leading to chaos and confusion. Not only that, but it speaks to a general immaturity and lack of professionalism of the development team and its members. That is, unless you iterate rapidly and continuously improve.
Poorly written code creates exceptions. Bad deployment extends service outages. SLA violated due to extreme acts of nature. Let’s face it, bad things happen as part of life, and also because people are human and mistakes are made along the way. An analogy may be made about going to restaurant and ordering some food, the entire dinner/experience depends on a chain of events starting from freshness of ingredients, to expertise in the kitchen, to something as simple as hospitality and greetings. When something goes awry at the restaurant… someone on the staff will take ownership and assume responsibility. Often with that, comes an apology and some immediate offer of amelioration. We’ve come to expect that as part of customer service. The same analogy quickly breaks down when applied to the technology industry. Ironically, when things go badly, we tend to blame the technology.
The psychology of the average software engineer is fascinating. Not only do they develop a sense of entitlement towards management, as well as an attitude of elitism toward Product and QA, they will also seek to segment themselves from other software engineers through the selection of their programming language. Following a fraternal instinct, the overwhelmingly male software engineering community clumps into programming language cliques, and once the bond is established it can last their entire career. Sadly, much like college frat-boys attempting to define themselves through Greek letters, defining yourself by your programming language is both pathetic and myopic.
A closing interview question I sometimes like to ask is, “Is writing software more like stacking bricks, or playing high speed chess?” Asked my own question, I might answer, “It’s like playing high speed chess in order to figure out which bricks to craft so that I can stack them.” Stacking bricks, with no caveat to explain their origin, couldn’t be a more incorrect metaphor for developing software. It is monotonous, predictable, and not mentally taxing in the slightest. Playing high speed chess, on the other hand, is a grueling intellectual process, and one that cannot be sustained indefinitely.
At a glance, it may not be immediately be clear how ops engineers and software engineer would see eye-to-eye on very many things. For the sake of uptime, ops tend to dislike change, while developers create and enhance, by continuously introducing refinements ( “change” ) — two seemingly opposing ends of the spectrum. However, the part that cause one group to resemble the other is the inconsistency of people. When it comes to being human, one type of engineer may be indistinguishable from the other. The reliance on people to “do the right thing” repeatedly, ironically, is the greatest threat in most organizations when it comes to productivity and efficiency. It is the people that’s most likely to break down. You, the people, are the weakest link.
There is a trend with my posts about software design where I write a lot about how it is a creative process. This is largely because for years, software design was considered to be busy work, and I take extreme issue with that philosophy. I find producing consistently effective software designs to be a very difficult task, and I deeply respect those who are good at it. I also find that those who are good at it have in common that they are all inspired workers.
One of the most common points of extreme contention I find between Agile and Waterfall practitioners involves a heated debate about the appropriate time in the project’s life cycle to be making important decisions about the software’s architecture. In one corner, we have the Waterfall approach of Big Design Up Front (BDUF), and in the other corner we have the Agile approach of Emergent Design. I’d like to outline some of the key differences between the two design philosophies because I think the right choice becomes obvious when they’re contrasted appropriately.
In 1995, there were approximately 5 million mobile connections. A short 15 years later, that number is closer to 5 billion a number that’s rapidly approaching and will handily surpass the actual population of the planet. Look around, and look on your person, it’s likely you’ll find a smart phone, maybe a tablet or even an older MP3 player. This isn’t even counting such antiquated computing resource like a laptop, and it’s easy to see why the calculation is going to average over four such devices per person. To say there will be growth in mobile is a bit of an understatement, once you consider all the other “widgets” yet to come — from embedding into appliances to health-related devices, like glucose meters, heart monitors or even plain old thermometers. My point is, lots of mobile (platforms) invites for lots of mobile apps. Seems like daily, that I hear about someone becoming a mobile application developer.
Speed and quality are universally contentious. Engineers always choose quality, but claim to value speed. Business owners always choose speed, but claim to value quality. Each side is skeptical of the other’s intentions when their actions seem to contradict their claims. Why does such an ironic and contentious relationship exist between the funder and the implementer? This post is not a referendum on either side: it’s a bi-directional concession letter, between Competent Business Owner and Competent Engineer.
Held captive in a status meeting, your moment of shame approaches. Vulnerable and exposed, like a naked baby laid before encroaching doom, you await your turn. Your peers’ bold claims, naive honesty, and feeble excuses are recorded for posterity by a stone faced facilitator. They work their way around the room, thinly masking disdain at any answer that is not a crisply delivered “I’m done”. Finally, the time of judgement is upon you. As you fumble to explain your lack of completion, you are interrupted by a question that has struck down even the strongest among us: “When will you be done?” Fighting back a swelling tide of emotion, you try desperately to think of what you can say that you haven’t already. Step aside my child, let me handle this.
While on site at a client the other day, I heard one of the FTEs make the statement about their brand new legacy application that “there was too much technical debt, so it needed to be rewritten”. I cringed a little bit, but I considered that I was just being pedantic, and decided not to nit-pick at the misuse of the term “technical debt”. Looking back, I’m not sure I should have let that statement go. I probably should have shed light on the important distinction that should be made between accruing technical debt and creating a mess.
If you’re a senior-or-above talent, you know how work longer hours without it significantly impacting your work product. For this, I’m talking about the 40-50 hour range. Most people start to fall off around the 50-60 hour range, and we all start producing crap around 60-80 hours. Beyond 80 hours, it probably time to look for a new job. Having said this, let’s focus on the 40-50 hour range. If your employer suggests or requires that you work these extra hours, it’s probably not going to destroy your home life, or wreck your sense of well-being, so the impact should be minimal, if not negligible. Why not work these extra hours? Because you’re not getting paid for them.
What do you suppose the average person thinks of when they hear the words “software engineer”? Perhaps they think of a famous (or infamous) entrepreneur who’s been forcing them to restart their PC for years. They might even think of a hip, well-built hacker with an earring who assembles software “worms” by racing to reconstruct some antagonistic digital Rubik’s Cube1. But I would bet that the majority of the population would think of a middle-aged man with high blood pressure and vitamin D deficiency. Well folks, I’m here to tell you that while that last group may have been right for years, the Age of Milton is over.
In yesteryear, in the time of our coding forefathers, ancient project managers would gauge a developer’s productivity my measuring the lines of code added per unit time. This was a convenient and intuitive metric, as there was seemingly a correlation between number of lines coded and number of features added. When companies then started rewarding developers according to this metric, it lead to a rash of bad programming habits – most notably copy and paste coding. In modern times, we consider ourselves enlightened, and claim to have realized that not only is there no direct correlation, but that attempting to measure developer productivity in this way is both meaningless and destructive. Yet, if you corner the average development manager and ask them to compare developer productivity, many of them would not be able to resist the urge to pull metrics from source control. This due to a lack of awareness of a complete reversal in the way in which lines of code reflect productivity.
Yesterday morning, along with hundreds of thousands others online, I watched the live HD video feed of the Mars Science Laboratory Curiosity successfully touch down at Gale Crater as planned/designed. It is a significant milestone, and quite the hallmark of success for a complicated mission. I couldn’t help but think about the dichotomy that is NASA — the rare mix of size, bureaucracy and performance. Over the years, we’ve witnessed their triumphs and their failures, as well as some epic recoveries from disastrous missteps that plague the largest of enterprises. One observation became clear to me, as I listened to the debriefing panel: if you’re not making something better than you’re not relevant.
In 2005, socioeconomic researchers at Harvard, in conjunction with the local police force, performed a social experiment in Lowell, Massachusetts. After identifying over 30 of the highest crime rate areas in Lowell, the police forces were instructed to clean up litter from the roads and sidewalks, replace broken street lights, and implement a series of other small “cleanup” initiatives for 15 of the 30 areas. In the other 15, the police were instructed to continue operating normally. The result was an almost immediate 20% drop in police calls in the cleaned up areas. This is an interesting case-study for Broken Windows Theory.
Software engineers have an uncommonly difficult job. Not only are they tasked with the monumental challenge of accurately modeling a business in digital interactions, they also need to be constantly permuting the different ways they could be solving their problems and the pros and cons of each possible implementations. To make matters more difficult, they’re having to constantly juggle the ever-evolving needs of the client and an employer who thinks having something done yesterday is still late. You’d think that with all of this, we’d find little time to resist the progressive development philosophies and methodologies that have proven themselves for years from entrenching themselves in our day to day lives.
It still amazes me, sometimes, that SQL Injection ever came into vogue, becoming one of the poster children of web application vulnerability. It’s outright jaw-dropping that in 2012, an iconic web company would fall victim to this technique. I could go on and on, about the number of things that went awry and/or should’ve been done. But this appears to be a chronic failure, with each generation of software engineering re-inventing the same bug all over again, like an endless nightmare of unlearned remedial lessons. SQL injection attack is a variation on one of the oldest secure computing tenet no-no’s, and that is the implicit granting of permissions.
I started out titling this as The Challenge With Mobile, but the thoughts that keep me awake and go bump in the night are really troubling. I wonder and worry if people even recognize this brave new, slightly dystopian, world of technology we have created for ourselves — one which the phone is never off. An always-on and always-connected digital frontier, full of irresponsible citizens who fail to exercise their civic responsibilities on minding their own perimeter defense… thus as a result, endangering my co-existence within that space. If the initial wave of personal computers joining the Web unleashed a wave of malice and destruction, you ain’t seen nothing yet.
A regular critique I hear of engineers who are new to projects or codebases is that they provide almost immediate negative feedback about what they see, and are in no position to do so. The feedback tends to be received as being premature, unsolicited, and out of context, and is therefore undesired. To further complicate things, the engineer typically genuinely wants to improve things, and isn’t merely interested in bitching. I’ve personally received this feedback multiple times, and frankly, I’ve had just about enough of it. Rather than meet this constructive criticism with hostility, I’d like to propose a different approach for the receivers of the feedback. I’d also like to offer a peaceful compromise.
As I watch the flight attendant go through the pre-flight safety speech, I cannot help but wonder how many people are paying attention, and more importantly, in a “real” emergency, if people will actually find their nearest exits. That’s not just a problem plaguing airline passengers. I routinely observe managers, developers and engineers ignore smart practices and safety procedures, and head blindly into tasks without proper planning, ill-informed, or worse yet… motivated by fear. It’s not wonder they, along with their code and systems, end up in a prison of their own creation — the kind of legacy scenario we retell like ghost stories, nonetheless, people continue to not heed this information. Knowing where the exits are will help you to avoid getting trapped in your burning jail cell.
Let’s briefly take a step back. Yes, we’re all very concerned with how best to drive software development. We love our methodologies and our philosophies — our little software calendar platitudes and acronyms. We love Martin Fowler, Kent Beck, Eric Evans, Bob Martin, so on and so forth. But really, what is the point? This article is for everyone who has ever been subjected to my ongoing ramblings and critiques of software, processes, and team dynamics. I’d like to try to offer you a moment — if ever so brief — of catharsis. These were my intentions. Here was my point.
Fresh out of college, you have no idea what you’re doing. This fact is true no matter how much you paid for you college education, where you went, what your GPA was, or what Latin title was bestowed upon graduation. You have no clue, because you’ve never had a real job – this is your first. The number one gauge of being good at a job is experience in doing that job, and you have none. You may think that your college education is an indicator of how quickly you will master your job, but there is no correlation. It might actually work against you, because if you believe you already know everything you won’t be open to learning anything. If you want to get good at a job and you have no experience, you’ll need to apprentice under someone who can show you the ropes.
Since I’ve already started the fire for more Agile in operations, it makes sense to actually discuss what exactly is involved in doing just that. After all, this isn’t just envy of my fellow software development brethren– then again, who wouldn’t want to be a hip and Agile developer? — these are real methodologies and enlightenment gleamed through blood, sweat and tears and savvy Ops should outright
borrow steal those tough-earned wisdom from the software teams. If nothing else, only to avoid doing any real work so that we may continue to be grumpy and misanthropic stonewalls that system administrators are known for. And play StarCraft.
In Working Effectively With Legacy Code, Michael Feathers defines “legacy code” as code without tests. I love this definition for two reasons: first, it provides a succinct, objective way to measure whether code is legacy or not; secondly — and this really is the main reason I love it — it asserts that any code, written by anyone, at any time, without accompanying tests, is legacy the moment it’s written.
The 2000 Presidential Election was mired in controversy over which candidate received the popular vote. Whether due to faulty hardware, or a state-wide psychomotor-disfunction epidemic, a large number of Florida ballots were unclear about which candidate they endorsed. “How could something as parochial and deterministic as aggregating multiple choice answers lead to such a quagmire?”, you might ask, and indeed, it makes for an entertaining story.
How good are you – really? Are you a master of your craft, merely mediocre, or do you suck? How would you know? Asking other people tends to yield nothing more but polite indirection and flattery, or at the other end of the spectrum rudeness masquerading as honesty. Within your organization, there are annual reviews, but they only gauge your performance against set expectations, not how you stack up to your peers. Without true visibility into other organizations, much less how their employees perform, you’re left with how they describe their employees: Typically infallible and elite. In truth, there is no yardstick to assess your own abilities compared to others, so what are you to do? How do you know if you need to improve, if you’re perfectly adequate, or if you’re so good that you need no further improvement?
Go to a developer-oriented gathering and you’ll hear this: “I have no interest in learning <xyz>” where <xyz> represent some kind of operational tasks or knowledge. Why should they? System administration is not really essential to software engineering, and conversely, ops teams have similar disinterest in writing code. Or they would be doing each other’s job already. That doesn’t mean there aren’t lessons to be learned from one another. In fact, the emergence of devops reflect just that recognition. It’s time for operations to adopt and apply the same discipline and knowledge that their brethren in the software camp have gradually refined over the years. It’s time for agile operations.
As problem solvers, we dream of a positive vision of the future, based upon the belief that the challenges and problems we face are solved by good design. OK, stop the playback. Back in the real world, the dominoes occasionally get knocked down in sequences un-imagined and worse, in ways where our complex and richly-integrated systems cannot address. Some of it will be act-of-god happenstance, some will be introduced by the ever fallible humans, while others are just malicious intent. It’s a harsh reality, but that is and have always been the Internet in spite of funny cat GIFs. When you are creating a solution – from authentication to authorization — make sure the design takes some consideration for all intents, not just the ones derived from planned and expected user stories.
The Boy Scouts of America have a rule, “Always leave the campground cleaner than you found it.” I was never a Boy Scout, but I found out about this rule when I read Bob Martin’s Clean Code. I thought it was such a splendid philosophy to apply to software engineering and codebases that I’ve probably referenced it a hundred times. Sure, it rolls off the tongue nicely, and teaches children to clean up after themselves, but I knew there was something subtle about it that I liked that wasn’t immediately apparent — something more fundamental, revolving around diligence, discipline, and dedication. It occurred to me that the underlying philosophy I was so attracted to was this revolutionary concept of giving a shit.
There’s a skill that I’ve rarely seen among recent college grads walking into enterprise codebases, and that’s maintaining legacy code. It must be shocking to expect to walk off campus and on to a green field project, only to be met with a “where the hell did these 1.5 million lines of code come from?” sentiment. I’ve observed three particularly difficult challenges that less-experienced engineers face when working with legacy codebases: rapid understanding, adapting design, and finally, implementing bug-free modifications.
I have a few a stories I’ve told over the years to explain my origins in software engineering. One of the more humorous ones is that I wrote down on a piece of paper “Doctor, Lawyer, Software Engineer”, and chose the one that didn’t legally require a college degree. While that story is actually true, the reason Software Engineer ended up on the paper to begin with was that I had an instant, lasting attraction to the notion of being able to model real world interactions for the cost of electricity. Potentially even more than that was the allure of creating something out of nothing.
If, as a developer, you care about security, you need to be constantly running pentests against your own code. Constantly – and I’m not talking about buying an off the shelf tool that will do the scanning for you. Those are important, but they’re something that QA or Operations can use to cross-check your work. What I mean is good, old fashioned, trying to break into the software you just wrote. This shouldn’t be too hard, you wrote it! You know where you usually slack off, so you’re in the best position to find vulnerabilities in your own code.
Terrorism comes in many forms. It’s commonly typified by leveraging hostage situations to undermine national policy and to facilitate foreign influence, or personal ulterior motives. An increasingly common incarnation of what I consider to be corporate terrorism manifests itself as so-called “knowledge anchors”: employees who have been with a company long enough to have acquired intimate knowledge about the business’ problem domain, and have subsequently outlasted any one else who might also have acquired the knowledge.
Work in software development long enough, and one day, you too will hear this phrase uttered by someone, “I used to code.” While the intention may be one of empathy or solicitation, as the identification of the supposed, shared, past is meant to build bonds. Inevitably, this utterance almost always forms a divisive line between those who write software and those who’ve stopped writing software to perform another role. The simple observation is this: developers seem to not respect careers paths past the immediate creation of code, while management — be it project, or product, or even executive — usually resort to this declaration, as some form of critique on effort, resourcefulness and most likely, timeliness of delivery.
I’ve never received a compliment for making something simple seem complex. I’ve never had a conversation that began “Hey John, I think you’re great, as evidenced by the fact that I never know how the $@%# to begin attempting to use your code!” That would, of course, be absurd. Yet, for the longest time, I rarely thought about my fellow engineers when designing my solutions that they would, inevitably, one day maintain — or refactor.
It’s extremely difficult to find talented software engineers — especially in today’s market. I have my own theories as to why that is, ranging from the fact that software engineering is still a relatively young profession, all the way to deeper philosophies about how competent people are at their jobs, in general. What’s more, companies that lack talented engineers typically have a very difficult time finding talented engineers, and this is not coincidental.
Well, we’re all writing services now. This is a service, that is a service. We need a persistence service, a web service, service generating service, and services that service other services. Please stop. Put the word, “service” down and think about what you’re doing and saying. Everything you’re writing ain’t a service. “Service” ain’t the new term we’re using for “code”. You’re not always writing a “service”, sometimes you’re just writing a class. When you’re writing a service, it’s got a higher purpose and meaning that your average piece of software. It’s got to service something, or someone.
Hand me your credit card, you idiot. No, when things start running slowly don’t just blame the hardware. Yes, hardware can affect performance, but if you just did a release, and at the same time you started to see a spike in system resources without a spike in load, the problem is in your software. “But hardware is cheap! We can just buy more hardware!” or the more contemporary, “Cloud instances are cheap! We can just spin up more instances!” You twit.
Maintenance sucks. I don’t care who you are, or what you’re into, maintaining legacy code sucks. You spend your days getting asked to fix other people’s bugs in a labyrinth of crappy code that was clearly written by angry monkeys who could inexplicably hurl feces at a keyboard in a manner that would yield compiling code. In fact, it’s not called “maintenance” if the code is good — that’s just normal development. If you’re on maintenance, then you have done something very wrong at some point in your life — or you’re just starting out. Much like anyone new to anything, you’ve got to pay your dues. In software development, that’s working maintenance.
Would quality be better if there were no Quality Assurance people? Let’s face it, most engineers abuse QA. They write a bunch of crap they’ve never tested, declare it’s done to much fanfare, and throw it over the wall to QA while they sit back knowing it’s going to take QA at least a day or two to start writing up bugs – more than enough time to get in some solid games of foosball. Furthermore, once they have *said* they are done, the pressure is off of them, and on QA to find them, at which point the engineer can sit back and pick and choose what they feel like fixing under the guise of “triage”.