A Year in Review

Today marks a full year working as a professional application security consultant.  Prior to this last year, I was a senior software engineer who grew up playing pranks online.  I tried to build reasonably secure applications, to the best of my ability, with as much time as I could afford.  Now I try to break them, basically under the same restraints.  I’ve faced the problem of secure applications from both sides of the coin.

All and all, I am humbled by the opportunities I have been blessed with over this year.  But I’ve done more than I should have, and less than I was capable.  I’ve spoken locally numerous times through OWASP Phoenix which I help co-run.  I’ve spoken nationally at conferences such as BsidesDFW, BSidesAustin, AppSecDC, and was recently accepted to talk at Defcon 19.  I’ve competed in 5 CTF competitions, and have released my first security advisory.  I was awarded MVP status from Microsoft for my work with Windows Azure community.  I’ve tested a wide range of clients, and was exposed to new technologies that challenged me to grow.  I’ve traveled the country and met many talented people in security and I’ve somehow even kept blogging.  However, the single greatest blessing through this time, has been the birth of my second son.

I’ve been crazy busy and I’m not sure it is a good thing.  The job of a penetration tester has proven to be incredibly taxing.  It’s hard on the brain, it’s hard emotionally, and it generally requires long hours.  We are asked to do the difficult task of looking at an application with new eyes– to learn, understand, and expose it’s weaknesses… over, and over, and over again.  These challenges can lead to self doubt (why didn’t I find this, why isn’t this working? what don’t I know? what did I miss?), self disillusion (I am so awesome I found x vulnerability *and likely missed 15 other vulns), bias, and quite frankly– vices.  This job can be especially hard to separate away from during off hours.

But I do not intend the above to come across as a lament, to the contrary I consider these opportunities to grow as joy.  Each day is a chance face my demons, to learn of my frailties, and to be disciplined.  In budo culture, there is a saying, “True victory is over your self.”  We have to learn to live with our selves and how we interact with the world.  This job has become a part of my martial and spiritual practice– and I am a better person for it.

As I look at the industry itself, I see lots of things– both good and bad.  I don’t intend the below to pick on anyone– just observations of what I see.

Why the transfer of decisions from those with personal experience and a stake in the outcome to those with neither can be expected to lead to better decisions is a question seldom asked, much less answered. – Intellectuals and Society

In my experience, analysts and those in positions of management often seem to lack the breadth of experience to validate or support their claims in security.  In contrast, those who do have that experience often seem to lack the language and or desire to show others what they know in ways that are cohesive.  This could be because they’ve been dismissed as merely testers, or perhaps even entirely un-intentional.

Many of those operating with such knowledge have not fully articulated such knowledge even to themselves, and so can hardly transmit it to others, even if they might wish to. – Intellectuals and Society

Either way, there is a very obvious and deep communication rift between the two groups, and no one is benefiting from it.  Some individuals are making improvements in this area with data and metrics– but often the metrics I’ve seen are pure and utter rubbish (oh yah, I said rubbish again).  I’ve heard claims that have no bearing in reality, and I’ve seen data and conclusions drawn from it that are just as bad.  The only stake some seem have in the game is the one that drives their paychecks.

That isn’t to say that paychecks & agendas are bad on their own, but it’s a dangerous thing when I had thought (perhaps naively) that security and it’s improvement was the real concern.  Tactics and strategies that aren’t proven, that have no basis of proof, and aren’t technically sound need to be avoided.  It’s one thing to have theories you intend to prove– it’s another thing to sell them as solutions.  We need better feedback loops that are reasonably unencumbered from bias to make accurate decisions against.

Systemic processes are essentially trial and error processes, with repeated or continuous– and consequential– feedback from those involved in these processes.  By contrast, political and legal processes are processes in which initial decisions are harder to change, whether because of the high cost to political careers of admitting a mistake or– in the law– the legal precedents that are set.  – Intellectuals and Society

It’s hard to let go of an idea (or even be able to see it objectively) when the echo chamber lavishes over your claims.  But sharing uninformed ideals, based on agenda and not proof, to an even less informed public is appalling.  It is disheartening to see that people remain praised as experts, despite being proven wrong in their claims.  So called thought leaders are held completely unaccountable for their suggestions and actions; which are often in conflict with each other.  Personalities are king, not ideas.  Opposing ideas are not good for the ego.

Finally (as far as the not so ideal stuff goes) I think that what many are telling developers to do as “solutions” is misguided.  We sell them on brushing their teeth and eating their vegetables to get strong– but none of that prepares them for the reality of being punched in the face.  They are ill-prepared for the world they release software in.  I’ve even seen security professionals boast about how even if you followed all the best practices we recommend– they’d still win.  So what’s the motivation to even bother to protect your system?  There needs to be a better strategy/message than this– otherwise defense is a meaningless waste of time.  And regretfully, other measures like firewalls, WAFs, and AV have many of the same dangers as they often contain bypasses or security problems of their own.  A false sense of security is almost as bad as no security at all.

It is my genuine prayer that I never worry more about the security industry more than I do about security itself.  I am not perfect, nor free from the trappings of being human.  If you believe that I am being hypocritical about any of the above, you are likely right.

Any who.

On the good side I see that there are people who are genuinely chasing growth.  People who have really impressed me this year are: Chris Nickerson, Chris Gates, HD Moore, Andre Gironda,  Attack Research, Thomas Ptacek, John Strand, Adam Shostack and Alex Hutton.  These guys each demonstrate a palpable understanding of technical reality, and can back up their claims.  They aren’t “thought leaders” but real leaders who offer suggestions based on real day to day work and experience.  They challenge me to grow.  Another honorable mention is 451Wendy– who, despite being an analyst, has amazing insight and genuine passion for her work. 😉

Although I didn’t mention my co-workers in that list, I earnestly feel lucky to be a part of such a talented team.  David Bryne and Kevin Stadmeyer have especially been a huge source of personal growth in how I approach my job professionally and technically.  I wouldn’t mention any of this if I didn’t believe it– but my boss put it perfectly  “quality testers who ‘get it’ are worth more than any thought leader in this industry.”  The people who really make a difference do it one company at a time.

Finally– I am flat out floored by the community around this industry.  People like Jack Daniels, Diami03, Jsokoly and others have such huge hearts and passion for getting people together.  There is little like this in the software engineering space– and we really should be proud of the small local conferences like the BSides movement.  Prior to my first Defcon, I’d never been to a conference where the speakers weren’t merely there for presentations and then gone.  Yet my experience to date has revolved more around openness and availability. Sharing and taking part in this community has been amazing.  If there is any chance for change in this industry– I suspect it will stem from this.

So what’s next?  I intend to finish out this year with some talks– then I need to step back for a while.  I want to spend more time with my family, and I have hordes of books and studying that I want to do.  I am reminded of this:

Finish your outdoor work and get your fields ready; after that, build your house. – Proverbs 24:27

I am likely to invest a great deal more time working on my personal practices and training for this job– tempered of course with lots of real work.  Other things will need to fall where they may.  This includes some community involvements and likely twitter for a while.  I hope this next year I am quieter, so I can hear and learn more.

Talk is cheap– never accept it.  We must have great faith and great doubt if we are going to be successful in this industry.


Post a comment or leave a trackback: Trackback URL.


  • wurkin  On July 14, 2011 at 1:54 pm

    Great read. Thank you. You are saying most Security Metrics are rubbish. Can you please provide any guidance that will improve that field instead of complaining about it? (No offense intended)

    • pinvoke  On July 14, 2011 at 2:58 pm

      No. But others in that area, specifically Alex Hutton and Adam Shostack, are doing lots of great work in this area. I am not in the position to affect change in this area, nor think myself sufficiently qualified to do so. I do not need to be an expert, however, to note that the data we collect as “proofs” does not often hold up under objective review. Conclusions drawn from data are also often merely bias and prejudice.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: