Monday, April 14, 2008

Research Obfuscated

Update 07-Sept-2008: Four months later after writing his open letter to me (see below), Christofer Hoff experienced on his own difficult it is for one to control the press, making sure it correctly reports what you say. In this blog entry he describes how he was terribly misquoted by a report after his Black Hat presentation and he also explicitly admits that "[I] was essentially correct in [my] assertion during our last debate that you cannot control the press, despite best efforts." and that "[he] humbly submit[s] to [me] on that point." :)

This article has been brought to my attention recently. It’s an “Open Letter to Joanna Rutkowska”, by Christofer Hoff over at the “Rational Survivability” blog. I decided to spend time reading and answering this piece as 1) reported the blog’s authority as above 100 which suggests it has a reasonable number of readers, and also 2) because I believe this is a good example of the social engineering techniques used by my opponents and I couldn’t refrain myself from not commenting about this. Besides I felt a bit flattered that some individual decided to write an “Open Letter” to me, sort of like if I was a prime minister or some other important person ;)

Let me now analyze the letter, point by point:
  1. Fire rules! The first thing that Hoff accuses me of in his letter is myself being an irresponsible individual, not caring about safety of my audience (not a joke!):
    “As the room filled to over capacity before your talk began, you were upset and couldn't seem to understand why the conference organizers would not let people spill over from seats and sit on the floor and in the aisles to hear you speak. The fact that fire and safety codes prohibit packing a room beyond capacity was something you attributed to people being "...crazy in America." Go figure.”
    Dear Christofer, if you only read my recent blog post about this very specific incident, read thoroughly shall I say, you would notice this paragraph undoubtedly:
    “Interestingly it was perfectly ok for the additional people to stay in the room, provided they arranged for additional chairs for themselves. In other words it was fine for people to sit and block the main aisle, provided they sit on chairs, but they couldn’t stay and sit on the same aisle without having a chair (maybe a "certificated" chair also), as that would be against the fire regulations!”
    Conclusion: I was not so much picking upon the fire regulations that forced people to leave the room, but rather on the idiotic rule, that allowed those same people to stay in this very same room, provided they also had additional chairs with them.

  2. Type I vs. Type II hypervisors confusion. Hoff then switches to the actual content of the presentation and writes this:
    “When I spoke to you at the end of your presentation and made sure that I understood correctly that you were referring specifically to type-2 hosted virtualization on specific Intel and AMD chipsets, you conceded that this was the case.”
    This simply is an incorrect statement! On the contrary, when describing the security implications of nested virtualization (which was the actual new thing I was presenting at the RSA), I explicitly gave an example of how this could be used to compromise type I hypervisors. Kindly refer to slides 85-90 of my presentation that can be downloaded here.

    I said that the code we posted on indeed targets type II hypervisors and the only reason for that being that it has been built on top of our New Blue Pill code that was designed as a Windows kernel driver.

  3. Shit not giving. Mr. Hoff goes even further:
    “When I attempted to suggest that while really interesting and intriguing, your presentation was not only confusing to many people but also excluded somewhere north of 80% of how most adopters have deployed virtualization (type-1 "bare-metal" vs. type-2 hosted) as well as excluding the market-leading virtualization platform, your response (and I quote directly) was: I don't give a shit, I'm a researcher.
    Now that was a hard blow! I understand that the usage of such a slang expression by an Eastern European female during an informal conversation with a native speaker must have made an impression on him! However, I couldn’t give such an answer to this very question, simply because of the reasons given in point #2 (see above).

    If I remember correctly, I indeed used this very American expression to answer somebody’s concern (undoubtedly our Christofer Hoff’s) that most of the type I hypervisors out there are based on monolithic hypervisor architecture, and not on the micro-hypervisor architecture (and that I should not try to convince people to switch to micro-hypervisor architecture). In that context it makes it more logical for me to use the “I’m a researcher” as an excuse for not caring so much that most people use monolithic based hypervisors. Obviously, the usage of micro-hypervisors would allow to better secure the whole VMM infrastructure. And I also said, that I don’t care what people are using today, because I try to help to build a product that would be secure in the future (Phoenix’s HyperCore).

  4. No obfuscation postulate. Hoff then comes up with some postulates that:
    “[I], as a researcher who is also actively courting publicity for commercial gain and speaking at conferences like RSA which are less technical and more "executive" in nature, you have a responsibility to clarify and not obfuscate (intentionally or otherwise) the facts surrounding your research.”
    This postulate is cleverly constructed because it also contains an embedded accusation of me being a commercially motivated researcher. Well, I never tried to hide that fact, and the reason for this is very simple: I consider security research as my job, and one of the primary goals of any job is to… bring commercial gain to the individual doing the job.

    Second, I really don’t understand what Hoff means by asking me to not obfuscate my research?! Maybe he was just disappointed that the presentation was too technical for an average CISSP to understand it? But, well, this presentation was classified as “Advanced Technical”, which was displayed in the conference program. I still did my best so that, say 70% of the material, was understandable to an average IT people, but, come on, there always must be some deep technical meat in any non-keynote-presentation, at least this is my idea for how a conference should look like.

  5. Commercially motivated. Hoff accuses me of presenting commercial product, i.e. the Phoenix’s HyperSpace, during my speech:
    “No less than five times during your presentation, you highlighted marketing material in the form of graphics from Phoenix, positioned their upcoming products and announced/credited both Phoenix and AMD as funding your research.”
    Well, let me tell you this – this was one of the main reasons why I decided to speak at the RSA – just to announce this very product that I try to help to secure. Why would that be wrong?

    BTW, I have no idea how Mr. Hoff concluded that AMD was founding my research. I never said that, nor did I have it in my slides. Needles to say, AMD has not been founding my research. NOTE: interestingly I consider this particular mistake by Hoff to be accidental – at least I don’t see how this could be connected to any PR campaign, in contrast to all the other incorrectness he made use of.

  6. Independence. Hoff, for some reason, apparently known only to him, tries to argue that I’m not an “independent researcher”:
    “I think it's only fair to point out that given your performance, you're not only an "independent researcher" but more so an "independent contractor." Using the "I'm a researcher" excuse doesn't cut it.”
    “I know it's subtle and lots of folks are funded by third parties, but they also do a much better job of drawing the line than you do.”
    Well, I found this one to be particularly amusing, as, for at least several years now, I have not claimed I have been an independent researcher.

  7. Final hit. You might have been wondering by now – why this gentleman, nah, I think “the guy” would fit better here, so why the guy decided to spent so much time to write all those points, all those quasi-arguments and why he made so many “mistakes”? Well he seems to give an answer right in this paragraph:
    “I care very much that your research as presented to the press and at conferences like RSA isn't only built to be understood by highly skilled technicians or researchers because the continued thrashing that they generate without recourse is doing more harm than good, quite frankly.”
    Aha, now all is clear. May I ask then, which virtualization vendor you write PR for? ;)

So, then Hoff quotes the Forbes article that was written after my presentation and accuses me that the article (written by some Forbes reporter) was too sensationalist. I definitely agree the article was very sensationalist (but correct) and when I saw the article I even got angry and even wanted to write a blog about it (but as the article was actually correct, I had no good arguments to use against it). And you know why I was so angry? Because I actually spent over 40 minutes with this very Forbes reporter in the RSA’s speaker’s lounge just after my speech, I spent that time on clarifying to that guy what my presentation was about and what it was not about and what was the main message of the presentation. Still, the reporter had his own vision of how to write about it (i.e. make it into a sensation) and I hardly, as it turned out, could do anything about it…

So, what was the main massage of my presentation? Interestingly Mr. Hoff forgot to mention that… Let me then remind it here (a curious reader might want to have a look at the the slide #96 in my presentation):
  • Virtualization technology could be used to improve security on desktop systems

  • However there are non-trivial challenges in making this all working well...

  • ... and not to introduce security problems instead...

Additionally, the message I was trying to pass during the whole presentation was:
“Keep hypervisors simple, do not put drivers there, as otherwise we would get to the same point where we are with current OSes these days, i.e. no kernel security at all!”
Now I wonder, maybe Christofer Hoff doesn’t do PR for any VMM vendor, maybe he just didn’t listen carefully to my presentation. Maybe he’s just one of those many guys who always know in advance what they want to hear and selectively pick up only those facts that match their state of mind? Otherwise, why would he not realize that my presentation was actually a pro-virtualization one and needed no (false) counter-arguments?


Anonymous said...

I do not know if you tell the truth nor do I know that Hoff`s tell the truth. In medio stat veritas nevertheless, and it's only my humble opinion, you should not give a shit about it, you _are_ researcher so do your research and olej to sikiem prostym ;).

Steven J. Murdoch said...

I can sympathize about your points on media sensationalism. When my research has made it into the press, especially the banking security work, it is almost always exaggerated. If you find out how to fix this, tell me how :-)

I do work hard to get the accurate message across, for example by producing press releases by talking to journalists, as well as FAQ pages for journalists. These help, but the end result is seldom as balanced as I would like.

However, I think the media coverage of security research is clearly beneficial. Managers don't make important business decisions based solely on media reports. What happens, in my experience, is that media coverage causes the manager to call their security staff.

Initially this makes corporate security staff unhappy at us. They'll be asked uncomfortable questions about why the problem exists and why the manager wasn't told about it (even when they were). When we control the media schedule, we try to help by giving the vendors advance notice and privately distributing technical details within the industry. It's still not nice to be called back from your holiday though.

But once the dust has settled, and people have calmed down, we can see the benefits of this approach. We don't tell the media a problem exists unless we think there is one. Security staff tell us that they were previously ignored by management but now have a mandate and budget to solve the problems we demonstrated.

It sometimes takes years after the initial media coverage, but I can point to concrete improvements in security policy and technology triggered by our media coverage. Coverage might be sensationalist, but that's what the mainstream media does. The alternative is to produce vapid statements which offend and inform nobody, which I don't think is the right approach.

Joanna Rutkowska said...

Hi Steven, thanks for dropping by! I think that maybe I should consider doing the formal press releases + FAQs too. Good luck with your research!

Anonymous said...


Thanks for taking the time to respond to my "letter."

Blog comments are like ping pong, so if you care to spend a few cycles reading my response on my blog, I hope that it clarifies a few things.

One day I hope to ask you the other questions I wanted to, but I'm not holding my breath ;)


I M Tired said...

Ever since your Blue pill research and pieces I have been a great fan of yours and envy your technical expertise.

It seems to me that Hoff is trying to goad you and pick at, what he sees, as incorrect or different to the way he thinks.
Regarding the media they always misquote and will always write what they remember which is precious little. I have found myself misquoted many times no matter how much I try to clarify.

Anyway just wanted to let you know that I thouroughly enhoy your articles and blog and would one day like to discuss with you clientless endpoiint control and management.

Anonymous said...

somehow.. after read all these bitter posts back and forth, I had feeling that some male researchers in IT security are very sensitive about being slammed by a female researcher in public, so they got all these unnecessary rebound. To the point that some of their posts are not purely about technical discussion anymore. Joanna, perhaps do not have a polished attitude like what a politician would behave, which is expected in this culture, accidentally triggered a little resentment. Shouldn't be that a big deal...unless you all enjoy it ;)

Peter Joseph said...

Well said Joanna! What's that guys problem? I think he didn't listen and/or didn't understand.

Realistically, if ones job involves running Nessus once a day your research is going to be hard to understand. ;)

Also how can you be held responsible for how reporters misunderstand/misrepresent what your saying?

He sounds like he's in the misguided "don't reveal exploits/new research as it gives the 'bad guys' an easier time" camp.

Michael Dundas said...

Hi Joanna

I took your / Alex's course last year at Blackhat 2007. I'll be there this year (although taking other courses). It is probably the best course I have ever taken in the security arena.

Yes your stuff is technical, but that is a good thing. Too much security information is watered down these days for the media or a manager that thinks they understand security but do not have the technical understanding to actually 'understand' it.

Last year in the course you used AMD / Vista as your test bed but make it very clear that Intel and other operating systems would be susceptible to the same types of attacks. If you actually understand the attack, this makes complete sense -- again I think it is just lack of understanding of some.

Keep up the good work.

Anonymous said...

Yes, I'm pretty late, but apropos "media sensationalism": nevertheless it's amusing stuff to read and see how even you people (and probably everyone else in this department of space striving for acceptance) suffer from an illness called "image complex". Far more amusing is how many people pat you on the back; obviously those who envy you for your most prominent capabilities.

Sigh. Well, I do not care since that fulfils exactly my expectation (as with quite closely everyone dealing with this and other matters, not to say mankind - oh, what a mess).

I'd just prefer you to keep up to facts, showing what goes wrong. I do not care for that "meta-level" stuff of most people in even outer parts of the headlight sounding something like "ping, pong, blah, bubblebubble, spam, spam, spam, spam, bacon and spam". Boring stuff to read, to be quite frankly.

But nevertheless I'd gladly appreciate and thank you for opening my eyes on virtualization and the threats associated with it nowadays. I decided to drop the project I was working on.

Thank you, with all my heart.

No offence meant, really. I'm just a locum of mankind who prefers keeping things down a bit.

Anonymous said...

"When my research has made it into the press, especially the banking security work, it is almost always exaggerated. If you find out how to fix this, tell me how :-)"
Well, avoiding oral interviews in favor of written interviews helps. After being screwed enough, I copied RMS's tactic: I set ground rules. I did this successfully with Bloomberg. Example: I asked them to verify that a breach had begun in 2005, contrary to reporting in other media that it had begun later, provided the proof, and said I'd answer their questions if they addressed the error. They agreed, and the result was a much better article.

Anonymous said...

P.S. Kudos to Steven Murdoch and his team, and to Joanna Rutkowska, on their excellent research.