Adam Caudill

Software developer & architect, security researcher, technical elitist.

DEFCAD & Freedom of Information

Freedom of speech is, in my opinion, the single most important and inalienable right that humanity has. This isn’t a privilege granted by a government, it’s an innate right of humanity. It’s a right that is essential to the preservation of many other freedoms - without it, there is no freedom, there is no liberty.

Earlier today a friend posted something on Twitter, I started to retweet it, but after some thought decided I needed to say more than I could fit in 140 characters. I feel rather strongly about this one.

This is of course about the mess going on around DEFCAD which was told(PDF) to remove 3D models of the Liberator gun and various other gun parts.

The DEFCAD site currently has this message up:

DEFCAD files are being removed from public access at the request of the US Department of Defense Trade Controls.

Until further notice, the United States government claims control of the information.

From reading the letter from the Office of Defense Trade Compliance, a part of the State Department, the issue is not that DEFCAD is distributing the information - but who can access it. By it’s very nature, any thing put on the internet is accessible by anyone around the world - the restrictions that could, and I believe will, be imposed will make DEFCAD’s mission impossible.

The restrictions that ITAR imposes makes it impossible to make this kind of material available - even with IP restrictions, there’s no way to know who is accessing this information.

In this case, ITAR has restricted an inalienable right - the right to free speech. Cody Wilson, as the face and voice of DEFCAD, has been told to withhold information - his free speech denied. This isn’t about shipping illicit guns to war torn countries - this is about making information available to the public. Theses aren’t stolen plans from a defense contractor, or the design of a new military weapon - it’s about a man that decided to share his knowledge of something he created and a government that told him to stop.

I should point out, I’m no fan of guns - I don’t own one and probably never will. I have mixed feelings about the proliferation of guns, but I have no mixed feeling about freedom of speech or attempts to suppress it. So this isn’t about guns for me, this is about an attempt to suppress knowledge.

Very, very, few people around the world have access to the type of 3D printer necessary to make a copy of the Liberator - the first item on the ODTC’s list and the design that put DEFCAD on everybody’s radar. So there really isn’t much risk of people making one - and the design is quite primitive as guns go. The Liberator is simple; it has only 15 plastic parts and one nail, less advanced than the gun it was named for - the FP-45 Liberator which was dropped in various countries during WWII (with less concern than what the US Government shows for its namesake). Does anyone really think that other countries are so far behind the US that they don’t have something more advanced than this?

An engineering student, in any country, with access to a CNC machine (which is cheaper and easier to get than the 3D printer used to make the Liberator) can make a weapon far more effective, and far more advanced. From a technology perspective, it’s simply not a threat. One can only speculate about what has really motivated the government’s actions.

This attempt to suppress information has had its usual effect - it’s widely available and has been downloaded more that it ever would have been had the government said nothing. Ah, the Streisand effect, will people ever learn? While the Liberator design and the other files on DEFCAD are safe (for now at least), it’s certain that this will have a chilling effect on Wilson and anyone else researching this.

Perhaps it’s not the Liberator that they wanted to stop - but whatever design that was going to be next. In that, it’s likely they have succeeded.

Password Hashing: No Silver Bullets

In the dark days of the web, if a service hashed your password instead of storing it in plain text, they were doing good. As sites were hacked, and credentials stolen, a silver bullet emerged: always hash and salt passwords when storing them. Many, many services were built with this design - LivingSocial is a great example. SHA1 hashing with a 40 byte salt - once upon a time, that was considered reasonable protection. Today, you’ll be blasted by industry insiders and the media for that.

We all so often rush to criticize companies for not being on the cutting edge of security - but how often do any of us look at why? Do we ever ask who’s really at fault? Is the company at fault for not putting the money into updating systems, is it the engineers and architects for not knowing what they should do, or could it be something else - could the advice companies receive be part of the problem?

Looking at some recent breaches and the overall security position of the companies involved - it’s easy to see which companies get it and which don’t. There’s one recent example where I really believe the company and its staff were simply incompetent. From top to bottom, they showed little thought for security, privacy, or user trust (for obvious reasons, I’m not naming names). LivingSocial and some others - I’m less sure about, when those systems were designed they were likely following “best practices” - or at least what developers thought were best practices.

For a number of years many developers saw SHA1 + a strong salt as a silver bullet. They implemented and stopped thinking.

The problem is, attackers are advancing far faster than companies can adapt, and best practices that may be right for today will soon fail the “reasonable protection” test. Engineers and architects see a best practices document and build to that standard - if that document is even a couple years old, what they build is hopelessly dated and won’t provide the security the need.

Once that system is built - unless there’s a breach, what incentive is there to change it? At many companies it’s a constant fight to fund security related changes; when the change is to fix a system that’s only broken in the eyes of experts, and (to management) only matters if some other layer of security fails, you can expect a fight for that money.

Fixing things that aren’t obviously broken is hard. For those signing the checks, it’s not obvious that what was “best practice” a few years ago is today a disaster waiting to happen.

There are many problems here, best practices are dated and inaccurate, there isn’t enough focus on secure storage, and there needs to be more push for better, more actionable recommendations that stand the test of time. While there are no silver bullets, surely as an industry we can do better than this.

Secure Storage

If I can get everything I need to start brute-forceing passwords from a simple SQL injection attack against your users table - your users are at risk. It doesn’t matter if your using SHA1 or scrypt - one day you’ll be breached, and passwords will be exposed. It’s that simple.

The salt, the hash, or (probably better) both should be encrypted. It won’t help you in case your server is breached, but it’ll help if you’re hit with a SQL injection attack or other attacks that allow access to data, but not the system itself. As always, it’s not a silver bullet, but it adds protection in some situations, and anything you can do to make life harder for an attacker is a good thing.

This of course opens up the age old question of storing the encryption key, which I’ll leave as an exercise to the reader. This can be taken to the next level, encrypt the salt & hash with an HSM - provides the best security, but that’s rarely an option because of expense.

There is some debate over this in the industry, but I’m a big fan of defense in depth - don’t make things easier for attackers.

Useful Recommendations

Crypto is hard, often confusing, and too often there is conflicting information. Implementing hash + salt is easy (and stupid), PBKDF2 is fairly simple - but you have to answer a couple questions:

  • Hash Function - SHA1-HMAC is a common option thanks to wide compatibility, though using a member of the SHA2 family would be far more appropriate.
  • Iteration Count - There’s no real consensus on this, some say it should double every two years (so this year it would be 96,000), others say it should be tuned to your server.

Iteration count here is a real issue - if you blindly double every two years you risk server stability, on the other hand if you tune it to your server, you risk not keeping up with attackers. There’s no winning, no easy answer. When it comes to password brute-forcing, the advantage goes to the attacker - having faster hardware and more optimized software is common. It’s the job of the defender to constantly play catch-up.

With scrypt the options are even more complex, with too little obvious documentation to get developers going in the right direction.

Dynamic Adjustment

For example, the Ruby scrypt implementation makes it quite easy to dynamically adjust cost settings - though if you run it on dated hardware, you run the risk of giving the attackers even more of an advantage. For those tied to PBKDF2 for some reason (i.e. compatibility), it would be easy enough to come up with dynamic adjustment code based on either performance benchmarking or date (following the somewhat risky double every two years approach).

Designing for dynamic adjustment gives defenders the best chance of staying current without costly upgrades. Again, not a silver bullet, but gives defenders a fighting chance of keeping up.

The Future

What I’ve been looking for is a strongly opinionated library in the spirit of NaCl - secure by default, and hard to make insecure. For now, it’s clear that it should be based on scrypt - but ideally it could be easily updated in the future to use the winner of the Password Hashing Competition.

Here’s what I’d like to see:

  • Based on scrypt.
  • Easily updated to support new algorithms (with backwards compatibility).
  • Single cost setting, ideally based on approximate execution time.
  • Function output encrypted with AES256.
  • Reasonable minimum settings to prevent insecure usage.

Something like that could be handed to a developer with minimal instruction, and allow them to provide reasonable protection today and in the (near) future. There are times when giving people what they want (options, control, etc.) does more harm than good - give developers just what they need, no more, no less.

The WikiLeaks We Deserve

I’ve been a (fairly quiet) critic of WikiLeaks for a long time, the core of the mission I agree with - information should be free, and should be preserved - but the implementation is deeply flawed. But then, that’s not really news is it? Two and half years ago when I last wrote about WikiLeaks, I pointed out that Julian Assange was the organizations biggest problem. So what do we have today? A grand idea perverted into a political machine, and twisted by ego and delusions of grandeur.

More than two years ago I wrote this:

Assange has become a threat to WikiLeaks not because of the attacks against him, but because people see him as WikiLeaks.

Today, I would say that perception is correct - WikiLeaks has become an embodiment of Assange’s personality. In the public’s perception, the man and the organization have become one; his role as the singular public face of WikiLeaks has clouded the publics vision and diminished the potential of the organization. While the reality maybe different, if the public sees Julian Assange as WikiLeaks - it’s fate is tied to his.

Maybe this is what we deserve - but it’s not what we need.

The WikiLeaks We Need

What we need is an organization with many vocal advocates, but no singular leader. What we need is information to be free, and openly accessible to all - a resource for legitimate journalists, unopinionated and apolitical. It should have a governance structure that allows simple leadership changes as needed. We need a resource for repressed information - either secret information from oppressive regimes or by overzealous laws such as the DMCA. There’s a wealth of information that is being lost while WikiLeaks focuses on political goals and public relations.

What we need is larger than any one person, and thus not tied to their faults or failings - what we need is a mission that’s bigger than the people involved, bigger than the organization.

Here’s how I see it:

Governance: The structure should allow members of a governance board to easily come and go; no member should be critical to continued operation. Should a member be unable to serve or potentially become compromised, it’s critical that their position be transferred with as little organizational impact as possible. All votes, actions, and minutes of the board should be published; the organization should be fully transparent. The governance board should be responsible for certain operational & organizational decisions:

  • Addition of new members to the board
  • Addition of ‘official’ (i.e. trusted) mirrors
  • Approval of new content (or delegation to trusted persons)
  • Control of finances

Technology: Publishing should be distributed as possible, and use a lightweight front-end to minimize server load. A technology like Jekyll could be used to make as many of the pages static as possible (for security, performance, and cost reduction). Once a site update is released, it should be made available to a set of official mirrors to handle spikes in traffic and help mitigate the impact of a server going offline.

Content: As much legitimate information as possible should be published, but care must be taken to avoid publishing false or misleading information, or information that is clearly politically motivated. If there’s a public perception of political or ideological motivation, then trust is put at risk. Content that is potentially controversial or has potential legal ramifications should be put before a quorum of the governance board to vote on how to proceed.

Content should cover all legitimate repressed information - from highly technical system details so often attacked via the DMCA’s anti-circumvention clause, to the emails and memos that built WikiLeaks’ early reputation. A hyper-focus on any one topic or type of information causes other valuable information to be ignored, a mistake that must be avoided.

Finances: Financial reports should be regularly released to disclose how all donations are used. When money is involved, so is suspicion and mistrust - spending should be extremely conservative and fully documented.

Public Relations: While there should be many vocal advocates for both the organization and the mission, there should be no single person that is entrusted with control. The focus should be on the goal - the freedom and preservation of information - not the individuals involved, or even the organization itself; the organization is just to tool, a means to an end.

Will it happen?

What I propose is an organization without ego, without politics, with adherence to a single mission and complete disregard for all other influences. In other words, what I propose is nearly impossible. There are few that believe so strongly in the ideals of freedom and preservation of information that they could be selfless enough to serve on the board of such an organization without letting personal opinions and motivations get in the way.

What I propose would be a haven for whistle-blowers and and those seeking freedom of speech - and no less of a target than WikiLeaks, and perhaps moreso due to expanding the focus to all legitimate information, not just the sole topic that WikiLeaks has adopted in the last couple years.

Someday, maybe…

We live in a time where information is produced at a nearly incomprehensible rate, and yet far too much of it repressed - by bad laws and worse governments. Without those working tirelessly to make it freely available, it would remain unseen by most, and possibly lost forever. Work always needs to be done, and WikiLeaks as we now know it isn’t the answer.

1Password, PBKDF2, & Implementation Flaws

…or “Crypto Is Hard, Vol. 479”

Earlier today a tweet about a new feature for oclHashcat-plus started a truly interesting debate on Twitter over the implications. The new feature is the ability to crack 1Password keychain files - at an impressive 3 million passwords per second.

To achieve this speed, two optimizations were used - the first is in precomputing ipad and opad for SHA1-HMAC, this effectively cuts the number of SHA1 calls in half. This is clever, but not the part I’m interested in, it’s the second optimization that I find interesting.

1Password uses AES128, in CBC mode to encrypt data - and it’s the design of CBC that enables the next optimization. If you have the wrong IV when you decrypt in CBC mode it corrupts the first block - but only the first block, the rest will be fine. It’s this fact that seems to have been overlooked when 1Password was designed.

The final encryption key is derived from the user’s password via PBKDF2. They generate a 320bit key (so two blocks or four calls to SHA1 after the precompute optimization); of those 320 bits, 128 bits go into the encryption key, another 128 bits go to the IV. For 1Password to work, it needs both the key and the IV - but attackers don’t.

An attacker can look at later blocks to see if they are valid and ignore the first block, by ignoring the IV generation and the first encrypted block, the number of calls to SHA1 is again cut in half.

To their credit, Jeffrey Goldberg of AgileBits (the maker of 1Password) responded very quickly and provided details about what they are doing to address it, and joined in a very open discussion about the impact of how this was implemented. Depending on who you ask, this ‘flaw’ costs either 1 or 2 bits of entropy.

For the end user, it means that an attacker only needs to perform 50% of the SHA1 calls that the 1Password software needs (maybe only 25%, depending on how optimized the 1Password code is). When it comes to password cracking, that certainly seems less secure than what was intended. As flaws go it could be far worse, but it’s likely less secure than intended.

And finally, after all of this, we come to the point:

Crypto is hard, even for smart people.

It’s so easy to accidentally introduce weaknesses like this - everything looks right, but because of how they used PBKDF2 in combination with AES128-CBC, the end result is weaker than was intended. It’s specific, fairly obscure (to many at least), details of how these two algorithms work, and how they were used together that led to this flaw.

AgileBits has made a lot of good decisions, and in general at least, seems to know what they are doing. I certainly don’t mean to beat up on them, they’ve handled this very well and the issue is fairly small. I applaud them for discussing this so openly, it’s hard to do - I know that from personal experience.

This should serve as a great example of why any new crypto implementation needs to be expert reviewed to make sure that it’s right. Nobody gets it right every time, every code base needs to be reviewed, this could (but doesn’t have to) happen to you.

Linode: Another Breach Notification Gone Wrong

Last night I received an email from Linode about a possible breach and mandatory password reset that reminded me of another recent email, in some disturbing ways.

Dear Linode customer,

Linode administrators have discovered and blocked suspicious activity on the Linode network.

Not too long ago, I received a similar email from Evernote - not just in it’s text, but in the errors made.

Dear Evernote user,

Evernote’s Operations & Security team has discovered and blocked suspicious activity on the Evernote network that appears to have been a coordinated attempt to access secure areas of the Evernote Service.

When I received the email from Evernote, I quickly noticed that the links were to a different site - was it a thinly veiled phishing attempt? With all the links going to links.evernote.mkt5371.com it took a few minutes to work out that the email was actually legitimate. Instead of being able to quickly reset passwords, users were forced to try to figure out if the email was real or not.

When you are sending out a breach notification, it’s critical that users are able to verify, as quickly as possible, that an email is legitimate, and the the call to action really is necessary. Linode, unfortunately, made a similar mistake.

The first thing that jumps out is that it’s from e2ma.net - not Linode directly, next is that the first link in the email points to e2.ma/message/.... These are immediate red flags, the focus instantly switches from taking necessary action to secure a user’s account to trying to figure out if they are being attacked. From comments on their blog post, I’m not the only one that was disturbed by how the email was sent.

Thankfully, they didn’t include a link to the password reset page, and they even told users not to click reset links in unsolicited emails - so it’s not all bad. Users shouldn’t be spending time trying to figure out if a notice is real instead of taking steps to secure their accounts. In situations where there may be a significant security issue, service providers should take every effort to minimize confusion - provide clear information, and make the source as clear as possible.