Adam Caudill's Blog » Web Development

Why Cringely is wrong about Java

Adam Caudill — Sat, 15 Oct 2011 19:47:52 +0000

A couple of days ago I was sent a link to Robert Cringely’s latest treatise: The second coming of Java – and to say I disagreed was a bit of an understatement. To me, it represents a fundamental flaw in his perception of developers, and more importantly the economics of software development.

The key to Cringely’s argument comes down to this:

When SSDs gain enough capacity there will be a shift from the Ruby world back to the Java world. Not for prototyping, because, well, it’s prototyping. But simply because the statement “Ruby is incredibly slow but I don’t care because my database is slower” will no longer be true.

What he’s missing here is the real reason people use frameworks like Rails; it’s not about it being Ruby, or being the latest cool thing – it’s about developer productivity. That’s it, and that’s all there is to it – Rails allows a developer to do more in less time. That’s one of the key reasons so many Java web developers jumped ship (though I can think of a few others), and what pushed Microsoft to invest so heavily in their MVC framework.

I could fully rehash the argument, but in what I consider to be one of Jeff Atwood’s best articles, Hardware is Cheap, Programmers are Expensive, he covers a key point to my argument – developer time is vastly more expensive than hardware. Atwood’s take on the issue is clear:

Clearly, hardware is cheap, and programmers are expensive. Whenever you’re provided an opportunity to leverage that imbalance, it would be incredibly foolish not to.

When there’s a choice between developer productivity, and spending money on hardware – the conclusion should be the same. It’s much cheaper to throw more hardware at a slower framework than it is to invest more developer time in a faster framework. For any non-trivial application, throwing more front-end servers at it will always be cheaper than slowing the development process down with a non-productivity-centric toolkit.

It’s simple economics; server hardware is getting faster and cheaper, developer time is only getting more expensive.

Pen-Testing Silverlight+RIA with SoapUI

Adam Caudill — Wed, 06 Jul 2011 21:39:37 +0000

I was recently given the task of ensuring that a Silverlight+RIA application that could contain private information was secure for deployment to a public web site. So I started searching for automated pen-testing tools that could work against Microsoft’s Binary SOAP protocol (msbin1, a.k.a “application/soap+msbin1“) and found only disappointment. For various reasons, it’s significantly more complex to pen-test a application using msbin1 than traditional SOAP + WSDL.

To properly test the services, I had to make a compromise: temporarily modify the application to expose a SOAP endpoint. While this changes the state of the application and thus reduces the validity of the tests, it does provide a reasonable way of testing the web services to ensure that they are behaving as intended.

The recently released SoapUI Pro 4 adds new security testing tools that makes this a viable (and attractive option). To get this working, there are a few small changes that need to be made to the solution:

First, you’ll need to add a reference to “Microsoft.ServiceModel.DomainServices.Hosting.EndPoints” which is part of the RIA Services Toolkit; this allows you to expose different End Points for the service such as SOAP and OData.

Next, you’ll want to add the following configSections entry to your Web.config:

...

Finally, to expose the SOAP end point:


 ...
 
  ...
  
   
    
   
  
  ...

Finally, just follow the instructions for SoapUI to setup your tests, and you can feel (just a little) more confident in your application. Passing with flying colors obviously doesn’t mean your application is bulletproof, but it helps to confirm that web service code is solid.

Now, while this does provide some insight into your application and should help find common issues, it’s not a replacement for a professional assessment by a qualified auditor. If you are handling credit cards or other highly targeted information, please consult a security specialist before a public deployment.

bbPress 2.0 (beta-1) released!

Adam Caudill — Sat, 21 May 2011 18:25:18 +0000

This morning, the latest version of my favorite forum software, bbPress, was released. While this is a beta and not a final release, it’s still a major event for the project: this release is in the form of a WordPress plugin, instead of a stand-alone application as it’s been in the past. In December 2009 I wrote about this decision – and at the time I was rather disappointed with the change in direction; but now my views have changed. I’m also excited to see that Matt and his company, Automattic have decided to invest the time and money needed to make this happen; there for a while it looked like bbPress really was dying.

Thanks to services like StackExchange and its numerous clones, it seems the days of standalone forums are numbered; when combined with the increasing strength of WordPress as a full-blown CMS, this move actually makes a lot of sense.

I’m excited both as a user and developer – this opens many new doors from both perspectives, and will certainly increase adoption of the software, as integration with the current version is a bit tricky.

Congratulations to the entire team on this release; it really is a major milestone for the project!

Errors on ‘gem install mysql2′

Adam Caudill — Mon, 16 May 2011 05:29:19 +0000

On my fresh Ubuntu 11.04 box running Ruby 1.9.2 instead of the standard Ruby 1.8, I ran into some undocumented errors while installing the mysql2 gem. Here’s what I was seeing:

$ gem install mysql2
Building native extensions.  This could take a while...
ERROR:  Error installing mysql2:
	ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.9.1 extconf.rb
          :29:in `require':
          no such file to load -- mkmf (LoadError)
	from :29:in `require'
	from extconf.rb:2:in `'

Gem files will remain installed in
  /usr/lib/ruby/gems/1.9.1/gems/mysql2-0.3.2 for inspection.
Results logged to
  /usr/lib/ruby/gems/1.9.1/gems/mysql2-0.3.2/ext/mysql2/gem_make.out

This obviously isn’t all that helpful, nor did I find anything all that useful on Google – thankfully the issue is easy to solve. All that’s needed is to ensure that the following packages are installed:

sudo apt-get install libmysqlclient-dev libmysql-ruby1.9 ruby1.9.1-dev

Then you can try installing the gem, it should work, hopefully

You can’t fix stupid…

Adam Caudill — Tue, 10 May 2011 06:06:05 +0000

For those outside of the IT field, developers are looked at as miracle workers – through us, business leaders think anything is possible (and they often see no reason why we can’t work our latest miracle by the next morning). In reality though, we do work miracles; we save companies vast amounts of money every year through increased worker efficiency and automation, we enable new lines of business that wouldn’t be possible otherwise, and reduce energy costs because we keep the office lights turned off. Well, that’s more or less how they see us.

But for all of the wonders we are responsible for, there is one thing we can’t do (no matter what amazing powers some executives think we have to make them look better or earn them more bonuses):

You can’t fix stupid.

I’ve often described development as being a professional problem solver, and we are often tasked with rather challenging problems to solve. Sometimes the problems are purely technical – making something new possible that previously was impracticable or even impossible, sometimes it’s all about efficiency, other times it’s about image and controlling how people see a company. When the problem is the user though, you know you’re in for a rough day.

I was recently given such a task – the users weren’t listening to their supervisors and they wanted the software to force these users to do whatever it was that management told them they should be doing. I was given less than a week to find ways to make people that don’t want to work, work.

Basically, users fall into three simple categories:

Power Users – these users understand software, and require little, if any instruction – more than anything, you give these users a tool and stay out of their way.
Average Users – odds are, your mother or father falls into this category. They understand enough to get by, and with a little instruction they should have no trouble.
Idiots* – odds are, you work with one of these users. You lead them by they hand, and show them exactly what to do – just in time for the boss to walk by and praise them for doing a good job (and 10 minutes later you find them playing in traffic, somehow defying Darwin in the process).

For users of the last category, there’s just not much you can do.

I always do my due diligence while building software; doing all I can to make it simple to use, flexible, and forgiving of user error. I always use extensive data validation, carefully worded instructions and dialogs, and do my best to follow the various best-practice guides for UI and UX; yet for all this effort and design – I can’t write software that thinks for people or makes judgement calls based on business rules that only they know (probably because they make it up as they go).

No matter how helpful or intelligent an application is, or how idiot-proof you think you’ve made it, reality is that you simply can’t fix stupid – you can’t take an incompetent person that refuses to think for themselves and turn them to into a great, productive asset. After years in this industry (which has made me just a little cynical [in the way that Sol only seems little when compared to Betelgeuse]), I’ve come to understand something rather disturbing: idiots keep getting better.

Somewhere, right now, idiots are working to build even better idiots – and that’s a really scary thought.

We can make a user more efficient by automating tasks, providing better information, or helping to manage their workload – what we can’t do is make them smarter, make them think through their actions, or force them to do what their managers tell them. Yet we are, at least on occasion, asked to fix this problem.

Despite our best efforts as professionals and passionate developers; if a user won’t think – we just can’t fix it.

* – I define an idiot about the same way I do someone that’s lazy; they have no medical issues or legitimate handicap. They just don’t want to think or work (probably both). Those that are handicap or have learning or medical issues are a very different story and not the target of this article; I donated time and services to charities that served the disabled for a number of years, I highly recommend that all developers do it – it’s a very rewarding experience to see your work make somebody’s life better and it teaches you quite a bit about how people interact with technology.

bbPress & WordPress 3.1 (Registration Issues)

Adam Caudill — Sun, 06 Mar 2011 08:39:49 +0000

While building a new website for a small ISV, I was asked to integrate a forum – should be simple I thought, I’ll just use WordPress as a CMS and bbPress. Simple? Not quite.

To make things easier, I used Platform Pro by Pagelines (a great platform IMHO) and so used their “deep integration” instructions (while they are intended for Platform Pro users, the steps are virtually identical for all other setups). While the setup went quite smoothly, it wasn’t until it came time to test user registrations that the issue came up: when a new user tried to register they would receive an error stating that “email is required,” even if they had entered a valid email address. This effectively made it impossible for users to register, and thus made the forum useless.

After a bit of research, it looks like there is a conflict between newer versions of WordPress (3.0+) and bbPress 1.0.2 (I’m not sure if older versions have the same issue, but I would assume so). The fix for this is actually quite simple: install a plugin. The plugin itself is quite simple – containing only around a half-dozen executable lines of code, and works around the issue with a simple hack.

Unfortunately, the plugin description is rather vague and required a fair bit of research to identify as the fix, and the official documentation makes no reference to the issue or that this plugin can be used as a workaround.

Overall, a simple workaround but a major issue if you don’t notice that it’s broke (as is quite easy to do).

Rails 3 & Dreamhost PS

Adam Caudill — Fri, 28 Jan 2011 07:47:01 +0000

I recently had an idea for a small web application, and seeing as I’ve not spent as much time as I’ve wanted to using Rails – I opted to build it the latest version of Rails. A decision that caused far more grief than I expected.

If you are using Dreamhost’s PS offering (a managed VPS for those that don’t know), the seemingly simple task of getting a Rails 3 application up and running is actually quite complex. The root cause of this is that Dreamhost’s OS image is based on Debian etch, which was released in April 2009 and has since been replaced; which means etch has become fairly outdated.

Here’s the process I used, and so far it seems to be working quite well:

Domain Setup:

When adding your domain to the Dreamhost panel, you’ll want to enable Passenger.

Once your application is uploaded to the server, you’ll be greeted with a particularly unhelpful error message (something like “uninitialized constant Bundler“) from Passenger (or perhaps just a 500 error page).

Server Updates:

This is where the work starts, and gets somewhat ugly. As a warning, it’s quite possible that you could damage your configuration doing this; though thankfully you can restore your server to a working state within a few minutes from the Dreamhost panel should something go wrong. You’ll also need to have an “admin user” for this task, as much of what needs to be done has to be done as root.

First step: Get your PS up to date; even after performing a restore on my server, there were a number of updates that are available to be installed. So let’s start off by getting those out of the way.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -f install

Once you get past those three commands, the next step is to update SQLite to the latest version, as the version Dreamhost uses is quite old and won’t work with Rails 3.0 (well, to be accurate it won’t work with the latest version of sqlite3-ruby, which is the default database provider for Rails 3).

wget http://www.sqlite.org/sqlite-autoconf-3070400.tar.gz
tar zxvf sqlite-autoconf-3070400.tar.gz
cd sqlite-autoconf-3070400
sudo ./configure --bindir=/usr/bin --libdir=/usr/lib
sudo make
sudo make install

If you don’t update SQLite you’ll get an error like this:

sudo gem install sqlite3
Building native extensions.  This could take a while...
ERROR:  Error installing sqlite3:
	ERROR: Failed to build gem native extension.

/usr/bin/ruby1.8 extconf.rb
checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... yes
checking for rb_proc_arity()... no
checking for sqlite3_initialize()... no
sqlite3-ruby only supports sqlite3 versions 3.6.16+, please upgrade!
*** extconf.rb failed ***

or if you install the updated version, but don’t force it to /usr/lib you’ll get an error like this:

sudo gem install sqlite3
Building native extensions.  This could take a while...
ERROR:  Error installing sqlite3:
	ERROR: Failed to build gem native extension.

/usr/bin/ruby1.8 extconf.rb
checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... no
sqlite3 is missing. Try 'port install sqlite3 +universal' or 'yum
install sqlite3-devel'
*** extconf.rb failed ***

Once that is taken care of SQLite, the rest is easy.

sudo gem update

At this point if you visit your new Rails site, it should be working!

Notes:

I’ve not tested this extensively, and I’ve no idea if this breaks anything. All I can say for certain, if that all of my sites still work, but your mileage may vary.
I was a fairly early Dreamhost PS adopter, and part way through this process I reset my server to get it back to a clean state. After resetting, I noticed some differences with the behavior of apt-get (404s on update and upgrade are gone), so for other early adopters it may be necessary to perform a reset to get your servers configuration in-sync with the latest official setup.
I can’t say for a fact that this is completely necessary, though you’ll likely need to selectively update a few packages if you skip this step. Also, for me, gem was broken until I ran sudo apt-get -f install.
Special thanks to Matt for helping me get this working; troubleshooting the SQLite install was more than a little time consuming.

Google Chrome and H.264

Adam Caudill — Wed, 12 Jan 2011 07:00:42 +0000

In a somewhat (but not entirely) surprising announcement, Google is removing support for H.264 video from Chrome. This change to their implementation of the often controversial HTML5 tag is both a major step by Google and a furtherance of the already complicated world of video online.

… To that end, we are changing Chrome’s HTML5 support to make it consistent with the codecs already supported by the open Chromium project. Specifically, we are supporting the WebM (VP8) and Theora video codecs, and will consider adding support for other high-quality open codecs in the future. Though H.264 plays an important role in video, as our goal is to enable open innovation, support for the codec will be removed and our resources directed towards completely open codec technologies.

When Google released WebM (a royalty-free codec which Google acquired as part of On2), it was clear that the intention was to take on H.264 and with this move there seems little doubt that the gauntlet has been thrown down. Although, now that H.264 has such a strong base (it’s included in Flash, IE9, Safari, Mobile Safari, and Android), it really makes one wonder if Google has picked a fight that has long since been lost.

So why would Google do this?

Here’s my best guess: Money (specifically, patent licensing).

H.264 is heavily encumbered by numerous patents owned by companies like Microsoft, Apple, and Cisco, and controlled by MPEG LA, the consortium charged with turning these patents into profit (here’s the 70 page list of patents for those interested). While I’m sure many people recall that MPEG LA made a very public pledge that H.264 would be free forever, as is often the case, things aren’t quite that simple.

Peter Csathy wrote a fairly detailed post on the matter, pointing out some key details that many in the media skipped. Here’s the core of what wasn’t discussed after the MPEG LA announcement (but should have been):

But, you say, MPEG LA recently announced that it will no longer charge royalties for the use of H.264. Yes, it’s true – MPEG LA recently bowed to mounting pressure from, and press surrounding, WebM and announced something that kind of sounds that way. But, I caution you to read the not-too-fine print. H.264 is royalty-free only in one limited case – for Internet video that is delivered free to end users. Read again: for (1) Internet delivery that is (2) delivered free to end users. In the words of MPEG LA’s own press release, “Products and services other than [those] continue to be royalty-bearing.”

Mike Shaver, Mozilla’s VP of Engineering offer’s a somewhat similar take in “Free as in Smokescreen:”

What MPEG-LA announced is that their current moratorium on charging fees for the transmission of H.264 content, previously extended through 2015 for uses that don’t charge users, is now permanent. You still have to pay for a license for H.264 if you want to make things that create it, consume it, or your business model for distributing it is direct rather than indirect.

What they’ve made permanently free is distribution of content that people have already licensed to encode, and will need a license to decode. This is similar to Nikon announcing that they will not charge you if you put your pictures up on Flickr, or HP promising that they will never charge you additionally if you photocopy something that you printed on a LaserJet.

I’m just waiting for one of the licensors to reinterpret the license and claim that ads constitute a form of payment or some similar excuse to exclude them from the exception they granted. I’ve yet to get my hands on the latest licensing agreement to see exactly what it says about this, but I wouldn’t be surprised at all to see this card played at some point to wring extra revenue from these patents.

Given that Google owns the massive video sharing site YouTube, which uses H.264, plus whatever unknown projects relating to Google TV – it stands to reason that Google would certainly save some money by moving away from such an encumbered technology; not to mention avoid future risk should rules change. Though personally, I also have to wonder if it could be fears of a repeat of the GIF patent debacle.

Now where does this leave us?

Right now HTML5 is a mess, at best. There is a war for which codec becomes the de facto standard, and there is a lot of money at stake depending on who wins. At this point there is no single codec that works across all major browsers; to get full coverage the best option now looks to be a combination of H.264, WebM, and Flash. Doesn’t really sound like the progress that was promised with HTML5 does it?

It’s worth noting though that Google isn’t the first browser developer to reject H.264; both Firefox and Opera have decided against including it in their browsers as well. As painful as the fragmentation is now in regard to who supports what, this move by Google actually does little to change the landscape. Support has been fragmented from the beginning, and all this really does it push H.264 a step away from being the de facto standard; a title that it has been very close to seizing.

Had Firefox added support for H.264, I think the fight would be over and would have made today’s announcement almost suicidal for the project. Though with such a major player holding out against it, Google’s move becomes a minor tactical shift in the short-term (though the long-term impact could be significant).

I could go on for pages about what works are where we are now, but Mark Pilgrim (an infinitely better writer than I) sums it all up here: “Dive into HTML5: Video on the Web” – well worth reading if you want to really understand what’s going on.

So in summary – video needs to be encoded to multiple formats, which today’s announcement does little to alter due to the fragmentation that was already in place. In the long run, WebM may be better for the community due to its license, though many of the internet’s biggest players have a vested interest in H.264. So when you factor in politics and propaganda between competing companies, distrust, and possible patent claims that haven’t been addressed yet; this all leads me to an even simpler summary:

Yesterday, HTML5 was a mess; tomorrow it will still be a mess.

Secure Password Storage

Adam Caudill — Mon, 01 Feb 2010 07:08:35 +0000

Do you use MD5 or SHA1 to store passwords? Think they are secure? Think again.

While generic hashing algorithms are certainly better than storing passwords in plain text, it’s still not as secure as it should be. Users place great trust in us to ensure that their credentials will be secure and treated with the utmost respect; it’s our responsibility to live up to these expectations.

With the simplicity and speed of these general purpose algorithms, it’s possible to generate hashes looking for collisions (or even the original value) extremely quickly. It’s this speed that introduces the fatal flaw; with a database dump containing MD5 hashed passwords, with a fairly small investment most could be recovered within a very small amount of time (mere days for a large database).

Many people are moving to bcrypt as a solution. In Coda Hale’s “How To Safely Store A Password” he covers this topic in more detail, complete with useful stats and links to implementations in languages from C# to Ruby (even Erlang is represented).

If you are looking for ways to better protect your user’s data, take a closer look at your password storage.

What’s your Code Legacy?

Adam Caudill — Mon, 21 Dec 2009 03:42:59 +0000

When you move on to your next challenge how will those that inherit your code think of you? Noble or notorious, innovator or insane? This is a question that all developers should ask themselves frequently; though too few ever do. You should always write with the assumption that someday a new developer will take over your code, and they will question every decision and assumption you’ve made. When this happens, what will they think of you?

Perhaps I’m more aware of this because I maintain an internally developed shared library that my company uses in every application; but regardless of the scope of the project you should always assume that someday you will hand the project off. Many developers think little about what happens to their code after it passes on to another; what other developers will have to deal with, or how their efforts will be perceived.

When I’m training a new developer there are a few points I try to reinforce as much as possible:

Code is only good if other developers can work on it without extensive training. If it takes days or weeks of introduction to get a new developer up to speed, then you’ve done something wrong¹.
Clever solutions are no better than an ugly hack if it’s not clear what you are doing. If the code isn’t clear then it’s not maintainable, if it’s not maintainable then it’s junk.
Assume you’ll be hit by a bus. Always write code with the assumption that you won’t have the opportunity to cleanly pass the code off to a new maintainer. Never assume that you’ll have time to come back and clean things up later.
Always perform design reviews, no matter the size of the project². Once you have a design in mind, talk it through with a at least two other developers. Just because you think it’s clean and clear doesn’t mean that others will see it that way as well.
Be consistent, always. I’ve seen more projects ruined by people doing things “their way” than anything else. Match style and design when working on an existing project. Be careful when adding new techniques, technologies, or methodologies to an existing project; unless you are willing to update the entire code-base, you can easily create a minefield without realizing it.

If you want your work to be seen positively after you move on, start thinking about your heirs today. The opinion they have of you will be almost entirely based on what they see in your code – not the stories or memories left behind.

¹ – There are always exceptions; these are generalized guidelines, not hard and fast rules.
² – This includes “throw away” projects, many projects that are intended to have a short life end up living far longer than intended. This is the most likely place that your heirs will find code that makes them question the quality of your work.