Hire The Professional

“If you think it’s expensive to hire a professional…
…wait until you hire an amateur.”
– Red Adair *

I recently changed the radiator in my Jeep.

I did it myself, in part to save money, in part because it was a project my son and I could do together, and in part just because I wanted to. There’s satisfaction in doing a task yourself, even if- sometimes especially if- it’s not the kind of task you normally do.

And with a 20-year-old car that I paid $500 for- what’s the worst that can happen?

On the other hand, there are tasks that I defer to the professionals on. Brain surgery, for one. Another is accounting. My business partner and I are smart guys- we could certainly figure out how to file our own corporate taxes if we wanted- but why? The cost of doing it wrong is huge. If you overpay, you’re out thousands of dollars. If you underpay, you could go to jail. All in all, the cost of doing it right is not much compared to the cost of doing it wrong. So I use a professional.

Sometimes people choose amateurs to do a job that calls for a professional. I see this in software development all the time.

At Calavista, we recently spoke with a prospect (we’ll call him “Mycroft”) who had contacted us after his (outsourced) development team failed to make its 3rd consecutive delivery deadline. His team had been working for 9 months on an application, and though they’d done “great work” so far, they had not been able to actually release the project.

When we asked what “great work” meant, he said that the product demoed cleanly, looked good, and clearly worked- it just wasn’t complete. There were a few minor bugs to work out, and he couldn’t understand why it was taking so long.

Personally, my experience is that when a team cannot put the nail in the coffin and finish a project, it’s usually because they’ve accumulated too much technical debt. That is they might find the quick, “demo-able” solution to some problem is X, while the real, “releasable” solution would be Y. Y is more complex and time-consuming than X, so they do X. Often the reasons for this are valid- e.g. the customer needs to see the functionality working so they can make decisions on other things- and sometimes the team is just lazy. Or ignorant. But regardless, the choice to do the simple thing is made again and again, and ultimately when the time comes to actually deliver, all those choices now have to be addressed. In some cases, there may be so many things to resolve that it’s an effective re-architecture of the product, and “finishing things off” would really mean re-writing much of the code.

When we warned Mycroft of this, he pooh-poohed the idea (“I’ve seen it work!”), and indicated that the code was solid, and he didn’t think it would take long (or cost much) for us to clean things up and ship his code. He made sure we knew that he was very experienced, having run many development groups, and having brought lots of products to market. He knew what things should cost in this industry, and (in so many words) put us on notice that he was nobody’s fool.

During the course of this conversation, it became apparent that Mycroft had a team of 8 people, which he’d cobbled together, clearly with price as the driving factor. Now the warning bells were really going off. But we did agree to look at the code, so that we could give him an estimate.

The code was frankly shocking. Shortcuts were taken everywhere, crippling the application. Account passwords were stored in plain text, credit card CCID numbers were actually stored in the DB, and worst of all, there were huge vulnerabilities to SQL injection attack. Taken together, this meant that a savvy attacker could easily spoof the application into revealing all of the customer names, account info (including credit card numbers and CCIDs), and passwords. These problems- signs of quick and easy implementations in order to get functionality to demo- were systemic and ubiquitous. This was not an enterprise application. Security, scalability, performance…everything had been sacrificed to make a demo work.

The analogy we used was to a Hollywood set. His developers had built a false town. The bank’s facade was complete, but inside, there was no safe- the money was lying around in piles. Same with the hotel, the saloon, the blacksmith’s shop… You could walk an investor through the town on a guided tour and it would look good, but if you opened the doors to the public, it was all over.

The upshot is that Mycroft’s company will need to start over completely.

In this case, choosing the cheapest option was a spectacularly bad decision. 6-figures of investment (even given the low hourly rate), down the drain. More importantly, 9 months of lost market opportunity.

I’m not actually a believer in “you get what you pay for”- I’ve seen large software firms charge exorbitant prices for middling work. But I do believe in hiring the professional. In this case, Mycroft’s team clearly had no experience in producing applications that were enterprise-class, followed institutional guidelines on credit card security, and just observing commonly-accepted (and necessary) standardized coding practices. They wrote a piece of software that could impress an individual in a demo, but which could never actually be used.

There are all sorts of lessons learned here. “If it looks too good to be true…,” “always interview your developers,” “always perform code reviews,” “build specific performance/security/scalability requirements in from the start,” etc. But Red summed it up. Hiring an amateur can be the most expensive mistake you’ll ever make.

*- Red Adair was a legendary oil well firefighter, whose company was hired to extinguish some of the most catastrophic oil well blazes in history: from Texas, to the North Sea, to Kuwait. He charged top dollar for his services, but his customers knew that no one would do the job better or faster.