What Is It, Really?

What Is It, Really?

“What is it, really?”

Those four words launch a boondoggle.

It starts with smart software engineers. Smart and bored. They’re using a software library or tool to solve a problem. They have a lot of options to choose from. Multiple open source and commercial solutions, high quality, lots of customers or users. But it just isn’t perfect. It’s missing a few things they need and has a few of things they don’t.

So the engineers begin to ponder the nature of this tool. And they start with a gross oversimplification.

  • What is a name server, really? It’s basically a lookup table.
  • What is an object cache, really? It’s… well, it’s another lookup table.
  • What’s an ORM, really? It’s a map between SQL result sets and object fields.

Everything is just a map! And those are all real examples. I’ve seen companies who:

  • ignored Open LDAP, Netscape Directory Server, and Active Directory to write their own name server
  • ditched Ehcache to write their own that crashed the app on any serious load, and nobody knew why (multiple offenders)
  • ditched ORM to write all their queries by hand, seemingly unaware that they could easily use SQL for the 20% of queries that needed optimization. And since they supported MSSQL, Oracle, and MySQL, they wrote their DAOs 3 times. Cut and paste, baby!
  • wrote their own version of Struts with some extra features; then they were stuck on a proprietary Struts 1 clone long after Struts 2, Spring MVC, etc. came out.
  • wrote their own terribly designed version of portlets/JSF/etc. that nobody in the company understood after the creator left (and even he was shakey on it)

I’ll admit, often it’s less boredom than intimidation. You request the feature and the maintainers respond, “That sounds great! The source is over there, let us know when you’ve added it.”

You don’t even look at the source. I mean, it’s gotta be crazy complex. It already does so much. You’re not sure where to start. The developer contribution guide is scant and/or years old.

So you start rationalizing. You’re using just part of this thing. How hard would it be to recreate that? You’d understand all that code because you wrote it. And you could add those extra features you needed.

But you’re vastly underestimating the problem. To start with, the corner cases. I remember a story from Jamie Zawinski about the Netscape/Mozilla rewrite.1 A couple devs were reimplementing the FTP functionality. They had taken a few weeks and had a question about an edge case. He helped them, but the real issue was that the original code was gnarly because it had taken them 6 months to find and handle all the edge cases. And they were ignoring the original code because it looked icky. The same has been said about search code, Unix utilities, ORM, caching, anything with serious concurrency, etc.

Enterprise software companies seem particularly prone to all this. Perhaps because the sales division loves proprietary tools and lock-in.

What I am not saying

I am not saying don’t innovate. Or that you can’t improve things or come up with better products.

  • If you want to create a new open source competitor, go for it. A number of ORMs came before and after Hibernate, both open source and commercial. More will come.
  • If you can build a product and sell it, even though there’s competition, go for it.
  • If you need a small piece of a bloated dependency, and you can knock this out with unit tests pretty quickly, go for it.
  • Are you brilliant, working with other brilliant folks who will vet this idea? And it’s for something of massive scale, like Google, FB, Amazon, or MS would need? Go for it.

What I am saying is that building a one-off of a sizeable, complex component, for just your project, will waste tons of money and become a huge regret for all involved. And it’s always done because of ignorance.

Another way

As a manager, if I have the budget for a new, complex subsystem, I have the money to go to the maintainers of the project causing you grief and say, “Hey, if you agree this is a good idea, how much would I have to pay you to implement it? Are there committers who are available and want to be paid fairly to make this better?” Almost certainly yes. Maybe there is commercial support. The work would be blessed in advanced and fast-tracked for review.

At a minimum, you can hire them to write a proper contribution guide and code walkthrough so your devs don’t crap their pants at the prospect of contributing.

This happens often in projects like Linux. It’s cheaper and causes fewer problems in the long run. But when it comes to developer frameworks and libraries, reinventing the wheel seems like too much fun to pass up.

If you liked this, you’ll appreciate What’s the Developer Experience?

Thanks to Dave Ford and Kiran Manur for hilarious, head-shaking discussions about this. And to Joel Spolsky for probably writing about this 15 years ago.


  1. I’m almost certainly misremembering this, but some fine young cannibal will correct me. I couldn’t find a reference, so it was probably in a book. Coders at Work? []
What’s the Developer Experience?

What’s the Developer Experience?

This is an excerpt from Enterprise Sofware Confidential, my white paper on buying enterprise software without making a huge mistake. 

In this article, you will learn:

  • solid business reasons to care about developer happiness – even when they’re not your employees
  • why so many enterprise vendors produce piles of garbage that still sell

When making a major software purchase, you have a lot of questions. But there’s one that nobody asks, yet is so critical: what’s the developer experience like?

As you read earlier, enterprise software is all about configuration and customization. Your company has proprietary processes to which the software must conform. Or it should. Granted, many times the software is in line with industry best practices and you are not. Changing your processes to match the software is a win/win. But if you’ve invested time and money to create competitive advantages, you don’t throw them away to save engineering dollars.

So no matter the core feature set, there will be a bunch of custom programming to get this thing off the ground. Yet nobody asks if it will be pleasant for the developers. Yes, pleasant! Maybe you don’t care if the programmers like their job. Maybe you don’t like your job. Misery loves company, right?

Kidding! Of course, you care about the happiness of your employees. But in most cases, these are not your employees. They’re at the system integrator or vendor, and you’re paying them a lot to power through it without complaining.

In reality, there are solid business reasons to care about developer happiness, even if they’re not “your” developers. (Note: I’m going to use the word integrator to denote anyone customizing your software. Short for system integrator, but often referred to as professional services or solutions consultants in the biz.)

Golden Handcuffs
Golden handcuffs are what we call the high salaries paid to good engineers so they don’t leave because the work sucks and they are not learning anything new. This cost is passed on to you, of course. And sooner or later, they are going to leave.  Maybe in the middle of your project. Probably to a better platform in the same industry. A lateral move pay-wise, but a raise in enjoyment.

Ill Communication
The experience could be bad because the vendor doesn’t support system integrators well. Poor documentation, no access to the bug tracker, and firewalls between the integrators and the product team. When something goes wrong, it’s on the integrator to prove it to the product team, which is a long and painful process when you can’t talk to them directly. Many eye-rolling questions. (“Have you tried turning it on and off again?”) And that time is billed to you!

A Pile of Garbage
Often times, the reason it’s a bad developer experience is because it’s bad software. It didn’t start that way (I assume). The 1.0 version was pretty good, used the latest and greatest frameworks and incorporated smart design decisions.

Then the sales division took over. If the vendor has to spend a lot on sales, and that team is credited with driving revenue, they amass power. When that mindset grows to the point where engineering can’t make a stand for quality, you’re on a path to a pile of garbage.

Sales will push for new features over fixing bugs. Marketing will mandate deadlines to meet their initiatives (trade shows, commercials, etc.), rather than how long it takes to do it right.

Have you heard of the term technical debt? That’s what you get when you rush developers instead of giving them time to do it right. Quick and dirty is expected every once in a while, but they need time to clean up the mess. In sales-driven organizations, that rarely happens.

Unfortunately, piles of garbage can have pretty long shelf lives. Prohibitive sales costs keep out challengers, and customers rely too much on social proof. Waaay too much.

From a technical perspective, however, I have not seen a product recover from pile-of-garbage status.1 They become notorious among engineers. The problem you’ll have is that engineers aren’t terribly confrontational. Notice that I have not called out any bad actors. Engineers love the truth and want to be honest, but they’re unlikely to openly bash products, especially the ones paying their salary. The good ones simply find better work.

What to do?
I started writing a comprehensive list of questions you want to ask when choosing software. But I don’t want to mislead you into thinking a list of questions off the internet prepares you for shelling out megabucks on software, with answers from dubious sources. If you have a software development background, and bought that type of software before, and oversaw its implementation and use, you’re in good shape. Otherwise, hire a consultant. Doesn’t have to be me, just someone with that experience. Then get them

Then get them to act like a seedy private eye. Wine and dine developers to get them to open up about how much they enjoy the platform. You will be shocked at how many vendors you eliminate this way.

You can comment on Hacker News.

  1. If you have, I’d love to hear about it. []
Enterprise Software Explained

Enterprise Software Explained

I recently wrote a hit article on buying enterprise software, and I was working on the followup when I realized that you, the manager tasked with choosing it, could benefit from an overview from an insider. I’ll explain how this software is packaged and delivered so you can discuss it with more confidence.

Most importantly, this will provide critical context and understanding to my final article, which will arm you with questions and talking points, both internal and external. That’s the one that will prevent you from going all Fahrenheit 451 on your company’s cash pile.

If you think you know all this stuff, please take a quick read and let me know if I’ve left anything out. Thanks!

What the heck is it?

I was a little surprised  the first time an exec asked me, “What exactly is ‘enterprise’ software?” But it’s a reasonable question. In my view, it is software designed to handle the needs of the world’s largest and most complex businesses. This might include:

  • thousands of transactions per second (Black Friday, returns after Christmas)
  • large or complex data1
  • highly complex business logic
  • integration (data sharing and process triggers) with similarly complex systems

How do you write software that can do all this? Well, you don’t create a software package, like Word or Excel. You create a software platform. Something that can be molded and extended to meet your business needs within that specific domain (ecommerce, ERP, CRM, marketing automation, etc.).

SaaS or On Premise?

Software as a Service includes vendors like Salesforce, Shopify, Mailchimp, Google Analytics, etc. They host it, and everyone is on the same system. It’s sometimes called “cloud based” because marketing thinks that’s cool. There might be different tiers of prices/features, and they will likely do something extraordinary for you if you throw enough money at them.

On Premise
The name implies it’s installed on your servers, but it doesn’t have to be. Could be hosted by the vendor, integrator, or anywhere you like. The key is that each installation is a one-off that requires separate installation and maintenance. Almost all on premise enterprise software includes the source code for you to customize, so it can also be a unique snowflake (for better or worse).

OOTB, Configuration, and Customization

OOTB is out of the box. It’s the features you get right after installation and minimal configuration. Like user management, and, um… well probably something else, you did just pay 6 figures for this.

Configuration requires no coding, just enabling features and modifying settings through the user interface. A product expert or business analyst (typically referred to as an administrator/admin in this case) can handle this, no need for a coder.

Note: Salesforce enables a ton of functionality via configuration, to the point where a Salesforce certified “developer” doesn’t do any actual software development (coding). Of course, there is coding you can do for Salesforce (Apex), but why not use confusing vernacular when you can? Welcome to enterprise software.

Customization requires coding. It’s often the bulk of the cost of the software. I frequently see professional services (what they call this work, along with the installation and configuration) running 1x-3x the license cost, which is often 6 figures.

Good enterprise software will have clear extension points (APIs) for customization. Most SaaS software has a web service API for integration. Web services enable your integration code to run anywhere, written in any language on any platform. You have a .NET coder, but the enterprise platform is Java/Linux, or some mystery? No problem. Through the API, you can download your orders, upload changes to your your customer database, destroy valuable data, etc.

A couple SaaS platforms, like Salesforce and Demandware, have a proprietary programming language, in addition to their web service. On premise software might have a web service interface, and/or a language API like Java or C#.

None of that is rocket surgery, but takes time to get the requirements right and cover all the edge cases. The problem comes when a core feature is implemented incorrectly for your business and there’s no way to customize it through the API. You have to change the core code (and maybe the database) to meet your requirements. I often suggest changing your process to match the software, if this is possible. But maybe you’re doing it differently because this is a competitive advantage. Or you’ve already changed your process to accommodate another, more expensive piece of software. Implementing this is possible with on-premise software that includes the source code.

Basic Architecture

It’s important to understand, at the 30K ft. level at least, the different parts of the software.

  • Core code: The creamy center. It has most of the functionality and underlying services. It includes the APIs to integrate with other software (say, ecommerce to ERP or CRM) and add/modify features (customization).
  • Admin interface: The interface to configure the software.
  • User interface: What your customers or employees see.
  • External integration code: This uses the web service API, FTP, or other external integration method. This code is stored and run completely separately.
  • Custom code: What you write to add/modify features. It’s typically stored somewhere it uses or overrides – not overwrites – the existing code. Typically. It could include…
    • modifying the user interfaces mentioned (changing the templates)
    • code using the Core APIs to add features and integrate with other software
    • changing (overwriting) the core code, if it’s really needed to support the business

As you’ll see in a second, modifying the core code is a lot like going back in time. You fix one thing, like, you didn’t total your classic car. But you somehow wiped out all the pandas. And we’re at war with Eastasia. (Luckily, we’ve always been at war with Eastasia.)


Upgrades can be very tricky depending on the platform and level of customization. With most SaaS, upgrades should be automatic and you get all the admin UI features and core fixes and improvements. The rest will require enabling through configuration, modifying the user interface, or adding code to access new features in the API. Code using old APIs should still work if they do it right.

With on premise, some of that will be the same. A new version means a new core. You drop it in, run regression tests, and you should get the fixes and performance improvements to the core code, plus any new admin screens.

In both cases, you may have modified the UI. That means your code is overriding theirs, so any features in the UI won’t be available until you add them to your custom templates. With on premise, you may have modified the admin screens as well, so same deal.

Finally, you may have modified the core code.  This means you have to merge your changes with the new core. Which means understanding all the new code, but even the old code may have been changed significantly. Improving existing code is called refactoring and is an industry best practice. Lucky for you, such things are rare in enterprise software. No matter, it’s still a pain, and error prone. When finished, run a smoke test: are there still pandas?

In all cases, upgrades will be vastly aided by running your regression test suite. These are all the end to end tests created by your QA team that prove everything is working as it should, and they should be automated. When a project is running late, QA is the first thing to go. Why not assume we got it right the first time? When it’s time to upgrade, if you scrapped these tests, you’re going to have to factor them into the cost of the upgrade, along with everything else. Also, scrapping these tests means you have bugs you don’t know about, which can be really embarrassing for you.

Or you can continue to ignore proper QA. Just have your interns and office admins run through some basic tasks to see if anything broke. If so, blame it on the vendor. The vendor will never admit it’s their fault, but that’s OK until your CEO wants to be on your conference call.

Welcome to enterprise software.

  1. I find most businesses don’t have true, valuable “big data” despite the hype, but it can be pretty scattered and complex. []
Ask a Repair Shop

Ask a Repair Shop

This is the story of how you’re buying your enterprise software the wrong way. Probably your appliances, too.

This is an excerpt from my guide, “Enterprise Software Confidential.”

This post is also available in Chinese thanks to Xu Zhi.

Some years ago, I had a broken GE washer. Pretty sure I knew the culprit, but if I was wrong I’d waste a bunch of money on a part I couldn’t return. So I called a local repair shop who serviced a number of brands, including mine. “Oh, we don’t repair GE anymore. They’re pretty much throwaways now. When they break, you just buy a new one.”

This shocked me, beyond learning of GE’s quality nosedive.1 If their products frequently broke, wouldn’t that mean a steady stream of work? Then I realized a flaw in human reasoning. The first time it breaks, you blame the manufacturer. The second time, and every time after that, you blame the repair shop. For a repair shop, servicing appliances from a bad vendor can damage your reputation. Upon realizing this, I called him right back. “If GE is bad, who do you recommend?” I got a full education on washers, including a lot of industry dirty laundry.

Since then, every time I make an appliance purchase – vacuum cleaner, dishwasher, car – I go to a repair shop. They’re always excited to talk to someone who will listen. They may even have a used model that will last you for years.  (more…)

  1. Could be better now, but I wouldn’t know. []