What Is It, Really?

What Is It, Really?

by | Nov 3, 2017 | Software

“What is it, really?”

Those four words launch a boondoggle.

It starts with smart software engineers. Smart and bored. They’re using a software library or tool to solve a problem. They have a lot of options to choose from. Multiple open source and commercial solutions, high quality, lots of customers or users. But it just isn’t perfect. It’s missing a few things they need and has a few of things they don’t.

So the engineers begin to ponder the nature of this tool. And they start with a gross oversimplification.

  • What is a name server, really? It’s basically a lookup table.
  • What is an object cache, really? It’s… well, it’s another lookup table.
  • What’s an ORM, really? It’s a map between SQL result sets and object fields.

Everything is just a map! And those are all real examples. I’ve seen companies who:

  • ignored Open LDAP, Netscape Directory Server, and Active Directory to write their own name server
  • ditched Ehcache to write their own that crashed the app on any serious load, and nobody knew why (multiple offenders)
  • ditched ORM to write all their queries by hand, seemingly unaware that they could easily use SQL for the 20% of queries that needed optimization. And since they supported MSSQL, Oracle, and MySQL, they wrote their DAOs 3 times. Cut and paste, baby!
  • wrote their own version of Struts with some extra features; then they were stuck on a proprietary Struts 1 clone long after Struts 2, Spring MVC, etc. came out.
  • wrote their own terribly designed version of portlets/JSF/etc. that nobody in the company understood after the creator left (and even he was shakey on it)

I’ll admit, often it’s less boredom than intimidation. You request the feature and the maintainers respond, “That sounds great! The source is over there, let us know when you’ve added it.”

You don’t even look at the source. I mean, it’s gotta be crazy complex. It already does so much. You’re not sure where to start. The developer contribution guide is scant and/or years old.

So you start rationalizing. You’re using just part of this thing. How hard would it be to recreate that? You’d understand all that code because you wrote it. And you could add those extra features you needed.

But you’re vastly underestimating the problem. To start with, the corner cases. I remember a story from Jamie Zawinski about the Netscape/Mozilla rewrite.1 A couple devs were reimplementing the FTP functionality. They had taken a few weeks and had a question about an edge case. He helped them, but the real issue was that the original code was gnarly because it had taken them 6 months to find and handle all the edge cases. And they were ignoring the original code because it looked icky. The same has been said about search code, Unix utilities, ORM, caching, anything with serious concurrency, etc.

Enterprise software companies seem particularly prone to all this. Perhaps because the sales division loves proprietary tools and lock-in.

What I am not saying

I am not saying don’t innovate. Or that you can’t improve things or come up with better products.

  • If you want to create a new open source competitor, go for it. A number of ORMs came before and after Hibernate, both open source and commercial. More will come.
  • If you can build a product and sell it, even though there’s competition, go for it.
  • If you need a small piece of a bloated dependency, and you can knock this out with unit tests pretty quickly, go for it.
  • Are you brilliant, working with other brilliant folks who will vet this idea? And it’s for something of massive scale, like Google, FB, Amazon, or MS would need? Go for it.

What I am saying is that building a one-off of a sizeable, complex component, for just your project, will waste tons of money and become a huge regret for all involved. And it’s always done because of ignorance.

Another way

As a manager, if I have the budget for a new, complex subsystem, I have the money to go to the maintainers of the project causing you grief and say, “Hey, if you agree this is a good idea, how much would I have to pay you to implement it? Are there committers who are available and want to be paid fairly to make this better?” Almost certainly yes. Maybe there is commercial support. The work would be blessed in advanced and fast-tracked for review.

At a minimum, you can hire them to write a proper contribution guide and code walkthrough so your devs don’t crap their pants at the prospect of contributing.

This happens often in projects like Linux. It’s cheaper and causes fewer problems in the long run. But when it comes to developer frameworks and libraries, reinventing the wheel seems like too much fun to pass up.

If you liked this, you’ll appreciate What’s the Developer Experience?

Thanks to Dave Ford and Kiran Manur for hilarious, head-shaking discussions about this. And to Joel Spolsky for probably writing about this 15 years ago.

 

  1. I’m almost certainly misremembering this, but some fine young cannibal will correct me. I couldn’t find a reference, so it was probably in a book. Coders at Work? [↩]
What’s the Developer Experience?

What’s the Developer Experience?

by | Oct 25, 2017 | Ecommerce, Software

This is an excerpt from Enterprise Sofware Confidential, my white paper on buying enterprise software without making a huge mistake. 

In this article, you will learn:

  • solid business reasons to care about developer happiness – even when they’re not your employees
  • why so many enterprise vendors produce piles of garbage that still sell

When making a major software purchase, you have a lot of questions. But there’s one that nobody asks, yet is so critical: what’s the developer experience like?

As you read earlier, enterprise software is all about configuration and customization. Your company has proprietary processes to which the software must conform. Or it should. Granted, many times the software is in line with industry best practices and you are not. Changing your processes to match the software is a win/win. But if you’ve invested time and money to create competitive advantages, you don’t throw them away to save engineering dollars.

So no matter the core feature set, there will be a bunch of custom programming to get this thing off the ground. Yet nobody asks if it will be pleasant for the developers. Yes, pleasant! Maybe you don’t care if the programmers like their job. Maybe you don’t like your job. Misery loves company, right?

Kidding! Of course, you care about the happiness of your employees. But in most cases, these are not your employees. They’re at the system integrator or vendor, and you’re paying them a lot to power through it without complaining.

In reality, there are solid business reasons to care about developer happiness, even if they’re not “your” developers. (Note: I’m going to use the word integrator to denote anyone customizing your software. Short for system integrator, but often referred to as professional services or solutions consultants in the biz.)

Golden Handcuffs
Golden handcuffs are what we call the high salaries paid to good engineers so they don’t leave because the work sucks and they are not learning anything new. This cost is passed on to you, of course. And sooner or later, they are going to leave.  Maybe in the middle of your project. Probably to a better platform in the same industry. A lateral move pay-wise, but a raise in enjoyment.

Ill Communication
The experience could be bad because the vendor doesn’t support system integrators well. Poor documentation, no access to the bug tracker, and firewalls between the integrators and the product team. When something goes wrong, it’s on the integrator to prove it to the product team, which is a long and painful process when you can’t talk to them directly. Many eye-rolling questions. (“Have you tried turning it on and off again?”) And that time is billed to you!

A Pile of Garbage
Often times, the reason it’s a bad developer experience is because it’s bad software. It didn’t start that way (I assume). The 1.0 version was pretty good, used the latest and greatest frameworks and incorporated smart design decisions.

Then the sales division took over. If the vendor has to spend a lot on sales, and that team is credited with driving revenue, they amass power. When that mindset grows to the point where engineering can’t make a stand for quality, you’re on a path to a pile of garbage.

Sales will push for new features over fixing bugs. Marketing will mandate deadlines to meet their initiatives (trade shows, commercials, etc.), rather than how long it takes to do it right.

Have you heard of the term technical debt? That’s what you get when you rush developers instead of giving them time to do it right. Quick and dirty is expected every once in a while, but they need time to clean up the mess. In sales-driven organizations, that rarely happens.

Unfortunately, piles of garbage can have pretty long shelf lives. Prohibitive sales costs keep out challengers, and customers rely too much on social proof. Waaay too much.

From a technical perspective, however, I have not seen a product recover from pile-of-garbage status.1 They become notorious among engineers. The problem you’ll have is that engineers aren’t terribly confrontational. Notice that I have not called out any bad actors. Engineers love the truth and want to be honest, but they’re unlikely to openly bash products, especially the ones paying their salary. The good ones simply find better work.

What to do?
I started writing a comprehensive list of questions you want to ask when choosing software. But I don’t want to mislead you into thinking a list of questions off the internet prepares you for shelling out megabucks on software, with answers from dubious sources. If you have a software development background, and bought that type of software before, and oversaw its implementation and use, you’re in good shape. Otherwise, hire a consultant. Doesn’t have to be me, just someone with that experience. Then get them

Then get them to act like a seedy private eye. Wine and dine developers to get them to open up about how much they enjoy the platform. You will be shocked at how many vendors you eliminate this way.

You can comment on Hacker News.

  1. If you have, I’d love to hear about it. [↩]
An Education on Education

An Education on Education

by | Oct 11, 2016 | Career

I had a very expensive education. But it was at a good school (RPI),1 in the right major at the right time.2 So upon graduation, I was employed, well-paid, and ready to avoid snow for the rest of my life.

However, many students have paid just as much or more, but they ended up with massive debt and terrible prospects. Here’s how you can have a great education and a great job, or at least avoid crippling debt. Fair warning: it also explains why we’ll never see free higher education in the US.

Forgive and Forget
Student loans are common and literally unforgivable, meaning bankruptcy doesn’t erase them. Why? Well, if you stop paying your mortgage, they take your house. Your car loan? They take your car. But there’s no lien on your education, and no way to repossess it. It’s yours for life, or at least until you forget it. In a dystopian system, they could make you forget it, but they still wouldn’t have it back. That’s why you always owe what you borrowed for education.

A forgiven student loan is essentially a scholarship or grant. Those are great, but have completely different qualification criteria. You typically have to have good grades and beat out other students. I got some grants and scholarships due to academic performance, but it only covered half my tuition. Those who performed better than I, or had greater financial need, got full rides. Makes sense!

Less Than the Paper It’s Printed On
Some think all higher education should be free, but that would be insane because of value. By value, I mean your earning potential after graduation. Some degrees are worthless. Actually, when you factor in opportunity cost – what you could have gained with the same time, effort, and money – they have negative value. This describes many degrees at for-profit universities. Some say those universities tricked people into going there, and I’m sure some did. The government is definitely doing a lousy job of protecting students there. Maybe forgiveness is reasonable in those extreme situations (fraud), but I’d rather see a class action or government lawsuit for fraud repay students, instead of our tax dollars.

Let’s say you’re going to a good school. Your major and the job market still have a huge impact on degree value, and it changes over time. Switch to a Democrat in the White House, defense spending goes down, as does the value of an aerospace engineering degree. Stringent (some would say overbearing) regulation on nuclear power plants has had the same effect on nuclear engineers. These unpredictable events can devalue a degree significantly. Disruptive technology can have a similar effect.

But even with good timing, is the right school and major enough? No, your degree matters a great deal. A friend who majored in bio had to get a masters degree just to be a lab tech or teach high school.3 Just about every scientific field other than computer science requires you to get a Ph.D. to have any career prospects. Many fields of engineering now expect a MS. It blows me away how many students enter college without realizing that.

Of course, you can still have the right major and degree at a reputable school, but it’s the wrong school! Most American universities are specialized in one field or another. Combined with reputation, it has a big impact on value. For instance, MIT is one of the best engineering schools in the world, if not the best. STEM4 degrees from there are quite valuable.

But did you know that MIT offers majors in all sorts of liberal arts fields, such as languages, cultures, and literature/writing? Given that MIT is also one of the most expensive schools in the world, do you think this is a wise investment? What is the expected return on a writing degree from here vs. NYU or the University of Iowa? (Hint: they are the MIT of creative writing.)

The Tuition is Too Damn High
So why don’t universities charge different prices for different degrees, when their value differs so wildly, and there’s clear data to prove it? Because the cost to educate them is roughly the same. You’ve surely heard about the soaring cost of education. Well, investigative reports show it’s not due to faculty salaries. Dependence on adjunct professors has skyrocketed, keeping educator payroll (vs. administrative) pretty stable.

Adjuncts are the bottom rung of professorship. They are part-time teaching jobs that not only have low wages, but educators are paid only for the hours spent in the classroom, not preparing lesson plans or grading papers. Many work multiple jobs to get by and are on welfare and food stamps.

One theory behind the rise of adjuncts is a flooded labor market. Lately, more college students are unsure of what to do upon graduation. They don’t like their options, especially if they have a liberal arts degree, so they stay in school. They get an MFA or PhD, take on more debt. Now they’re qualified for a job that has a flooded market and the worst prospects for tenure we’ve ever seen.5

So if the actual teaching costs aren’t to blame, what is? Buildings. Turns out, your obscene tuition – and alumni gifts – go to new buildings and campus beautification/maintenance. Why? Because prospective students rank campus beauty and amenities very highly when choosing a school.

My friend’s daughter is very smart and sensible. They toured several schools, but when they got to the Ivy League schools, she was smitten. “It looks like Hogwarts!” She’s right, and you know what? I did the same thing. I toured some schools with computer labs that were just a bunch of Macs and PCs scattered around a room. Then I visited RPI and saw rows and rows of high-end workstations housed here:

VCC

A big, beautiful chapel of stone and stained glass. It was like a sign from God.

Teenagers may be very smart and sensible (or not), but we’re asking them to make major life decisions – high risk investments – without the benefit of wisdom and hindsight. They are emotional thinkers, and architecture makes us emotional. So they weigh it much higher than they should, and colleges are using it to win them over.

A friend pointed out that administrative costs are also rising. That’s true, but I’ve read that most are related to buildings – it takes a lot of people to plan, build, maintain, and staff a $100M building. You know, like the arts building at my alma mater, an engineering school. Of course, it went way over budget.

Indiscriminately handing out loans makes it easy for schools to raise tuition. Since Reagan, politicians have been buying votes with easy money for houses, elevating homeownership to a basic human right, and causing the housing bubble and economic collapse of 2008. The same kind of thinking, applied to student loans, has created a tuition bubble. Unless we put that in check, tuition will continue to rise and Americans will be saddled with ever more debt.

Solutions

The first is mandatory mentoring. Before teens pick a school, force them to interact with professionals who have the job they think they want.6 Tell them which schools in their field are most respected, and what degrees are required. In other words, tell them up front what the minimum requirements are. We assume they know this, but they don’t, and neither do their parents. That wastes tons of time and money on the wrong school, major, degree, or all three.

Next, make schools collect data on employment rates and salaries by major. That’s the basic info a student needs to calculate his/her return on investment (ROI) for their chosen degree program. It should be part of the accreditation process and used for student loan qualification. People hate hearing this, but for the foreseeable future, computer science majors at top engineering schools should qualify for much bigger student loans than art history majors at the same school. Or, for that matter, CS majors at Bojangles College of Book Learnin’ and Taxidermy.

That’s smart investing. You don’t reward schools for skyrocketing tuition by having taxpayers give everyone a free ride. That’s part of what’s caused this mess. Schools will erect gilded palaces with 24/7 concierges and we’ll foot the bill.

And those bills will only get bigger. People point to Scandinavian countries as a working model for free education, but ask someone from there how many “permanent students” they know. People studying into their 30s, getting multiple degrees, happily living off the stipend with no intention to graduate. Then consider that the US population is over 12 times larger than all of Scandinavia combined. Heck, California alone is 40% bigger. Imagine the tax burden this would cause. Actually, you don’t have to, just ask someone from Scandinavia what their tax rate is.

At the same time, a top student from a poor family should never think they can’t afford Harvard or MIT. It should be guaranteed!7 That’s how we raise the impoverished. Reward both students and universities who perform well.

Finally, is there an alternative to college? Mike Rowe has been pointing out for years that our society suffers from a false belief that colleges are the only good answer to employment, and learning a trade is a recipe for hard times. In reality, compared to many college graduates, welders and machinists make way better money, have better job security, and they spent less time and money on their education.

I know it’s not easy work, but it could be a lucrative path to artist or author. I’m willing to bet that many tradesmen have great stories. I know veterans do;  just look at Hemingway. Pulitzer Prize? Check. Nobel Prize? Check. College degree? Nope.

The Future

It’s fun to think about the future and what might disrupt education. The first, which is already happening, is distance learning and MOOCs (massive open online courses). A great example is the Online Master of Computer Science offered by Georgia Tech. For at least 25 years, Georgia Tech has been considered a top 10 engineering school and by all accounts their distance learning program is legit. It’s just as rigorous as the one on campus, but cheaper and more convenient.

Given all the cheap adjuncts and grad students available to act as teaching assistants, you can have the star professors giving the lectures and scale this out to many students across the world.

You can also have educational tourism, in the same way you have medical tourism. That could play out in a few ways. First, note that undergraduate education, in STEM fields at least, is largely homogeneous. It’s there to prepare you for a graduate degree, where the real differentiation happens.

Therefore, a school abroad could gain a good reputation by licensing a curriculum from a top school. MIT Open Courseware has most of their material online for anyone to learn from. If a school could also license the assignments, tests, and grading policy, it would gain instant credibility.8

There has already been press about Americans attending German colleges for “free.” I can even see the same for us, brain draining the rest of the world because we don’t have enough students who get can into, say, top medical schools. Seems like a great alternative to a doctor shortage. I’m all for attracting the world’s best to America and fast-tracking their citizenship and contribution to society (including taxes).

 

 

  1. Less famous for being the oldest engineering school in the country than for its 4:1 gender ratio. [↩]
  2. Computer science during the dot com boom. [↩]
  3. Some states, including NY, require both a MS degree and a teaching credential to teach science. Despite the fact that a semester of college science teaches you more than a year of HS science. [↩]
  4. Science, Technology, Engineering, and Math [↩]
  5. But even before this, there was a sign: elbow patches. I always thought they were cool, a symbol of knowledge and authority. Then a friend explained that they weren’t a fashion statement but a sign of poverty. Professors were required to wear suits and had their elbows on desks while writing all day. They wore out the fabric, but couldn’t afford to buy new suits, so they sewed leather patches on them to keep it from happening again. This is why I won’t buy a new jacket with elbow patches, just as I don’t buy jeans with holes in them. It’s poverty as a fashion statement. [↩]
  6. Yes, force, because they will probably be shy and insecure about it, and not realize adults get a big ego boost just from being consulted. Heck, I have to hammer this home to young professionals who already have a job. [↩]
  7. I know Ivy Leagues are making sure that happens, but only very recently, and only because they’ve been shamed by alumni donors. [↩]
  8. I have several other ideas along those lines, especially for computer science. If you’re an international school who wants input from an authority on the tech hiring process, reach out. [↩]