Notes From Thee Linux Conference (SCALE 14)

I attended SCALE for the first time this past weekend and highly recommend it. They had not only talks, but all-day training sessions for which you would reasonably expect to pay hundreds of dollars. I also got to meet Jenkins founder Kohsuke Kawaguchi, Apache Bigtop founder Konstantin Boudnik, and Randal L. Schwartz, whose Learning Perl I referenced countless times back in my Perl days. Plus I got to spend quality time with good friends.

Here are my notes from talks. I’m no longer in admin/DevOps, so I picked talks that piqued my interest.

Scaling – from Joel Salas of HauteLook
They are an ecommerce site that has to deal with high burst traffic. It’s LAMP, so not entirely applicable to my stack (Grails), but there were some good tips that many high traffic sites could use:

  • Tuned, a Linux daemon that collects data and allows you to set optimal system defaults that can have a big difference.
  • He prefers Gatling over JMeter for load testing. (I’ll note I’ve had good success with the latter, but never hurts to learn about new tools.)
  • Varnish, in addition to a CDN.
  • SolrCloud for search.

Postgresql EXPLAIN PLAN – from Josh Berkus of Red Hat

Postgresql for big data/data science – from Jim Nasby

  • Slony and Londiste should no longer be used
  • Can scale to 80+ cores
  • pg_shard – for sharding/replicating PG tables
  • cstore – columnar data store
  • PGStrom – allows PG to utilize the GPU
  • PostgREST –  give any PG DB a REST API
  • Toro DB – Mongo DB compatible JSON document DB, but built on Postgresq. Can save you a ton of space and I/O.

Dart by Randal Schwartz

  • Looking really good, lots of progress made, in use on big projects (100+ devs) at Google
  • Works on Node.js
  • Lot of work into making it compile into readable JavaScript
  • Great features taken from Java: types, reflection, concurrency, async futures, streams
  • Removes unused functions on packaging (smaller downloads, e.g., won’t include all of jQuery)
  • Pub is where everyone shares open source packages:
  • Free book to learn, kept up to date:

Painless Documentation (What If You Won the Lottery?)

Abstract: Sometimes good things happen to good people. When something good takes away someone you depend on, will you be prepared to go on without them?  With a little low effort documentation and knowledge transfer, they can leave you without torpedoing the schedule.

People often ask, “What if you got hit by a truck?” to pose a hypothetical around losing you. But what an awful thought! And in the corporate world, highly unlikely. Some form of winning the lottery, however, is not only possible, but probable. What if a developer on your project…

  • goes on maternity/paternity leave?
  • has to move?
  • goes on extended vacation or sabbatical?
  • goes to grad school?
  • retires?
  • gets a new job?

Really, that last one (along with the errant bus) is the most dangerous and represents management’s true fear: someone leaves NOW. With no time to aid in a smooth transition. I should note that in the US, two weeks notice is a courtesy typically given by both employee and employer, but not a law.1 And many tech departments have  a policy of paying you for two weeks, but walking you to the door as soon as you declare your intent to quit.2

That, my friends, is what drives many project management policies, especially those regarding documentation. I think it’s so obvious to management that they never explain it to employees this way. “We’re afraid you’ll leave us.” If they said that, perhaps you’d start wondering if there was someplace better. But really, most employees would totally understand if you flipped it on them:

  • Do you want to go on vacation and not have to give us a ton of notice (or lie and call it sick time)?
  • When you leave, do you want your coworkers, who have to take over your work, to hate you for leaving such a mess?
  • Do you want to own that piece of code that nobody wants forever?

OK, the second one might be a loaded question, but odds are you don’t. If you don’t do continuous documentation, you have a ton of work to do in your last two weeks. You perform this duty with all the work ethic of a graduating senior who already got into college. Maybe you convince your boss to accept a code walk-through in lieu of written docs. They’ll think they understand everything, but when it breaks or needs modification, it takes them forever and they silently curse you. Wait, no they don’t! You’re not here! They bad mouth you to the boss directly. Hope you weren’t counting on either as a reference.

Of course, by “you” dear reader, I’m using the royal you. You would never do that. Why, you’ve had that done to you!

I get that nobody enjoys documentation. Most devs don’t enjoy anything that isn’t coding, or maybe designing. Heck, I’ve seen people get fussy about filling out a time card, and that takes what? 2 minutes a day? They know the effort is low, but something about the task generates psychological resistance. I think a big part of it is not seeing any direct benefit, or putting a low value on time delayed benefits. Hopefully the preceding alleviates some of that. Still, I’m going to make this as painless as possible. What follows are the simple guidelines to provide redundancy while maintaining productivity and code quality.

Bug/Issue Tracker
I would think this goes without saying, but let’s say it anyway. All new features, bugs, and improvements go in your issue tracker. Each entry should be considered a live document containing both the requirement (user story, etc.) and design.

Pro tip: keep it classy in the issue tracker. If they don’t have access already, the client eventually will and you don’t want to be caught badmouthing anyone. Not even the vendor, because someone picked them, and be it the client or your company, mud slinging won’t win you any friends. My general rule is to never leave a paper trail when blowing off steam.

Version Control System
Shoot, if I’m mentioning the issue tracker, better mention this. Not using version control is one step away from not saving your work, or taking backups.3 Make your commit messages meaningful. In multi-file commits, I list each file and a brief explanation of the change.

Most importantly, I integrate the code repository with the issue tracker. Most systems make this dead simple; you just type the issue ID and it’s connected automatically. In Eclipse,4 I use Mylyn and plugins for my issue tracker and code repo, combined with a commit template, to make this automatic (it all works out of the box). If you need to fix a bug for that feature later, you’ve got all the commits and files for it listed in the issue tracker and can usually click on the commit message to see the source diff between versions. Very handy for the new person taking over (which may be you if you haven’t touched the code in a few months).

Note: if you can’t integrate your VCS and issue tracker, you probably have a lousy issue tracker. Actually, I bet someone chose a general project management tool instead of a real bug tracker. I use JIRA (I also like FogBugz), but there are so many choices and they all do this. Heck, the super simple issue tracker that comes with Github does this. You can’t get a lower effort, higher value documentation practice than adding the bug ID to your commit message. Even when you forget, git lets you amend it pretty easily.

Design Reviews
Your design does not have to be exhaustive, but should provide a development road map for that feature and give you confidence in the estimate. Have at least one other person review it (and your estimate) for omissions and general sanity.

Code Reviews
Similar to the above, get another pair of eyes on the code. If it’s just one or two people instead of the whole team, make sure they do it solo and take notes. That’s the only true test for readability, which is the most important aspect of your code.5 If you walk them through it, you’ll never know if your code is clear enough for someone to take over when necessary.

Note: for both design notes and code comments, don’t forget to document surprises and gotchas. We’ve all had the experience of starting with a sane, obvious solution, only to discover some obscure, insane obstacle, usually in some system we can’t touch. Be a hero and document that in the issue and in the code. In fact, if you only documented gotchas, you’d still save everyone a number of headaches down the road.

Bonus: Group Training
While in Kyiv, I saw a great talk about code clarity by Venkat Subramaniam.6 In his presentation, he pointed out the difference between readability and familiarity. All languages have idioms, and if you code in them long enough, or read enough good books on them, you’ll pick them up.7 The thing with idioms, though, is that their intent is often unclear to the uninitiated, who are unfamiliar with the language constructs being employed. To a practitioner, however, it’s perfectly readable. We may seek to correct this by banning idioms, but the better choice is to make everyone a better language practitioner.

One way to do this is through code reviews, as mentioned. But another is through group training. If you’re making a platform switch, the best way to do this is to hire a language and/or framework expert to coach the team, because the point is to get everyone on the same page, writing code that is familiar to everyone. But it could be something as simple as a company sponsored study group, or even a short, daily discussion of a single idiom. Basically, you want to idiom-proof your team. (Sorry.)

What does your team do to foster knowledge sharing? Please, share your thoughts in the comments.

  1. Well, not a law in all states. See at-will employment. Even that has exceptions, especially during layoffs. Consult an attorney if you really need to know this stuff. []
  2. Presumably to prevent sabotage or industrial espionage, based on the assumption that you didn’t know this policy existed before you gave notice and didn’t implement anything nefarious before you gave notice. []
  3. I personally think of commits as off-site backups of my code. []
  4. Note: I’ve since switched to IntelliJ. []
  5. Well, other than correctness, but if it’s not readable there’s a good chance it’s incorrect, or soon will be. []
  6. I highly recommend attending one of his talks if you get a chance; he’s a very dynamic speaker. []
  7. A good example are the “Effective” books for C++, Perl, and Java. All have different authors, but all teach you the same thing: how to best your structure code, including patterns and idioms. []

GGTS vs. IntelliJ IDEA

For years, I’ve been hearing about the superiority of IntelliJ over Eclipse from its enthusiastic fan base. However, I couldn’t find any evidence to persuade me to switch. I’ve had my frustrations with Eclipse, to be sure. I’d love to find a great replacement for any of my tools. But nobody could give me a clear reason. From its fans, I heard, “It’s just better.” Same endorsement I hear for OS X. And JetBrains’ feature list doesn’t having anything over GGTS. I needed a real reason to switch.

Well, earlier this month I got one. I won an IntelliJ Ultimate license at an LAJUG raffle. I now agree that IntelliJ is better, and I have clear reasons why!

The folks at JetBrains have always had a reputation at being language gurus. They’ve been able to provide support for languages faster than others, and frequently more comprehensively, even for dynamic languages. They’ve even invented a language. While I knew this, I assumed that for Java and Groovy GGTS would be on par due to it’s heritage. Unfortunately, it’s not.

To give you a clear example, I recently generated static scaffolding for a domain class in Grails 2.3. I noticed the controller uses the respond method instead of render, and the respond docs are not comprehensive. I tried opening the declaration to look at the code and it doesn’t work – GGTS has no idea where the code for controller methods reside.

I also tried some static imports of an embedded enum and again, GGTS had issues. Autocomplete (ctrl-space) didn’t work, nor did highlighting. In other parts of the code, it couldn’t locate the declaration for classes where a dynamic finder (e.g., User.findAllByName) was being invoked! In some classes it did work, but overall it was inconsistent. To be clear, there were neither compilation nor runtime errors – the code was fine, and to be honest there wasn’t much of it. This is a pretty new project.

As a test, I decided to import it into IntelliJ. The first attempt failed. I had it import it from Git, then parse the Eclipse project file when that was offered as an option. I was close to giving up as it didn’t even seem to recognize it as a Grails project, and there was no clear way to fix it in the configuration. But instead of giving up, I deleted the project and reimported it, this time ignoring the .project files and letting it discover the Grails nature on it’s own. Much better!

I opened the troublesome controller and noticed everything was properly highlighted, including those pesky enums. It understood the code and could open the declaration of all the classes, even those using a finder (and autocomplete was pretty nice). But best of all, it had no problem opening the declaration of those Grails controller methods like respond and render! This is a big win and clear advantage over GGTS.

I then checked out running/debugging. I’m a big fan of the debugger and had heard that IntelliJ’s method involves attaching to a separately run/managed Tomcat instance. This may have been the case in the past, but I was able to run and debug my Grails app the same way I do with GGTS – right from the IDE with “run-app -https”. No extra hoops to jump through.

Couple things I wish were better with IntelliJ:

  • Key bindings are not mnemonic. I’m sure they can be updated, but the choices strike me as odd.
  • Help window usability is bad, which is odd considering overall usability is good. Examples:
    • mouse scrolling is page-up/down, not smooth scrolling
    • searching highlights phrase – but you can’t turn it off!
    • back/forward buttons have no effect

Considering that help pages are the last refuge of the hopeless, making that a good experience is really important to customer satisfaction.

Couple tips for Eclipse/GGTS/STS users:

  • Search for “CamelHumps” in settings to navigate camel case like Eclipse
  • Play around with the Groovy/Java code style, in particular indentation, to match what you’re used to.


Dropwizard, JDBI, and Metrics

I recently attended the SBJUG talk on Dropwizard, which is less of a framework than a collection of libraries for developing self-contained REST web services. I had been hearing a lot of buzz about it recently and wanted to see how it compared to Grails. It is definitely lightweight and I’d encourage you to check it out. However, if you are an experienced Spring developer, your time might be better spent learning Spring boot and using that collection of libraries. If you know Grails, you might just be able to start using 2.4, which allows the @CompileStatic annotation in controllers and domain classes, which could really speed things up. For various reasons you may not be a fan of Spring, and if that’s the case definitely check out Dropwizard.

Although it’s not for me, I did learn about two cool libraries that could be incorporated into any project:

JDBI is like the sequel to iBATIS. You use straight SQL – no HQL or equivalent – with convenience features for writing DAOs to map to classes. No XML. Seems appropriate for groups where most of the queries are hand written for performance or religious reasons.

Metrics is a performance measurement/profiling library. Looks handy. Like Dropwizard, it’s a Yammer project.

What Stifles Startup Growth in OC?

Last night I attended an Orange County Tech Startups event on the future of innovation and startups in OC. Shervin Talieh put together great panels and I heard some interesting stats provided by journalist Chris Casacchia of the Orange County Business Journal (we need an infographic!). We heard how there are a number of billion dollar business in OC, but they’re in diverse fields. As most know, hardware, aerospace/defense, biotech, and action sports are more prevalent than software.

Here are my reasons for OC’s failure to launch a big startup culture: (more…)