Upgrading to Grails 3

Preface

I’m upgrading an app from Grails 2.4 to 3.1 and thought I’d take share my thoughts and notes.

First, I want to say that I’m very grateful to the maintainers for all their hard work. I know this wasn’t easy.

Second, I’m not a full time coder anymore, but a consulting CTO. So although I’ve been doing Grails for a few years now, and Spring/Hibernate before that, consider me a casual user. Still, I have opinions – don’t we all? I’m sharing them because I want Grails to thrive.

As you know, Grails began as a way to make Spring/Hibernate development much easier. Over time, it grew a wonderful ecosystem, but it’s still Spring/Hibernate at its core.

When Spring 4 was launched, I attended an event where all the new features were explained, along with Spring Boot. The Spring people were very excited. But I kept thinking, “This is nice, but it’s still well behind Grails.”

The problem is, SpringSource made a lot of changes under the hood to deliver those features. And Grails, being a layer on top, had to be rewritten to remain compatible and up to date.

That means this upgrade is a necessary evil. Because I left Spring for Grails a few years ago, for me it means work and frustration with no benefit. That’s not meant to be an insult, but rather a compliment to Grails 2.x, which worked so well for me and thousands of others. I know the maintainers didn’t have a choice, but at no point during my Grails dev did I ever think, “Gosh, I wish Grails had a completely different architecture under the hood. It’s really holding me back!” Most of the Grails devs I talk to feel the same way. So I’m sorry this post isn’t quite ebullient.

I’m doing this now because I know this is the future,1 and doing it with a project still in development will be way easier. I have another an app in production and I want to take the lumps with a pre-launch project first.

Docs on upgrading are decent, but I had a lot of questions it didn’t answer. Hopefully this is helpful.

New Tech

Grails 3 incorporates the following technologies you’ll have to understand:

  • Gradle (replacing Gant)
  • YAML
  • logback (replacing log4j)
  • Asset Pipeline plugin

Gradle does seem to be a improvement over Ant (and I never liked Maven), but the Grails build system always worked great for me. YAML and logback seem simple enough, but I don’t see the value added, especially after hosting a talk on log4j 2, which looks great.

Asset Pipeline is a hands down win, and something you should be using in 2.x if you’re not already. I know a lot of people are still using the Resources plugin, but it’s time to move on.

I’ll add that the command line expects you to be running with JDK 8. run-app crashed immediately with a bytecode error and the answer was to set the GRAILS_OPTS environment variable to “-XX:-UseSplitVerifier -Xverify:none”. Later, I simply changed my JAVA_HOME to JDK 8, which doesn’t need that. I think we’re all ready to move on to Java 8 now anyway.

IntelliJ

Update: I want to share some inside info from a JetBrains employee. JetBrains uses votes and social media (Twitter complaints) to prioritize work. The Grails community has been pretty lax in voting for IntelliJ issues, hence we’re seeing core functionality like reload/debug half-broken for Grails 3. Please upvote issues, and comment if an issue is holding you back.

The Grails community is unanimous in it’s support for IntelliJ, and I previously wrote about why it’s better than GGTS (Eclipse). I believe GGTS has been discontinued as it was maintained not by the Grails team, but by the SpringSource tools team that makes STS. If your employer was too cheap to buy you IntelliJ before, you have an ironclad argument now.

I had to upgrade to the latest version to use Grails 3. I had put this off because I knew the Grails view was missing. Now it’s there for Grails 2 projects, but not 3. I didn’t realize how helpful it was until I lost it.

I used it to create my project, since in Grails 3 you create a blank project and copy your old project into it. First thing you need to do is pull up the Gradle tool window and click the refresh icon. As brilliant as IntellliJ is, it’s too dumb to realize this is a Grails project, even though it just created it. It thinks it’s only a Gradle project until you pull up the Gradle tool window and click the refresh icon. After this, you’ll see the Grails menus and your app available in the run configuration menus. Absolutely critical, yet not in the docs or automated for you, leaving some confusion.

Once Gradle is refreshed you can use the usual way of running it, but that has a glaring bug – environment variables you set in the run configuration are not passed on! This is a big deal, as you’ll read below, because environment variables are the standard way to do external config now that specifying files is not supported out of the box.

To get around this, use the Application run configuration. Go to ../grails-app/init and right click on Application.groovy to run/debug. Note: this is not your config file. That’s under conf, with a lowercase ‘a,’ see below.

However, there’s another problem – if you need to test SSL connections, you need run-app -https, and as mentioned that doesn’t work with environment variables. Since it’s local, I’m running with an insecure channel, but I’d like that fixed.

environments {
    development {
        grails.plugin.springsecurity.secureChannel.definition = [
                [pattern: '/**', access: 'REQUIRES_SECURE_CHANNEL']
        ]
    }
}

IntelliJ is so smart it notices when you’re using something without the source and docs, and suggests you grab it.  That is terrific.

But like the Gradle refresh, it also does things that are confusing. If I click on just about any file, including settings.gradle, the editor tab shows the file name. But when I click on build.gradle, the tab has the app name. I know I’ll get used to this, but it adds to the learning curve. And I’m chuckling as I write this and realize this is trivial compared to the fact that I now have to learn Gradle. Hey, at least it’s not Maven 🙂

Update: auto-reload broken?
I find that auto-reload on edit is now broken. Not sure why, but as linked there is a similar issue. Auto-reload works fine when I run ‘grails run-app’ on the command line. I can run that in the background and edit with IntelliJ, but I can’t debug. Losing the edit/test cycle with the debugger means troubleshooting is much slower now, especially with controllers, which are a real pain to unit test.

Update 2: Workaround (Thanks Josh!)
I’ve done more investigating. Basic reload via the grails run-app configuration works. However, there is a significant disconnect between the editor window and compiled code – the editor uses the code built on startup when determining if a line can have a breakpoint. So if you change the code and want to set a new breakpoint, it won’t work

In the comments, Josh pointed out that it was working for him when run via Gradle. Follow the docs to run/debug via Gradle. It worked, but I couldn’t find the console output. To see it, in the Console window that is showing Gradle tasks/times, click the toggle button with the tooltip “Toggle tasks executions/text mode.” Now you’ll see the console output.

Note that this still does not allow you to run via HTTPS like with run-app. It’s good enough for now, but please upvote this issue to make it better for everyone.

Configuration

Grails 3 introduces an application.yml file that is used by default. Even though the docs describe it as “an alternative format,”I have found at least one setting that must be set in application.yml:

https://github.com/grails/grails-core/issues/10005

We do not have a definitive list of what must be defined in the YAML file. That is the only one I noticed, but there certainly could be more. If you’re going to use application.groovy, you may want to add a check in BootStrap.groovy and exit on missing settings.

So why was YAML chosen? Because Spring Boot uses it, and according to Graeme Rocher it’s fully machine writable where Groovy sometimes isn’t (or not easily). But for whatever reason, the helpful comments in Config.groovy are gone. If you’re upgrading, you can just copy them all over.

If you’re converting to Groovy or vice versa, this SO question may help:

http://stackoverflow.com/questions/30057329/convert-configuration-file-application-yml-to-application-groovy-in-grails-3-x

One thing you’ll need to change is your defaultPackage, since Grails create-app doesn’t ask for it. I had originally converted all the YAML to Groovy, then found defaultPackage is ignored when in application.groovy. For now, this is my whole application.yml, but expect I’ll need to add more:

grails:
    codegen:
        defaultPackage: com.madeupname.web

Which takes precedence if you use both? Short answer is “don’t do that!” Make sure there is zero overlap between the two config files. You won’t get an error or even a warning, and in my test YAML consistently overrode Groovy, but you can’t assume that will always be the case. The recommended approach is to stick with application.yml and use Groovy whenever you need to programmatically set values. You’ll also have to keep an eye out for plugins modifying either file.

If you need external configuration files, you’ll have to do some hacking. I recommend reading this thread on the dev list. But the general recommendation is to use system variables/command line arguments (which take precedence) or environment variables (see Spring PropertySource). This is probably what you need to do if you’re deploying to a PaaS like Elastic Beanstalk, Cloudbees, Jelastic, etc.

Update: external config
Thanks again to Josh in the comments, I found very clear instructions from Haki on how to enable external config in Grails 3, and it’s pretty easy. As instructed, I added the VM option to my run config, updated my Gradle config file, and made sure the format was YAML. Maybe you can use Groovy config for this, but I played it safe.

With reload not working properly for debug in the Application and run-app run configs, you have to use Gradle, which requires external config files because IntelliJ doesn’t support setting environment variables for Gradle tasks.

So it would be great if you could upvote this issue to support environment variables.

Plugins

Because of the sweeping changes to the architecture, most (if not all) plugins need to be updated to work with Grails 3. In many cases the change is small, but you’ll want to check what the latest compatible version is.

Unfortunately, the Grails plugin site, while improved, still has issues. The good news is that it separates Grails 3 compatible plugins. The bad news is that the current version could be way off. For example, Spring Security UI needs a lot of supporting libraries, including Hibernate4 plugin. You can find it here:

https://grails.org/plugin/hibernate4

Problem is, the actual latest version is 5.0.7 and definitely not 5.0.0.RC1. There’s also no clear list of required and optional dependencies, you need to look to the plugin docs for that. If an official/core plugin has the wrong version listed, I’m forced to distrust every version on the plugin site. I’m also wondering how that info is added, and I suspect it’s manually, since the fields plugin has its source and issues links reversed.2

Another concern is that after a year, the Grails 3 compatible Mail plugin is still a release candidate, and Spring Security UI is just a milestone release. This is going to give a lot of people pause when considering upgrading.

Mail
The MailService class changed, but it’s no longer needed. Config is largely unchanged, except for the environment variable placeholders. This is for Gmail:

grails {
    mail {
        host = '${MAIL_HOST}'
        port = '${MAIL_PORT}'
        username = '${MAIL_PASSWORD}'
        password = '${MAIL_USERNAME}'
        props = ["mail.smtp.auth":"true",
                 "mail.smtp.socketFactory.port":'${MAIL_PORT}',
                 "mail.smtp.socketFactory.class":"javax.net.ssl.SSLSocketFactory",
                 "mail.smtp.socketFactory.fallback":"false"]
    }
}
grails.mail.default.from = "[email protected]"
grails.mail.default.to = "[email protected]"

Scaffolding
You have to change very instance of:

static scaffold = true

to

static scaffold =

Classname

I also found a potential bug where the configuration setting domainSuffix is only changing the model name and not what is expected in the view. The workaround is to remove the domainSuffix setting and update all your controller responses to remove the word “Instance.” So companyInstanceList would be changed to companyList, and the same with count, etc.

Spring Security
I use S2 core and UI plugins, which must be updated. For S2 UI, you must specify plugin and library dependencies. In particular, you need to specify a number of GORM libraries, and it’s not clear why. The docs do not list in one place every plugin you (likely) need. Here’s my build.gradle config for both plugins:

compile 'org.grails.plugins:spring-security-core:3.1.1'
compile 'org.grails.plugins:spring-security-ui:3.0.0.M2'
compile 'org.grails:grails-datastore-core:5.0.7.RELEASE'
compile 'org.grails:grails-datastore-gorm-support:5.0.7.RELEASE'
compile 'org.grails:grails-datastore-gorm:5.0.7.RELEASE'
compile 'org.grails:grails-datastore-simple:5.0.7.RELEASE'
compile 'org.grails:grails-datastore-gorm-hibernate4:5.0.7.RELEASE'
compile 'org.grails:grails-datastore-gorm-hibernate-core:5.0.7.RELEASE'
compile 'org.grails.plugins:mail:2.0.0.RC6'

Other

Traits
It’s not config, but I had a class in src/groovy that used the @Validateable annotation, and now wouldn’t compile. You need to make it implement the Validateable trait (interface) instead.

Looking back, it does not seem that bad, but without knowing any of the above it was an exercise in frustration. Hope I save you some trouble!

  1. Kind of like when I moved from Subversion to Git. []
  2. While the main Grails site has a button to fix any page, the plugin sub-site does not, and there’s no docs to fix plugin data. []

GGTS vs. IntelliJ IDEA

For years, I’ve been hearing about the superiority of IntelliJ over Eclipse from its enthusiastic fan base. However, I couldn’t find any evidence to persuade me to switch. I’ve had my frustrations with Eclipse, to be sure. I’d love to find a great replacement for any of my tools. But nobody could give me a clear reason. From its fans, I heard, “It’s just better.” Same endorsement I hear for OS X. And JetBrains’ feature list doesn’t having anything over GGTS. I needed a real reason to switch.

Well, earlier this month I got one. I won an IntelliJ Ultimate license at an LAJUG raffle. I now agree that IntelliJ is better, and I have clear reasons why!

The folks at JetBrains have always had a reputation at being language gurus. They’ve been able to provide support for languages faster than others, and frequently more comprehensively, even for dynamic languages. They’ve even invented a language. While I knew this, I assumed that for Java and Groovy GGTS would be on par due to it’s heritage. Unfortunately, it’s not.

To give you a clear example, I recently generated static scaffolding for a domain class in Grails 2.3. I noticed the controller uses the respond method instead of render, and the respond docs are not comprehensive. I tried opening the declaration to look at the code and it doesn’t work – GGTS has no idea where the code for controller methods reside.

I also tried some static imports of an embedded enum and again, GGTS had issues. Autocomplete (ctrl-space) didn’t work, nor did highlighting. In other parts of the code, it couldn’t locate the declaration for classes where a dynamic finder (e.g., User.findAllByName) was being invoked! In some classes it did work, but overall it was inconsistent. To be clear, there were neither compilation nor runtime errors – the code was fine, and to be honest there wasn’t much of it. This is a pretty new project.

As a test, I decided to import it into IntelliJ. The first attempt failed. I had it import it from Git, then parse the Eclipse project file when that was offered as an option. I was close to giving up as it didn’t even seem to recognize it as a Grails project, and there was no clear way to fix it in the configuration. But instead of giving up, I deleted the project and reimported it, this time ignoring the .project files and letting it discover the Grails nature on it’s own. Much better!

I opened the troublesome controller and noticed everything was properly highlighted, including those pesky enums. It understood the code and could open the declaration of all the classes, even those using a finder (and autocomplete was pretty nice). But best of all, it had no problem opening the declaration of those Grails controller methods like respond and render! This is a big win and clear advantage over GGTS.

I then checked out running/debugging. I’m a big fan of the debugger and had heard that IntelliJ’s method involves attaching to a separately run/managed Tomcat instance. This may have been the case in the past, but I was able to run and debug my Grails app the same way I do with GGTS – right from the IDE with “run-app -https”. No extra hoops to jump through.

Couple things I wish were better with IntelliJ:

  • Key bindings are not mnemonic. I’m sure they can be updated, but the choices strike me as odd.
  • Help window usability is bad, which is odd considering overall usability is good. Examples:
    • mouse scrolling is page-up/down, not smooth scrolling
    • searching highlights phrase – but you can’t turn it off!
    • back/forward buttons have no effect

Considering that help pages are the last refuge of the hopeless, making that a good experience is really important to customer satisfaction.

Couple tips for Eclipse/GGTS/STS users:

  • Search for “CamelHumps” in settings to navigate camel case like Eclipse
  • Play around with the Groovy/Java code style, in particular indentation, to match what you’re used to.

 

Web Framework Performance Benchmarks

This is making the rounds and is pretty interesting:

http://www.techempower.com/blog/2013/03/28/framework-benchmarks/

My framework of choice, Grails, doesn’t look spectacularly fast, although I already knew that. However, it is nice to see that it is notably faster than RoR (which appears to be a complete dog) and Django, and will probably improve quite a bit with v2.2 on  Groovy 2.0 and Java 7.

However, it seems a big part of the performance lag is because it’s built on Spring, and to my surprise Spring isn’t a great performer, either. It is nice to see that straight up Servlets and Vert.x are beasts, so another score for our dear JVM.

Another thing that caught my attention is that Grails’s ranking improves quite a bit once you move it from EC2 to dedicated hardware. IaaS and PaaS save you some admin work, but also costs you some dev time, but at some point it looks like you’ll get a nice gain from dedicated hardware. It could make something like Rackspace managed hosting on dedicated hardware look reasonable.

With the attention it’s getting, I also expect a shakeup as framework providers help the benchmarkers improve performance. It will be interesting to see how this plays out.

UPDATE 2/2/2014

Round 8 of the web performance benchmarks are out and Grails is looking good. (more…)

Companies Using Grails

Companies Using Grails

I’m pretty excited about Grails, but am often asked for social proof. Well, by every measure,1 Groovy is the #1 JVM language after Java, and Grails is the #1 full stack web framework.2 But for some, that’s not enough. They want to know who is making it #1? Major companies. Below I’ve listed the household names. I could not get details for some companies, but a Google search will verify the claim (e.g., the Disney developer posting a Grails question to a mailing list).

Last update: August 20th, 2015

Entertainment/Gaming

  • DISH
  • Disney
  • Epic Games
  • ESPN
  • Industrial Light & Magic
  • LiveNation – TicketWeb
  • Live Gamer
  • Lucasfilm
  • MTV (and Viacom by extension)
  • NBC Universal
  • Netflix – Asgard, their Amazon EC2 cloud management app, now open sourced.
  • Riot Games (makers of League of Legends)
  • Rumble Entertainment (maker of Kings Road)
  • Time Warner Cable

Finance

  • American Express
  • Bank of the West
  • Citi
  • Credit Suisse
  • Federal Reserve Bank of San Francisco
  • Federal Reserve Bank of Kansas City
  • Fidelity Investments
  • First Data Corporation
  • Freddie Mac
  • H&R Block – mobile site (now defunct)
  • Intuit – Intuit Payment Network
  • Janus
  • JPMorgan Chase
  • MoneyGram
  • OppenheimerFunds
  • PayPal – Bill Me Later
  • UBS Wealth Management
  • Visa
  • Wells Fargo

Retail/Ecommerce

  • 7-11 (back office systems)
  • Big Lots – core site
  • Jostens
  • Macy’s
  • Nordstrom (HauteLook)
  • Panera Bread
  • Target
  • Walmart – MP3 streaming site (now defunct)

Tech

  • Apple
  • Atlassian
  • Cloudera
  • CollabNet – web interface to SVN server
  • CyberSource
  • Dell – Enstratius
  • Ericsson
  • HP – Application Information Optimizer
  • IBM – Kenexa
  • IEEE
  • Infusionsoft
  • Intel – New Devices Group
  • Nokia
  • Qualcomm
  • Rackspace
  • Rally Software
  • Seagate
  • Symantec
  • T-Mobile – internal apps
  • Toshiba- Toshiba America Business Solutions

Other

  • CARFAX
  • Commission Junction (part of Conversant)
  • CondĂ© Nast
  • eHarmony
  • GEICO – mobile web
  • HealthPartners – Virtuwell
  • Humana
  • Infusionsoft
  • Jet Propulsion Laboratory
  • Johnson & Johnson
  • LifeLock
  • LinkedIn – Recruiter application
  • Lockheed Martin
  • Lyris
  • NetJets
  • Nissan – Leaf site
  • Orbitz
  • Pacific Gas and Electric Company
  • Precor – Preva product
  • Raytheon
  • ReachLocal
  • Reinsurance Group of America
  • Time Inc.
  • Trip Advisor
  • UnitedHealth Group
  • ValueClick
  • Virtual Tourist – core site (parent is Trip Advisor)
  • Volkswagen – UK site
  • The Washington Post
  • Wired.com
  • The XO Group (parent of The Knot / WeddingChannel.com)
  • Zynx Health (Hearst Health Group)
These are household names in some countries outside the US:
  • MACSF – (France) insurance company, internal systems
  • Mercado Libre – (South America) core site
  • National Film Board of Canada
  • Sky – (UK) TV listings, movies, and showbiz sites
  • Vodafone Music – (UK) core site
Colleges and universities:
  • Carnegie Mellon University
  • Colgate University
  • Columbia University
  • Harvard University
  • NYU (Langone Medical Center)
  • SUNY
  • UCSF
  • University of Alaska
  • University of California, Irvine
  • University of Hawaii
  • University of Minnesota
  • University of New Mexico
  • University of Oklahoma
  • University of Pennsylvania
  • University of San Francisco
  • Yale University

There are lists of many lesser known companies/apps found here and here.3 Another good place to check is Find Grails Jobs and indeed.com. I used all of them to compose this piece.

  1. Tiobe, job listings, etc. []
  2. Granted, I can only think of 3 others, Play, Roo, and Seam, but full stack is where you get the most productivity. []
  3. Apologies in advance that I’m intentionally leaving off startups and smaller companies. That’d be too much to manage and it won’t sway managers and architects, which is the point of this piece. []