Unfortunately, it’s not so good when it comes to generating moderately complex XML files. By moderately complex, I mean lists of lists like this:

<products>
  <product id="1">
    <categories>
      <category id="1" />
      <category id="2" />
    </categories>
    <skus>
      <sku>12345</sku>
      <sku>67890</sku>
    </skus>
  </product>
<products>

Talend can do this, it’s just obscenely slow for larger file sizes. By “larger” I mean a few MB. It appears this is due to their use of DOM4J instead a SAX parser. Why a few megs of XML data takes up so much memory I don’t know, but that’s the case.¹

Talend converts columnar data to XML, so you have to do it in two passes:

Products/Categories (category is the loop element):

<products>
  <product id="1">
    <categories>
      <category id="1" />
      <category id="2" />
    </categories>
  </product>
<products>

Products/SKUs (sku is the loop element):

<products>
  <product id="1">
    <skus>
      <sku>12345</sku>
      <sku>67890</sku>
    </skus>
  </product>
<products>

I noticed when when generating these files individually, Talend is really fast, around 1,000 rows/second on my machine.² But when you instruct it to combine the files (really append the second file to the first, joining on the product ID), it slows down to about 1 row/sec for a 5,000 row job. Yes, 1,000 times slower. Again, it’s all due to the file size, as when I restricted it to 350 rows it ran at ~120 rows/s. The problem was that in production I need to process about 18K rows and it gets exponentially slower.

The solution is to generate two separate files, then merge them using a SAX parser. I’m just starting to use Groovy, which Talend supports, and was assuming it would be faster to develop in that language over Java. Well, if it didn’t require a ton of trial and error to overcome poor documentation, maybe it would have. Hopefully this heavily commented code makes it easier for the next guy.

package com.madeupname
package com.madeupname

import groovy.util.slurpersupport.GPathResult;
import groovy.xml.StreamingMarkupBuilder;

// If there is no match in the products/categories file, use this empty node
def emptyCategoriesXML = '''<sites>
<categories>
	<category />
</categories>
'''

// Uses a SAX parser, less memory and overhead than a DOM parser (XmlParser)
// parse() method returns a GPathResult, which allows you to traverse and
// manipulate  an XML file or snippet using dot (.) notation.
def xs = new XmlSlurper()
// Main products file with SKU list
def productsSKUs = xs.parse(new File('/Data/ProductSKU.xml'))
// Products file with list of categories
def productsCategories = xs.parse(new File('/Data/ProductCategory.xml'))
def emptyCategories = xs.parseText(emptyCategoriesXML)
// Output file
File output = new File('/Data/Products.xml')

// Store category nodes into a Map for fast retrieval later. Key is product ID.
HashMap<String, GPathResult> productsCategoriesMap = new HashMap<String, GPathResult>()
// Loop through all the products. Note that the roote category is products
// (plural), but the GPathResult you get from XmlSlurper assumes you're already
// in the root category. That's why it's not productsCategories.products.product.each
productsCategories.product.each {
	// Note you must put the id in a String (Groovy style shown here)
	// in order to have a String key.
	productsCategories["${it.@id}"] = it
}

// This allows you to use a DSL to write the file. Note that you are not
// actually doing the work specified in the closure until you start writing it.
new StreamingMarkupBuilder().bind {
	// mkp is a special markup namespace for use within this closure. There
	// are other methods as well, see the docs.
	mkp.xmlDeclaration(["version":"1.0", "encoding":"UTF-16LE"])
	// My root category
	products {
		// Loop through each product and append (insert) the categories
		// node to the product node with the same product id.
		productsSKUs.product.each {
			if (productsCategoriesMap["${it.@id}"] != null) {
				it.appendNode(productsCategoriesMap["${it.@id}"].sites)
			} else {
				it.appendNode(emptyCategories)
			}
			// Note this is not System.out, it merely ensures the
			// GPathResult is printed when written.
			out << it
		}
	}
// Here we actually write the file, executing the above closure.
// Note how I specify the character set to match the declaration.
} .writeTo(output.newWriter("UTF-16LE"))

To save you some time, here are the links to the relevant documentation:

http://groovy.codehaus.org/api/groovy/util/XmlSlurper.html
http://groovy.codehaus.org/api/groovy/util/slurpersupport/GPathResult.html
http://groovy.codehaus.org/gapi/groovy/xml/StreamingMarkupBuilder.html
http://groovy.codehaus.org/api/index.html?groovy/xml/MarkupBuilderHelper.html

Apologies for complaining about the docs, but I like Groovy and want to see adoption spread. To do that, it has to make the hard things easy. None the docs I read (above JavaDocs, all the XML walk throughs on the official site, and relevant chapters of Programming Groovy) went go beyond the basics of generating XML from scratch (and the JavaDocs are particularly lacking). Groovy could really benefit from a good cookbook site (maybe nowadays that’s Stack Overflow) and most of all, annotated API documentation like PHP has had for years. I found those user contributed notes to be priceless when I was learning it. I think a wiki with comments would be a great home for the Groovy API reference docs.

1. I note that loading an 8MB XML file has caused Chrome to use tons of memory and crash the tab, so there must be a complexity I’m missing.
2. Core i7 quad 1.6, 6GB RAM, SSD.

Well, it had slipped my mind that I had updated my 3.6 eclipse.ini settings with those I had found in an excellent Stack Overflow answer from VonC on optimal JVM settings for Eclipse. It hasn’t been updated for 3.7 (nor does it mention STS), but after some experimenting and research it appears to work well for it. Here are my settings, and below I add some commentary on what they do, which is missing from the original answer (although I still suggest you read that, as it covers other situations/issues that may affect you). Keep in mind I’m not a JVM tuning expert, YMMV, etc. Here are the contents of my sts.ini:

-vm
C:/Java/SDKs/jdk1.6.0_24x64/bin/javaw.exe
-startup
plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.100.v20110502
-product
com.springsource.sts.ide
–launcher.defaultAction
openFile
–launcher.XXMaxPermSize
384M
-vmargs
-Dosgi.requiredJavaVersion=1.6
-Xmn128m
-Xms256m
-Xmx768m
-Xss4m
-XX:PermSize=128m
-XX:MaxPermSize=384m
-XX:CompileThreshold=1000
-XX:+CMSIncrementalPacing
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+UseFastAccessorMethods

-XX:CompileThreshold=1000
This is the number of method invocations/branches before compiling. This is normally set to 10,000, so we’re changing it dramatically, but the original suggestion was leading to errors so I raised it. You will notice on startup that it takes longer, and your CPU usage jumps. However, your performance after that is much better. Those 10s save times for web.xml? Gone after this. I’m willing to take a hit at the beginning for better productivity while coding.

-XX:ReservedCodeCacheSize=64m
Related to the above, I was getting the error “Unhandled event loop exception / out of space in CodeCache for adapters” due to setting the compile threshold to 5. This is another solution to that problem, and may be redundant.

-Xss4m
This is stack size, and was previously set to 1MB, now up to 4MB per thread. Doing this will increase the overall memory used.

-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+UseFastAccessorMethods
These enable parallel garbage collection. I saw my CPU utilization reach 100% after this, which is rare on a Core i7 Quad. It felt like I was finally using it to its potential.

Again, I’m not an expert. I’ve found it’s more sluggish at first, but response times quickly improve. For me, it’s a clear net gain. Not documented are the things I turned off in preferences because I wasn’t using them (Maven is disconnected, etc.). Visit Windows >> Preferences and filter on startup, see if there’s anything you can get rid of. Finally, I must give credit to my sources outside the original article:

http://ugosan.org/speeding-up-eclipse-a-bit-with-unlockexperimentalvmoptions/

http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

http://performance.netbeans.org/howto/jvmswitches/

1. I’ve had issues with things like web deployment assemblies, not prompting for workspace on startup, etc.

I’m looking for a clone of the Tivo Wishlist. The difference is that instead of recording, you get email alerts. I imagine if you have a DVR/PVR that is internet programmable, the service could take advantage of that, but I’ve got my cable company’s DVR (Scientific Atlanta) like most people and must program it with the remote. So this provides a wishlist feature for everyone without a Tivo, which I think is compelling.

The search features of current TV listings sites are missing critical fields for a wishlist to work (not to mention the email reminder part). Filtering (both inclusive and exclusive) by genre and channel are required.

Here are a couple strong (IMHO) use cases:

• You want to be notified if anyone on a list of people is scheduled to be on a talk show. You enter description:”Quentin Tarantino, Kevin Smith, Judd Apatow” and genre: talk and every time any of them appear on a talk show you’re notified. If Tarantino’s Reservoir Dogs is played on HBO, nothing happens.
• You’re planning a vacation and you want to record travel shows about various places. You enter keywords:”Prague,Tokyo,Paris” and interest:travel (or perhaps channels:travel,discovery,tlc,pbs) and you get notified for any travel shows relevant to you.

Of course, the above would be done via a nice GUI/query builder.

When you get your email, there would be links to hide/exclude shows in the future, which is useful for anything that gets rerun frequently (especially basic cable shows).

You can monetize this through targeted ads, since the user is telling you what he/she wants.

Another service would be to send a post-mortem email that includes links to the shows you want on Hulu, YouTube, the network’s website, etc. after they’ve been uploaded.  At that point you’re much closer to a real Tivo service and could possibly charge for it. Possibly.

I should point out that Tivo’s own advanced search is great and includes categories (and subs) and is open to the public.

And if you are only interested in the talk show part, you can set a calendar reminder to check the talk show lineups page once a week. However, I’d much rather have something automated that allows me to set it and forget it.  I could probably whip up a script to parse that page and run it as a service/cron job to notify me when there’s a match, but still, it would only work for talk shows. And parsing poorly formed HTML is a pain.

No, the easiest solution is to convince someone else to implement it for me

Update: If you want to see Yahoo TV implement this, upvote it here.

My idea is for a directory of Java example projects.¹ It would make it easy to search projects by stack: IDE, tools, frameworks, libraries, and app server. Projects would include the minimum to show the integration points. The directory would also have basic social features: a way to vote up projects and a forum for discussing integration, setup, and architecture best practices. This is probably something a CMS expert could tackle fairly easily.

I admit there are a few problems with this idea. People expect to thrash around during project startup. They learn from it. When it’s over, they start doing real work. They probably don’t want to share their lessons learned. Those that do are typically evangelists for a particular framework. Spring, as an integration framework, comes to mind with its Pet Clinic and Pet Store examples. Hibernate’s Caveat Emptor is probably the best example app I’ve seen. AppFuse also demonstrates integration, although earlier versions seemed more explicit (the new version built with Maven seems to hide a lot of the details).

However, these apps don’t cover IDE configuration, and the IDE/build tool mismatch problem is where most of the difficulty lies. The most important IDE features, IMHO, are incremental building for syntax checking, and debugging. So you have this natural duplication of effort. The IDE/build script conversion tools (in either direction) seem narrow in scope; granted, I’m limiting myself to Eclipse/Flex Builder to/from Ant. I don’t know if full integration between the IDE and the build tool is the answer, but it would certainly go a long way in making things easier.

In the meantime, however, a project directory would be a welcome resource.

1. You don’t need to host the projects themselves when there are already sites like code.google.com and Source Forge.