Unfortunately, it’s not so good when it comes to generating moderately complex XML files. By moderately complex, I mean lists of lists like this:

<products>
  <product id="1">
    <categories>
      <category id="1" />
      <category id="2" />
    </categories>
    <skus>
      <sku>12345</sku>
      <sku>67890</sku>
    </skus>
  </product>
<products>

Talend can do this, it’s just obscenely slow for larger file sizes. By “larger” I mean a few MB. It appears this is due to their use of DOM4J instead a SAX parser. Why a few megs of XML data takes up so much memory I don’t know, but that’s the case.¹

Talend converts columnar data to XML, so you have to do it in two passes:

Products/Categories (category is the loop element):

<products>
  <product id="1">
    <categories>
      <category id="1" />
      <category id="2" />
    </categories>
  </product>
<products>

Products/SKUs (sku is the loop element):

<products>
  <product id="1">
    <skus>
      <sku>12345</sku>
      <sku>67890</sku>
    </skus>
  </product>
<products>

I noticed when when generating these files individually, Talend is really fast, around 1,000 rows/second on my machine.² But when you instruct it to combine the files (really append the second file to the first, joining on the product ID), it slows down to about 1 row/sec for a 5,000 row job. Yes, 1,000 times slower. Again, it’s all due to the file size, as when I restricted it to 350 rows it ran at ~120 rows/s. The problem was that in production I need to process about 18K rows and it gets exponentially slower.

The solution is to generate two separate files, then merge them using a SAX parser. I’m just starting to use Groovy, which Talend supports, and was assuming it would be faster to develop in that language over Java. Well, if it didn’t require a ton of trial and error to overcome poor documentation, maybe it would have. Hopefully this heavily commented code makes it easier for the next person.

package com.madeupname
package com.madeupname

import groovy.util.slurpersupport.GPathResult;
import groovy.xml.StreamingMarkupBuilder;

// If there is no match in the products/categories file, use this empty node
def emptyCategoriesXML = '''<sites>
<categories>
	<category />
</categories>
'''

// Uses a SAX parser, less memory and overhead than a DOM parser (XmlParser)
// parse() method returns a GPathResult, which allows you to traverse and
// manipulate  an XML file or snippet using dot (.) notation.
def xs = new XmlSlurper()
// Main products file with SKU list
def productsSKUs = xs.parse(new File('/Data/ProductSKU.xml'))
// Products file with list of categories
def productsCategories = xs.parse(new File('/Data/ProductCategory.xml'))
def emptyCategories = xs.parseText(emptyCategoriesXML)
// Output file
File output = new File('/Data/Products.xml')

// Store category nodes into a Map for fast retrieval later. Key is product ID.
HashMap<String, GPathResult> productsCategoriesMap = new HashMap<String, GPathResult>()
// Note: if you're using Talend, you can't statically type this and must use this:
// def productsCategoriesMap = new HashMap<String, GPathResult>()
// Loop through all the products. Note that the root category is products
// (plural), but the GPathResult you get from XmlSlurper assumes you're already
// in the root category. That's why it's not productsCategories.products.product.each
productsCategories.product.each {
	// Note you must put the id in a String (Groovy style shown here)
	// in order to have a String key.
	productsCategoriesMap["${it.@id}"] = it
}

// This allows you to use a DSL to write the file. Note that you are not
// actually doing the work specified in the closure until you start writing it.
new StreamingMarkupBuilder().bind {
	// mkp is a special markup namespace for use within this closure. There
	// are other methods as well, see the docs.
	mkp.xmlDeclaration(["version":"1.0", "encoding":"UTF-16LE"])
	// My root category
	products {
		// Loop through each product and append (insert) the categories
		// node to the product node with the same product id.
		productsSKUs.product.each {
			if (productsCategoriesMap["${it.@id}"] != null) {
				it.appendNode(productsCategoriesMap["${it.@id}"].sites)
			} else {
				it.appendNode(emptyCategories)
			}
			// Note this is not System.out, it merely ensures the
			// GPathResult is printed when written.
			out << it
		}
	}
// Here we actually write the file, executing the above closure.
// Note how I specify the character set to match the declaration.
} .writeTo(output.newWriter("UTF-16LE"))

For Talend users, I had to use tGroovyFile instead of tGroovy because it was complaining about a missing library.³ You’ll also want to change the hard coded file paths to use context variables.

To save you some time, here are the links to the relevant documentation:

http://groovy.codehaus.org/api/groovy/util/XmlSlurper.html
http://groovy.codehaus.org/api/groovy/util/slurpersupport/GPathResult.html
http://groovy.codehaus.org/gapi/groovy/xml/StreamingMarkupBuilder.html
http://groovy.codehaus.org/api/index.html?groovy/xml/MarkupBuilderHelper.html

Apologies for complaining about the docs, but I like Groovy and want to see adoption spread. To do that, it has to make the hard things easy. None the docs I read (above JavaDocs, all the XML walk throughs on the official site, and relevant chapters of Programming Groovy) went go beyond the basics of generating XML from scratch (and the JavaDocs are particularly lacking). Groovy could really benefit from a good cookbook site (maybe nowadays that’s Stack Overflow) and most of all, annotated API documentation like PHP has had for years. I found those user contributed notes to be priceless when I was learning it. I think a wiki with comments would be a great home for the Groovy API reference docs.

1. I note that loading an 8MB XML file has caused Chrome to use tons of memory and crash the tab, so there must be a complexity I’m missing.
2. Core i7 quad 1.6, 6GB RAM, SSD.
3. Yes, I tried adding tLibraryLoad, but it didn’t help and I got no response on the forums.

A good friend of mine was asking for advice about business cards. He was going to be traveling in Europe for 6 weeks, meeting a ton of people, and wanted something that would stand out, something creative and memorable. But he also didn’t have a lot of time. My answer was simple. First, when you’re traveling for an extended period of time, hitting a lot of locations, you want to keep it light. The last thing you need is to lug around is a box of business cards.

My advice was to create a single, sturdy business card that simply had a QR code on it. People would scan it with their phones and you’d take it back. It’s both memorable and green, which I think a lot of Europeans would respond positively to. Especially the, ahem, female Europeans whose acquaintance he wanted to make. Moreover, it goes right into their contacts, saving them the trouble of transferring it, which won’t happen if it gets lost (even at the bottom of a purse).

Then another thought hit me – why do you need the card? You can have it as an image on your phone! Their phone photographs your phone and you’re done. To test this out, I went to an online QR code generator, capable of making a vCard/meCard. I took out my relatively new HTC Evo 4G and photographed the screen.

Nothing happened.

Turns out, even though Japanese cell phones have had built-in QR code readers for several years,¹ Google and Apple still want you to download a separate barcode reader app for this.² I’ve been seeing these codes all over the place: business cards, movie posters, real estate signs. I’m sure you have as well, although maybe you didn’t notice them or know what they were called. I was quite surprised to learn that they all rely on a 3rd party app.

All you need to read a QR code is a camera and bit of processing power. There are several free readers for iPhone, Android, Blackberry, Windows Phone 7, Palm OS, and probably several others. You can create several images representing each virtual business card you want to share: work, personal, work + personal, etc.. You can store them in a folder in your photo gallery app, or use it as your phone’s background image so it can be viewed and captured without even unlocking the phone.

Breaking Down The Branding Defense
I know, many people use business cards as part of their branding. My brother David is a talented graphic designer who has done this for many clients. But as I sort through a stack of about 30 collected business cards, very few people are doing this. What I’m seeing:

• 1/3 are nice. Here I include the traditional Fortune 500 business cards. Those have good layout, fonts, print and paper quality, and enforce the brand message (solid, traditional), but don’t differentiate them. Maybe 2 or 3 actually looked kinda cool, but none blew me away.
• 1/3 are meh. They don’t look like they were created by experienced designers, more like professional amateurs. Or, as is often the case, the client’s choice overruled the designer’s.
• 1/3 are just bad. Cheap paper and printing, ugly design. These actually hurt the person who hands them out.

While it’s a small sample size, it feels about right. Most people think their card helps them, but most people are wrong. The best defense for paper business cards is that your potential clients are primarily dumbphone owners. For most professionals, that’s a pretty small group. If you take down their info instead of giving yours, you gain a measure of control over the transaction. If that’s not feasible, you can ask your designer about small batch printing and eco-friendly materials.

In contrast, what does the paperless business card say about you or your company? At a minimum, it says you’re tech savvy, even cutting edge, and that you’re environmentally conscious. Some don’t care about the environment, but I can’t think of cases where that mindset makes you look bad.

Resources
You’re sold, right? So, where to go from here? First, make one or more QR codes. You can do that here:³

QR Code and 2D Code Generator
Allows you to create many different QR codes, including vCards and meCards. You can also choose format. I chose PNG (an image file, like GIF or JPEG), then saved it and mailed to to my phone.

ZXing QR Code Generator
From the maker of the free, open source Barcode Reader app. I tested this with a couple generators and it works fine with contacts. It also generates QR codes for you from your contact list, although I don’t know how compliant they are with the vCard or meCard formats.

Google Infographics
Allows developers to create QR codes with a simple HTTP GET or POST request.

Google will find you many more options. After that, you need to find one for your phone. Instead listing them here, just go to your favorite app store. There are many quality, free apps to choose from. My only caveat is that I first tried Google Goggles and discovered it can’t read embedded phone numbers (I tried two different generators, and both vCards and meCards). Pretty major limitation for getting contact info.

How to Help
If you want to help, request that your phone maker or carrier provide this feature natively.

Android barcode reader integration – You can directly vote for this feature in Android by “starring” this request.

iPhone Feedback Form – Ask Apple for QR code reader to be integrated into the camera.

Postscript: Alternatives
There are alternatives to QR codes. One is near field communication (NFC), but most phones, including my relatively new HTC Evo 4G, don’t have an NFC chip/antenna. Another is the app Bump. Bump’s mechanism is extremely clever, and the company appears to have some brilliant, highly credentialed people working there. I think it’s a good idea and it’s on my phone. However, it’s only available for iOS and Android, and the Android version is missing some critical features like multiple contact cards or custom contact cards (it uses Android’s contact info, which doesn’t have a field for your web site). And, of course, neither works with printed advertisements like movie posters.

1. This article, published in 2004, stated that 60% of all mobile phones came with QR code readers: http://www.hypulp.com/entries/more_qr_codes.php
2. Some carriers provide this. A friend of mine has a Windows Phone 7 device and AT&T bundled a barcode reader with it.
3. I’m not associated with any of these companies, people, or products.

Well, it had slipped my mind that I had updated my 3.6 eclipse.ini settings with those I had found in an excellent Stack Overflow answer from VonC on optimal JVM settings for Eclipse. It hasn’t been updated for 3.7 (nor does it mention STS), but after some experimenting and research it appears to work well for it. Here are my settings, and below I add some commentary on what they do, which is missing from the original answer (although I still suggest you read that, as it covers other situations/issues that may affect you). Keep in mind I’m not a JVM tuning expert, YMMV, etc. Here are the contents of my sts.ini:

-vm
C:/Java/SDKs/jdk1.6.0_24x64/bin/javaw.exe
-startup
plugins/org.eclipse.equinox.launcher_1.2.0.v20110502.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.100.v20110502
-product
com.springsource.sts.ide
–launcher.defaultAction
openFile
–launcher.XXMaxPermSize
384M
-vmargs
-Dosgi.requiredJavaVersion=1.6
-Xmn128m
-Xms256m
-Xmx768m
-Xss4m
-XX:PermSize=128m
-XX:MaxPermSize=384m
-XX:CompileThreshold=1000
-XX:+CMSIncrementalPacing
-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+UseFastAccessorMethods

-XX:CompileThreshold=1000
This is the number of method invocations/branches before compiling. This is normally set to 10,000, so we’re changing it dramatically, but the original suggestion was leading to errors so I raised it. You will notice on startup that it takes longer, and your CPU usage jumps. However, your performance after that is much better. Those 10s save times for web.xml? Gone after this. I’m willing to take a hit at the beginning for better productivity while coding.

-XX:ReservedCodeCacheSize=64m
Related to the above, I was getting the error “Unhandled event loop exception / out of space in CodeCache for adapters” due to setting the compile threshold to 5. This is another solution to that problem, and may be redundant.

-Xss4m
This is stack size, and was previously set to 1MB, now up to 4MB per thread. Doing this will increase the overall memory used.

-XX:+UnlockExperimentalVMOptions
-XX:+UseG1GC
-XX:+UseFastAccessorMethods
These enable parallel garbage collection. I saw my CPU utilization reach 100% after this, which is rare on a Core i7 Quad. It felt like I was finally using it to its potential.

Again, I’m not an expert. I’ve found it’s more sluggish at first, but response times quickly improve. For me, it’s a clear net gain. Not documented are the things I turned off in preferences because I wasn’t using them (Maven is disconnected, etc.). Visit Windows >> Preferences and filter on startup, see if there’s anything you can get rid of. Finally, I must give credit to my sources outside the original article:

http://ugosan.org/speeding-up-eclipse-a-bit-with-unlockexperimentalvmoptions/

http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

http://performance.netbeans.org/howto/jvmswitches/

1. I’ve had issues with things like web deployment assemblies, not prompting for workspace on startup, etc.

This is similar to telemarketers who give phone surveys and, because of some stupid rule set up by their management, must tell you what the numbers 1 through 5 represent for every single question. At that point, I’m thinking 1 for slightly annoyed, 2 for really annoyed, 3 for angry, 4 for hanging up right now…

And offering a chance of winning a single $100 Amazon gift card (which seems to be a new survey standard) is really no incentive at all. If you really want to incentivize, why not say 100 people will get a free month of Rhapsody to Go? Wouldn’t that improve your image without costing you much, since it’s your survey to begin with?

Look, for many topics, I’m a guy who actually cares. I’m happy to give you my opinions and insights. Please stop making me care less.

I’m looking for a clone of the Tivo Wishlist. The difference is that instead of recording, you get email alerts. I imagine if you have a DVR/PVR that is internet programmable, the service could take advantage of that, but I’ve got my cable company’s DVR (Scientific Atlanta) like most people and must program it with the remote. So this provides a wishlist feature for everyone without a Tivo, which I think is compelling.

The search features of current TV listings sites are missing critical fields for a wishlist to work (not to mention the email reminder part). Filtering (both inclusive and exclusive) by genre and channel are required.

Here are a couple strong (IMHO) use cases:

• You want to be notified if anyone on a list of people is scheduled to be on a talk show. You enter description:”Quentin Tarantino, Kevin Smith, Judd Apatow” and genre: talk and every time any of them appear on a talk show you’re notified. If Tarantino’s Reservoir Dogs is played on HBO, nothing happens.
• You’re planning a vacation and you want to record travel shows about various places. You enter keywords:”Prague,Tokyo,Paris” and interest:travel (or perhaps channels:travel,discovery,tlc,pbs) and you get notified for any travel shows relevant to you.

Of course, the above would be done via a nice GUI/query builder.

When you get your email, there would be links to hide/exclude shows in the future, which is useful for anything that gets rerun frequently (especially basic cable shows).

You can monetize this through targeted ads, since the user is telling you what he/she wants.

Another service would be to send a post-mortem email that includes links to the shows you want on Hulu, YouTube, the network’s website, etc. after they’ve been uploaded.  At that point you’re much closer to a real Tivo service and could possibly charge for it. Possibly.

I should point out that Tivo’s own advanced search is great and includes categories (and subs) and is open to the public.

And if you are only interested in the talk show part, you can set a calendar reminder to check the talk show lineups page once a week. However, I’d much rather have something automated that allows me to set it and forget it.  I could probably whip up a script to parse that page and run it as a service/cron job to notify me when there’s a match, but still, it would only work for talk shows. And parsing poorly formed HTML is a pain.

No, the easiest solution is to convince someone else to implement it for me

Update: If you want to see Yahoo TV implement this, upvote it here.