Libraries, libraries everywhere

[jducoeur]

So on the one hand, I notice that the WAR file for CommYou is now 12,345 K in size. The numbers geek in me find this pointlessly entertaining.

OTOH -- 12 *Meg*?!? Jeez, it is easy for an app to bulk up these days. I mean, IIRC that's something like twice the code size of Thief, a project that consumed many programmers for many years.

Of course, the actual code for CommYou is only a tiny, tiny fraction of that. The rest is all libraries. We're currently including, what? -- a cursory inventory finds 38 libraries so far, ranging from a few dozen K to the better part of a Meg. Libraries for XMPP, for XML, for logging, for JSP, for DB, for all *their* dependencies, and so on. Indeed, I'd be surprised if we're even calling 5% of the code that's included here, since in most cases I'm only using a small chunk of a big library.

It's not really a big deal, of course, but the old-fashioned programmer in me is kind of appalled. (And uploading the bloody WAR does take a fair while.) Here's a question for those more hip to Java development than I -- is there any sort of mechanism for trimming all this bulk down? In the old C++ days, the linker would trim away the paths that were never called. (Indeed, occasionally causing me problems in doing so.) Is there anything comparable now, or does everyone just live with library bloat?

Comments

[doubleplus.livejournal.com]

I've never dealt with that specifically, I guess because my apps that used lots of libraries were deployed in-house. I do vaguely recall going through some of them and figuring out entire lower-level dependent libraries that I didn't need at runtime because I wasn't using the functionality in the upper-level libs that required them, but I think that was more of a trial-and-error process, not anything systematic.

It looks like IBM AlphaWorks used to have a tool (http://www.alphaworks.ibm.com/tech/jax) to do just that, but they seem to have incorporated it into WebSphere and then made it impossible to find out anything about it, and messed up their website so that most of the links from the original page inexplicably take you to a page about their "Lotus Expeditor" customer care application. Sigh.

This tool (http://www.zelix.com/klassmaster/index.html) appears to have a 'trim' function that claims to do that, as does this one. (http://www.retrologic.com/retroguard-main.html) Apparently the only tools that include this function are code obfuscators.

[jducoeur]

Hmph. I wonder why this isn't a common tool? I mean, a naive version of it would seem to be straightforward to write: figure out the root entry points of the program, recursively trace the classes that actually get referenced, and slice away any that aren't. That wouldn't eliminate *all* of the code, of course, but it certainly seems like it would get a lot.

I guess one explanation is dynamic class loading. That isn't super-common, but it's certainly not hard to do in Java, and would interact extremely badly with such a tool. Since there's always a danger that your constituent JAR files might contain dynamic loading, the tool can't be used safely enough to be worth it...

[doubleplus.livejournal.com]

That's definitely one factor. At least one of those tools claims to go through configuration files looking for class names to include in the package. I suspect the main culprit is that because everything is dynamically linked, the mindset of what parts are "the application" isn't as strong as it is in statically linked languages.

Yeah, for web applications, the WAR is a definite unit and includes some configuration, but for standalone Java applications, packaging as a JAR is just an optional convenience. The only real drawback is upload/download time, and while that's a pain for you if it's in the development cycle, it's a one-time thing for a user downloading an application, and not even that for a user of a webapp. So it becomes much more of a specialized tool than a traditional linker.

[rickthefightguy.livejournal.com]

By WAR, do you mean the new warhammer MMO?

[jducoeur]

"WAR" == "Web ARchive". It's the standard file extension for, essentially, an application that runs on a web server...

Upload times

[metageek.livejournal.com]

(And uploading the bloody WAR does take a fair while.)

Dumb question: Can you upload just your own jarfile and build the WAR on the server? (I haven't done any J2EE.)

Re: Upload times

[jducoeur]

In principle, probably, but I don't love the approach -- it makes deployment significantly more complex. Currently, I simply add the libraries on my dev machine and check them into SVN along with everything else; when I build on the build machine, it all gets pulled together into one neat bundle that gets shipped up to the live site in Texas.

To just upload my own JAR would require that the libraries get shipped up as a separate matter, and automating that reliably so that it doesn't require too much manual intervention and updates probably when versions change -- well, it would almost certainly take much longer than the time I'm going to spend waiting for the WARs to upload in the next year...

Re: Upload times

[metageek.livejournal.com]

You could put the libraries into version control; then the deployment server could just do an update before building the WAR.

Personally, I'd go that way, rather than trusting a tool to strip out just the right bits.

Re: Upload times

[jducoeur]

Well, there are other costs as well, of course -- for instance, the memory footprint of the application is presumably much larger than I'd like. (And Java isn't always kind to large memory footprints, especially inside Tomcat.)

But it's a possibility, at least eventually. For the time being, though, it would still be a pretty considerable deployment rewrite, and I have bigger fish to fry...

Re: Upload times

[doubleplus.livejournal.com]

I could be wrong, but I would kind of doubt that there's much relationship between library bloat and memory footprint, actually. I think Java classloaders are pretty smart about only loading what you're using. From the point of view of the classloader, it doesn't much matter whether it's individual class files on disk or a library jar or a library jar within a war; the class is the basic unit that gets dynamically loaded when it's used.

[metahacker.livejournal.com]

Seems like you ought to be able to, but the only package I could find was this:
http://www.retrologic.com/rg-docs-trim.html

Now, they do have a free GPL'd library, but the commercial version is slightly pricey.