Enterprise Integration Zone is brought to you in partnership with:

Nicolas Frankel is an IT consultant with 10 years experience in Java / JEE environments. He likes his job so much he writes technical articles on his blog and reviews technical books in his spare time. He also tries to find other geeks like him in universities, as a part-time lecturer. Nicolas is a DZone MVB and is not an employee of DZone and has posted 229 posts at DZone. You can read more from them at their website. View Full User Profile

Selecting Development Infrastructure

08.06.2009
| 11790 views |
  • submit to reddit

This week, I was tasked to create a development infrastructure with the following components:

I added 2 more components:

  • A database in order to store data from each application in the same datastore
  • A LDAP in order to authentify oneself in each application with the same credentials

Since we had no server to begin from, I had to install the infrastructure on my development machine, meaning Windows and 1 Gb RAM. The following is a personal feedback I have from this week.

Subversion

Subversion means Apache HTTP Server. Since I already installed Apache a few times, installing (and configuring it) one more time went like a breeze. I really love this software.

Likewise, once one’s got the Subversion installer, installing it and configuring the underlying Apache instance can be done in less than 1 hour.

Trac

Now comes the fun part. Trac was a choice not made by myself (I even didn’t know this product until this week). This choice was made from Trac’s features. Little did we know that Trac runs on Python.

I (and my company) have no Python skills, so please, don’t make fun of me because of what follows.

I installed Python and then configured Apache to use it. That was the easy part.

First lesson, Trac doesn’t run on the latest Python (v3.2). I learnt that the hard way.

Second lesson, Trac needs a whole stack of component connectors:

  • since Trac can display you Subversion repository, it needs to communicate with Subversion. It is done through Python so here come Python svn bindings. And if you installed Python v2.6 which is compatible with Trac, that’s too bad because bindings are only avalaible for Python 2.5
  • Trac uses SQLLite (first time I heard of this, I’m beginning to feel like a complete idiot) out-of-the-box to store its data. Since all the other applications can run on MySQL and I happen to know a little about MySQL, I wanted Trac to use MySQL (a logical step). Not surprisingly, Python needs drivers to use MySQL, like in Java. This is done through MySQL Python. Completely unlike in Java, drivers are not put in the classpath of the running application, but installed in the Python distribution. Python 2.6+ is not supported though Trac supports it. Life is fun.
  • Trac needs a installation component called setuptools that manages Python eggs (a specific format for Python extensions).
  • Trac uses a templating engine. Version 0.10 of Trac needed ClearSilver, version 0.11 needs Genshi. Since I used Trac v0.11 and completely forgot to install Genshi, it complained about… not finding ClearSilver. Thanks for the misleading error message, I lost about 1 hour.

Third lesson: in order to use Eclipse Mylyn (an Eclipse plugin that manages to connect to a bugtracker and displays issues in Eclipse), Trac needs the XML-RPC plugin to be installed. Using the previously installed setuptools, it only needs a command-line instruction.

Fourth lesson, a project is linked to an environment. An environment is created using a command-line tool and then configured in Apache.

Fifth lesson, it is impossible to create sub-projects though a patch exists to modify the code in order to do so.

Hudson

Hudson comes into two flavors:

  • Hudson standalone as a Java Web Start installer
  • a war file

The standalone runs nicely even though I can only guess at the underlying complexity. It can be configured to run as a Windows service. I am amazed: this is too good to be true. My deepest respect to the developement team. Finally, I uninstall Hudson (equally as easy) because…

I will have to run 3 Java applications: Hudson, Sonar and Nexus. If each runs in standalone mode, this means 3 JVM running concurrently and I’m bound to my crappy 1Gb RAM. Since each of these applications is delivered in a WAR archive, better have a lightweight server such as Tomcat (or Jetty). Since I’v used Tomcat since the dawn of time and since I see Jetty more like an embedded container, I installed Tomcat (this is a no-op, even for first time users).

Deploying Hudson in Tomcat is easy enough. Just don’t forget to configure Tomcat’s JVM with the -DHUDSON_HOME Java property. This tells Hudson to use this folder for storing its files (runs, builds, etc.).

Sonar

Sonar was deployed easily in Tomcat. I configured it to use MySQL and everything went fine. The only comment I have about Sonar is that it comes in a whooping 45+ Mb archive. I mentioned this to the development team: it will likely be curbed in future versions.

Nexus / Artifactory

Nearly finished I thought and went on to deploy Nexus. Deploying Nexus war in Tomcat went fine. Configuring Nexus went not. I wasn’t able to find a single line of documentation regarding using Nexus not run in standalone mode, even though it is available for download in war format. Having lost too much time with Track, I decided to use Artifactory instead. I already installed this Maven repository on a customer’s site and it gave me complete satisfaction.

Just don’t forget to configure Tomcat’s JVM with the -artifactory.home Java property. This tells Artifactory to use this folder for storing its internal files.

Yet, I ran into an error trying to use MySQL instead of JavaDB. Whatever the configuration I used, Artifactory was not able to use the password I provided to acces MySQL: I kept having Access denied for user 'artifactory'@'localhost' (using password: NO) messages. Having spent a considerable time trying to correct the thing, I went back to using JavaDB. Since Artifactory provides an repository backup functionality out-of-the-box, I can live with it.

Finally, having Artifactory up and running, I kept running into OutOfMemoryError: PermGenSpace errors. Tomcat was using 512Mb (which is a really low value for 3 important applications) but it has no effect on permanent generation space. I tried different -XX:MaxPermSize configurations without any luck. So in a bold move, I switched from a Sun JVM to a JRockit. Since then, I got no such errors anymore, though I have very high Garbage Collection time. Nothing a little more RAM and correct tuning can cure.

OpenDS

I wanted all my applications to use the same authentication component. I used OpenLDAP a couple of times before but it was always ankward to install and use (I’m no LDAP guru) . I browsed the web a little to find OpenSource and free (as a beer) LDAP servers and stumbled upon OpenDS. Easy to install, easy to configure and last but not least, easy to manage LDAP entities. Everything done through its GUI.

Configuring Apache and Tomcat to use LDAP is done through XML editing, for Artifactory and Hudson, it is done through the webapp. Artifactory even provides a test button to check whether the connection is correct.

Conclusion

All in all, I’m pretty happy with the components installed, save one. Trac does provide many functionalities but it has not its place in our infrastructure since Python is not my friend and it needs to many components I won’t be able to debug if we run across a problem. JIRA would be the bugtracker of choice, but since it is not free, it has no chance. I’ve already used MantisBT, and though it uses a PHP stack (and not Java), I feel more at ease with it. In its latest version, it can even integrate third party PHP Wikis. Any thought about it ?

References
Published at DZone with permission of Nicolas Frankel, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

David Gilbert replied on Thu, 2009/08/06 - 5:55am

JIRA would be the bugtracker of choice, but since it is not free, it has no chance.

Is that because you require open source licensing on principle, or just that you don't want to pay over any cash? If it's the latter, then I don't get it (unless JIRA is really really expensive, which is not the impression I have of that product).

Artur Biesiadowski replied on Thu, 2009/08/06 - 8:08am

You want JIRA Professional or Enterprise (Standard is too crippled for any serious work). For LDAP, you need Enterprise, same goes for per-component leaders and few other nice-to have features.

This means $2400 or $4800 license. Somehow I feel that if entire project infrastructure has to run on 1GB RAM developer machine, even $2400 might be a prohibitive cost.

Rodrigo Carvalh... replied on Thu, 2009/08/06 - 9:15am

Great post!

About bugtracking, did you tested Redmine? It's a new bugtracking tool that aims to be a "Trac better than Trac". I've been testing it and I'm enjoying very much!

Best regards!

Robert Enyedi replied on Thu, 2009/08/06 - 10:09am

Off topic: I can only imagine the money your company loses because of sticking to 1 GB of RAM: startup, compile, run, debug can take several times more if you are short on memory, not considering the effects of frustration.

In my opinion, these days a Java EE developer's machine should have at least 3-4 GB of RAM (thanks Sun), with the only limit being a 32 bit OS and the time/effort required to switch to the 64 bit version.

On topic: I wanted to try out Trac myself, but I was intimidated by the installation process. Thanks for sharing your experiences!

Mladen Girazovski replied on Thu, 2009/08/06 - 10:24am

Off topic: I can only imagine the money your company loses because of sticking to 1 GB of RAM: startup, compile, run, debug can take several times more if you are short on memory, not considering the effects of frustration.

 Yeah, save a Penny and spend a pound...

I'm using a very similar setup (Mantis/Maven2/Sonar/SVN/Hudson/Artifactory), had similar issues with trac, gonna look at Redmine, seems a like a great tool.

However, i stopped installing it on physical machines, i prefer virtual machines (currently vmware server [free], since the VMs run on Windows & Linux), that way, if you find out (which you probably will) that you'll need more resources, it's easy & fast to add more.

Andy Gibson replied on Thu, 2009/08/06 - 11:39am

Just out of interest, in terms of general development infrastructure, what about backup of at least the Subversion repositories? Personally, I've been using online subversion repos more and more as it lets me access work from anywhere, and someone else is (I hope) taking care of backups (hourly in my case).

 Cheers,

 Andy Gibson

Developer Dude replied on Thu, 2009/08/06 - 3:45pm

The experience with Trac doesn't surprise me - not because it is Trac (which I have heard good things about), but because there is a lot of software out there, especially OSS s/w, which has the same kind of installation headaches and dependency hell. If you think this is frustrating for an experienced developer, try walking in the shoes of a total computer neophyte. It shouldn't be this hard regardless of what the software is - it really doesn't need to be this hard, we can do it, we have the technology.

As for the 1GB of memory. Totally unrealistic constraint - unless you are totally broke, unemployed, starting some OSS project totally on your own and this is the only machine you have. Otherwise upgrade your RAM or get another machine that can be upgraded. If an employer asked me to do any dev task on a machine with only 1 GB of RAM I would be polishing off my resume right then and there - or, breaking out my credit card to buy my own machine. RAM is cheap - time is not.

BTW - I am just about ready to buy a 17" MacBookPro, now that it takes 8+ GB of RAM, but I am still holding out for 4 cores. My personal home dev machine, an 8 core MacPro, currently has 10GB of RAM - specifically purchased for its capacity for RAM so I can run various server processes in a realistic environment. It is not unusual for me to see memory consumption of 6 GB or more. I figure Eclipse itself needs at least 1 GB of RAM, Tomcat with any servlets at least 1 GB of RAM, most DBMSs at least 1 GB of RAM (SQL Server at least 2 GB of RAM), and then most modern OSes need at least 1 and preferably 2 or more GB of RAM, not to mention all the other things I might have installed, and not to mention having multiple instances of something running in a VM (itself a memory hog because you are running another copy of an OS) to test against (say how something works on SQL Server v. PostGreSQL v. mySQL). I am not talking about bare bones minimums, I am referring to reasonable configs where you get adequate performance and don't spend (waste) time stopping one process in order to start another.

Most employers will skimp at least somewhat on a dev computer purchase that, if they bought the right machine, would be less than what they pay a single dev in a month - usually much less. The amount they would save by not skimping would be recouped in less than a month. But go figure.

You have my sympathies. I guess the fact that they didn't give you a server to do this on should have been a clue.

Shantanu Kumar replied on Thu, 2009/08/06 - 4:05pm

SVN for SCM? Maybe you can look at Git or Mercurial. As a Trac alternative, there is jTRAC -- http://www.jtrac.info/

Ronald Miura replied on Fri, 2009/08/07 - 8:09am

Git/Mercurial IDE integration sucks hard. Sorry, command line isn't an option, period.

Subversion's isn't wonderful in this area, but at least there are working plugins.

Gilbert Herschberger replied on Fri, 2009/08/07 - 7:42pm

I appreciate how important and difficult it is to share information with other developers. It is important because a development environment determines your capacity. It is a limiting factory of what you are capable of producing.

It is difficult because a development environment is very personal. What works for one might not work for another. Developers are sometimes afraid of writing about it because it reveals something very personal.

One very important part is keeping a written record of what you are doing. How do you go about developing a development environment? How often do you examine your environment? How often do you replace it?

Your machine has become a de-facto server, part of the infrastructure. By coincidence, you happen to also use it for your development. There are well-documented benefits and costs of choosing a multiple-purpose machine. Life (and technology) is filled with difficult choices.

Brian Fox replied on Mon, 2009/08/10 - 11:23am

Nearly finished I thought and went on to deploy Nexus. Deploying Nexus war in Tomcat went fine. Configuring Nexus went not. I wasn’t able to find a single line of documentation regarding using Nexus not run in standalone mode, even though it is available for download in war format.

The configuration of Nexus is 100% done via the UI, so there is no difference between the configuration once you have it running in Tomcat vs the Standalone mode. Was there something in particular you were looking for that you couldn't find?

Nicolas Frankel replied on Wed, 2009/08/12 - 4:57pm

Thanks for your contribution, I will try to enlighten the points you raised the most / are the most pertinent regarding to what I wanted to tell:

  • First thing first: that was not a personal project but my company's
  • 1Gb sucks. Oh yes it does! But the point was to prototype an environment, not run on such a machine in a long run.
  • JIRA is not free, so we won't use it. Alas, I must say that the cost was a factor. I had no budget: no penny, no pound, no nothing. Only sweat and toil ;-). Products could be closed source but free (Oracle Lite would have fit... if not for PHP).
  • I looked at RedMine briefly, liked what I saw but had not time to test. I will put it under my 'To evaluate' bookmarks.
  • Virtualization is a very good idea and thought about it. Unfortunately, my conclusion was that was not compatible with my crappy configuration.
  • As for online SCM, it could be compatible with a personal or OpenSource development but not for a private or closed one. For me, it's a no go there. Just like putting companies' private data in the cloud. Feels like the beginning of a flame war, doesn't it?

Walter Bogaardt replied on Thu, 2009/08/13 - 6:22pm

Why not use Continuum for CI? It's simple works great with maven. Setup is no time. There are other uses beyond just simple CI. Carlos worked on a project for us doing just this and this is the result of his work. http://www.eu.apachecon.com/presentation/materials/171/Enterprise_Build_and_Test_in_the_Cloud_-_Carlos_Sanchez.pdf

 Oh try to do this infrastructure in .NET, yikes.

Matt Raible replied on Wed, 2010/01/13 - 12:02am

I wrote about my preferred development infrastructure stack. In short, I think there's a number of best-of-breed applications you can use to build a sweet development infrastructure stack.
  • Source Control: Git
  • Source Viewer: FishEye
  • Wiki: Jive SBS
  • Bug Tracker: JIRA
  • Continuous Integration: Hudson

Carla Brian replied on Sun, 2012/07/01 - 5:51pm

Thank you for these tips. I have the same problem as well. Good thing  I saw your post. - Garrett Hoelscher

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.