DevOps Zone is brought to you in partnership with:

Kris Buytaert is a long time Linux and Open Source Consultant. He's one of instigators of the devops movement, currently working for Inuits He is frequently speaking at, or organizing different international conferences and has written about the same subjects in different Books, Papers and Articles He spends most of his time working on bridging the gap between developers and operations with a strong focus on High Availability, Scalability , Virtualisation and Large Infrastructure Management projects hence trying to build infrastructures that can survive the 10th floor test, better known today as the cloud while actively promoting the devops idea ! Kris is a DZone MVB and is not an employee of DZone and has posted 27 posts at DZone. You can read more from them at their website. View Full User Profile

How I like my Java

12.20.2011
| 12514 views |
  • submit to reddit

This is a repost of my article earlier posted at Jordan Sissel's awesome SysAdvent

 

Be sure to tell us how you like your Java in our latest Ask DZ post


After years of working in Java-based environments, there are a number of things that I like to implement together with the teams I`m working with - the application doesn't matter much, whether it's plain java, Tomcat, JBoss, etc, these deployment strategies will help your ops and dev teams build more managable services.

Packaging

The first step is to have the native operating system packages as build artifacts rolling out of your continuous integration server - No .ear, .war or .jar files: I want to have rpms or debs. With things like fpm or the maven rpm plugin this should not be an extra hassle, and the advantages you get from doing this are priceless.

What advantages? Most native package systems support dependency resolution, file verification, and upgrades (or downgrades). These are things you would have to implement yourself or cobble together from multiple tools. As a bonus, your fellow sysadmins are likely already comfortable with the native package tool used on your systems, so why not do it?

Proxied, not running as root

Shaken, not stirred

Just like any other daemon, for security reasons, I prefer to run Tomcat or JBoss as its own user, rather than as root. In most cases, however, only root can bind to ports below 1024, so you need to put a proxy in front. This is a convenient requirement because proxying (with something like Apache) can be used to terminate SSL connections, give improved logging (access logs, etc), and provides the ability to run multiple java application server instances on the same infrastructure.

Service Management

Lots of Java application servers have a semi functional shell script that allows you to start the service. Often, these services don't daemonize in a clean way, so that's why I prefer to use the Java Service wrapper from Tanuki to manage most Java based services. With a small config file, you get a clean way to stop and start java as a service and even the possibility to add some more monitoring to it.

However, there are some problems the Java Service wrapper leaves unsolved. For example, after launching the service, the wrapper can return back with a successful exit code while your service is not ready yet. The application server might be ready, but your applications themselves are still starting up. If you are monitoring these applications (e.g for High Availability), you really only want to treat them as 'active' when the application is ready, so you don't want your wrapper script to return, "OK," before the application has been deployed and ready. Otherwise, you end up with false positives or nodes that failover before the application has ever started. It's pretty easy to create a ping-pong service flapping scenario on a cluster this way.

One application per host

I prefer to deploy one application per host even though you can easily deploy multiple applications within a single Java VM. With one-per-host, management becomes much easier. Given the availability and popularity of good virtualization, the overhead of launching multiple Linux VM's for different applications is so low that there are more benefits than disadvantages.

Configuration

What about configuration of the application? Where should remote API urls, database settings, and other tunables go? A good approach is to create a standard location for all your applications, like /etc/$vendor/app/, where you place the appropriate configuration files. Volatile application configuration must be outside the artifact that comes out the build (.ear , .jar, .war, .rpm). The content of these files should be managed by a configuration management tool such as puppet, chef, or cfengine. The developers should be given a basic training so they can provide the systems team with the appropriate configuration templates.

Logs

Logs are pretty important too, and very easy to neglect. There are plenty of alternative tools around to log from a Java application: Log4j, Logback, etc .. Use them and make sure that they are configured to log to syslog, then they can be collected centrally and parsed by tools much easier than if they were spread all over the filesystem.

Monitoring

You also want your application to have some ways to monitor it besides just checking if it is running - it is usually insufficient to simply check if a tcp server is listening. A nice solution is to have a simple plain text page with a list of critical services and whether they are OK or not (true/false), for example:

    someService: true
    otherService: false

This benefits humans as well as machines. Tools like mon, heartbeat or loadbalancers can just grep for "false" in the file. If the file contains false, it reports a failure and fails over. This page should live on a standard location for all your applications, maybe a pattern like this http://host / servicename/health.html and an example "http://10.0.129.10:8080/mrs-controller/health.html". The page should be accessible as soon as the app is deployed.

This true/false health report should not be a static HTML file; it should be a dynamically generated page. Text means that you can also use curl, wget, or any command-line tool or browser to check the status of your service.

The 'health.html' page should report honestly about health, executing any code necessary to compute 'health' before yielding a result. For example, if your app is a simple calculator, it should verify health by doing tests internally like adding up some numbers before sharing 'myCalculator:true' in the health report.

The 'health.html' page should report honestly about health, executing any code necessary to compute 'health' before yielding a result. For example, if your app is a simple calculator, then before reporting health it should put two and two together and get four.

This kind of approach could also be used to provide you with metrics you can't learn from the JVM, such as number of concurrent users or other valid application metadata for measurement and trending purposes.

Conclusion

If you can't convince your developers, then maybe more data can help: Check out Martin Jackson's (presentation on java deployments) Automated Java Deployments with RPM

With good strategies in packaging, deployment, logging, and monitoring, you are in a good position to have an easily manageable, reproducible, and scalable environment. You'll give your developers the opportunity to focus on writing the application, they can use the same setup on their local development boxes (e.g. by using vagrant) as you are using on production.


Source:  http://www.krisbuytaert.be/blog/how-i-my-java
Published at DZone with permission of Kris Buytaert, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Sandeep Bhandari replied on Tue, 2011/12/20 - 9:30am

I wish Java had AOP for configuring logging which would let us get rid of those logging statements in methods and that too without using any framework like Spring AOP.Tips to increase Java application performance

John David replied on Tue, 2011/12/20 - 10:40am

I want to comment on Service Management of web and application server. It has lot of issues like ports are already allocated.

 

My suggestion is that Tomcat and JBoss should have a pool of ports and it should first look for available ports instead of throwing any exception. T should automatically connected with the available port and at the end display connected port.

Log4J is very good to do logging with out any hactic. It provide ease of loggin with configurable options :)

Loren Kratzke replied on Tue, 2011/12/20 - 4:26pm in response to: John David

Port management is a sys admin responsibility. You need to configure each Tomcat service to run on a dedicated port so that Apache may proxy requests to the proper instance.

It is not even possible to encounter an unavailable port if you are using service wrappers and dedicated ports. And if you do, then you have bigger problems like a hung process.

Steven Baker replied on Tue, 2011/12/20 - 5:41pm in response to: Sandeep Bhandari

You could always put in compile annotations and apply some byte code manipulation in your build.

Mladen Girazovski replied on Wed, 2011/12/21 - 7:28am

I`m working with - the application doesn't matter much, whether it's plain java, Tomcat, JBoss, etc, these deployment strategies will help your ops and dev teams build more managable services.

...

 The first step is to have the native operating system packages as build artifacts rolling out of your continuous integration server - No .ear, .war or .jar files: I want to have rpms or debs. With things like fpm or the maven rpm plugin this should not be an extra hassle, and the advantages you get from doing this are priceless.

 I'm really curious to learn what use a rpm or deb file is for a windows client or server.

I think your statement is to general, it might be a valid option for server applications that run on linux servers, but it certainly does matter if the application is a "plain java" application for clients or something else like a Server App.

Apart from that, you're missing out some of the server provided features for EARs and WARs if you choose another format, but that depends on the admins. More than once i've witnessed an administrator trying to manage a Java Server App (WAR/EAR) with linux tools only...

 

 

Jammer Man replied on Wed, 2011/12/21 - 3:30pm

RPMs ?   LOL.  Oh wait ... you're serious?  Sorry ... anyone recommending RPM's is not someone I take seriously.

 

Steven Baker replied on Wed, 2011/12/21 - 6:02pm in response to: Jammer Man

Jammer, I am wary of it too. But I would like to try it out before I judge.
I think it might be useful for the versioning and dependency control (upgrading Tomcat, etc).

Still, I think it might limit how the application is layed out and deployed. But this might be a good thing, just like obeying the conventions of Maven suits in the long run.

Philip Andrew replied on Wed, 2011/12/21 - 9:31pm

Hi there,

You don't need a proxy in front to get port 80 to your Tomcat, JBoss.

Use iptables port forwarding. There are many articles on how to do this.

http://fclose.com/b/linux/816/port-forwarding-using-iptables/

 Thanks!

Philip Andrew replied on Wed, 2011/12/21 - 9:56pm

I wouldn't do this in a company for the reason that the employee's have to understand rpm and the deployment process, which costs more money for them to learn. (time = money).

I would get the employee's to use something java based and popular, so when I hire new employee's, its likely they already understand it. 

Liran Mendelovich replied on Thu, 2011/12/22 - 3:29am

Hye

Can you, or anyone else, please give details on how is this monitoring being done ?

Like you said, suppose I have 3 different servers, and each of them runs 2 different processes (web services which listen to some port generally), and I want to create one place, e.g. a file, which contains true / false values for each service, whether it is running or not. And notify me when there is 'false' somewhere.

Thanks

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.