Cloud Zone is brought to you in partnership with:

Trevor Parsons is Chief Scientist and Co-founder of Logentries, 'the log management and intelligence platform'. Trevor has over 10 years experience in enterprise software and in particular has specialised in developing enterprise monitoring and performance tools for enterprise systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland. Trevor is a DZone MVB and is not an employee of DZone and has posted 73 posts at DZone. You can read more from them at their website. View Full User Profile

How to Easily Get All Your Logs from Amazon Web Services EC2

10.23.2013
| 3238 views |
  • submit to reddit

This post was originally written by Benoit Gaudin.

Let’s say that you, like many of your colleagues, are hosting your application on Amazon’s EC2 cloud infrastructure. You’re chugging along at a steady rate of growth when BAM! One day you get a spike of traffic and have to scale up quickly. “Good job,” you think as you pat yourself on the back in your mind, “this choice to host in the cloud means we can easily handle this load spike without a problem. We’ve set it to auto-scale, so we’ll have all the instances we need.” But is everything all good? How are you going to get logs off of those new instances? Even if you’re under the impression that you only need logs when things go wrong, do apps tend to misbehave less or more under heavy load?

It used to be that using a 3rd party logging tool on Amazon’s EC2 cloud was tough. You’d have to manually discover your log files and set each one up to go into your solution of choice. If you cloned an instance or spun up new instances to scale up, you’d then have to go and do the process again for the new ones. One of the main reasons to us Amazon Web Services has always been the ability to scale effortlessly…so why does such a critical piece of your app such as logging have to be so hard?

Well today we’re happy to tell you that it doesn’t have to be that hard anymore! With the Logentries AWS connector you can near-instantly detect all your EC2 logs and configure them to send data to Logentries. All your logs in one place. Easily. With alerts, search, tags, graphs and more.

What does it do?

Le_aws is a command line tool that makes it possible to automatically generate RSyslog configuration files and deploy them onto your Linux EC2 instances. Install it from our script, enter your AWS IAM credentials and Logentries account key. That’s it! Then simply run aws_setup to automatically forward your EC2 instance logs to Logentries.

Screenshot_AWS_Setup_Log

What do I need?

A computer running Linux (apt-get or yum required) or Mac OS X (Xcode and command line tools required). Python >= 2.6 is also required as well as python development headers and tools (for Linux users this can be installed from our installation script if it is not already available on your system). Finally, the AWS key pairs associated to your EC2 instances need to be available on your computer.

From there, our installation script installs le-aws commands in a directory called le_aws (created in the folder where the installation runs). Nothing is installed globally in your system. So if you wish to un-install it, simply remove the le_aws directory!

How does it work?

Our AWS conne ctor uses your AWS credentials in order to retrieve basic information about your EC2 instances (whether it is running or not, the name of the associated key pair, etc). For each running instance le-aws looks for the corresponding key pair on your machine (in ~/.ssh by default but a list of directories can be provided instead) and creates a SSH config like configuration file.

Le-aws uses this file to connect to your EC2 instances and retrieve paths to your log files (under /var/log by default but different paths can be provided). From there, an RSyslog configuration file is automatically generated for you and deployed to your EC2 instances. In the meantime, corresponding hosts and logs are created in Logentries and you simply need to log into your account to start seeing data coming in!

Screenshot_AWS_Provisioning

Where do I go from there?

By default, the connector assumes that your EC2 key pairs are in your ~/.ssh directory and configures RSyslog on all the EC2 instances so that all the log files located under /var/log are forwarded to Logentries. You can keep these default settings at first, look at the output in Logentries and decide on how to refine them then. Update the le-aws configuration file, run le-aws again and the changes will be automatically picked up!

So what are you waiting for? Get your AWS EC2 instances set up for logging on Logentries and make your life a little bit easier.
Published at DZone with permission of Trevor Parsons, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)