Re-using an old laptop as monitoring screen

This article describes, how to set up an old laptop with a monitor connected to it as a two-screen monitoring dashboard

monitoring

We want to install Ubuntu 14.04 LTS on the laptop, assuming, that he doesn’t have a builtin CDROM anymore. So, we need to prepare an USB stick first:

1. Download initrd.gz and linux from http://archive.ubuntu.com/ubuntu/dists/trusty-updates/main/installer-i386/current/images/netboot/ubuntu-installer/i386/

2. Format the stick (assuming, it is /dev/sdb):

mkdosfs /dev/sdb1

3. Transform the stick into a bootable device:

syslinux /dev/sdb1

4. Mount the stick and copy the two previously downloaded files initrd.gz and linux into the root directory of the stick

5. Create (again in the root of the stick) a file syslinux.cfg with the following contents:

default vmlinuz
append initrd=initrd.gz video=vesa:ywrap,mtrr vga=788

6. Download your desired ISO image (e.g. ubuntu-14.04-desktop-i386.iso ) and copy it onto the stick

7. Install a master boot record on the stick with

install-mbr /dev/sdb

 

Now, let your laptop boot from the USB stick and follow the installation dialogue.

Be sure to tick the automatic security updates, if you don’t plan to manually check for updates.

After setting up the machine, there’s not much to do any more but installing a few additional packages, such as

  • openssh-server
  • google chrome
  • xdotool

The, enable auto-login for your user and disable any kind of screensaver.

Now, write a small shell script in ~/bin/monitoring:

#!/bin/bash

google-chrome --start-fullscreen --window-position=0,0 --window-size=1680,1050 --app=http://your.monitoring.page.one.html &
sleep 10
xdotool key F11
sleep 5
google-chrome --window-position=1680,0 --window-size=1680,1050 --app=http://your.monitoring.page.two.html &
sleep 5
xdotool windowfocus --sync
xdotool getactivewindow windowmove 1680 0
xdotool key F11

and insert that script into the list of autostart programs (“Startprogramme” in the dash) in unity.

As a further enhancement, you can add a cronjob, which shuts down the laptop at the end of your working day. This means, that on each working day, you have to power on the laptop manually, but due to the startup script, the windows will be placed automatically, and there’s nothing, you have to do manually.

Troubleshooting Java applications – Part 1: jmap

If you encounter abnormal behaviour in your Java application, Java offers you a set of tools to effectively identify the culprit. The first one, which I want to present, is jmap.

jmap

jmap, which is part of the JDK, is the tool of choice, if you get the impression, that you have a resource leak. With a simple command, you can create a histogram of all Java objects on the heap of your process:

jmap -histo PID

(where PID is the process ID of your Java process)

The output will look like this:

 num   #instances      #bytes  class name
----------------------------------------------
 1:       2031082   185199616  [C
 2:       2013868    64443776  java.lang.String
 3:        656635    21012320  java.util.HashMap$Entry
 4:        614312    14743488  de.christophlorenz.foo.Culprit
 5:         81443    12737392  <constMethodKlass>
 6:        123267    11564632  [Ljava.util.HashMap$Entry;
 7:         81443    11083656  <methodKlass>
...
3263:          1          16   sun.reflect.GeneratedMethodAccessor63
3264:          1          16   de.christophlorenz.foo.SomeFactory
Total    7182799   426077152

As you can see in number four, there are no less than 614312 instances of the Culprit class, consuming a little less than 15MB in total. It’s now up to you to decide, whether it’s the desired beaviour or not.

(Don’t worry about the large number of [C and [I entries, they are native character and ints, you will certainly use a lot.)

Now, as an other example, imagine, your RabbitMQ server is stressed by too many connections, maybe even blocking clients to connect, and someone has the suspicion, that your application might be running wild. With a simple jmap call, you can check your application and verify (or deny), that it is the cause:

jmap -histo 12345 | grep -i rabbitmq
 ...
 130:    14360   344640 [Lcom.rabbitmq.client.Address;
 131:    14356   344544 com.rabbitmq.client.Address
 188:     6250   100000 com.rabbitmq.client.impl.LongStringHelper$ByteArrayLongString
 212:      625    75000 com.rabbitmq.client.impl.AMQConnection$MainLoop
 226:      625    65000 com.rabbitmq.client.impl.AMQConnection
 235:      625    60000 com.rabbitmq.client.impl.ChannelN
 ...

Now, are you sure, you really need 625 connections to your RabbitMQ? No? Just go ahead and fix it 🙂

Of course, with jmap, you have many more options, like generating full heap- and thread dumps, which can later be analyzed by jvisualvm, but I’ll talk about that later.

Application monitoring with JMX and Jolokia

(or: It’s the inner values, that count)

Remember last time, when your application was all green in your monitoring suite, but you got complaints, because it did not do, what it was expected to? Or have you ever been in the situation, where you wanted to measure what your application does, without going through megabytes of logfiles? Do you need some KPI based monitoring? Don’t want to reinvent the wheel?

For any of these cases, the following monitoring approach, using the standard Java JMX approach together with Jolokia as HTTP bridge, will be perfect!

At first, let’s take a look at JMX, the Java Management Extensions:

JMX is a Java API for ressource management. It is a standard from the early days (JSR 3: JMX API, JSR 160: JMX Remote API), got some overhaul recently (Java 6: Merge of the both APIs into JSR 255 – the JMX API version 1.3) and since Java 7, we have the JMX API version 2.0. Basically, JMX consists of three layers, the Instrumentation Layer (the MBeans), the Agent Layer (the MBean Server) and the Distributed Layer (connectors and management client).

Although you can use JMX for managing virtually everything (even services), we just contentrate here on using JMX for monitoring purposes.The same we do for MBeans.

What are MBeans?

Generally spoken, MBeans are resources (e.g. a configuration, a data container, a module, or even a service) with attributes and operations on them. Everything else, like notifications or dynamic structures are out of scope for us now.

Technically, a MBean is a class, which implements an Interface and uses a naming convention, where the Interface name is the same as the class name plus “MBean” at the end:

class MyClass implements MyClassMBean

Now, let’s create a sample counting MBean:

public interface MyEventCounterMBean {
  public long getEventCount();
  public void addEventCount();
  public void setEventCount(long count);
}
package my.monitoring;
public class MyEventCounter implements MyEventCounterMBean {
  public static final String OBJECT_NAME="my.monitoring:type=MyEventCounter";
  private long eventCount=0;

  @Override
  public long getEventCount() {
    return eventCount;
  }

  @Override
  public void addEventCount() {
    eventCount++;
  }

  @Override
  public void setEventCount(long count) {
    this.count = count;
  }
}

Before we can use the bean, we have to make it available. For that, we need to wire it with the MBean server. The MBean server acts as a registry for MBeans, where each MBean is registered by its unique object name. Those object names consists of two parts, a Domain and a number of KeyProperties. The Domain can be seen as the package name of the bean, and one of these KeyProperties, the type, is its class name. if you use the “name” property, it denotes one of its attributes.

So, for our example above, the ObjectName would be:

my.monitoring:type=MyEventCounter

In every JVM, there’s at least one standard MBean server, the PlatformMBeanServer, which can be reached via

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();

In theory, you could use more than one MBean server per JVM, but normally, using only the PlatformMBeanServer is sufficient.

Next step: Accessing MBeans

To access our MBean, we can either use Spring and its magic, or we do it manually.

The manual way looks the following:

We once have to register our bean, e.g. in an init method:

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName myEventCounterName = new ObjectName(MyEventCounter.OBJECT_NAME);
MyEventCounter myEventCounter = new MyEventCounter();
mbs.registerMBean(myEventCounter, myEventCounterName);

And for every access, we have to retrieve it from the MBeanServer so that we can invoke the methods:

MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName myEventCounterName = new ObjectName(MyEventCounter.OBJECT_NAME);
mbs.invoke(myEventCounterName, "addEventCount", null, null);

Have you seen the second argument of the invoke method? It’s the name of the operation, you want to invoke. If you want to pass arguments, you pass their values as an object array as third, and their signature as fourth string array, e.g.

mbs.invoke(myEventCounterName, "setEventCount", new Object[] {number}, new String[] {int.class.getName()});

If we’re lucky, and our whole application is managed by Spring, it’s sufficcient to work with configuration and annotation only.

The MBean needs to be annotated as @Component and @ManagedResource with the object name as parameter:

@Component
@ManagedResource(objectName="my.monitoring:type=MyEventCounter")

and the attributes need a @ManagedAttribute:

@Override
@ManagedAttribute
public void addEventCount() {
  eventCount++;
}

In your spring configuration, besides the <context:component-scan> tag, you need one additional line for exporting the MBeans:

<context:mbean-export>

And those classes, which want to use the bean, just have to import it with the @Autowired annotation:

@Autowired
private MyEventCounterMBean myEventCounterMBean

Accessing MBeans from outside, using Jolokia

Of course, with the jconsole, you can access your MBeans, but a more elegant and more firewall-friendly way is use an HTTP bridge, which allows you to access the MBeans over HTTP. That’s, where Jolokia joins the game.

Jolokia is JMX-JSON-HTTP bridge, which allows access to your MBeans over HTTP and returns their attributes in JSON. Nice, isn’t it? Besides that, it allows bulk requests for improved performance, has got a security layer to restrict access and is really easy to install.

If you want to monitor your webapp, which runs inside tomcat, all you need is, to deploy the Jolokia agent webapp (available as a .war file) into your tomcat.

For a standalone Java application, just apply the Jolokia JVM agent (which in fact acts as an internal HTTP server) as javaagent in your start script:

java -javaagent:$BASE_DIR/libs/jolokia-jvm-1.2.1-agent.jar=port=9999,host=*

And if you build with gradle, apply the following line to your build.gradle:

runtime (group:"org.jolokia", name:"jolokia-jvm", classifier:"agent", version:"1.2.1")

Helpers – jmx4perl

Now, that we can access our MBeans from outside, it would be nice to have a tool available to just read the values on the comand line. The best tool for that is jmx4perl, which is available on github at https://github.com/rhuss/jmx4perl

The installation reminds of the good old Perl days with CPAN. If you’ve never worked with CPAN, just install jmx4perl according to the documentation and ACK all questions.

Now, let’s get an overview of all available MBeans:

jmx4perl http://your.application.host:9999/jolokia list

And if you want a decicated bean, run:

jmx4perl http://your.application.host:9999/jolokia read my.monitoring:type=MyEventCounter

Your output is in JSON and will be like:

{
 EventCount => 234,
 Name => 'MyEventCounter'
}

And finally, if you just need one attribute, run:

jmx4perl http://your.application.host:9999/jolokia read my.monitoring:type=MyEventCounter EventCount

In that case, you’ll get nothing but the value as a result.

Let’s go!

With these tools and figures, you can monitor virtually everything inside your application. All you have to do now is to provide the data (and you, as the developer of your application know, what exactly shall be monitored) and to monitor it with Nagios, OpenTSDB, whatever you want. All these tools are able, either directly, or with helpers like jmx4perl, to access, process and monitor the data.

Configuring Eclipse Kepler

By default, Eclipse Kepler comes with a way too large font for the package explorer and a pretty weird tab order.

Fixing both is pretty easy:

1. Setup a file called .gtkrc-eclipse in your home directory with the following contents:

style "eclipse" {
font_name = "Lucida Grande 8"
GtkButton::default_border={0,0,0,0}
GtkButton::default_outside_border={0,0,0,0}
GtkButtonBox::child_min_width=0
GtkButtonBox::child_min_heigth=0
GtkButtonBox::child_internal_pad_x=0
GtkButtonBox::child_internal_pad_y=0
GtkMenu::vertical-padding=1
GtkMenuBar::internal_padding=0
GtkMenuItem::horizontal_padding=4
GtkToolbar::internal-padding=0
GtkToolbar::space-size=0
GtkOptionMenu::indicator_size=0
GtkOptionMenu::indicator_spacing=0
GtkPaned::handle_size=4
GtkRange::trough_border=0
GtkRange::stepper_spacing=0
GtkScale::value_spacing=0
GtkScrolledWindow::scrollbar_spacing=0
GtkExpander::expander_size=10
GtkExpander::expander_spacing=0
GtkTreeView::vertical-separator=0
GtkTreeView::horizontal-separator=0
GtkTreeView::expander-size=8
GtkTreeView::fixed-height-mode=TRUE
GtkWidget::focus_padding=0
}

class "GtkWidget" style "eclipse"

style "gtkcompactextra" {
xthickness=0
ythickness=0
}
class "GtkButton" style "gtkcompactextra"
class "GtkToolbar" style "gtkcompactextra"
class "GtkPaned" style "gtkcompactextra"

2. Modify your eclipse start script to use this resource file by prepending it with

GTK2_RC_FILES=$GTK2_RC_FILES:/home/yourname/.gtkrc-eclipse ./eclipse

3. Install the Eclipse CSS editor and modify the appearence according to http://wiki.eclipse.org/Eclipse4/CSS , especially the part

.MPartStack {
    swt-mru-visible: true;
}

and restart eclipse

Migration to gradle

Last week, I re-vitalized a six year old struts web application, I once wrote for maintaining my music collection. It was neccessiary, since with the last Ubuntu upgrade, many changes to my system were made, among them a new PostgreSQL version.

Because of that, an annoying bug, caused by an old driver jar, came up and wanted to be fixed.

The application was created long ago with ant as build tool and manual dependeny management. Updating the jars was not really funny, working on the code was also pretty easy, but there was no real reliable build and installation process for the new webapp war file.

Since I’m using gradle on my daily job, the decision was soon made to transform my project into a nice gradle project, and I was really fascinated, how fast and easy the transition was. After 30 minutes, I got a clean gradle project, which not only does a proper and reliable dependency management, but also builds now in a reliable way.

Setting up the build.gradle file was so easy – here are the most important parts:

apply plugin: ‘java’
apply plugin: ‘war’

Instruct gradle, that we have a Java project here and want to get a war file as artifact

repositories {
mavenCentral()
}

All dependencies will be resolved by jars from the MavenCentral repository. That’s pretty standard, and unless you host your own repository, you can just keep it that way

dependencies {
compile group: ‘commons-beanutils’,   name: ‘commons-beanutils’, version:   ‘1.9.1’
compile group: ‘commons-collections’,   name: ‘commons-collections’, version: ‘3.2.1’

This is the actual list of first-level dependencies. All transitive dependencies of these jars are resolved automatically!

war {
from ‘WebRoot’ // adds a file-set to the root of the archive
}

In my application, there are some static jsps in a subdirectory called “WebRoot”. The Java source resides by default in src/main/java

task deployToTomcat(type: Copy) {   from war
into “/opt/tomcat/webapps”
}

This is the last step, which just deploys the war artifact into the tomcat.

Now, with the magical line “gradle clean build war deployToTomcat” my whole application is build and re-installed into my tomcat, which (by default) automatically updates the web application.

Pretty easy and straightforward, isn’t it?