Code analysis with SonarQube, jacoco and gradle

When you work on a Java project, you want to get an idea of your code quality.

Of course, “good” code doesn’t mean, the code is error-free, but on the other hand, if your code is seen as “bad”, you can be pretty sure, that it will become unmaintainable very soon.

Because of this, tools like SonarQube can be helpful to give an unbiased insight into how well your code might be, according to established coding standards.

First, you have to set up a SonarQube server, which is a very easy task, if you’re on an Ubuntu system:

Add the following line to your /etc/apt/sources.list:

deb http://downloads.sourceforge.net/project/sonar-pkg/deb binary/

and then run the well known and to-be-expected

apt-get update
apt-get install sonar

commands.

Assuming, that you already have got a PostgreSQL database running, create a user “sonar” with password “sonar” and enable the few PostgreSQL-related parts in /opt/sonar/conf/sonar.properties.

Finally, as root, start SonarQube with

/etc/init.d/sonar start

and maybe add it to /etc/rc.local

The next step is now, to prepare your project’s build.gradle script to ensure, that not only the SonarQube is filled with data, but also at least measures your test coverage.

The relevant parts are:

apply plugin: "sonar-runner"
apply plugin: "jacoco"

sonarRunner {
        sonarProperties {
                property "sonar.host.url", "http://localhost:9000"
                property "sonar.jdbc.url", "jdbc:postgresql://localhost:5432/sonar"
                property "sonar.jdbc.driverClassName", "org.postgresql.Driver"
                property "sonar.username", "sonar"
                property "sonar.password", "sonar"
                property "sonar.projectName", "rmmusic"
                property "sonar.jacoco.reportPath", "build/jacoco/test.exec"
                property "sonar.java.source property", "1.8"
        }
}

jacoco {
    reportsDir = file("build/tmp/jacoco.exec")
}

Additionally, log in as admin user into your SonarQube instance and in Settings->System->Update Center, add a few plugins:

  • Java
  • Checkstyle
  • Sonargraph
  • PMD
  • Timeline
  • Findbugs

and restart SonarQube.

As admin user, you should then set now a quality profile, e.g. the FindBugs profile

Now, when you run the gradle target sonarRunner, all those tests will be executed automatically and you’ll get detailed insights into your code and its quality.

Using nagios4 as a babysitter for your envionment

Out of many different monitoring solutions, nagios is one of the most used solutions. The following article shall describe in brief, how to set up some useful monitoring tasks:

Defining custom checks

in /etc/nagios4/commands.cfg, you can define a custom command. Let’s do this with a http check, which evaluates the result and checks in this result for a regex:

define command {
 command_name check_http_host_regex
 command_line $USER1$/check_http -H $HOSTADDRESS$ -p $ARG1$ -u $ARG2$ -r $ARG3$ -t30
}

This check is to be called from your /etc/nagios4/$cfg_dir/your_host.cfg:

define service {
 use generic-service
 host_name your.host.name
 service_description nginx_http_result
 is_volatile 0
 notification_options c,r
 check_command check_http_host_regex!80!/foo.css!your_regex
}

After that, you can run nagios4 -v /etc/nagios4/nagios.cfg to verify, that your configuration is valid.

Machine monitoring with OpenTSDB

(Partially updated in July 2016)

Inspired by a nice presentation at http://de.slideshare.net/oliverhankeln/opentsdb-metrics-for-a-distributed-world, I wanted to set up an OpenTSDB environment on my machine to replace the old munin monitoring, I’m still using and fighting with.

The following guide shall describe the steps for setting up OpenTSDB monitoring on an Ubuntu machine.

A word on disk space

According to the SlideShare presentation, referenced above, any data point consumes less than 3 Bytes on the disk if compressed and less than 40 Bytes, if uncompressed.

With that number in mind, you shall be able to give an estimation, how much data you will gather the next year(s).

Installing HBase

Follow https://hbase.apache.org/book/quickstart.html, download the latest binary (at the time of writing: hbase-1.2.2-bin.tar.gz ) and install it e.g. in /opt:

clorenz@machine:~/Downloads $ cd /opt
clorenz@machine:/opt $ tar -xzvf ~/Downloads/hbase-1.2.2-bin.tar.gz
clorenz@machine:/opt $ ln -s hbase-1.2.2 hbase

Next, edit conf/hbase-site.xml:

<configuration>
 <property>
  <name>hbase.zookeeper.quorum</name>
  <value>127.0.0.1</value>
 </property>
 <property>
  <name>hbase.rootdir</name>
  <value>file:///opt/hbase</value>
 </property>
 <property>
  <name>hbase.zookeeper.property.dataDir</name>
  <value>/opt/zookeeper</value>
 </property>
</configuration>

If you already have a running zookeeper instance, you must instruct OpenTSDB not to start its own zookeeper. For that, please add the following configuration to conf/hbase-site.xml:

<property>
 <name>hbase.cluster.distributed</name>
 <value>true</value>
</property>

And in conf/hbase-env.sh set the following line:

# Tell HBase whether it should manage it's own instance of Zookeeper or not.
export HBASE_MANAGES_ZK=false

Now, in any case, regardless of zookeeper, continue and edit conf/hbase-env.sh:

...
export JAVA_HOME=/opt/java8
...

Be careful to ensure, that your local hostname is resolved properly, the best is:

clorenz@machine:/opt/hbase $ ping machine
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.042 ms
^C

Finally, start hbase by running

clorenz@machine:/opt/hbase $ ./bin/start-hbase.sh

With ps -ef | grep -i hbase you can ensure, that your hbase instance is running properly:

clorenz@machine:/opt/hbase/logs $ ps -ef | grep -i hbase
root 31701 2795 0 14:37 pts/2 00:00:00 bash /opt/hbase-0.98.9-hadoop2/bin/hbase-daemon.sh --config /opt/hbase-0.98.9-hadoop2/bin/../conf internal_start master
root 31715 31701 44 14:37 pts/2 00:00:07 /opt/java7/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/opt/hbase-0.98.9-hadoop2/bin/../logs -Dhbase.log.file=hbase-root-master-ls023.log -Dhbase.home.dir=/opt/hbase-0.98.9-hadoop2/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start

Congratulations: You’ve finished your first step. Let’s take the next one:

Installing OpenTSDB

At first, download the latest source code from github:

clorenz@machine:/opt/git $ git clone git://github.com/OpenTSDB/opentsdb.git
Klone nach 'opentsdb'...
remote: Counting objects: 5518, done.
remote: Total 5518 (delta 0), reused 0 (delta 0)
Empfange Objekte: 100% (5518/5518), 27.09 MiB | 6.39 MiB/s, done.
Löse Unterschiede auf: 100% (3704/3704), done.
Prüfe Konnektivität... Fertig.

Next, build a debian package:

clorenz@machine:/opt/git/opentsdb (master)$ sh build.sh debian

If you encounter an error (e.g. like ./bootstrap: 17: exec: autoreconf: not found ), it’s likely possible, that you’re missing the prerequisite packages. Be sure to install at least the following ones:

  • dh-autoreconf
  • gnuplot

If everything went well with the debian build, you can install it:

clorenz@machine:/opt/git/opentsdb (master)$ sudo dpkg -i build/opentsdb-2.2.1-SNAPSHOT/opentsdb-2.2.1-SNAPSHOT_all.deb

Initial preparings for OpenTSDB

Before you can run OpenTSDB, you have to create the hbase tables:

clorenz@machine:/opt/git/opentsdb (master)$ env COMPRESSION=GZ HBASE_HOME=/opt/hbase ./src/create_table.sh

and at least in the beginning, it is helpful, that OpenTSDB creates the metrics automatically. For that, you have to set the following line in /etc/opentsdb/opentsdb.conf:

tsd.core.auto_create_metrics = true

Starting OpenTSDB

sudo service opentsdb start

When you access http://localhost:4242/ you will see the OpenTSDB GUI.

Now it’s time to start gathering data. We’ll use TCollector for the most basic data:

Installing TCollector

Again, we’re fetching the sourcecode from github:

clorenz@machine:/opt/git $ git clone git://github.com/OpenTSDB/tcollector.git

Let’s configure tcollector, so that it uses our own OpenTSDB instance by adding one single line to /opt/git/tcollector/startstop :

TSD_HOST=localhost

Starting tcollector is pretty easy:

clorenz@machine:/opt/git/tcollector (master *)$ sudo ./startstop start

It’s done!

Now, you can access your very first graph in the interface by selecting a timeframe and the metric df.bytes.free. You shall see now a graph!

Writing custom collectors

Any collector writes one or more lines with the following format:

metric timestamp value tag1=data1 tag2=data2

With in the subdirectory collectors of your tcollector installation, there are numerical subdirectories, which denote, how often a collector is executed. A directory name of 0 stands for “runs indefinitely, like a daemon”, values greater zero stand for “runs each n seconds”.

With that in mind, it shall be fairly easy to write own collectors now, like on the following example. Note, that these collectors are not neccessiarly written in Python, but you can basically use any language

#!/usr/bin/python
import os
import sys
import time
import glob

from collectors.lib import utils

def main():
 ts = int(time.time()) 
 yesterdayradio = glob.glob('/home/clorenz/data/wav/yesterdayradio-*')
 
 print "wavfiles.total %d %d type=yesterdayradio" %( ts, len(yesterdayradio))
 
 sys.stdout.flush()


if __name__ == "__main__":
 sys.stdin.close()
 sys.exit(main())

You can test your collector by executing it on the shell:

PYTHONPATH=/opt/tcollector /usr/bin/python /opt/tcollector/collectors/300/mystuff.py

Pretty straightforward, isn’t it?

If you for some reason generated wrong data, you can delete it, but beware, that this command is very dangerous, so the “1h ago” parameter in the following script actually means “now”, since the resolution is about one hour:

/usr/share/opentsdb/bin $ sudo ./tsdb scan --delete 2h-ago 1h-ago sum wavfiles.total type=*

Find more about manipulating the raw collected data at http://opentsdb.net/docs/build/html/user_guide/cli/scan.html

Let’s now polish the whole installation with a nicer frontend to get a real dashboard:

Installing Status Wolf as frontend

Since the standard GUI of OpenTSDB is a little raw, it’s a good idea to install an alternative for it, the one which is currently best looking (not only visually, but also in terms of features, like anomaly detection) is StatusWolf. To install StatusWolf, you need only a few steps:

  • install apache2
  • install libapache2-mod-php5
  • install php5-mysql
  • install php5-curl
  • ensure, that mod_rewrite is working:
    sudo a2enmod rewrite
    sudo a2enmod actions
    sudo service apache2 restart
  • download StatusWolf and install it into /opt
  • install pkg-php-tools
  • install composer ( https://getcomposer.org/download/ ):
    sudo mkdir -p /usr/local/bin
    sudo chown clorenz /usr/local/bin
    curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
  • install mysql-server (remember the root user password for later creation of the database)
  • Follow the StatusWolf setup instructions (be sure to remove all comment lines in the JSON configuration!)
  • Ensure, that all files belong to the www-data user:
    clorenz@machine:/opt $ sudo chown -R www-data StatusWolf
  • Create /etc/apache2/sites-available/statuswolf.conf:
    Listen 9653
    <VirtualHost your.host.name:9653>
     ServerRoot /opt/StatusWolf
     DocumentRoot /opt/StatusWolf
     <Directory /opt/StatusWolf>
     Order allow,deny
     Allow from all
     Options FollowSymLinks
     AllowOverride All
     Require all granted
     </Directory>
    </VirtualHost>
  • Link this file to /etc/apache2/sites-available
  • Ensure, that in /etc/apache2/mods-available/php5.conf, the php_admin_flag is disabled:
    # php_admin_flag engine Off
  • Create an user in the database (please use different values unless you want to create a security hole!):
    mysql statuswolf -u statuswolf -p
    INSERT INTO auth VALUES('statuswolf',MD5('statuswolf'),'Statuswolf User');
    insert into users values(2,'statuswolf','ROLE_SUPER_USER','mysql');

Upgrading Joomla 2.5 -> 3.3.x

Upgrading Joomla 2.5 (LTS) to 3.3.x (STS) is pretty hard, if you’re using Plugins or have reconfigured your system. I haven’t yet found out the complete way how to do the upgrade, but I’m trying to do it on a staging system. Here’s a guide, how to prepare the staging system:

1. Make a backup of your web contents, e.g. with gftp. Gzip it and put it in a safe, because you’ll most certainly need it again for a backup, since the Joomla update will fail for sure! After having the backup, install it locally:

clorenz@christoph ~ $ sudo rm -rf uhrenbastler
tar -xzvf uhrenbastler.tar.gz
sudo chown -R www-data uhrenbastler

2. Make a backup of your mysql database and install it locally, too

#!/bin/bash

JOOMLA_DB="xxxxxxxxxx"
JOOMLA_DB_PASSWORD="yyyyyyyyy"
echo "Copying remote joomla db to localhost"

mysqldump $JOOMLA_DB --verbose --add-drop-table --host=www.christophlorenz.de --user=$JOOMLA_DB --password=$JOOMLA_DB_PASSWORD > /tmp/$JOOMLA_DB.dump
mysqladmin -f drop $JOOMLA_DB --user=$JOOMLA_DB --password=$JOOMLA_DB_PASSWORD
mysqladmin -f create $JOOMLA_DB --user=$JOOMLA_DB --password=$JOOMLA_DB_PASSWORD
mysql $JOOMLA_DB --user=$JOOMLA_DB --password=$JOOMLA_DB_PASSWORD < /tmp/$JOOMLA_DB.dump

If your user doesn’t exist before, you must create it the following way:

$ mysql --user=root --password mysql
mysql> CREATE USER '$JOOMLA_DB'@'localhost' IDENTIFIED BY '$JOOMLA_PASSWORD';
mysql> GRANT ALL PRIVILEGES ON *.* TO '$JOOMLA_DB'@'localhost' WITH GRANT OPTION;
mysql> flush privileges;
mysql> \q

3. Deactivate the “Remember Me” plugin (filename “remember”)

4. Deinstall the following plugins:

  • CacheControl plugin
  • Include Content Item (NOT, ouch!)
  • n3t template
  • widgetkit
  • Advanced Google Analytics
  • Xmap (ouch! But there’s a replacement)
  • Mavik Thumbnails (maybe)
  • EasyImageCaption (ouch!, but optional)

5. Update patched “languagedomains” from localhost

6. Select “Options” -> “Short Term Support”

7. “Erweiterungen” -> “Aktualisierungen”: Cache leeren

8. Do the Upgrade (“Site” -> “Kontrollzentrum”)

9. Reinstall the following plugins

  • Mavik Thumbnails (3)
  • EasyImageCaption oder Multithumb
  • Advanced Google Analytics
  • Tooltips
  • mod_news_pro gk5
  • include_content_item (and when installed, apply the patched file plugins/content/include_content_item/include_content_item/include_content_item.lib.php (where lang.id is replaced by lang.lang_id and lang.code by lang.lang_code)
  • Phoca Gallery

10. Reconfig

  • News Show Pro GK5

11. New Template and lots, really lots of work!