Thursday, 22 September 2016

How to approach a memory leak

Steps to take to get a picture of the memory dump:
  • Get a historic look at the logs.
  • Open the heap dump with VisualVM
  • Look at the objects that using the most memory.
  • Query those objects for information using QQL.

If there are lots of objects then you need to distinguish if the object have the same type of information or they are referenced from the same type of object. It will give you a clearer picture of what type of object is causing the problem and where it is coming from.

QQL has the ability to count objects based on a criteria. For example, you query to count the number of hashmaps that have a certain size.

I have added a OQL project to GitHub. It has advanced OQL queries which have been tested with a sample program in the project. I have expanded the queries past what is in the visualvm OQL userguide.

Sunday, 11 September 2016

Making a simple maven plugin

I had a look around for a how to create simple maven 3 plugins. There is a mixture of maven 2 and maven 3 tutorials doing it different ways. Here is one that just does a simple printout. It can be expanded to anything you can dream of.
The project below creates a maven 2 setup but I have added extra steps to make it maven 3 setup

Running this command to create a plugin project
mvn archetype:create -DgroupId=com.naughton \ 
-DartifactId=simple-maven-plugin \ 
-DarchetypeGroupId=org.apache.maven.archetypes \ 

It will create this project structure for you

The pom file should look like this. It will include the maven plugin api as a dependency and packaging will be maven-plugin. That is the only different from a default pom.

 <project xmlns="" xmlns:xsi=""  
  <name>simple-maven-plugin Maven Mojo</name>  

Add in the following to the pom.xml to complete the project

This will let you add annotations in your Mojo files

Change the file to this. I have gotten rid of the Javadoc entries. Previously in maven 2 it got the goal name and phase from the Javadoc. Now you can use annotations in maven 3.
  * Goal which prints a sentence.  
 @Mojo(name = "print",defaultPhase = LifecyclePhase.PACKAGE)  
 public class MyMojo extends AbstractMojo  
   public void execute() throws MojoExecutionException  
     getLog().info("It executed the goal by printing this sentence.");  

You can see in the build that it will find no javadoc descriptors but one annotation descriptor. That is the annotation named print above.
 [INFO] --- maven-plugin-plugin:3.4:descriptor (default-descriptor) @ simple-maven-plugin ---  
 [WARNING] Using platform encoding (UTF-8 actually) to read mojo metadata, i.e. build is platform dependent!  
 [INFO] Mojo extractor with id: java-javadoc found 0 mojo descriptors.  
 [INFO] Mojo extractor with id: java-annotations found 1 mojo descriptors.  

Add this to generate the descriptor plugin.xml file automatically

This is what a plugin.xml file looks like. It includes the goals that the plugins supports. This is the file maven reads to understand your plugin.
  <name>simple-maven-plugin Maven Mojo</name>  
    <description>Goal which touches a timestamp file.</description>  

Structure of the plugin jar file

You can see that the plugin.xml is generated by the pluging plugin.

How to test your new plugin

Quickest way to test is your plugin it though a command line call to one of the goals defined in the new plugin. You must first install the plugin in to your local .m2 repo by running this command in the root plugin project.

mvn install

mvn com.naughton:simple-maven-plugin:1.0-SNAPSHOT:print

 [INFO] Scanning for projects...  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Building simple-maven-plugin Maven Mojo 1.0-SNAPSHOT  
 [INFO] ------------------------------------------------------------------------  
 [INFO] --- simple-maven-plugin:1.0-SNAPSHOT:print (default-cli) @ simple-maven-plugin ---  
 [INFO] It executed the goal by printing this sentence.  
 [INFO] ------------------------------------------------------------------------  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Total time: 0.238 s  
 [INFO] Finished at: 2016-09-11T19:38:39+01:00  
 [INFO] Final Memory: 7M/110M  
 [INFO] ------------------------------------------------------------------------  

How to test your plugin in another project.

Your plugin should now be in your local .m2 repository after mvn install.  To use the simple plugin use this in plugin entry in the other project pom.xml


I have added the simple maven plugin to github to hack around with: 


Thursday, 8 September 2016

A look at haskell

Recently I got some time on my hands so I started looking at programming in Haskell. I have heard of haskell a few times from academic people but never truly understood the syntax. Last week I was at a programming language debate. Haskell did have some good points so I thought I would at least understand the basics of the syntax. Just in case it was useful in the future and improve my pure functional programming skills

How I approached learning it
First I always try the hello world example. I learn two things this way. How to compile and execute  and how to print out to standard out. Printing to standard out helps with debugging. When learning a new language putting a few debugging message helps you see the flow of the program. It also shows where the program was before the failure and the values of the variables.

Next I try a few simple maths equations. Then I jump in to calling functions or methods.
  • First with no return type or arguments
  • Then with a return type. Usually returning an integer for simplicity.
  • Then with a return type and arguments. The are not linked. Just to avoid any computation problem. More concerned with getting the in and out correct first.
  • The with a return and arguments. The return is based on the arguments supplied.
Once I mastered the above then I have a basic know the of the syntax and how everything runs. I am no master at this point but a good foundation.

Thoughts on haskell while learning from scratch:
  • Has 2 ways of defining and calling functions. Makes it confusing when looking at different tutorials
  • For a beginner haskell has a lot of builtin functions. There is no way to tell the difference between defined function and a builtin which knowing the builtin functions. In java you can check were the import came to see if it came from the core library.
  • The compiler errors are hard to decipher for a beginners. It is better to start with a correct program with customizing and constant compiling. 
  • I  am not a big fan white spaces to separate arguments. I prefer to have parentheses for better readability. Also I prefer curly braces to group information instead of indentation. You can do these in haskell but white spaces and indentation seem to be the normal for less verbose code.

Here are 2 resources that I found useful.

 I uploaded my small haskell programs. It might help a beginner.


Saturday, 16 July 2016

Findbug : Unusual equals method (EQ_UNUSUAL)

I came across this find bug issue in my code recently. It appeared on the all the equals methods that had overidden the objects equals. The equals method that was there look fine to me.

Definition of the issue.
This class doesn't do any of the patterns we recognize for checking that the type of the argument is compatible with the type of the this object. There might not be anything wrong with this code, but it is worth reviewing.

What patterns does find bug look for? 

Here is the source code for the equals pattern detection.

In the method visit() there is a nested if else statement. This method is checking what equals pattern was found by method sawOpcode(). At the end of the nested if statement is this code:

 else {  
         if (AnalysisContext.currentAnalysisContext().isApplicationClass(getThisClass())) {  
           .reportBug(new BugInstance(this, "EQ_UNUSUAL", Priorities.NORMAL_PRIORITY).addClassAndMethod(this));  

This code will flag the "Unusual equals method" issue since it has checked your code against every pattern it knows.

Does your code have an issue?

Probably not since it has checked all other possible issues you could of had with your code. Like the 2 issues below. It will make you think about your equals method thought. Wondering for ages why FindBug does not like your equals method.

Some patterns it is looking for

Patterns it does not like
EQ_ALWAYS_FALSE and EQ_ALWAYS_TRUE pattern is when it is returning a boolean and the previous opcode was ICONST_0 or ICONST_1. That means the equals method always returns the same boolean regardless of input so this is flagged as EQ_ALWAYS_FALSE or EQ_ALWAYS_TRUE in findbug.

EQ_GETCLASS_AND_CLASS_CONSTANT is when you are comparing the class to another class but you are using the class name instead of "this" variable. Subclasses will be broken if they are inherit this equals.

Patterns findbug likes in an equals method
A method call to compare
An Instanceof check
A method call to getClass

Saturday, 9 July 2016

Hand carving your own wireless configurations

After Debian 6 "squeeze" network manager does not handle interfaces that  are in /etc/network/interfaces by default. You update the network manager configuration to inform it to handle those interfaces. Network Manager is included in many Linux distros. Its reason for living is to make Linux networking just work.

Network Manager will infinitely display a loading logo sign in Gnome if it does not handle the wireless interface.

There is no fun network manager handling all the juicy configuration.

What is involved when making your own wireless configurations:
  • Scanning all the wireless networks on the command line using iw
  • Looking at the details of wireless access point I want to connect to. I need to know if the authentication process it uses such as if its PSK (Pre shared key) or EAP (Extensible Authentician Protocol). I also need to know if there is a cipher required on top of the protocol.
  •  Once you have the information you can then add it to the wpa_supplicant.
 wpa_supplicant is just for the WPA protocol. For the WEP (Wired Equivalant Protocol) you use another application.  Reason is that nobody wants to touch the WEP application again. No one is using it because of the way you can sniff the packets and figure out the pre shared key being used which makes WEP unsecured.

Network Manager code

You can browse the code to see how it decides what information it needs for every network configuration.

What are the cipher used for?

TKIP and CCMP are used to encrypt your data from the computer to access point. They make sure no see what you are browsing. They use the pre shared key to encrypt the data to the router.

Here is an example of wireless printout from iwlist. It is a WPA2 router.
      Cell 06 - Address: 5C:DC:96:56:4F:F3  
           Frequency:2.462 GHz (Channel 11)  
           Quality=54/70 Signal level=-56 dBm   
           Encryption key:on  
           Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 11 Mb/s; 18 Mb/s  
                24 Mb/s; 36 Mb/s; 54 Mb/s  
           Bit Rates:6 Mb/s; 9 Mb/s; 12 Mb/s; 48 Mb/s  
           Extra: Last beacon: 28ms ago  
           IE: Unknown: 000B4254487562352D4A354A37  
           IE: Unknown: 010882848B962430486C  
           IE: Unknown: 03010B  
           IE: Unknown: 0706474220010D14  
           IE: Unknown: 2A0106  
           IE: Unknown: 2F0106  
           IE: IEEE 802.11i/WPA2 Version 1  
             Group Cipher : CCMP  
             Pairwise Ciphers (1) : CCMP  
             Authentication Suites (1) : PSK  

 Information you need from this print out
  • Group Cipher
  • Pairwise Cipher
  • Authentication Suites
You can then add this to /etc/wpa_supplicant/wpa_supplicant.conf
     psk=<your psk passphrase produced by wpa_passphrase>  

That is a general over view to make your own wireless configurations.

I GIT you now

My relationship with GIT is a love/hate relationship. Comparing it with SVN I have all love for GIT.

I remember the first time setting up SVN that I wanted to use with my project. I did not want the project online and I did not have a running server so I had to host the repository on my computer.  On my windows machine I thought this would be no problem but from memory you could not have the client and repository on the same machine with out a hack. SVN did not like the way I was going to set it up. I just remember giving up on SVN then.

Then came GIT with its local check in which I thought was a god send. Now development was easier to keep track of.

GIT is at version 2.9.0 now and it has a lot of features. If you look through the man pages for GIT there is a lot of options. This makes it very powerful but also colossal. Like my oven that has 20 different features, I only use 2 of them from day to day. I do not use all the  fancy features of GIT.

I like the positive flow of GIT. When all you checkins are correct. I just hate when I have a negative flow. A negative flow is when you have messed up a commit message, accidentally checked in, merged from the work branch etc. They are simple negative flows. There are harder negative flows to come back from which you need google and some one that deeply knows git.

I was at a conference recently and there was a GIT presentation. The presenter focused how to fix negative flows to return to a positive flow. Most presentations about GIT are what it is and what it does. This was a refreshing presentation. What was thought provoking was even though GIT was improving all the time the foundation GIT is built on has not changed. Blob objects, tree objects and commits are the foundation. Understand those deeply and you can figure out how to come back to a positive flow. You just need to find the command that allows to do it. Instead of not know how to come back and finding a post to explain it to you along with the command to do it.

In the future I will post my GIT negative flow conquests.

Saturday, 4 June 2016

Building Eclipse from Scratch

I read a mailing list that outlines the reason why Debian unstable still has a Eclipse version that is 4 years old. No one has been able to package it up fully and maintain it since version 3.8 of Eclipse. The latest version of eclipse is 4.5 (Mars)

Then I thought that I have never built Eclipse from scratch previously. I wondered how hard it would be.

Thankfully they have made it easy by using a git repo that has a list of sub modules. Everything you need is in the platform aggregator repo.

By follow Section 13.4 in this tutorial you can build Eclipse 4.x. Previously the build was done differently. The new way to build is with Tycho.

Run this command and you will see the GIT project is made up of all other projects that together form the Eclipse Java Develop Tool IDE.

 ~/eclipsebuild/eclipse.platform.releng.aggregator$ git config --file .gitmodules --name-only --get-regexp path  

Monday, 25 April 2016

Query your firefox browsing history with sqlite


I was search for a number that I had rang from an advertisement I seen on the internet. A few months later I needed that number. I went to the url and found the ad had expired. There was no number available. I have an itemized phone bill but the number could be any where with in the last 6 months. I needed to narrow down the search.

Information that I knew:
  • The url of the advertisement
  • I rang the number around lunchtime.

Firefox history

I checked my firefox browser history. I thought that if i knew the date I visited the advertisement it would reduce the number of telephone numbers I had to choose from. The browser history page was not detailed enough for that.

2 issues with the firefox history browser
  • It organizes the website you visited into what month you visited it if it older than a month. There is no column for exact date or time of the visit.
  • When using the searchbox the url would appear in the results but it would not tell you what month,date or time you visited the url.

Firefox history database

In your home directory firefox keeps all the history in a sqlite file.
~/.mozilla/firefox/<random directory>/places.sqlite
To find your database run this command
find ~/.mozilla/firefox/ -name places.sqlite
In the moz_places table it has an epoch in microseconds of the exact time you last visited the website. Unfortunately if you visited the site more recently you only have that date. Firefox does not keep a record of every time you visited a url. By chance I visited this url on 2 different computers so one computer still had my first visit time while the other had the time that I visited the expired advertisement.

I was then able to get the exact url and query the table. I converted from microseconds to a date in one query.
select datetime((select last_visit_date from moz_places where url='<EXACT URL>')/1000000,'unixepoch', 'localtime');

I found the date I visited the url. I was able to see on my phone bill the next day that I rang a number I did not know. I rang the number and it turned out to be the person I was looking for. The query helped me narrow down my search.

Thursday, 14 April 2016

Java nio move under the hood

Moving files with java.nio it will figure out if you are moving a file on the same partition or to a different partition.
  • If you are moving on the same file system then it will do a move which is faster than a copy.
  • If you moving to another filesystem then it will do a copy.
Do not always assume move is fast in java nio. If there is issue with the move it will can mask the issue and do a copy of the file instead.
By looking through the code below you can see.

This is the move method in the Files class. It decides if should do a move or
copy based on if the file is moving with in the same file system or to another  one.
  public static Path move(Path source, Path target, CopyOption... options)  
     throws IOException  
     FileSystemProvider provider = provider(source);  
     if (provider(target) == provider) {  
       // same provider  
       provider.move(source, target, options);  
     } else {  
       // different providers  
       CopyMoveHelper.moveToForeignTarget(source, target, options);  
     return target;  

FileSystemProvider is an abstract class that provides methods for concrete classes to provide a move method that is associated with the file system. 
You will have a different move implementations for every file system.

For linux filesystems this class provides the move method that just calls UnixCopyFile.move()
In this class it does a rename on the files from source to target. If it can not complete that job then it will just do a copy which will transfer the bytes. Look at the code below. The exception can be ignored.
If the rename exception is not bad enough it will ignore and continue with a copy.
 // first try rename  
     try {  
       rename(source, target);  ######  
     } catch (UnixException x) {  
       if (x.errno() != EXDEV && x.errno() != EISDIR) {  
         x.rethrowAsIOException(source, target);  
     // copy source to target  
     if (sourceAttrs.isDirectory()) {  
       copyDirectory(source, sourceAttrs, target, flags);  
     } else {  
       if (sourceAttrs.isSymbolicLink()) {  
         copyLink(source, sourceAttrs, target, flags);  
       } else {  
         if (sourceAttrs.isDevice()) {  
           copySpecial(source, sourceAttrs, target, flags);  
         } else {  
           copyFile(source, sourceAttrs, target, flags, 0L);  ######  
This C file has the native JNI rename method for linux

In the end you could be faster to do a rename instead of a move if you know your application is working on the same file system. Less decisions will be made by java for you. 

Sunday, 10 April 2016

Sqlite 2 databases can not be correctly read by sqlite3

If you try to open a sqlite file that is created with sqlite 2 with sqlite3 command then you can get an error like this when running commands on that database

Error: file is encrypted or is not a database

The database file must be created and read by the same version of sqlite. Sqlite3 will not be able to see the tables created in the sqlite2 database.
This can also cause a problem if you are using the sqlite 3 jdbc connector with a sqlite database file that was created with sqlite2. Queries on this file will say that the table you are trying to access does not exist.