Saturday, 12 April 2014

Java Garbage Collection

I re-read "Java Performance" by Charlie Hunt and here is some general information about garbage collection - mainly for my own reference:

The Java heap size is the size of the young generation and the old generation spaces (does not include the permanent generation) (p.111)

The app memory is the heap size + the perm generation + the threads stack size (p.277)

There are three aspects to garbage collection (p.262)
  • Throughput: How much time is spend in garbage collection vs. how much time is spend on executing application code
  • Latency/Responsiveness: How much pause time in executing application code does GC introduce, i.e. how is the response time affected by GC
  • Memory: The amount of memory used

GC Monitoring

Switch on GC monitoring using: -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<filename> -> Analyze using GCHisto (p.121)

To shows how long the applicaton runs and how long GC takes add: -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime (p.120)


The occupancy of the young generation after a minor GC is the survivor space occupancy (p.111) 

The live data size is the amount of memory of long-lived objects in old and perm generation. It is the size of the heap after a full GC - calculate an average of multiple full GCs (p.113, 268, 274)


The parallel garbage collector is using adative heap sizing, i.e. the HotStop VM initially uses explicit young generation sizing settings (-Xmn, -XX:NewSize, ...) and the automatically adjusts young generation spaces sizes from those settings. This can be disables using the -XX:-UseAdaptiveSizePolicy flag. (p.105 & http://docs.oracle.com/javase/8/docs/technotes/guides/vm/gc-ergonomics.html)

If you wanted to know what GC ergonomics was thinking, try adding -XX:+PrintAdaptiveSizePolicy or -XX:AdaptiveSizePolicyOutputInterval=1. The later will print out information every i-th GC about what the GC ergonomics to trying to do. (https://blogs.oracle.com/jonthecollector/entry/the_unspoken_the_why_of)


Heap size starting points (p.276)
  • Set -Xms & -Xmx to 3 to 4 times the live data size of the old generation
  • Set -XX:PermSize and -XX:MaxPermSize to 1.2 to 1.5 times the live data size of the perm generation
  • The young generation should be 1 to 1.5 times the live data size of the old generation
  • The old generation should be 2 to 3 times the live data size of the old generation

If young GC is takes too long reduce young generation size - if it occures too frequently increase it (p.280)

A parallel (throughput) garbage collector overhead of near 1% is considered well tuned. With >3% overhead tuning may improve the applications performance. It should be less than 5%. The larger the heap the better the opporunity for lower gc overhead - but this will also increase the max. stop-the-world time. p. 122/314

If worst case full GC time is unacceptable switch to a concurrent GC (Concurrent Mark Sweep or G1) (p.287)

Thursday, 10 April 2014

Samaxes Minify Maven Plugin & Eclipse - Deployment to Tomcat

I started using the Samaxes Minify Maven Plugin to minify my JavaScript and CSS files.The usage is pretty easy and I ended up with the following config in my POM


<plugin>
 <groupId>com.samaxes.maven</groupId>
 <artifactId>minify-maven-plugin</artifactId>
 <version>1.7.2</version>
 <executions>
  <execution>
   <id>default-minify</id>
   <phase>generate-resources</phase>
   <configuration>
    <charset>UTF-8</charset>
    <cssSourceDir>css</cssSourceDir>
    <cssSourceFiles>
     <cssSourceFile>x.css</cssSourceFile>
     <cssSourceFile>y.css</cssSourceFile>
     <cssSourceFile>z.css</cssSourceFile>
    </cssSourceFiles>
    <cssFinalFile>style.css</cssFinalFile>
    <jsSourceDir>js</jsSourceDir>
    <jsSourceFiles>
     <jsSourceFile>x.js</jsSourceFile>
     <jsSourceFile>y.js</jsSourceFile>
     <jsSourceFile>z.js</jsSourceFile>
    </jsSourceFiles>
    <jsFinalFile>script.js</jsFinalFile>
    <jsEngine>CLOSURE</jsEngine>
    <webappTargetDir>${basedir}/src/main/webapp</webappTargetDir>
   </configuration>
   <goals>
    <goal>minify</goal>
   </goals>
  </execution>
 </executions>
</plugin>

This worked fine when I ran 'mvn clean install' from my command line.

But I am also using Eclipse (with the whole ecosystem of  m2e / WTP / m2e-wtp) and Tomcat from within Eclipse. And I wanted to be able to publish the generated resources "script-min.js" and "style-min.css" to Tomcat using Eclipse.

Here is what I did in the end:

Did you notice the line in the plugin configuration.


<webappTargetDir>${basedir}/src/main/webapp</webappTargetDir>


By default the Minify Maven Plugin generates the resource into the 'target' folder. By setting this parameter I let the plugin generate the resources into the same folder as the source js and css files.

Now one simply has to run 'mvn generate-resources' either from the command line or from within Eclipse and then do a "Refresh" (F5) on the project and afterwards call 'Publish' on the Tomcat from within Eclipse. If Tomcat is already running it will even auto-publish the change.

This is it and this is what I do.

Note 1:

As an alternative to running 'mvn generate-resources' one can also call "Clean" on the project and let m2e rebuild it. There is just one additional thing if one want to go with this alternative: By default m2e does not execute the "Minify Maven Plugin" so you have to tell it to do so in your pom.xml:


<pluginManagement>
 <plugins>
  <plugin>
   <groupId>org.eclipse.m2e</groupId>
   <artifactId>lifecycle-mapping</artifactId>
   <version>1.0.0</version>
   <configuration>
    <lifecycleMappingMetadata>
     <pluginExecutions>
      <pluginExecution>
       <!-- We need to add this so that m2e (maven2eclipse) will execute the minify-maven-plugin's goal 'minify' when it builds the eclipse project -->
       <pluginExecutionFilter>
        <groupId>com.samaxes.maven</groupId>
        <artifactId>minify-maven-plugin</artifactId>
        <versionRange>[1.0.0,)</versionRange>
        <goals>
         <goal>minify</goal>
        </goals>
       </pluginExecutionFilter>
       <action>
        <execute />
       </action>
      </pluginExecution>
     </pluginExecutions>
    </lifecycleMappingMetadata>
   </configuration>
  </plugin>
 </plugins>
</pluginManagement>

That's it.

Note 2: Within the 'm2e lifecyce mapping' configuration I also tried to set


<execute>
 <runOnIncremental≶true</runOnIncremental>
</execute>

which would generate "script-min.js" and "style-min.css" each time one made a change to any of the source js or css files. This worked nicely when I manually called 'Publish' but I encountered a problem when Tomat was running and trying to auto-publish the changes:

When I changed one of the JS/CSS files Eclipse would get into an endless loop of automatically republishing and building the project.

As I do not very often change the JS/CSS files in this project I decided it was best to go with the manual steps of running 'mvn generate-resources' (from Eclipse), refreshing by hitting F5, and then publishing.

Monday, 7 April 2014

Oracle HotSpot JVM Command Line Flags

Find information about all the Oracle HotSpot JVM Command Line Flags:

One can use -XX:+PrintCommandLineFlags to see the default command line flags. For example


C:\>java -XX:+PrintCommandLineFlags -version

-XX:InitialHeapSize=263467392 -XX:MaxHeapSize=4215478272 -XX:+PrintCommandLineFlags 
-XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation 
-XX:+UseParallelGC
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)

tells us that Java 8 uses the Parallell Garbage Collector as a default (-XX:+UseParallelGC) garbage collector, that the default min heap size is 1/64 (263467392 bytes - about 250MB) of my machines RAM (16GB) and the default max heap size is 25% (4215478272 bytes - about 4 GB), of my machines RAM.

One can also find out the value of a specific command line flag using the tool jinfo. For example


C:\>jinfo -flag MaxHeapSize 3216

-XX:MaxHeapSize=4217372672

tells us again the value of the max heap size. 3216 is the process id of the Java process you are interested in. You can find out the process ids of locally running Java processes using the tool jps


C:\>jps

3216 Bootstrap
5872 Jps
2944 org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar

Tuesday, 18 March 2014

Setting HTTP Cache-Control header on Spring MVC contoller methods via annotations

I started using Spring MVC in a project and was really surprised to find that I could not specify HTTP-Cache-Control settings on my controller methods via annotations.

There are more people who miss this feature in Spring MVC and there are currently two issues open regarding this:

- https://jira.spring.io/browse/SPR-7129
- https://jira.spring.io/browse/SPR-8550

I was just about to implement this functionality myself when I found that thankfully Scott Rossillo already had: https://github.com/foo4u/spring-mvc-cache-control

Using "spring-mvc-cache-control" one simply registers a Spring MVC HandlerInterceptor in the Spring Dispatcher context file:

 <mvc:interceptors>
  <bean class="net.rossillo.spring.web.mvc.CacheControlHandlerInterceptor" />
 </mvc:interceptors>

Then one can happily annotate ones controller methods:


@Controller                                        
public final class MyTestController {

    @RequestMapping 
    @CacheControl(maxAge=300, policy = { CachePolicy.PUBLIC})
    public void test() {         
        System.out.println("In test method");   
    }
    
    @RequestMapping 
    @CacheControl(policy = { CachePolicy.NO_CACHE})
    public void test2() {         
        System.out.println("In test2 method");   
    }      
}

That's basically it - but as a bonus here are a few interesting things about caching:

1) When a user reloads the current page (e.g. hits F5) the browser will send a conditional request (using a If-Modified-Since or If-None-Match header in the request when a Last-Modified or ETag header was specified in the response) to validate the cached page has not changed. Only when a user gets to a page via a link will the browser not send a conditional request but directly display the cached page.

2) If there is not Cache-Control header but a Last-Modified header then Firefox calculates an expiration value as specified in the HTTP 1.1:
Also, if the response does have a Last-Modified time, the heuristic
expiration value SHOULD be no more than some fraction of the interval
since that time. A typical setting of this fraction might be 10%.
3) Imagine the scenario where you send your content gzip encoded to the browser and this gets cached by a proxy server. Now a different browser requests the same page from the proxy but does not support gzip encoded content. Dilemma. The solution is that one can use the Vary header to tell a proxy to cache different versions of your page depending on one or more header values specified by the browsers, e.g. Vary=Accept-Encoding

Wednesday, 29 January 2014

SonarQube analysis with Maven for a JavaScript project with multiple source directories

Here is how I convinced SonarQube to analyze a Maven JavaScript project with multiple source directories.

As SonarQube by default is using the standard Maven source directory (src/main/java) (and also ignoring the property "sonar.sources" when running from Maven) I had to specify the source directory of my webapp and then use the property "sonar.inclusions" to specify in which subdirectories of my webapp directory my JavaScript source files reside. Here is an excerpt from my POM:


 <properties>
  ...
  <sonar.language="">js</sonar>
  <sonar.inclusions="">app/**,astCommon/**</sonar>
  ...

 <build>
  <sourcedirectory>src/main/webapp</sourcedirectory>
  ...

Tuesday, 17 December 2013

Unit Testing Sencha Touch (or ExtJS) applications using Karma Test Runner, Jasmine, Maven and Jenkins


I wanted to be able to run the unit tests for my Sencha Touch 2.3 application locally as well as being able to run them in my continous integration process using Jenkins.

I decides to  use "Karma - Spectacular Test Runner for JavaScript" because it nicely integrates with Jasmine and offers headless execution using PhantomJS out of the box. If PhantomJS is not enough for you, you can also specify other Browsers.

Karma runs on NodeJS. I am not going to describe how to set it up. This information can be found on the Karma site.

Run the tests locally

Here is the 'karma.config.js' I use for local testing.

I bootstrap the Sencha Touch app using the 'development.js' microloader. Because the microloader is loading all required files dynamically I need to specify a proxy element in the Karma config pointing to a local webserver hosting the development version of the application JS files as well as the Sencha Touch JS files.


module.exports = function (config) {
    config.set({
        // base path, that will be used to resolve files and exclude
        basePath: '',

        // frameworks to use
        frameworks: [ 'jasmine' ],

        // list of files / patterns to load in the browser
        // Files for Sencha Touch development microloader
        files: [
            'src/main/webapp/.sencha/app/microloader/development.js',
            'src/test/javascript/setUp.js',
            'src/test/javascript/**/*.js' 
        ],

        // list of files to exclude
        exclude: [
        ],

        proxies: {
            '/': 'http://localhost:8080/'
        },

        // test results reporter to use
        // possible values: 'dots', 'progress', 'junit', 'growl', 'coverage'
        reporters: [ 'progress'],

        // web server port
        port: 9876,

        // enable / disable colors in the output (reporters and logs)
        colors: true,

        // level of logging
        // possible values: config.LOG_DISABLE || config.LOG_ERROR ||
        // config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
        logLevel: config.LOG_DEBUG,

        // enable / disable watching file and executing tests whenever any file
        // changes
        autoWatch: true,

        // Start these browsers, currently available:
        // - Chrome
        // - ChromeCanary
        // - Firefox
        // - Opera (has to be installed with `npm install karma-opera-launcher`)
        // - Safari (only Mac; has to be installed with `npm install
        // karma-safari-launcher`)
        // - PhantomJS
        // - IE (only Windows; has to be installed with `npm install
        // karma-ie-launcher`)
        browsers: [ 'PhantomJS' ],

        // If browser does not capture in given timeout [ms], kill it
        captureTimeout: 60000,

        // Continuous Integration mode
        // if true, it capture browsers, run tests and exit
        singleRun: false
    });
};

If you do not want to use the 'development.js' microloader to bootstrap Sencha Touch one can also point to the single JS file created by a Sencha Command test or production build. Then one does not need the proxy configuration.


        // Files for Sencha Touch test build
        files: [
            'src/main/webapp/build/testing/mae/app.js',
            'src/test/javascript/setUp.js',
            'src/test/javascript/**/*.js'
        ],

The advantage of the 'development.js' microloader approach is that you can changes your JS files and test them immediately. The drawback is that you need a web server running.

The advantage of pointing to the result of a Sencha Command build is that you do not need to have a webserver running. The drawback is that you need to rebuild after every change to your JS files to be able to test them.

You have probably noticed the file 'setup.js' that I include before the real test files.

1) This creates an element in the DOM which Sencha Touch expects
2) Delays the start of the tests till the Sencha Application is ready


// We need to add an element with the ID 'appLoadingIndicator' because [app.js].launch() is expecting it and tries to remove it
var myDiv = document.createElement("div");
myDiv.setAttribute("id", "appLoadingIndicator");
document.getElementsByTagName("body")[0].appendChild(myDiv);

// Karma normally starts the tests right after all files specified in 'karma.config.js' have been loaded
// We only want the tests to start after Sencha Touch/ExtJS has bootstrapped the application.
// 1. We temporary override the '__karma__.loaded' function
// 2. When Ext is ready we call the '__karma__.loaded' function manually
var karmaLoadedFunction = window.__karma__.loaded;
window.__karma__.loaded = function () {};

Ext.onReady(function () {
    console.info("Starting Tests ...");
    window.__karma__.loaded = karmaLoadedFunction;
    window.__karma__.loaded();
});

Another nice thing about Karma is that there is integration into the Webstorm IDE, i.e. you can run and even debug you tests from within Webstorm

Run the tests on Jenkins

I am using a different 'karma.config.js' for running tests within Jenkins.

It uses a "junitReporter" and has "singleRun=true" specified. It loads the "app.js" file  created by Sencha Command containing the Sencha Touch and my application JS files.


module.exports = function(config) {
 config.set({

  // base path, that will be used to resolve files and exclude
  basePath : '',

  // frameworks to use
  frameworks : [ 'jasmine' ],

  // list of files / patterns to load in the browser
  files : [ 'javascript/app.js','javascript/setUp.js',
    'javascript/**/*.js' ],

  // list of files to exclude
  exclude : [ 
  ],

  // test results reporter to use
  // possible values: 'dots', 'progress', 'junit', 'growl', 'coverage'
  reporters : [ 'dots', 'junit' ],

  junitReporter : {
   outputFile : 'karma-test-results.xml'
  },

  // web server port
  port : 9876,

  // enable / disable colors in the output (reporters and logs)
  colors : true,

  // level of logging
  // possible values: config.LOG_DISABLE || config.LOG_ERROR ||
  // config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
  logLevel : config.LOG_INFO,

  // enable / disable watching file and executing tests whenever any file
  // changes
  autoWatch : false,

  // Start these browsers, currently available:
  // - Chrome
  // - ChromeCanary
  // - Firefox
  // - Opera (has to be installed with `npm install karma-opera-launcher`)
  // - Safari (only Mac; has to be installed with `npm install
  // karma-safari-launcher`)
  // - PhantomJS
  // - IE (only Windows; has to be installed with `npm install
  // karma-ie-launcher`)
  browsers : [ 'PhantomJS' ],

  // If browser does not capture in given timeout [ms], kill it
  captureTimeout : 60000,

  // Continuous Integration mode
  // if true, it capture browsers, run tests and exit
  singleRun : true
 });
};

I am using Maven to build my app:

1) Sencha Command builds the app.js in the 'generate-resources' phase
2) I specify where my "test resources" are so Maven can copy them to the target folder
3) In the "test" phase Karma runs the tests.

Here is a snippet from my POM:


<build>
 <!-- Specify the location of the test resources and JS files -->
 <testResources>
  <testResource>
   <directory>${basedir}/src/test/resources</directory>
  </testResource>
  <testResource>
   <directory>${basedir}/src/main/webapp/build/${senchaBuildEnvironment}/mae</directory>
   <includes>
    <include>app.js</include>
   </includes>
   <targetPath>javascript</targetPath>
  </testResource>
  <testResource>
   <directory>${basedir}/src/test/javascript</directory>
   <targetPath>javascript</targetPath>
  </testResource>
 </testResources>
 <plugins>
  <!-- Sencha Command will build our web-application -->
  <plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>exec-maven-plugin</artifactId>
   <version>1.2.1</version>
   <executions>
    <execution>
     <id>SenchaCommand</id>
     <phase>generate-resources</phase>
     <goals>
      <goal>exec</goal>
     </goals>
     <configuration>
      <workingDirectory>${basedir}/src/main/webapp</workingDirectory>
      <executable>/sencha/Cmd/4.0.0.203/sencha</executable>
      <arguments>
       <argument>app</argument>
       <argument>build</argument>
       <argument>${senchaBuildEnvironment}</argument>
      </arguments>
     </configuration>
    </execution>
  <!-- Karma runs the JS tests in the 'test' phase -->       
    <execution>
     <id>Karma Test Runner</id>
     <phase>test</phase>
     <goals>
      <goal>exec</goal>
     </goals>
     <configuration>
      <workingDirectory>${basedir}/target/test-classes</workingDirectory>
      <executable>node</executable>
      <arguments>
       <argument>${node.modules.home}/karma/bin/karma</argument>
       <argument>start</argument>
      </arguments>
     </configuration>
    </execution>
   </executions>
  </plugin>
  ...

I have different profiles for test and production builds, .e.g.
     
<profile>
 <id>production</id>
 <activation>
  <property>
   <name>env</name>
   <value>production</value>
  </property>
 </activation>
 <properties>
  <senchabuildenvironment>production</senchabuildenvironment>
 </properties>
</profile>

Now Jenkins simply calls "mvn -Denv=production clean test".

I configured Jenkings to pick up the "karma-test-results.xml" file to display the test results:

**/target/surefire-reports/*.xml, **/target/test-classes/karma-test-results.xml

Tuesday, 3 December 2013

Sencha Touch 2 production builds and delta updates


A production build by Sencha Cmd (sencha app build production) will create a minfied version of your application. 

It uses a microloader (embedded in index.html) which caches the JavaScript (e.g. app.js) and CSS (e.g. app.css) files specified in “app.json” into the Browser's local storage (using a hash of the file content to identify the “version” of the file).
 
This enables the micoloader on further loads of the application to only download the app.json file, because this file contains the information which versions of the JS and CSS files need to be loaded.

If the JS and CSS files have not changed they can be read from local storage. If a file has changed it will have a different “version”-hash in app.json. The action then taken by the Microloader depends on the value of the “update” attribute associated with the respective file:
·         "update": "delta"
o   A delta-file with the changes is downloaded by the microloader and patched into the file stored in local storage
·         "update": "full"
o   The file will be loaded via Ajax call and stored in local storage


How delta-updates work



Archive Folder:
·         During a production build Sencha Cmd copies each file into the “build/archive”-folder
·         Each file goes into a folder named like the file; the file itself will be named like the hash of its content
·        Therefore the “archive”-folder contains the history of all the versions of all the files which have been built in any production build, e.g.
o   archive\app.js\     <<<  The folder named like the file
-  88c900a6530d892d21fbae08670c074e727c35ba  <<< 1st built
-  28688b22ae4da508dc6deb3e6e90473ee525d1c1 <<< 2nd built
-  727eebf10074ef70fd1584c3ced713b610197094  <<< current built
Deltas-Folder:
·         Sencha Cmd creates a “deltas”-folder for the web application (build/[appName]/production/deltas)
·         For each of the previously built files (stored in the archive folder) a delta-json-file (compared to the current file) is created, e.g.
o   deltas\
-  88c900a6530d892d21fbae08670c074e727c35ba.json
-  28688b22ae4da508dc6deb3e6e90473ee525d1c1.json
-  Each of these files contains the delta to the current file: “727eebf10074…”
·         When the Microloader loads “app.json” it sees that the version “727eebf10074…” is the current version of “app.js”
·         Depending on the version stored in local storage it loads the correct delta file, patches the stored file and stores it in local storage with the updated version number ”727eebf10074…”