Friday, January 31, 2014

Spring Data on GAE - Part 1 - Basic JPA

I would like to create an applicaton using AngularJS as front-end and Spring as back-end. The data access layer will be implemented by Spring Data JPA with DataNucleus JPA provider. The whole application will be running on Google App Engine.

There was debate over JPA vs JDO on app engine (see here). Personally I prefer to use JPA due to its widely adoption and also its support by Spring Data. However, we will see some design issues later due to the App Engine Datastore owned relationship feature.

1. Data model


Firstly I need to define a simple data model for this experiment. Here I created a POJO named Player to represent the players on the game server:

@Entity
public class Player {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String name;
    private String rank;

    public Player() {
    }

    public Player(String name, String rank) {
        super();
        this.name = name;
        this.rank = rank;
    }

    // setters and getters skipped
}

Due to the design of Datastore, we will soon see problem of using Long as the primary key. However, for simplicity we just use Long here temporarily.

2. JPA Configurations


As I use Spring MVC and Spring Data to implement the server side presentation and data layer, I add the following dependencies in build.gradle:

build.gradle
dependencies {
    compile "org.springframework:spring-webmvc:3.2.6.RELEASE"
    compile "org.springframework.data:spring-data-jpa:1.3.0.RELEASE"
    compile "com.fasterxml.jackson.core:jackson-databind:2.3.1"
    compile "org.datanucleus:datanucleus-enhancer:3.1.1"
    compile "org.codehaus.jackson:jackson-mapper-asl:1.9.9"
    compile "org.slf4j:slf4j-simple:1.7.5"
    ...
}

Note that the older spring-data-jpa 1.3.0 is used instead of the latest release because of a bug in DataNucleus (http://www.datanucleus.org/servlet/jira/browse/NUCJPA-250).

To specify DataNucleus as the JPA provider, the persistence unit is set as follows:

META-INF/persistence.xml
<persistence version="2.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns="http://java.sun.com/xml/ns/persistence" 
    xsi:schemalocation="http://java.sun.com/xml/ns/persistence
    http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
    <persistence-unit name="jpa.unit">
        <provider>org.datanucleus.api.jpa.PersistenceProviderImpl</provider>
        <class>com.angularspring.sample.domain.Player</class>
        <exclude-unlisted-classes />
        <properties>
            <property name="datanucleus.ConnectionURL" value="appengine" />
            <property name="datanucleus.appengine.ignorableMetaDataBehavior" value="NONE" />
        </properties>
    </persistence-unit>
</persistence>

3. Spring Data Repository


We need something to actually interact with the database. Here I do not use native JPA API (i.e. EntityManager) for database operations, instead I use Spring Data JPA. Spring Data can greatly reduce boilerplate codes for DAO implementation. We just need to define a player repository interface:

package com.angularspring.sample.repository;
public interface PlayerRepository extends JpaRepository<Player, Long> {
}

To complete the JPA repository configuration, we need to specify the followings in a Spring configuration:

applicationContext.xml
<jpa:repositories base-package="com.angularspring.sample.repository" />

<tx:annotation-driven transaction-manager="transactionManager" />

<bean class="org.springframework.orm.jpa.JpaTransactionManager" id="transactionManager">
    <constructor-arg ref="entityManagerFactory" />
</bean>

<bean class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" id="entityManagerFactory">
    <property name="persistenceUnitName" value="jpa.unit" />
    <property name="packagesToScan" value="com.angularspring.sample.domain" />
    <property name="loadTimeWeaver">
        <bean class="org.springframework.instrument.classloading.SimpleLoadTimeWeaver" />
    </property>
</bean>

4. Spring MVC Restful Service


The Spring MVC is used to implement RESTful service. Firstly, the web.xml is setup to include the DispatcherServlet and the context loader (loads the Spring configuration in previous section):

web.xml
<servlet>
  <servlet-name>dispatcher</servlet-name>
  <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
  <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
  <servlet-name>dispatcher</servlet-name>
  <url-pattern>/service/*</url-pattern>
</servlet-mapping>
<listener>
  <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>


And the following configuration for the dispatcher:

dispatcher-servlet.xml
<mvc:annotation-driven />
<context:component-scan base-package="com.angularspring.sample" />
<context:annotation-config /> 

5. Spring MVC Controller


Finally, we need to add a Spring MVC Controller to test the application from browser. A PlayerController class is configured to map the path "/service/players" to a service returning all players, and map path "/service/players/{id}" to return a particular player, both in JSON format. This object to JSON conversion is handled automatically by the jackson-mapper-asl library if we put that in classpath.

@Controller
@RequestMapping("/players")
public class PlayerController {
    private static Logger logger = LoggerFactory.getLogger(PlayerController.class);

    @Autowired
    PlayerRepository playerRepository;

    @RequestMapping(method = RequestMethod.GET)
    public @ResponseBody List<player> getAllPlayers() {
        return playerRepository.findAll();
    }

    @RequestMapping(value = "/{id}", method = RequestMethod.GET)
    public @ResponseBody Player getPlayer(@PathVariable("id") Long id) {
        return playerRepository.findOne(id);
    }

    @RequestMapping(value = "/initDB", method = RequestMethod.GET)
    @Transactional(readOnly = false, isolation = Isolation.READ_COMMITTED)
    public ResponseEntity<string> initDB() {
        List<player> players = new ArrayList<player>();
        players.add(new Player("Snoopy", "9p"));
        players.add(new Player("Wookstock", "9p"));
        players.add(new Player("Charlie", "1d"));
        players.add(new Player("Lucy", "4d"));
        players.add(new Player("Sally", "5d"));
        playerRepository.save(players);
        return new ResponseEntity<string>("5 players inserted into database", HttpStatus.OK);
    }
}

I have also added an initDB() method to initialize the datastore using HTTP request. The above will add 5 players into the database.

6. Test the application


We could test the application by using the Eclipse Google plugin. Start the google web application and try with the following in your browser (assume port is 8888). I used Firefox instead of IE because Firefox will output JSON directly.

    http://localhost:8888/service/players

It should return nothing, as expected, because we have no records in the database. So what we see is an empty array in JSON format (i.e. [ ]).

We can then initialize the database by:

    http://localhost:8888/service/players/initDB

Query the players again. It should give us something like the following:

[{"id":4573968371548160,"name":"Sally","rank":"5d"},{"id":4925812092436480,"name":"Snoopy","rank":"9p"},{"id":5488762045857792,"name":"Charlie","rank":"1d"},{"id":6051711999279104,"name":"Wookstock","rank":"9p"},{"id":6614661952700416,"name":"Lucy","rank":"4d"}]

7. Problem


It looks simple to use JPA with GAE.

However, when we test the application in browser, one would easily discover something unusual: not all entities are displayed immediately after the database initialization! It may display 2 records at the first time, 3 records at the second time, and finally get all 5 records displayed if we submit the HTTP request enough times. The reason is about the eventual consistency characteristic of Google App Engine. All those entities are unowned entities, so they are not belonging to the same entity group. In App Engine Datastore architecture, transactionality and data consistency only apply to entity group level.

For details please read the owned relationship of Datastore entities. Also note the unsupported features of JPA on Datastore (e.g. we cannot do many-to-many mapping) when you do the data modelling.

The handling of owned relationship and transactionality on GAE will be discussed in next part.

The source can be found at GitHub (tagged v0.1).



Tuesday, January 21, 2014

Gradle project template for Eclipse with GAE nature

I have been using Maven for years, but after some brief readings about Gradle, I am quite impressed by its flexibiliy. Under gradle, the build customization is straightforward using script language, provided that user has some basic Groovy knowledge.

I am going to create a project template using gradle. The following has been tested against Eclipse 4.3 (Kelper), Gradle 1.10 and Google App Engine SDK 1.8.9. It is also assumed that the Google GAE plugin has been installed in Eclipse. With this project template one should be able to:
  • Build a war file by running
       gradle war
    
  • Create an eclipse project with Google App Engine (GAE) nature by running
       gradle cleanEclipse eclipse
    
Note: there is an eclipse plugin for gradle provided by Spring (https://github.com/spring-projects/eclipse-integration-gradle/wiki). However, at the time of writting, it supports Java project nature only. Thus I still prefer to use gradle directly to generate all eclipse + GAE stuff for my project.

1. Apply the plugins


In build.gradle, firstly we need to add the required plugins:
apply plugin: 'war'
apply plugin: 'eclipse-wtp'
apply plugin: 'gae'
The eclipse-wtp is added as the Eclipse GAE plugin will create an GAE project as web project, so I generate an eclipse web project too.

2. Copy artifacts from local App Engine SDK folder


Originally I want to pull the GAE artifacts from google repositories, just like other 3rd party maven dependencies. So I configure the following maven repository:
repositories {
   mavenCentral()
   maven {url 'https://oss.sonatype.org/content/repositories/google-releases/'}
}

However, eclipse will complain about the different jar size with the App Engine SDK for datanucleus-appengine-2.1.2.jar. Therefore, if there is already App Engine SDK installed locally, I would recommend to copy the artifacts from the SDK folder; otherwise, just specify the maven dependencies as usual.

In this template, I assume the SDK exists locally and an environment variable APPENGINE_HOME must be set to point to the SDK location. The dependencies are listed as follow,
ext {
   springVersion = '3.2.6.RELEASE'
   appEngineHome = "$System.env.APPENGINE_HOME"
   appEngineSdkVersion = '1.8.9'
}

dependencies {
   providedCompile "javax.servlet:servlet-api:2.5"
   
   testCompile "junit:junit:4.+"
   
   compile "org.springframework:spring-webmvc:$springVersion"

   // AppEngine dependencies, copy from APPENGINE_HOME lib folders
   compile files(
      "$appEngineHome/lib/user/appengine-api-1.0-sdk-${appEngineSdkVersion}.jar", 
      "$appEngineHome/lib/opt/user/appengine-api-labs/v1/appengine-api-labs.jar",
      "$appEngineHome/lib/opt/user/appengine-endpoints/v1/appengine-endpoints.jar",
      "$appEngineHome/lib/opt/user/appengine-endpoints/v1/appengine-endpoints-deps.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/asm-4.0.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/datanucleus-core-3.1.3.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/datanucleus-api-jpa-3.1.3.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/datanucleus-api-jdo-3.1.3.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/datanucleus-appengine-2.1.2.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/geronimo-jpa_2.0_spec-1.0.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/jdo-api-3.0.1.jar",
      "$appEngineHome/lib/opt/user/datanucleus/v2/jta-1.1.jar",
      "$appEngineHome/lib/opt/user/jsr107/v1/jsr107cache-1.1.jar",
      "$appEngineHome/lib/opt/user/jsr107/v1/appengine-jsr107cache-${appEngineSdkVersion}.jar")

   // runtime dependencies
   runtime "javax.servlet:jstl:1.1.2"
}
As I would like to setup Spring later, the Spring MVC dependency is added.

I would also like to populate the WEB-INF/lib folder by copying the dependencies so I do not need to check-in the jars to source control system, so I add the following:
// task to clean WEB-INF/lib folder
task cleanWarLibDir(type: Delete) {
   delete fileTree(dir: "war/WEB-INF/lib")
}

// task to populate WEB-INF/lib folder from the compile dependencies
task populateWarLib(type: Copy) {
   into('war/WEB-INF/lib')
   from configurations.compile
}

// populate the WEB-INF/lib
tasks.eclipse.dependsOn('checkenv','populateWarLib')
tasks.populateWarLib.dependsOn('cleanWarLibDir')

The checkenv task is just used to check the APPENGINE_HOME environment variable. If the variable is not set, it will throw an exception.

3. Setup the eclipse project configurations


The Eclipse GAE plugin is picky about the project settings. Although it is kind of web project, it does not expect the existence of web container library. Thus the jars in WEB-INF/lib will not be added to the build path automatically in Eclipse, and we need to add the needed jars as external libraries manually in eclipse IDE.

Here is the eclipse part of the gradle configuration:
eclipse {
   // add build commands and specify the project natures
   project {
      natures.clear()
      natures 'org.eclipse.jdt.core.javanature',
         'com.google.appengine.eclipse.core.gaeNature',
         'org.eclipse.wst.common.project.facet.core.nature'
      buildCommands.clear()
      buildCommand 'org.eclipse.wst.common.project.facet.core.builder'
      buildCommand 'org.eclipse.jdt.core.javabuilder'
      buildCommand 'com.google.gdt.eclipse.core.webAppProjectValidator'
      buildCommand 'com.google.appengine.eclipse.core.gaeProjectChangeNotifier'
      buildCommand 'com.google.appengine.eclipse.core.projectValidator'
      buildCommand 'com.google.appengine.eclipse.core.enhancerbuilder'
   }

   classpath {
      defaultOutputDir = file("${project.projectDir}/war/WEB-INF/classes")

      // correct classpaths as needed by GAE eclipse plugin
      containers.clear()
      containers.add 'com.google.appengine.eclipse.core.GAE_CONTAINER'
      containers.add 'org.eclipse.jdt.launching.JRE_CONTAINER'
      file {
         whenMerged { classpath ->
            classpath.entries.removeAll {
               entry -> entry.kind == 'lib' || 
               (entry.kind == 'con' && entry.path == 'org.eclipse.jst.j2ee.internal.web.container')
            } 
            classpath.entries.findAll {
               entry -> entry.hasProperty('exported') 
            }*.exported = false
         }
      }
   }

   // GAE application needs Java 1.7
   wtp {
      facet {
         facet name: 'java', version: '1.7'
      }
   }
}
We also need to add two google specific properties files in the .setting folder:
// fix the GAE eclipse prop file
tasks.eclipse.doLast {

   // write settings file 'com.google.appengine.eclipse.core.prefs'
   ant.propertyfile(file: ".settings/com.google.appengine.eclipse.core.prefs") {
      ant.entry(key: "eclipse.preferences.version", value: "1")
      ant.entry(key: "gaeDatanucleusVersion", value: "v2")
      ant.entry(key: "gaeHrdEnabled", value: "true")
   }

   // write settings file 'com.google.gdt.eclipse.core.prefs'
   ant.propertyfile(file: ".settings/com.google.gdt.eclipse.core.prefs") {
      ant.entry(key: "eclipse.preferences.version", value: "1")
      ant.entry(key: "warSrcDir", value: "war")
      ant.entry(key: "warSrcDirIsOutput", value: "true")
   }
}
Without the above files the Eclipse GAE plugin will complain about the version of jars in WEB-INF/lib.

Okay, that's it. One should be able to import this project after running "gradle cleanEclipse eclipse". If you have already created an application in google appengine console, you should be able to deploy it in Eclipse using the GAE plugin.

Note that I also include the gradle gae plugin, so even there is no eclipse one should be able to do the deployment using gradle directly.

The project template is available on GitHub.