Wednesday, March 18, 2015

Agile - Tips From Live Projects

I recently had a chance to learn about a live agile project as part of agile training programme.
Below are some of the tips I picked up, while interacting with the key members on the team.

The Big Picture
The overall release is divided into several phases as mentinoned below.
Notice the SIT and UAT are run independently, towards end of release.
This is time consumimg but does reduce risk of any bugs in release.
This is more relevant for softwares which needs extremely thorough testing.

  1. Pre foundation Phase (1 week) 
  2. Foundation Phase (2 weeks) 
  3. Sprint 1 (2 weeks) 
  4. Sprint 2 
  5. Sprint 3 
  6. Sprint 4 
  7. Sprint 5 Final / Hardening Sprint
  8. SIT Test (1 month) 
  9. UAT Test (1 month) 
  10. Release 


During foundation phase high level estimation done using technique called T-Shirt sizing
(S, M, L, XL, XXL, XXXL). This helps in deciding scope of sprints.

Plan for sprints to progessively gain velocity

Balancing RnD and delivery
Instead of running 2 week Spikes like Sprints, only Spike Tasks are run to balance RnD efforts with delivering usable functionality.

1 Story = Spike Task1 + Task2 + Task3....

Sprint includes a sensible mix of techincal stories and functional stories.
Stubs are used for technical components planned for future technical stories.


Estimation 
Stories estimated as story points using estimation poker
Task are always estimated / tracked at hours spent.
There is provision in tool, to estimate task hours and also put in actual hours.
Over several successive sprints, a good average estimate of 1 Story point = xyz hours crystalizes


Sample Story Status: 
New
Analyzed
Approved
Dev WIP
Dev Blocked
Dev Done
Test WIP
Test Blocked
Test Done

Dev Done includes code review
Test Done includes show-n-tell, given by tester to convince the  BA



Very succint definition of done

Sample DOD for dev complete:

  • Impact Analysis document done
  • Code complete
  • Code reviewed
  • NFR, performance, security tests pass
  • Regression tests pass
  • All known bugs raised by test team resolved

About Code and CI

Release-wise branch in SVN (version control)
No branches per sprint
No parallel sprints
SVN checkin format:
<Release>_<Module>_<Story>_<Task/Defect>: Description
Automated regression testing in place


Other Takeaways: 
1.
Very detailed acceptance criteria, Yes No questions, fill blanks, unambigious answers
Is xxx panel visible as seen in logical diagram? Yes/No
Does the table/grid contain 13 rows Yes/No
Quality of story determined by how detailed is acceptance criteria

 2.
Story Status like "Test Blocked", if tester cannot test a story, calls up dev immediately or writes email.
All Blockers get mentioned in standup

 3.
Always testing team is lightly loaded at start of sprint and overloaded towards end of sprint, to reduce this pain point ....
Have stories at very fine-grained level, eg each story should add up to only few hours of dev task.
This way, dev always has something testable for tester throughout the sprint Idle time for testers (waiting for testable functionality) is reduced.


Friday, February 6, 2015

Rapid development of REST webservices with persistence

Development of modern business applications, has to become faster and easier, if we are to meet business delivery demands, spanning over ever shorter cycles. Also we are now required to deliver softwares continuosly and incrementally, in an agile manner.

Jump-starting the development process with the right tools and technolgies is the need of the hour.

Rapid development, using currently popular technology, has always been in vogue. Right now exposing middle tier services as REST APIs is all the rage. So how do we develop REST services with persistence, in a rapid manner, with least effort?

I had written an earlier post on grails 2_3 providing out of box support for REST CRUD , writing only a simple java bean (DTO), using the grails framework. But that would mean using an entirely new framework like grails only to exploit its REST CRUD functionality. The engineer in me, thought that was an unoptimized use of grails. Why shoudlnt we have this, out of box REST CRUD using spring, hibernate the technologies that we know best already :-)

Well more recently, advancements made in the Spring Data JPA framework, makes it very very easy to develop CRUD REST services out of the box, writing only the entity classes with JPA annotations.
Well here is how...

Effectively, we will see, how to write only a simple java bean, annotated with JPA annotations for persistence and using spring data jpa and spring boot, we can automatically generate REST CRUD services for our bean / entity. All this using very little code :-)



Create a typical maven java project (not a web project) 
eg. using maven-archetype-quickstart

We will use spring boot based spring data jpa and spring data rest

Use a pom.xml similar to below

https://github.com/ganeshghag/RestSpringDataJpa/blob/master/pom.xml



As per spring data jpa, create persistent entity class say Person as a POJO and annotate it with JPA annotations

package com.ghag.rnd.rest.domain;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

@Entity
public class Person {

@Id
@GeneratedValue(strategy = GenerationType.AUTO)
private long id;

private String firstName;
private String lastName;
private String email;
private String address;
private String mobile;
private Integer employeeId;

//getters and setters go here
}

Again as per spring data jpa framework, create an interface which will tell the spring data jpa framework, to create hibernate jpa based CRUD functionality with respect to the entity class Person we defined above

package com.ghag.rnd.rest.repository;

import java.util.List;
import org.springframework.data.repository.PagingAndSortingRepository;
import org.springframework.data.repository.query.Param;
import org.springframework.data.rest.core.annotation.RepositoryRestResource;
import com.ghag.rnd.rest.domain.Person;


@RepositoryRestResource(collectionResourceRel = "people", path = "people")
public interface PersonRepository extends PagingAndSortingRepository<Person, Long> {

List<Person> findByLastName(@Param("name") String name);

}

The methods in the interface like findByLastName, impart filtering capability on the the entity queries by entity attributes like lastName.


The annotation below
@RepositoryRestResource(collectionResourceRel = "people", path = "people")
ensures that, the CRUD rest services for the Person entity are auto-magically generated for us. This is the main fetaure, I wanted to publicize in this article.


Thats it, now using Spring Boot and its default sensible configurations, you can just right a class similar to below and you should be able to start up the applicatoin with an embedded tomcat server.

package com.ghag.rnd.rest;


import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.data.rest.webmvc.config.RepositoryRestMvcConfiguration;

@Configuration
@EnableJpaRepositories
@Import(RepositoryRestMvcConfiguration.class)
@EnableAutoConfiguration
@ComponentScan  //important and required
public class Application {

public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}


The application will expose the CRUD REST services for persistent entity Person at
http://server:port/people

you can easily test the services using curl commands like following:
curl -i -X POST -H "Content-Type:application/json" -d @insert.cmd http://localhost:8080/people
where file insert.cmd contains json for the Person object like thus:
{
"firstName" : "Ganesh",
"lastName" : "Ghag",
"email":"some@some.com",
"address":"flower valley, thane",
"mobile":"3662626262",
"employeeId":8373
}

In order to customize the application, writing custom REST controllers is as easy as writing below POJO

package com.ghag.rnd.rest.controllers;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

import com.ghag.rnd.rest.repository.PersonRepository;


@RestController
@RequestMapping("/custom")
public class MyCustomController {

@Autowired
PersonRepository personRepository;

@RequestMapping(value="/sayHello/{input}", method=RequestMethod.GET)
public String sayHello(@PathVariable String input){

System.out.println("findall="+personRepository.findAll());
personRepository.deleteAll();
return "Hello!"+input;
}
}


To further enhance the CRUD application, we can easily
  • use above application as a microservice, using spring boot as is
  • export above application as a war and deploy it on traditional app server
  • use various logging for debugging
  • use spring based validations
  • use spring security
  • switch CRUD database from current default H2 to any of our choice using spring boot config file application.properties


Please refer spring boot documentation for accomplishing the above mentioned enhancments.
http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/
http://docs.spring.io/spring-data/jpa/docs/current/reference/html/


By using hibernate tools reverse engineering, given an existing database, auto-generating JPA annotated entities is a no-brainer and with the above knowledge of generating REST CRUD services, it means, you can expose a database, as  REST services, with less than a days work.

Tuesday, September 23, 2014

ORM Tips

First Off, ORM is tougher than writing declarative SQL, make sure you have experienced developers, who are willing to "really understand" the data model, rather than developers who want to just "code-break-fix"


When you look at any persistent entity attributes think of them as following 3 categories:

  • "flat" attributes, that are stored in the table that the parent entity is mapped to
  • "referred" attributes, these merely refer to other entities, whose life cycle is independent of parent entity. Typically entities shown in dropdowns, lookups, master data entities etc
  • "cascade" attributes, refer to entities that may have CRUD cascades w r t the parent entity

The more the number of cascade attributes, the more the complexity in persisting the current entity
(this can be used for estimations)

Inserts are easy, updates are tricky. Allocate enough time for testing entity update, with different combinations of "cascade" attributes. (null, existing, new entities)

Complex cross entity (readonly) queries, need not go through the ORM. (caution: there are risks of stale/out-of-sync data, if not done properly)

The service code is not tested, till all lazy initialization has been exercised. Developers release the code as working too soon, but only when the services layer is invoked from the controllers, where-in the json serializers try to access the entire depth of the object graph, the code starts throwing data access exceptions. Working with shallow hibernate proxies often lulls us into false sense of security temporarily.

To translate entity objects into VO / TO objects and back, use object oriented code, rather than procedural, so let each VO / TO take care of its own mapping to corresponding entity object.


To test REST services, you can use simple HTML with jquery, but better is JUnit tests which use Spring's RestTemplate as REST Client and make REST service calls. This kind of test is very authentic (exercises controllers realistically)
and more importantly...the test code is strongly typed, so in case your VO changes, the compiler will break your test and catch your attention, this makes refactoring smooth.

Monday, September 8, 2014

Database multi-tenancy using Hibernate 4

With many business applications being written, with intent to be deployed on cloud, there is increasingly a need to design multi-tenancy into your application.


For persistence layer multi-tenant design offers a few options:
http://msdn.microsoft.com/en-us/library/aa479086.aspx


  1. separate database per tenant
  2. separate schema, same database per tenant
  3. same schema, same database for all tenant with discriminator column for tenant rows in same table


In this article, I am going to talk more about implementing the options 1 and 2 using the support in hibernate 4, for multi-tenancy.

Also for the database connections, we have choice of having a common database connection pool, from which connections are allocated to each tenant request, or separate connection pools for each tenant.
For the common connection pool approach, each connection before being used for data access, needs to be "primed" for usage against a tenant database or schema, using statements like "use <tenant_schema>" or "use <tenant_database>"


Multi-tenancy in Hibernate


For the "separate connection pool per tenant approach"

First we need to implement a custom connection provider as follows

import java.sql.Connection;
import java.sql.SQLException;

import org.apache.commons.dbcp2.BasicDataSource;
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;

public class ConnectionProviderImpl implements ConnectionProvider {
 
 private final BasicDataSource basicDataSource = new BasicDataSource();
 
 public ConnectionProviderImpl(String database){
                //this should be read from properties file
  basicDataSource.setDriverClassName("com.mysql.jdbc.Driver");
  basicDataSource.setUrl("jdbc:mysql://localhost:3306/"+database);
  basicDataSource.setUsername("myuser");
  basicDataSource.setPassword("mypassword");
  basicDataSource.setInitialSize(2);
  basicDataSource.setMaxTotal(10);
 }

 @Override
 public boolean isUnwrappableAs(Class arg0) {
  return false;
 }

 @Override
 public  T unwrap(Class arg0) {
  return null;
 }

 @Override
 public void closeConnection(Connection arg0) throws SQLException {
  arg0.close();
 }

 @Override
 public Connection getConnection() throws SQLException {
  return basicDataSource.getConnection();
 }

 @Override
 public boolean supportsAggressiveRelease() {
  return false;
 }

}




Next we need to implement AbstractMultiTenantConnectionProvider of hibernate, as follows
Here I am maintaining a map of database identifiers against, connection providers.
When hibernate invokes the selectConnectionProvider( ) method with the tenant identifier, we use the tenant identifier to return the "appropriate" connection provider from the map.
It is necessary to implement a method like getAnyConnectionProvider( ), which should return, a sensible - default connection provider.

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import org.hibernate.engine.jdbc.connections.spi.AbstractMultiTenantConnectionProvider;
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;

public class MultiTenantConnectionProvider extends AbstractMultiTenantConnectionProvider {
 
 private HashMap connProviderMap = new HashMap();
 
 public MultiTenantConnectionProvider(){
  

  List providerNames = new ArrayList();
  providerNames.add("default_db");
  providerNames.add("db1");
  providerNames.add("db2");
  //need to get above from properties file
  
    for (String providerName : providerNames) {
       connProviderMap.put(providerName, new ConnectionProviderImpl(providerName));
    }
  
 }
 
 @Override
 protected ConnectionProvider getAnyConnectionProvider() {
  System.out.println("inside MultiTenantConnectionProvider::getAnyConnectionProvider");
  return connProviderMap.get("default_db");
 }

 @Override
 protected ConnectionProvider selectConnectionProvider(String tenantId) {
  ConnectionProvider connectionProvider = connProviderMap.get(tenantId);
  if(connectionProvider == null)
   connectionProvider =  new ConnectionProviderImpl("default_db");
  
  return connectionProvider;
 } 

}



For common connection pool for all tenants.

We need to implement MultiTenantConnectionProvider of hibernate, as follows
Since we are using the same connection pool, in getConnection(), we need to prime the connection using a SQL statement like 'use <database>', so that the connection's further use will be in database as per tenant_id.
Also in releaseConnection( ), we have a 'use <default_db>' as a fallback.



import java.sql.Connection;
import java.sql.SQLException;

import org.hibernate.HibernateException;
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;
import org.hibernate.engine.jdbc.connections.spi.MultiTenantConnectionProvider;

public class MultiTenantConnectionProviderWithSingleDBPool implements
  MultiTenantConnectionProvider {
 
 private final ConnectionProvider connectionProvider = new ConnectionProviderImpl(CurrentTenantIdentifierResolver.DEFAULT_TENANT_ID); 
 

 @Override
 public boolean isUnwrappableAs(Class arg0) {
  return false;
 }

 @Override
 public  T unwrap(Class arg0) {
  return null;
 }

 @Override
 public Connection getAnyConnection() throws SQLException {
  System.out.println("inside MultiTenantConnectionProvider::getAnyConnection");
  return connectionProvider.getConnection();
 }

 @Override
 public void releaseAnyConnection(Connection connection) throws SQLException {
  connectionProvider.closeConnection( connection );
 }

 @Override
 public Connection getConnection(String tenantIdentifier) throws SQLException {
  final Connection connection = getAnyConnection();
  try {
     connection.createStatement().execute( "USE " + tenantIdentifier );
  }
  catch ( SQLException e ) {
     throw new HibernateException(
       "MultiTenantConnectionProvider::Could not alter JDBC connection to specified schema [" +tenantIdentifier + "]",e);
  }
  return connection;
 }

 @Override
 public void releaseConnection(String tenantIdentifier, Connection connection) throws SQLException {
  try {
   connection.createStatement().execute( "USE default_db" );
  }
  catch ( SQLException e ) {
     throw new HibernateException(
     "Could not alter JDBC connection to specified schema [" +
   tenantIdentifier + "]",e);
  }
  connectionProvider.closeConnection( connection );
 }

 @Override
 public boolean supportsAggressiveRelease() {
  return false;
 }

}





refer hibernate docs
http://docs.jboss.org/hibernate/orm/4.1/devguide/en-US/html/ch16.html



Now, to persuade hibernate to use tenant identifier before any database access, we can
  1. manually set the tenant identifier on hibernate session factory
  2. we can implement hibernate tenant identifier resolver


Manually set the tenant identifier on hibernate session factory
This can be done, in a web application in any suitable interceptor where hibernate session is available
Session session = sessionFactory.withOptions()
        .tenantIdentifier( yourTenantIdentifier )
        ...
        .openSession();


Hibernate tenant identifier resolver can be implemented as follows.
Here I am using a ThreadLocal to get the tenant identifier. The threadlocal itself can be set early on in the http request thread in say, a servlet filter.


import org.hibernate.context.spi.CurrentTenantIdentifierResolver;

public class CurrentTenantIdentifierResolver implements
  CurrentTenantIdentifierResolver {
 
 public static ThreadLocal _tenantIdentifier = new ThreadLocal();
 public static String DEFAULT_TENANT_ID = "default_db";

 @Override
 public String resolveCurrentTenantIdentifier() {
  System.out.println("from inside resolveCurrentTenantIdentifier....");
  String tenantId = _tenantIdentifier.get();
  if(tenantId == null)
   tenantId = DEFAULT_TENANT_ID;
  
  System.out.println("threadlocal tenant id ="+tenantId);
  return tenantId;
 }

 @Override
 public boolean validateExistingCurrentSessions() {
  return true;
 }

}



Finally, the hibernate configuration for specifying that we are using hibernate mutli-tenancy, in hibernate.cfg.xml





That's it people,
Happy multi-tenanting :-)

Cheers!