Sunday, June 11, 2017

Spark Quick Review : Study Notes

The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it. Users may also ask Spark to persist an RDD in memory, allowing it to be reused efficiently across parallel operations. Finally, RDDs automatically recover from node failures.

Driver program's in memory array can be converted to distributed data structure using parallelize
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
JavaRDD<Integer> distData = sc.parallelize(data);

Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc.

JavaRDD<String> distFile = sc.textFile("data.txt");

RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset.
For example, map is a transformation that passes each dataset element through a function and returns a new RDD representing the results. On the other hand, reduce is an action that aggregates all the elements of the RDD using some function and returns the final result to the driver program (although there is also a parallel reduceByKey that returns a distributed dataset).


Since RDDs are distributed data stuctures, something special needs to be done to count elements inside RDD.

// Wrong: Don't do this!!
rdd.foreach(x -> counter += x);

rdd.collect().foreach(println)  //can give out of memory since all data brought to drivernode
rdd.take(100).foreach(println)


Total Character Count Example

JavaRDD<String> lines = sc.textFile("data.txt");
JavaRDD<Integer> lineLengths = lines.map(s -> s.length());
int totalLength = lineLengths.reduce((a, b) -> a + b);


The following code uses the reduceByKey operation on key-value pairs to count how many times each line of text occurs in a file:

JavaRDD<String> lines = sc.textFile("data.txt");
JavaPairRDD<String, Integer> pairs = lines.mapToPair(s -> new Tuple2(s, 1));
JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);

Filter lines in a file for occurence of string "ganesh"

JavaRDD<String> logData = sc.textFile(logFile).cache();
JavaRDD<String> retData = logData.filter(s -> s.contains("ganesh"));


Different types of transformations and actions
https://spark.apache.org/docs/2.1.0/programming-guide.html


Word Count
JavaRDD<String> words = lines.flatMap(s->Arrays.asList(s.split(" ")).iterator()); JavaPairRDD<String, Integer> pairs = words.mapToPair(s -> new Tuple2(s, 1)); JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);



Shared Variables
Broadcast<int[]> broadcastVar = sc.broadcast(new int[] {1, 2, 3});

broadcastVar.value();
// returns [1, 2, 3]

LongAccumulator accum = jsc.sc().longAccumulator();

sc.parallelize(Arrays.asList(1, 2, 3, 4)).foreach(x -> accum.add(x));
// ...
// 10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s

accum.value();
// returns 10








Thursday, April 6, 2017

Simple security mechanism for token based auth

Almost every application we try to secure requires a security scheme for simple reliable token based authentication. I want to describe a simple yet effective scheme for token based authentication, which can be used.

Please review the security scheme below and comment, if it suitable and if any additional vulnerabilities and enhancements can be made. Please forward the blogpost link to, security experts/enthusiasts from your acquaintances, and ask them to review and comment.




User enters username and password on browser and clicks submit button
username and password travel over TLS onto web application's login handling resource URL

Session Initiation

On the webapp:
security filter uses the username and password, and authenticates user using application service
application creates  new security token
application stores token, token_time,etc against username, in app database
application encrypts token using its private RSA key
application sends back encrypted token as sync reply of http login request to client browser

On the client:
Client decrypts token using RSA public key. This ensures server's authenticity.
Client stores the raw token in session storage

Subsequent Session based calls

On the client:
Client sends in "pre-decided header field" or "Authorization header as Bearer token", the following:
HMAC(rawToken, sessionIdAsKey) - the hmac also ensures user or session authenticity

On the server:
Server security filter, checks if bearer token is present and also if it is valid as per follows:
server retrieves username's raw token, it does HMAC(rawToken, sessionIdAsKey)
server compares calculated HMAC with HMAC in bearer token, if both match token is valid


Vulnerabilities:

Client storage of RSA public key can be compromised and misused
Man in middle is still a risk, need to ensure TLS for all communications
Man in browser / Session hijacking is still a risk
Need to additionally guard against CSRF/replay attack

Enhancements

The simple raw token can be enhanced as a JWT token or digitally signed token

Thursday, November 19, 2015

A gentle introduction to java based electronic trading using FIX

I recently had the chance to work on a java based web application which provides electronic trading functionality using FIX(Financial Information eXchange). Wanted to share some of my experiences in this post :-)

Lets start out with the basics, though...

Electronic Trading the big picture, where does FIX fit in?
Trade life cycle can be divided into the following phases:
  1. Pre-Trade - analytics and price discovery
  2. Trade - order creation and execution
  3. Post Trade - clearing, allocation and settlement
  4. Post Settlement - risk mgmt, profit/loss account, position monitoring
FIX is mostly used in the Phases 1 and 2, whereas SWIFT is used in Phase 3 and beyond.

FIX protocol features like quote(RFQ), Indication of Interest (IOI) are used for price discovery, whereas features like DealAtQuote, Market Order and Limit Order are used for order placement.

How FIX is relevant  | Who Uses FIX
FIX® has become the way the world trades. Virtually every major stock exchange and investment bank uses FIX for electronic trading, alongside the world's largest mutual funds, money managers and thousands of smaller investment firms. Leading futures exchanges offer FIX connections and major bond dealers either have or are implementing them. Identifying an exact number of users is impossible, as FIX is a free and open standard, but it is very clear that the world’s financial community now speaks FIX.

Enterprise apps - Using Open Source QuickFixJ, the java library for FIX
Most enterprise applications, act as the BUY side of FIX protocol ie these applications place trade requests, to the SELL side of FIX, which is provided by vendors like winterflood, which have connectivity to stock markets, traders, etc and enterprise applications require to, send FIX messages to vendor's FIX engine.

In order to be able to interface with a FIX engine, of a provider like winterflood, using java code, there is an excellent open source framework called QuickFixJ. This is a java library which can be used as a JVM embedded FIX engine. Detailed documentation can be found at their website.

In following article, the java code I will  be show-casing code that uses the QuickFixJ APIs

Intro to common FIX flows
First, your application needs to communicate with a vendor who provides electronic trading services. Such vendors have a FIX server running with electronic trading capabilities and DMA(direct market access).

The whole idea is that our application will send a FIX message asking for a trade to be executed, to the vendors FIX engine. The vendor service will execute the trade and send back a response to our application. The responses are always asynchronous and can come in delayed or not at all.

Such trade instructions between our application and vendor FIX engine, using FIX messages format and protocol are called logically as FIX trade flows.

There are typically huge number of trade flow, sub-flows, combinations, varied responses, but the trick is to narrow down the number of trade flows by pre-deciding with vendor, which trade flows we will consider and also clamp down on the possible requests and responses.

This logical ring fencing of the possible trade flows will ensure, you can provide robust, deterministic functionality related to electronic trading, in your own application.

Quote with Deal At Quote Order
The most common trade interaction which can be done during market hours is the Quote and Deal functionality.

As per this trade flow, user sends out a quote request, for a ticker/symbol in the form of a FIX Message of type R. Asynchronously, but within a few milli seconds, a Quote response is received by the application. This quote response contains the market price of the ticker/symbol as a quote. This price in quote is applicable only for a a few seconds say for 30 seconds

The user now can send another FIX message within 30 seconds, asking to book order for above mentioned stock, with certain number of shares at the price agreed upon in the quote above

A more interactive and verbose narrative is mentioned below, along with the sample FIX messages, good luck deciphering :-)

Quote and Deal
Quote (R) - give me quote for stock with symbol VOD
<20150923-07:47:20, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=194 35=R 34=2 49=APP 50=D2C 52=20150923-07:47:20.027 56=VEND 115=12345 131=1442994439993 146=1 55=VOD.L 48=GB00BH4HKS39 22=4 207=XLON 54=2 38=10 64=20150925 60=20150923-07:47:20.026 15=GBP 120=GBP 10=009 )

Quote Response (S) - Unit market price for VOD is 2.445 pounds, valid for next 30 seconds
<20150923-07:47:20, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=245 35=S 49=VEND 56=APP 34=2 52=20150923-07:47:20 131=1442994439993 117=X90FZI23H4719001 55=VOD.L 22=4 48=GB00BH4HKS39 167=CS 207=XLON 132=2.1325 134=3750 62=20150923-07:47:50 64=20150925 15=GBP 120=GBP 152=21.325 76=VENDOR 60=20150923-07:47:20 10=088 )

Deal At Quote (D) - book an order for VOD @ 2.445 pounds for 100 shares
<20150923-07:47:20, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=238 35=D 34=3 49=APP 50=D2C 52=20150923-07:47:20.575 56=VEND 115=12345 11=1442994440565 15=GBP 21=1 22=4 38=10 40=D 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=4 60=20150923-07:47:20.575 63=3 64=20150925 117=X90FZI23H4719001 120=GBP 207=XLON 10=153 )

Exceution Report (8) for VOD @ 2.445 pounds for 100 shares with order status=FILLED
<20150923-07:47:20, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=359 35=8 49=VEND 56=APP 34=3 52=20150923-07:47:20 57=D2C 128=12345 1=WINTERG0 6=2.1325 11=1442994440565 14=10 15=GBP 17=b90FDI23H4720002 20=0 22=4 29=4 30=XLON 31=2.1325 32=10 37=b90FDI23H4720002 38=10 39=2 40=D 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=4 60=20150923-07:47:20 63=6 64=20150925 76=VENDOR 110=0 119=21.33 120=GBP 150=2 151=0 167=CS 207=XLON 10=244 )

other possible order status are: CANCELLED,  REJECTED, PARTIALLY_FILLED, etc

A quote can also be rejected, in which case a FIX message of type b is received by the application

Market Order
A market order is an order request which asks to buy/sell a certain number of shares of a symbol at the best price, at the time of request.

Market Order Request - 35=D order, 40=1 market,  54=2 sell, 55=VOD symbol, 38=110 num shares
<20150923-08:21:39, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=234 35=D 34=2 49=APP 50=D2C 52=20150923-08:21:39.383 56=VEND 115=12345 11=1442996499366 15=GBP 21=3 22=4 38=110 40=1 48=GB00BH4HKS39 54=2 55=VOD.L 59=6 60=20150923-08:21:39.380 63=3 64=20150925 120=GBP 126=20150925-08:21:39.381 207=XLON 10=110 )

Execeution Report (8) with order status=FILLED
<20150923-08:21:39, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=357 35=8 49=VEND 56=APP 34=2 52=20150923-08:21:39 57=D2C 128=12345 1=WINTERG0 6=2.1325 11=1442996499366 14=110 15=GBP 17=0D0FDI23I2139010 20=0 22=4 29=4 30=XLON 31=2.1325 32=110 37=0D0FDI23I2139010 38=110 39=2 40=1 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=3 60=20150923-08:21:39 63=6 64=20150925 76=VENDOR 119=234.58 120=GBP 150=2 151=0 167=CS 207=XLON 10=120 )


Limit Order
A limit order is an order request which asks to buy/sell a certain number of shares of a symbol, when the stock price of the symbol, reaches a certain specified limit price.

Limit Order Request - 35=D order, 40=2 market,  54=2 sell, 55=VOD symbol, 38=110 num shares
<20151009-09:05:22, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=206 35=D 34=229 49=APP 50=D2C 52=20151009-09:05:22.933 56=VEND 115=12345 11=1444381522933 15=GBP 21=2 22=4 38=110 40=2 44=7.885 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:05:22.933 63=3 120=GBP 207=XLON 10=112 )

Exceution Report (8) with order status=NEW (Early ACK for limit order)
Limit orders will be filled only when limit is reached
<20151009-09:05:23, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=296 35=8 49=VEND 56=APP 34=229 52=20151009-09:05:23 57=D2C 128=12345 6=0 11=1444381522933 14=0 15=GBP 17=y6u2743950 20=0 21=2 22=4 29=1 31=0 32=0 37=y6u2743950 38=110 39=0 40=2 44=7.885 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:05:23 63=3 120=GBP 150=0 151=110 167=CS 198=y6u2743950 207=XLON 10=065 )

when limit price is reached, Exceution Report (8) with order status=FILLED
<20151009-09:22:41, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=342 35=8 49=VEND 56=APP 34=268 52=20151009-09:22:41 57=D2C 128=12345 6=7.4 11=1444381522933 14=110 15=GBP 17=1040601735 20=0 21=2 22=4 29=4 30=XLON 31=7.4 32=110 37=y6u2743950 38=110 39=2 40=2 44=7.4 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:22:41 63=3 64=20151013 76=VENDOR 119=814 120=GBP 150=2 151=0 152=814 155=0 167=CS 207=XLON 10=208 )



The Java Code (finally!)

Quote request and deal-at-quote
Writing java code to submit a quote is as easy seen below

quickfix.fix42.QuoteRequest quote = new quickfix.fix42.QuoteRequest(new QuoteReqID(quoteReqId));

//optionally set header feilds that your Trade Provider may require
quote.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
quote.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

//create a new 'repeating group' as per FIX spec
quickfix.fix42.QuoteRequest.NoRelatedSym group = new quickfix.fix42.QuoteRequest.NoRelatedSym();

group.set(new Symbol("VOD"));
group.set(new SecurityID(GB00BH4HKS39 )); //value of ISIN: GB00BH4HKS39
group.set(new IDSource("4"));  //4=ISIN, 2=SEDOL
group.set(new SecurityExchange("XLON"));  //stock exchange ID
group.set(Side.BUY);   //this could just as well be Side.SELL

group.set(new OrderQty(10));  //u can buy 10 shares of ticker VOD
//group.setField(new CashOrderQty(4000.00));    //u can buy shares worth 4000 instead of 10 shares

group.setField(new SettlCurrency("GBP"));
group.set(new Currency("GBP"));
group.set(new TransactTime(new Date()));

quote.addGroup(group);

System.out.println("getQuoteForVEND FIX42 message "+quote);
Session.sendToTarget(quote, sessionId);

Deal@Quote
Writing java code to submit a Deal@Quote is as easy seen below
NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.PREVIOUSLY_QUOTED));

//optionally set header feilds that your Trade Provider requires
order.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
order.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

order.set(new SecurityID("GB00BH4HKS39"));   //ISIN for ticker VOD
order.set(new IDSource("4"));  //4=ISIN, 2=SEDOL
order.set(new SettlmntTyp(SettlmntTyp.T_PLUS_2));

order.set(new SecurityExchange("XLON"));  //stock exchange london

order.set(new QuoteID(origQuoteId));  //original quote ID
order.set(new Price(request.getPrice())); //price of previous quote

//order.setField(new OrderQty(10)); //u can buy 10 shares of ticker VOD
order.set(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10

order.set(new TimeInForce(TimeInForce.DAY));  //can be other values like
order.set(new SettlCurrency("GBP"));
order.set(new Currency("GBP"));

System.out.println("sending out DealAtQuotye FIX42 message "+order);


Session.sendToTarget(order, sessionId);


Market Order
Writing java code to send a market order is as easy as seen below

NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.MARKET));

//optionally set header feilds that your Trade Provider requires
order.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
order.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

order.set(new SecurityID("GB00BH4HKS39"));   //ISIN for ticker VOD
order.set(new IDSource("4"));  //4=ISIN, 2=SEDOL
order.set(new SettlmntTyp(SettlmntTyp.T_PLUS_2));

order.set(new SecurityExchange("XLON"));  //stock exchange london

order.setField(new OrderQty(10)); //u can buy 10 shares of ticker VOD
//order.set(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10

order.set(new TimeInForce(TimeInForce.DAY));  //can be other values like
order.set(new SettlCurrency("GBP"));
order.set(new Currency("GBP"));

System.out.println("sending out MarketOrderForVEND FIX42 message "+order);
Session.sendToTarget(order, sessionId);


Limit Order
Writing java code to send a Limit order is as easy as seen below.
Only change from above code is using the OrdType.LIMIT

NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.LIMIT));



Using the above concepts, frameworks and codes, provides some idea of the  electronic trading functionalities, that can be integrated into your java based enterprise application.

Cheers!

Friday, August 28, 2015

REST APIs for Simpler Payment Gateway Integration into your web and mobile apps

I recently had the chance to integrate the current web application I am working on with a popular UK based payment gateway for credit and debit cards, named Worldpay Online . Carrying out the above integration was surprisingly easy and below are the details.

Our requirement was that our end users should be able to pay for our application services using their debit / credit cards. I embarked on the payment gateway integration journey initially thinking of traditional ways of integration. We did not wish to incur the additional overhead of any PCI DSS compliance that occurs, if you choose to store any card details on your site. Hence, I started considering more traditional approaches like our site will redirect payment requests to the payment gateway hosted "payment pages".

These typically gives the end user the ability to choose their payment method, input their card details and ask for their card to be charged for relevant amounts.

Even worldpay as our preferred payment gateway provider had these options under the integration type of "HTML Redirect". However the main drawbacks of this kind of "payment page" based integration were as follows:

  1. No control and limited customisation of the payment pages
  2. Lack of a strong application context while the user was on the payment pages
  3. The mechanism of gateway provider letting us know that the card was charged successfully made the whole process asynchronous, as it was a post back to our application servers
Well, it turns out there is a much simpler approach available in most modern payment gateways. Using the payment gateways REST APIs. I used the worldpay online REST APIs, to quickly integrate the payment gateway into our application. Here I will talk a bit about my experience using the worldpay online payment APIs.

Though we use the payment REST APIs, we still do not need to have an PCI-DSS compliance. The reason is that we never capture or store the card details on our application. The entire process is broken down into 2 steps:
Step 1: Our application displays inline or as pop-up a worldpay online screen. This worldpay screen captures the card details and gives back a temporary token to the application.

Step 2: The application uses the above temporary token and actually calls the REST payment API. The token acts as a proxy for the card details like card number, expiration date, CVV etc. Application never knows about card details, they are known only to worldpay online!



The best part is that:

  • Except for a small portion of page which captures card details, rest of webpage is under applications control and hence customizable
  • The actual REST API call is synchronous so error handling effort is reduced by a huge amount



The actual payment REST call is as simple as shown below

Example Request

curl https://api.worldpay.com/v1/orders
    -H "Authorization:your-service-key"
    -H "Content-type: application/json"
    -X POST
    -d
'{
    "token" : "your-token",
    "orderDescription" : "order-description",
    "amount" : 500,
    "currencyCode" : "GBP"
}'

Example Response

{
    "orderCode": "worldpay-order-code",
    "token" : "your-token",
    "orderDescription" : "your-order-description",
    "amount" : 500,
    "currencyCode" : "GBP",
    "customerOrderCode":"my-customer-order-code",
    "paymentStatus": "SUCCESS",
    "paymentResponse": {
        "type": "ObfuscatedCard",
        "name": "name",
        "expiryMonth": 2,
        "expiryYear": 2015,
        "cardType": "VISA_CREDIT",
        "maskedCardNumber": "**** **** **** 1111",
        "billingAddress": {
            "address1": "18 Linver Road", 
            "postalCode": "SW6 3RB", 
            "city": "London", 
            "countryCode": "GB"
        },
        "cardSchemeType": "consumer",
        "cardSchemeName": "VISA CREDIT",
        "cardIssuer": "LLOYDS BANK PLC",
        "countryCode": "GB",
        "cardClass": "credit",
        "cardProductTypeDescNonContactless": "unknown",
        "cardProductTypeDescContactless": "unknown",
        "prepaid": "false"
    },
    "shopperEmailAddress": "name@domain.co.uk",
    "environment": "TEST",
    "riskScore": : {
        "value": "1" 
    }
}


For more information please visit Worldpay online documentation

Worldpay as a gateway provides various features, (many of which I implemented successfully) like

  • Recurring Payments
  • Card-on-File (for re-using previously entered card information)
  • 3d-Secure integration
  • Refunds
  • Dispute Defence
  • Telephone orders
  • .. and so on 

Happy integration, people :-)