Agent AI Workflows, Deja Vu?

LLMs have had a great run, up the hype cycle. After the initial euphoria, things seem to plateauing off in terms of potential usage, the burst of Generative AI not withstanding. Also the LLMs are not making much inroads into the super lucrative market of enterprise development. They have spent billions on evolving the models of these LLMs, the incremental evolution from here on can hardly pay the bills.

Refer my earlier post on the difficulties of LLM getting to learn about custom enterprise data models. In fact the only way to make inroads into the enterprise market and get a piece of the pie, is by somehow making enterprise business process development, dependent on the LLM.

As things stand right now, it would be very difficult and time consuming to enable LLMs to directly understand custom business processes, so what is the next best strategy now? Enter Agentic workflows ....

Every few years or so, this century, we have had technologies like EJBs, BPM, BPEL, ESB try to convince enterprises that they were the way to model business logic inside business processes. Well there is a new alternative which will now try to model business logic and processes, while being orchestrated by none other than the LLMs.

The #AgenticAI workflows with their nodes, edges, human-in-the-loop, sub-graphs for sub-processes, workflow designing studios, deployment servers will have the ability to loop with exit conditions contrasting them from DAG based BPM flows. The agentic workflow is a classical state machine implementation controlled to an extent by the LLM, but dictated primarily by business logic any which ways.

Meanwhile the inherent complexity of any business process stays just the same, irrespective of the technology used. Many of the lower intricacies of the business process may still need to be modelled as service implementations with persistent domain data models. The autonomous decision making part is left completely at the risk of designers who design the agents, agentic workflows, with LLM providing limited control or <ahem> intelligence.

The amount of sematic modelling involved in an advanced state machine, which will act as autonomous agents, is no trivial design exercise and maybe base agent reference implementations will evolve in due course to make business application development easier and less riskier. 

Are there less expensive options of leveraging LLMs intelligence, in the enterprise, as pin-point integrations points within a custom business process, apart from RAG based approach? Comment below.

Meanwhile, I am rushing off to continue learning #LangGraph, #Autogen and #CrewAI, since we all need to be in the game, right?

Why I am starting to believe that time is an illusion

We have refined our understanding of gravity from being a force of nature that attracts 2 masses, to gravity being merely the curvature of space-time to gravity being interpreted as time slowing down in space.

The other important ongoing discovery has to do with dark matter and dark energy which constitutes around 95% of our observable universe. Commensurate with evolving insights of dark matter and energy, dark energy at least seems to be the omnipresent energy in the vacuum of space, spread out over such large spaces that its detection is faint due to its density being very less.

I imagine, that dark matter may also on similar lines differ from ordinary matter, in that, it is much less dense and spread out over the vast expanse of inter galactic spaces. The current tug of war between our expanding universe v/s gravity, could then be seen as a war of equilibrium between normal matter/energy vs dark matter/energy, and clearly normal matter is getting rarefied, on the overall scale of the universe.

Next, lets for a moment assume that the nature of reality(matter and energy), truly is made up of E8 crystals or any other unit of reality, for that matter. So ordinary matter could be nothing but "densely packed E8 crystals" and dark matter could be "very rarefied" E8 crystals. 

Time as we know it, is perceived by us, as beings made up of matter. Our perception of time primarily is from the point of view of how it appears in the "ordinary matter" portion of the universe. In the vacuum of space and in the dark matter part of the universe, is time just as relevant?

The perception of time is linked to causality, as experienced by us in the macro universe. 

Time in the macro universe might seem ubiquitous but at sub atomic and quantum scales, where wave nature, duality, uncertainty and randomness are the norm, does the linear flow of time, as in the macro universe even hold relevance?

Time needs a frame of reference and an observer, without these, being clear to us, over the expanse of the universe and dark matter, is time a relevant variable for study, of particle physics itself?

Now can you blame me for tending to think, that time might be an illusion of our perception of the macro universe.

-GG

Throttling Kafka Consumption - Dispelling Myths

Kafka as a messaging broker was designed for high throughput publish and subscribe. At times this very fact can prove to be an issue, when being used in more traditionally designed transactional systems. For example if an application is consuming messages from a kafka topic and inserting, updating in a relational database, the high rate of consumption of kafka messages, can put pressure on transactional persistence in the RDBMS and cause exceptions. Hence there arises a need to throttle message consumption from a kafka topic.

Starting out, lets assume, that we are only considering the message consumption inside a single instance of a consumer (which may be a part of a bigger consumer group). So we have a single consumer instance, picking up kafka messages from a single partition of a topic.

Most kafka client frameworks, will implement a decent kafka consumer poll-loop, for reading kafka messages from the partition as soon as possible and then hand these messages over to client application, as fast as possible, but without waiting/blocking for the client application to process each message.

For single thread technologies like node js, the above can be an issue,in the sense that, the result is that the application starts processing large number of messages in parallel. Many will resort to kafka configuration parameters like below, to control the rate of consumption of messages from kafka, with very little success, for the reasons I mention below:

fetch.min.bytes - This property allows a consumer to specify the minimum amount of data that it wants to receive from the broker when fetching records. If a broker receives a request for records from a consumer but the new records amount to fewer bytes than fetch.min.bytes, the broker will wait until more messages are available before sending the records back to the consumer. 

fetch.max.wait.ms - By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer.If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.

max.partition.fetch.bytes - This property controls the maximum number of bytes the server will return per partition.

session.timeout.ms - The amount of time a consumer can be out of contact with the brokers while still considered alive defaults to 10 seconds.

max.poll.records - This controls the maximum number of records that a single call to poll() will return.

receive.buffer.bytes and send.buffer.bytes - These are the sizes of the TCP send and receive buffers used by the sockets when writing and reading data.

If we manipulate above values, messages will be fetched more frequently OR less frequently. But if  already 1000s of messages are lying in the topic, it wont matter a lot, given that fetch from topic to consumer is very quick(micro seconds). The consumer will still feel a steady supply of almost parallel messages.

Even playing around with commit offsets, will never block consumption of messages but only influence recovery, guaranteed delivery, once only delivery, etc. This will not be able to restrict super-fast, reading of messages by the consumer. (Its not like late acknowledgements, block, consumption of messages in traditional message brokers like JMS, MQ, etc)

The bottom line is that, the challenge of parallel consumption of messages in a kafka instance consumer, needs to be handled by the process/application and not by kafka configurations.

Consumer Side Technology

A much more important factor while dealing with throttling of kafka messages, is which tech/architecture you use, on the consumer side application.

Using Java / SpringBoot based consumer, will ensure that the consumer process runtime will be multi-thread-based and will be synchronous first or synchronous by default. The use of thread-pools in Java/Spring and given the fact that almost all calls in Java are synchronous by default, it is much easier to control the kafka message consumption in Java

Node JS based consumers are inherently single threaded an hence asynchronous first or asynchronous by default, using await, promises and callbacks in a way, that near-parallel consumption of kafka messages will be challenge, especially for less-experienced developers.

Process Mining - Practical Uses

Process mining is a family of techniques that support the analysis of business processes based on event logs. During process mining, specialized data mining algorithms are applied to event log data in order to identify trends, patterns and details contained in event logs, to auto-discover and paint process definitions of processes. Process mining can be used to discover processes from events or improve process efficiency by identifying bottlenecks and exceptions

By itself process mining has several applications in large enterprises, to discover or validate business processes (in finance, manufacturing and retail) merely from events and timestamps, but the exciting thing is that the same technique can be used to get insights into user/customer behaviors on your website and also aid in customer segmentation.

Consider a simplistic apache web log (though much more detailed application logs can also be used)
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /login.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /catalog.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /checkout.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /checkstatus.html HTTP/2" 200 1479

The important pieces of information are the user id "peter", the date timestamp and the URL visited which can proxy up for the user's intended action. The above log can be easily transformed into

Sample CSV file

<user>,<action>,<action date>, <user attribute 1>, <user attribute 2>, <user attribute n>
UserId,Action,StartTimeStamp,EndTimeStamp,PremiumCustomer,UserLocation,UserOccupation
Peter,Logged In,12-20-2017  10:20:00 AM,12-20-2017  10:25:00 AM,premium_customer,US,architect
Bob,Logged In,12-20-2017  10:20:00 AM,12-21-2017  10:25:00 AM,premium_customer,US,architect
Peter,Browsed Catalog,12-20-2017  10:35:00 AM,12-20-2017  10:40:00 AM,premium_customer,US,architect
Alice,Browsed Catalog,12-21-2017  10:35:00 AM,12-21-2017  10:40:00 AM,premium_customer,US,architect
Bob,Browsed Catalog,12-21-2017  10:35:00 AM,12-21-2017  10:40:00 AM,premium_customer,US,architect

The above CSV data can now to be fed into a process mining tool to generate the "user/customer's" interaction process, as he navigates and uses the website. This can give many insights into how customers are using our websites and what we can do to make things more convenient for them.

Statistics like what time duration is spent by each user in any of the activities can also be readily obtained as a report from the tool. The major deviations in the main process flow can also be denoted. Animations of the process flow, over a period of months can be played out within minutes to get insight into the activities that are bottle-necking the overall process.

Instead of discovering processes for individual users we can generalize processes for say "platinum customers" by filtering the data within the process mining tool itself.

Process Mining deals with Petrinet and BPMN notations of process definitions and some popular algorithms for process mining are alpha miner, inductive visual miner and fuzzy miner.

Notes on Enterprise Application Development Strategy

At the end of the day, technology is an enabler for extracting business value. Some sure fire ways to burn money needlessly, during enterprise application development are noted below. Use it as a checklist to guard against, in your own enterprise development. As always, use your own context, while weighing-in, any generic advice.

• Don't care why and what the end user actually wants. Its more important to impress top management, with a good-looking demo
• Using a particular tech because it is popular at google, netflix or facebook, irrespective of its applicability within your own context, especially NFR context
• No clarity about the WHY, obsessed with only the HOW
• Not able to contain scope creep and needless escalation of complexity
• First thought that comes up becomes the definitive design, no choices, no alternatives considered
• Resume driven or Ego driven, development by tech leads
• Why use a suitable tech, when a tougher, riskier, newer tech is at hand, no matter how irrelevant
• Underestimating integration and dependencies
• Committing to technology choices without understanding the full scope of functionality and the non-functional requirements(NFR) in detail
• Forgetting that no application is an island. Its important that an application can "fit into" the enterprise landscape in terms of security standards, IDAM (identity and access management), audit standards, deployment, reliability and business continuity standards
• No thoughts given for migration path from existing application/s
• Disproportionate emphasis on UI look and feel rather than UX and information models
• Re-invent the wheel, under the guise of tech bravado
• Dev Teaming mistakes
• Thinking that employing 10 mediocre developers, instead of 3 crack devs, will be better
• Believe the myth that application devs, still write framework level code
• Never acknowledge, that app devs now, require to use proper frameworks and APIs, rather than belt out algorithms and complex programs

Blockchain - Usecases in supply chains

Supply Chains, are the lifeline of businesses globally. Most likely, your business will find itself between upstream suppliers and downstream sales channels/distributors/customers. The figure below indicates this topology succinctly.

Generic Supply Chain connected via a Blockchain

Your organizations ability to fulfillment of its sales orders, totally depends on how efficient, agile your supply chain is. Some of the areas where blockchain technology can add great business value are:

Optimizing Inventory Costs
Though organizations have fairly automated internal processes, most still rely on antiquated ways of interacting with their suppliers and business partners, Orders can be placed over voice calls, email, logging into each other's portals, etc. Tracking of these orders status is never, in real time, mostly left to voice calls, email reminders, etc. In fact, it would not be uncommon for your organization to get notified of a supplier's delay in order fulfillment, close to your own, downstream, fulfillment commitments. By having, deep, blockchain-based integrations, with your supplier's ERP, you can ensure almost real-time notifications of any potential delays, in your order fulfillment. You can automate balancing of orders across your other suppliers as well.

If your distributors/sales partners are also on the blockchain, any weakness in demand, can also be notified to your organization in real-time, so as to re-balance supplier-side orders. This will lead to highly optimized inventories and substantial reduction in inventory costs. De-risking from supply side outages due to a specific supplier is another collateral benefit.

By having most of your suppliers on the chain, you will get unprecedented clarity about the big picture on the supply side, across all your suppliers. This can help you in coordinating receipt of goods from different suppliers, with a consolidated scheduling system based on blockchain data. This can further optimize the timing and logistics aspects of the inventory.

Minimizing Quality Costs using the blockchain
With the blockchain in place, your organization can have selective access to your supplier's ERP Data, including their quality control data. Having this level of accurate data, you can choose to minimize redundant quality checks, at your end. This is the good example of using a blockchain to enable trust and optimize costs.

Suppliers of a Supplier can also be brought onto the blockchain, to enable deeper and far reaching integration via the blockchain.

A blockchain can be used very effectively to "track the state" of a logical unit of delivery across supplier's parts. For example, if your organization was making refrigerators, every part, being supplied, tracked with the various suppliers, can be co-related with the logical entity "refrigerator 123" on the blockchain, hence, at any given point in time, you could not only know that 30 compressors, 26 coolant tubes are being supplied to me, but also that, I will be able to get out 26 complete refrigerators to my customers.

Bidding based supplier order management
The same trust based blockchain technology can be used to operate a bidding platform, to manage orders being issued out to the suppliers

Why traditional integration technologies like API, will just not work?
Point to point integrations with multiple suppliers not are just not scalable, in the supply chain ecosystem. imagine a supplychain with 5 nodes, each talking to other via point to API calls. Blockchains, along with providing a decentralized datastore, to track "shared state", also provide a standard stack for integration, at all peers, not to mention "consistent versions" of same distributed smart contracts for processing the data.

The suppliers that you bring onto the blockchain, may be direct competitors, inherently having a lack of trust about using processes, business logic, databases and codebases, belonging to each other, but the decentralized blockchain, will make it possible, to bring all such parties onto a common platform, sharing data/state, business processes and code.

Countering, Counterfeiting

While researching into the practical applications of blockchain tech, in provenance, tracking and traceability, I came across claims that blockchain can help to tackle counterfeiting, which is a serious issue in the fashion, luxury and branded products industry.

The general opinion seemed to be that its tough to fight counterfeiting because establishing authenticity of an original product item, over a cleverly counterfeited one, was a tough nut to crack (at affordable technologies).

Lets get into the mind of counterfeiter then...Typically counterfeiters will reproduce a near-duplicate of the original, at a fraction of the cost, then package it exactly like an original and send it out to be sold. It would be next to impossible to catch a cleverly made and packaged counterfeit item.

The barcodes and qrcodes on the item package can be copied to the T, as well and then pasted onto the duplicates package. Standard barcodes (EAN-13) contain only the manufacturer code and product category information, which is popularly recognized as a GS1 standard GTIN, like is typically used to scan retail products as POS terminals. But the real problem like I said is that, the entire barcode can be copied and put onto a duplicate item, making it indistinguishable from the original. Authenticity is sadly not guaranteed even if additional attributes about the product item were embedded into extended barcode versions like the GS1 Databar or GS1-128.

For authenticity checking, the trick is to use 2 factor authentication. Imagine a qr code encoding a random number, printed on the outside packaging of a luxury product. Now another QR code with yet another encoded number present inside the product itself (say on the product lining or inside of the cap of the bottle). Now we have 2 numbers, one on the outside and one on the inside(can be accessed, only after breaking the seal of the product).

A simple web page/mobile app can be developed to validate the compatibility of the inside number and the outside number. As long as the compatibility checking algorithm is secret (maybe using a magic number/s), the end user can just scan the outside and inside QR codes/numbers, to let the website/mobile app, determine if the product is authentic or counterfeit.

Even if the counterfeiter manages to exactly copy, the outer packaging, he can never "quess" what the inside number should be, since he does not know the exact algorithm, that is used to derive/correlate to the inside number.

The last problem to solve, is the possibility that the counterfeiter may get hold of a few good samples of "valid" number pairs and use them with all the counterfeits repeatedly ! This can be easily tracked, since the website/app is centralized and can keep track how many times the query for checking authenticity is being received and from what sources, mobiles, geo-locations etc

Let know in the comments, any loopholes, I may have missed or any improvements you can think of.

Integrating existing enterprise apps with blockchain

Lowering the barrier to entry, is key for increased adoption of blockchain based technologies into enterprises. Irrespective of the business value, that blockchains bring, there is a need to decrease the cost, pain and risk of integrating blockchains with existing applications in the enterprise.

Most large enterprises today, have established ERPs, CRMs, or cloud native custom applications, running 24 x 7. It makes sense to interface the new blockchain tech, harmoniously into the enterprise architecture landscape.

Its quite practical to assume that, most organizations, while they would want to participate in a blockchain and get all the benefits, will most likely be reluctant to do any major modifications to their existing enterprise apps. The need of the hour is, to ease ingestion of data onto the blockchain, without requiring expensive UI based / operational alternatives.

Presented below are a few ideas, for ingesting data onto blockchains, in an automated and batched manner, with minimal integration development and risk. Refer my earlier article about, wrapping up blockchain clients inside REST webservices, the below ideas can be used in conjunction with the REST adapters.

ERP Application Logs >> Logstash >> Http Post to Blockchain REST adapters >> Blockchain

Existing Application JDBC DB >> Database Polling >> Http Post to Blockchain REST adapters >> Blockchain

Existing Application publish message >> Topics (Message Provider/Kafka) >> Logstash >> Http Post to Blockchain REST adapters >> Blockchain

When the need for capturing data onto the blockchain is critical, more reliable and resilient, pipeline with Kafka based topics can be used.

Blockchain alternative to API integration across enterprises

What is the go-to solution when 2 enterprises wish to communicate with each other. Right! Its via APIs, that too using REST APIs, with appropriate security in place.

Typically this involves one of the companies developing webservices and exposing those services, to be consumed by another enterprise. There are several constraining factors for this kind enterprise integration:

The webservices paradigm is uni-directional, that is, if org A, invokes a webservice hosted by org B, for org B, to confirm that desired state changes have occured, org B must call another webservice hosted by org A. This is essentially the issue, that the webservice calls are stateless. There is no state sharing at all between the organizations.

For true leverage and reuse of the webservices, there needs to be orchestration at the consumer side. This orchestration is not transparent to the provider of webservices.

The communication is primarily point to point. So the interactions, cannot be scaled easily if newer participants join in. Imagine 4 organizations, each one trying to call services into others and trying to keep track of some shared state, using and exposing status calls

All the above issues are elegantly solved using the "shared state" paradigm of blockchain. All participants talk to the blockchain almost as if it were a logical messaging bus, but more importantly, the bus is not just a physical connection, but maintains state in distributed ledger and also has smart contracts that can house, common and transparent pieces of code, that can be shared across the un-trusting organizations aka partners

Server to server API calls Security

Calling Partner APIs from our application servers is a common use-case, a quick summary of options for securing such Server to server API calls

Cloud, Elasticity and Microservices

Many enterprise customers are being seen talking about moving to the cloud and adopting microservices based architectures. This is especially seen in the Indian BFSI domain. Heres my two cents at how the things are evolving logically and trade offs that all BFSI Businesses should be aware of.

The incentive to move to the cloud
Conventional wisdom is slowly but surely tilting towards moving IT infrastructures to secure public clouds like AWS, Azure and IBM Bluemix. The reasons are many and varied and well established by now, I wont regurgitate the same here again. Irrespective of the particular reason for moving the newer enterprise applications to the cloud, it makes sense to ensure that the underlying applications can "leverage" the cloud based infrastructures to the fullest.

Elasticity is important
The cloud provides for dynamic provisioning of resources like processing power, storage, network bandwidth, etc. It then follows logically, that enterprise applications that are deployed on the cloud should be fully capable of "leveraging" this elasticity of the cloud to "scale out" over resources.
Most importantly since clouds are pay-as-you-go model, if the underlying architecture is able to "scale down" automatically, when customer demand is low, the cloud costs for running applications can be optimized. Such architectures which can optimize operating costs of the cloud are very desirable. Mind you, in the cloud, being able to "scale down" as per demand is not just a good-to-have but rather an imperative that has direct cost implications.

Enter Microservices
In a monolithic application based approach including app servers or ESBs or service orchestration containers, the size of the monolith restricts the extent to which the scale-down can occur. Hence the architectural approach of microservices which are much more granular lends itself to better scale down capabilities, along with various important technical advantages of microservices

Containerization and choices
OK so microservices lend themselves well to cloud deployments, due to their granularity! But managing microservices, monitoring them, ensuring fault tolerance, reliability, scalability, is now a challenging task. Using light weight and proven container technologies like docker becomes inevitable. Docker also provides the much needed portability.
But managing the containers can itself become a complex task. Using a container management tool like docker swarm is good for less complex deployments and if you need to a single out of the box solution for container management. For more complex enterprise wide deployments a open source and portable and pluggable docker management framework like kubernetes is  much more practical.
For a quick compare between swarm and kubernetes

In addition, the docker container will need some deep integrations with your cloud provider, remember all this is running on the cloud ;-)
Here too, we have cloud provider based solutions like EC2's Container service which can be considered to be alternatives to kubernetes. But be aware that choosing a cloud provider based container management solution will mean no portability and vendor lock-in with your cloud provider


The architectural shift to microservices, is just the tip of the ice berg when considering your move to the cloud. In subsequent articles I will be talking about other aspects of your move to the cloud, which will affect the way your enterprise applications interact with the world. Some topics I intend to cover are Security and using API Gateways for integrations.

Spark Quick Review : Study Notes

The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it. Users may also ask Spark to persist an RDD in memory, allowing it to be reused efficiently across parallel operations. Finally, RDDs automatically recover from node failures.

Driver program's in memory array can be converted to distributed data structure using parallelize

List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
JavaRDD<Integer> distData = sc.parallelize(data);

Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc.

JavaRDD<String> distFile = sc.textFile("data.txt");

RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset.
For example, map is a transformation that passes each dataset element through a function and returns a new RDD representing the results. On the other hand, reduce is an action that aggregates all the elements of the RDD using some function and returns the final result to the driver program (although there is also a parallel reduceByKey that returns a distributed dataset).


Since RDDs are distributed data stuctures, something special needs to be done to count elements inside RDD.

// Wrong: Don't do this!!
rdd.foreach(x -> counter += x);

rdd.collect().foreach(println)  //can give out of memory since all data brought to drivernode

rdd.take(100).foreach(println)


Total Character Count Example

JavaRDD<String> lines = sc.textFile("data.txt");
JavaRDD<Integer> lineLengths = lines.map(s -> s.length());
int totalLength = lineLengths.reduce((a, b) -> a + b);

The following code uses the reduceByKey operation on key-value pairs to count how many times each line of text occurs in a file:

JavaRDD<String> lines = sc.textFile("data.txt");
JavaPairRDD<String, Integer> pairs = lines.mapToPair(s -> new Tuple2(s, 1));
JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);

Filter lines in a file for occurence of string "ganesh"

JavaRDD<String> logData = sc.textFile(logFile).cache();
JavaRDD<String> retData = logData.filter(s -> s.contains("ganesh"));


Different types of transformations and actions
https://spark.apache.org/docs/2.1.0/programming-guide.html


Word Count
JavaRDD<String> words = lines.flatMap(s->Arrays.asList(s.split(" ")).iterator()); JavaPairRDD<String, Integer> pairs = words.mapToPair(s -> new Tuple2(s, 1)); JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);



Shared Variables

Broadcast<int[]> broadcastVar = sc.broadcast(new int[] {1, 2, 3});

broadcastVar.value();
// returns [1, 2, 3]

LongAccumulator accum = jsc.sc().longAccumulator();

sc.parallelize(Arrays.asList(1, 2, 3, 4)).foreach(x -> accum.add(x));
// ...
// 10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s

accum.value();
// returns 10

A gentle introduction to java based electronic trading using FIX

I recently had the chance to work on a java based web application which provides electronic trading functionality using FIX(Financial Information eXchange). Wanted to share some of my experiences in this post :-)

Lets start out with the basics, though...

Electronic Trading the big picture, where does FIX fit in?
Trade life cycle can be divided into the following phases:

1. Pre-Trade - analytics and price discovery
2. Trade - order creation and execution
3. Post Trade - clearing, allocation and settlement
4. Post Settlement - risk mgmt, profit/loss account, position monitoring

FIX is mostly used in the Phases 1 and 2, whereas SWIFT is used in Phase 3 and beyond.

FIX protocol features like quote(RFQ), Indication of Interest (IOI) are used for price discovery, whereas features like DealAtQuote, Market Order and Limit Order are used for order placement.

How FIX is relevant  | Who Uses FIX
FIX® has become the way the world trades. Virtually every major stock exchange and investment bank uses FIX for electronic trading, alongside the world's largest mutual funds, money managers and thousands of smaller investment firms. Leading futures exchanges offer FIX connections and major bond dealers either have or are implementing them. Identifying an exact number of users is impossible, as FIX is a free and open standard, but it is very clear that the world’s financial community now speaks FIX.

Enterprise apps - Using Open Source QuickFixJ, the java library for FIX
Most enterprise applications, act as the BUY side of FIX protocol ie these applications place trade requests, to the SELL side of FIX, which is provided by vendors like winterflood, which have connectivity to stock markets, traders, etc and enterprise applications require to, send FIX messages to vendor's FIX engine.

In order to be able to interface with a FIX engine, of a provider like winterflood, using java code, there is an excellent open source framework called QuickFixJ. This is a java library which can be used as a JVM embedded FIX engine. Detailed documentation can be found at their website.

In following article, the java code I will  be show-casing code that uses the QuickFixJ APIs

Intro to common FIX flows
First, your application needs to communicate with a vendor who provides electronic trading services. Such vendors have a FIX server running with electronic trading capabilities and DMA(direct market access).

The whole idea is that our application will send a FIX message asking for a trade to be executed, to the vendors FIX engine. The vendor service will execute the trade and send back a response to our application. The responses are always asynchronous and can come in delayed or not at all.

Such trade instructions between our application and vendor FIX engine, using FIX messages format and protocol are called logically as FIX trade flows.

There are typically huge number of trade flow, sub-flows, combinations, varied responses, but the trick is to narrow down the number of trade flows by pre-deciding with vendor, which trade flows we will consider and also clamp down on the possible requests and responses.

This logical ring fencing of the possible trade flows will ensure, you can provide robust, deterministic functionality related to electronic trading, in your own application.

Quote with Deal At Quote Order
The most common trade interaction which can be done during market hours is the Quote and Deal functionality.

As per this trade flow, user sends out a quote request, for a ticker/symbol in the form of a FIX Message of type R. Asynchronously, but within a few milli seconds, a Quote response is received by the application. This quote response contains the market price of the ticker/symbol as a quote. This price in quote is applicable only for a a few seconds say for 30 seconds

The user now can send another FIX message within 30 seconds, asking to book order for above mentioned stock, with certain number of shares at the price agreed upon in the quote above

A more interactive and verbose narrative is mentioned below, along with the sample FIX messages, good luck deciphering :-)

Quote and Deal
Quote (R) - give me quote for stock with symbol VOD
<20150923-07:47:20, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=194 35=R 34=2 49=APP 50=D2C 52=20150923-07:47:20.027 56=VEND 115=12345 131=1442994439993 146=1 55=VOD.L 48=GB00BH4HKS39 22=4 207=XLON 54=2 38=10 64=20150925 60=20150923-07:47:20.026 15=GBP 120=GBP 10=009 )

Quote Response (S) - Unit market price for VOD is 2.445 pounds, valid for next 30 seconds
<20150923-07:47:20, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=245 35=S 49=VEND 56=APP 34=2 52=20150923-07:47:20 131=1442994439993 117=X90FZI23H4719001 55=VOD.L 22=4 48=GB00BH4HKS39 167=CS 207=XLON 132=2.1325 134=3750 62=20150923-07:47:50 64=20150925 15=GBP 120=GBP 152=21.325 76=VENDOR 60=20150923-07:47:20 10=088 )

Deal At Quote (D) - book an order for VOD @ 2.445 pounds for 100 shares
<20150923-07:47:20, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=238 35=D 34=3 49=APP 50=D2C 52=20150923-07:47:20.575 56=VEND 115=12345 11=1442994440565 15=GBP 21=1 22=4 38=10 40=D 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=4 60=20150923-07:47:20.575 63=3 64=20150925 117=X90FZI23H4719001 120=GBP 207=XLON 10=153 )

Exceution Report (8) for VOD @ 2.445 pounds for 100 shares with order status=FILLED
<20150923-07:47:20, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=359 35=8 49=VEND 56=APP 34=3 52=20150923-07:47:20 57=D2C 128=12345 1=WINTERG0 6=2.1325 11=1442994440565 14=10 15=GBP 17=b90FDI23H4720002 20=0 22=4 29=4 30=XLON 31=2.1325 32=10 37=b90FDI23H4720002 38=10 39=2 40=D 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=4 60=20150923-07:47:20 63=6 64=20150925 76=VENDOR 110=0 119=21.33 120=GBP 150=2 151=0 167=CS 207=XLON 10=244 )

other possible order status are: CANCELLED,  REJECTED, PARTIALLY_FILLED, etc

A quote can also be rejected, in which case a FIX message of type b is received by the application

Market Order
A market order is an order request which asks to buy/sell a certain number of shares of a symbol at the best price, at the time of request.

Market Order Request - 35=D order, 40=1 market,  54=2 sell, 55=VOD symbol, 38=110 num shares
<20150923-08:21:39, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=234 35=D 34=2 49=APP 50=D2C 52=20150923-08:21:39.383 56=VEND 115=12345 11=1442996499366 15=GBP 21=3 22=4 38=110 40=1 48=GB00BH4HKS39 54=2 55=VOD.L 59=6 60=20150923-08:21:39.380 63=3 64=20150925 120=GBP 126=20150925-08:21:39.381 207=XLON 10=110 )

Execeution Report (8) with order status=FILLED
<20150923-08:21:39, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=357 35=8 49=VEND 56=APP 34=2 52=20150923-08:21:39 57=D2C 128=12345 1=WINTERG0 6=2.1325 11=1442996499366 14=110 15=GBP 17=0D0FDI23I2139010 20=0 22=4 29=4 30=XLON 31=2.1325 32=110 37=0D0FDI23I2139010 38=110 39=2 40=1 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=3 60=20150923-08:21:39 63=6 64=20150925 76=VENDOR 119=234.58 120=GBP 150=2 151=0 167=CS 207=XLON 10=120 )

Limit Order
A limit order is an order request which asks to buy/sell a certain number of shares of a symbol, when the stock price of the symbol, reaches a certain specified limit price.

Limit Order Request - 35=D order, 40=2 market,  54=2 sell, 55=VOD symbol, 38=110 num shares
<20151009-09:05:22, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=206 35=D 34=229 49=APP 50=D2C 52=20151009-09:05:22.933 56=VEND 115=12345 11=1444381522933 15=GBP 21=2 22=4 38=110 40=2 44=7.885 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:05:22.933 63=3 120=GBP 207=XLON 10=112 )

Exceution Report (8) with order status=NEW (Early ACK for limit order)
Limit orders will be filled only when limit is reached
<20151009-09:05:23, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=296 35=8 49=VEND 56=APP 34=229 52=20151009-09:05:23 57=D2C 128=12345 6=0 11=1444381522933 14=0 15=GBP 17=y6u2743950 20=0 21=2 22=4 29=1 31=0 32=0 37=y6u2743950 38=110 39=0 40=2 44=7.885 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:05:23 63=3 120=GBP 150=0 151=110 167=CS 198=y6u2743950 207=XLON 10=065 )

when limit price is reached, Exceution Report (8) with order status=FILLED
<20151009-09:22:41, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=342 35=8 49=VEND 56=APP 34=268 52=20151009-09:22:41 57=D2C 128=12345 6=7.4 11=1444381522933 14=110 15=GBP 17=1040601735 20=0 21=2 22=4 29=4 30=XLON 31=7.4 32=110 37=y6u2743950 38=110 39=2 40=2 44=7.4 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:22:41 63=3 64=20151013 76=VENDOR 119=814 120=GBP 150=2 151=0 152=814 155=0 167=CS 207=XLON 10=208 )

The Java Code (finally!)

Quote request and deal-at-quote
Writing java code to submit a quote is as easy seen below

quickfix.fix42.QuoteRequest quote = new quickfix.fix42.QuoteRequest(new QuoteReqID(quoteReqId));

//optionally set header feilds that your Trade Provider may require
quote.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
quote.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

//create a new 'repeating group' as per FIX spec
quickfix.fix42.QuoteRequest.NoRelatedSym group = new quickfix.fix42.QuoteRequest.NoRelatedSym();

group.set(new Symbol("VOD"));
group.set(new SecurityID(GB00BH4HKS39 )); //value of ISIN: GB00BH4HKS39
group.set(new IDSource("4"));  //4=ISIN, 2=SEDOL
group.set(new SecurityExchange("XLON"));  //stock exchange ID
group.set(Side.BUY);   //this could just as well be Side.SELL

group.set(new OrderQty(10));  //u can buy 10 shares of ticker VOD
//group.setField(new CashOrderQty(4000.00));    //u can buy shares worth 4000 instead of 10 shares

group.setField(new SettlCurrency("GBP"));
group.set(new Currency("GBP"));
group.set(new TransactTime(new Date()));

quote.addGroup(group);

System.out.println("getQuoteForVEND FIX42 message "+quote);
Session.sendToTarget(quote, sessionId);

Deal@Quote
Writing java code to submit a Deal@Quote is as easy seen below
NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.PREVIOUSLY_QUOTED));

//optionally set header feilds that your Trade Provider requires
order.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
order.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

order.set(new SecurityID("GB00BH4HKS39"));   //ISIN for ticker VOD
order.set(new IDSource("4"));  //4=ISIN, 2=SEDOL
order.set(new SettlmntTyp(SettlmntTyp.T_PLUS_2));

order.set(new SecurityExchange("XLON"));  //stock exchange london

order.set(new QuoteID(origQuoteId));  //original quote ID
order.set(new Price(request.getPrice())); //price of previous quote

//order.setField(new OrderQty(10)); //u can buy 10 shares of ticker VOD
order.set(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10

order.set(new TimeInForce(TimeInForce.DAY));  //can be other values like
order.set(new SettlCurrency("GBP"));
order.set(new Currency("GBP"));

System.out.println("sending out DealAtQuotye FIX42 message "+order);

Session.sendToTarget(order, sessionId);

Market Order
Writing java code to send a market order is as easy as seen below

NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.MARKET));

//optionally set header feilds that your Trade Provider requires
order.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
order.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

order.set(new SecurityID("GB00BH4HKS39"));   //ISIN for ticker VOD
order.set(new IDSource("4"));  //4=ISIN, 2=SEDOL
order.set(new SettlmntTyp(SettlmntTyp.T_PLUS_2));

order.set(new SecurityExchange("XLON"));  //stock exchange london

order.setField(new OrderQty(10)); //u can buy 10 shares of ticker VOD
//order.set(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10

order.set(new TimeInForce(TimeInForce.DAY));  //can be other values like
order.set(new SettlCurrency("GBP"));
order.set(new Currency("GBP"));

System.out.println("sending out MarketOrderForVEND FIX42 message "+order);
Session.sendToTarget(order, sessionId);

Limit Order
Writing java code to send a Limit order is as easy as seen below.
Only change from above code is using the OrdType.LIMIT

NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.LIMIT));



Using the above concepts, frameworks and codes, provides some idea of the  electronic trading functionalities, that can be integrated into your java based enterprise application.

Cheers!

REST APIs for Simpler Payment Gateway Integration into your web and mobile apps

I recently had the chance to integrate the current web application I am working on with a popular UK based payment gateway for credit and debit cards, named Worldpay Online . Carrying out the above integration was surprisingly easy and below are the details.

Our requirement was that our end users should be able to pay for our application services using their debit / credit cards. I embarked on the payment gateway integration journey initially thinking of traditional ways of integration. We did not wish to incur the additional overhead of any PCI DSS compliance that occurs, if you choose to store any card details on your site. Hence, I started considering more traditional approaches like our site will redirect payment requests to the payment gateway hosted "payment pages".

These typically gives the end user the ability to choose their payment method, input their card details and ask for their card to be charged for relevant amounts.

Even worldpay as our preferred payment gateway provider had these options under the integration type of "HTML Redirect". However the main drawbacks of this kind of "payment page" based integration were as follows:

1. No control and limited customisation of the payment pages
2. Lack of a strong application context while the user was on the payment pages
3. The mechanism of gateway provider letting us know that the card was charged successfully made the whole process asynchronous, as it was a post back to our application servers

Well, it turns out there is a much simpler approach available in most modern payment gateways. Using the payment gateways REST APIs. I used the worldpay online REST APIs, to quickly integrate the payment gateway into our application. Here I will talk a bit about my experience using the worldpay online payment APIs.

Though we use the payment REST APIs, we still do not need to have an PCI-DSS compliance. The reason is that we never capture or store the card details on our application. The entire process is broken down into 2 steps:

Step 1: Our application displays inline or as pop-up a worldpay online screen. This worldpay screen captures the card details and gives back a temporary token to the application.

Step 2: The application uses the above temporary token and actually calls the REST payment API. The token acts as a proxy for the card details like card number, expiration date, CVV etc. Application never knows about card details, they are known only to worldpay online!

The best part is that:

• Except for a small portion of page which captures card details, rest of webpage is under applications control and hence customizable
• The actual REST API call is synchronous so error handling effort is reduced by a huge amount

The actual payment REST call is as simple as shown below

Example Request

curl https://api.worldpay.com/v1/orders
    -H "Authorization:your-service-key"
    -H "Content-type: application/json"
    -X POST
    -d
'{
    "token" : "your-token",
    "orderDescription" : "order-description",
    "amount" : 500,
    "currencyCode" : "GBP"
}'

Example Response

{
    "orderCode": "worldpay-order-code",
    "token" : "your-token",
    "orderDescription" : "your-order-description",
    "amount" : 500,
    "currencyCode" : "GBP",
    "customerOrderCode":"my-customer-order-code",
    "paymentStatus": "SUCCESS",
    "paymentResponse": {
        "type": "ObfuscatedCard",
        "name": "name",
        "expiryMonth": 2,
        "expiryYear": 2015,
        "cardType": "VISA_CREDIT",
        "maskedCardNumber": "**** **** **** 1111",
        "billingAddress": {
            "address1": "18 Linver Road", 
            "postalCode": "SW6 3RB", 
            "city": "London", 
            "countryCode": "GB"
        },
        "cardSchemeType": "consumer",
        "cardSchemeName": "VISA CREDIT",
        "cardIssuer": "LLOYDS BANK PLC",
        "countryCode": "GB",
        "cardClass": "credit",
        "cardProductTypeDescNonContactless": "unknown",
        "cardProductTypeDescContactless": "unknown",
        "prepaid": "false"
    },
    "shopperEmailAddress": "name@domain.co.uk",
    "environment": "TEST",
    "riskScore": : {
        "value": "1" 
    }
}

For more information please visit Worldpay online documentation

Worldpay as a gateway provides various features, (many of which I implemented successfully) like

• Recurring Payments
• Card-on-File (for re-using previously entered card information)
• 3d-Secure integration
• Refunds
• Dispute Defence
• Telephone orders
• .. and so on 

Happy integration, people :-)

Databases for HA and DR

During a recent participation in a bid, responding to an RFP, I had to wear my ops hat and it was a refreshing change, from my dev endeavours in recent past.

One of the things, that I was required to do was to come with a solution that includes HA and DR. When considering the database options for HA and DR, I had to brush up my basics related to database replications, options and also discuss and validate with other senior tech architects who specialize in ops. Following a brief of the learnings that came out of the experience.

To know the basics of replication, the wikidpedia link for replication is very good. But first some basics.

Background: why database synchronization is required?
Scalability in the middle tier (for example from app servers) is easy to achieve through horizontal scaling, all you would need is mutiple app servers, all symmetrical and fronted by a load balancer.But in the database tier achieving scalability is lot tougher. Especially for data centric applications like OLTP, the database is the single most stressed tier. Though scale-up is an option, it does not provide for a good high availability option. Hence we have to think of 2 or more copies of database, to provide scalability as well as high availability.

Multiple databases can be a tricky option. We can have a cluster of databases all symmetric masters, fronted by a load balancer, but synchronizing data across the databases, is a huge overhead and has potential for failures such as deadlocks, during synchronization.

But to think up other options such a single master, multiple read-only slaves, one must appreciate that, not all data access is equal. If in our application, we can visualize data access as being divided into readonly and readwrite, we may find that only small percentage of application's data access is read write, say 30%. In such cases having a single master database which caters to read-write and multiple slave databases which cater to read-only can be resorted to.

In any case, having mutiple copies of the database and trying to synchronize data across those databases is something you will have to inevitably deal with, either for database load balancing, high availability(HA) or on a more remote note for disaster recovery (DR).

So what are the options for database synchronization?
There are 3 broad options for replication:

• Storage based replication
• File based replication
• Database replication: this is usually supported natively by the specific database

Storage based replication
Active (real-time) storage replication is usually implemented by distributing updates of a block device to several physical hard disks. This way, any file system supported by the operating system can be replicated without modification, as the file system code works on a level above the block device driver layer. It is implemented either in hardware (in a disk array controller) or in software (in a device driver).

When storage replication is done across locally connected disks it is known as disk mirroring. A replication is extendable across a computer network, so the disks can be located in physically distant locations. For replication, latency is the key factor because it determines either how far apart the sites can be or the type of replication that can be employed.

Basic options here are synchronous replication, which guarantees no data loss at the expense of reduced performance and aysnchronous replication where-in remote storage is updated asynchronously hence it cannot guarantee zero data loss.


File based replication
File-based replication is replicating files at a logical level rather than replicating at the storage block level. There are many different ways of performing this. Unlike with storage-level replication, the solutions almost exclusively rely on software.

File level replication solution yield a few benefits. Firstly because data is captured at a file level it can make an informed decision on whether to replicate based on the location of the file and the type of file. Hence unlike block-level storage replication where a whole volume needs to be replicated, file replication products have the ability to exclude temporary files or parts of a filesystem that hold no business value. This can substantially reduce the amount of data sent from the source machine as well as decrease the storage burden on the destination machine.

On a negative side, as this is a software only solution, it requires implementation and maintenance on the operating system level, and uses some of machine's processing power (CPU).

File based replication can be done by a kernel driver that intercepts calls to the filesystem functions, filesystem journal replication or batch replication, where-in source and destination file systems are monitored for changes.

Database replication
Database replication can be used on many database management systems, usually with a master/slave relationship between the original and the copies. The master logs the updates, which then ripple through to the slaves. The slave acknoledges the updates stating that it has received the update successfully, thus allowing the master, sending (and potentially re-sending until successfully applied) of subsequent updates.

Also database replication becomes difficult when it scales up to support either larger number of databases or increasing distances and latencies between remote slaves.

Some common aspects to consider while choosing database replication:

• Do you wish to cater to HA only or you want load balancing as well?
• What is latency for replicaton? Are participant servers in replication, local or remote(site)?
• Are you ready to contend with multiple masters or is single master good enough?
• Can you split your database access into read-write and read-only queries?

The actual process of database replication usually involves shipping database transaction logs, from master to slave. This shipping can be done as file based transfer of logs or logs can be streamed from master to slaves. Databases can support "specialized transaction logs" which are solely for purposes of replication and are hence optimized.

If you wish only HA or  redundancy, than you can have slave/s in warm standby mode. In this case, master sends all updates to slave but the slave is not used even for read-only access, it is only an up-to-date standby.For HA, using shared storage for active master and inactive standby can be resorted to. But the shared storage becomes the SPOF(single point of failure). 

If you intend to have single master and mutiple read-only slaves, either your application must be smart enough to balance, database access as readwrite connections and read-only connections. Some databases like postgresql provide third party middle-ware libraries which bifurcate client's database access and send read-write request to master and read-only request balanced across multiple slaves.

If your application has need to load balance even the read-write requests, then you might think of going in for multiple masters. Many databases refer to this as database clustering or mutli-master support. Here, read-write commands are sent out to one of the masters and masters are synchronized through mechanisms like 2 phase commits. Multi master replication has challenges such as reduced performance and increased probability of transactional conflicts and related issues like deadlocks.

In addiiton, databases can support features like ability to load balance read-only queries, by allowing parallel execution of single query across multiple read-only slaves.

Synchronous replication implies, the synchronizations across slaves is guaranteed at the cost of commit performance against the master

For DR, remoteness introduces high latency, hence log shipping or async streaming of logs, to a remote warm standby is acceptable.