Process Mining - Practical Uses

Process mining is a family of techniques that support the analysis of business processes based on event logs. During process mining, specialized data mining algorithms are applied to event log data in order to identify trends, patterns and details contained in event logs, to auto-discover and paint process definitions of processes. Process mining can be used to discover processes from events or improve process efficiency by identifying bottlenecks and exceptions

By itself process mining has several applications in large enterprises, to discover or validate business processes (in finance, manufacturing and retail) merely from events and timestamps, but the exciting thing is that the same technique can be used to get insights into user/customer behaviors on your website and also aid in customer segmentation.

Consider a simplistic apache web log (though much more detailed application logs can also be used)
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /login.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /catalog.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /checkout.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /checkstatus.html HTTP/2" 200 1479

The important pieces of information are the user id "peter", the date timestamp and the URL visited which can proxy up for the user's intended action. The above log can be easily transformed into

Sample CSV file

<user>,<action>,<action date>, <user attribute 1>, <user attribute 2>, <user attribute n>
UserId,Action,StartTimeStamp,EndTimeStamp,PremiumCustomer,UserLocation,UserOccupation
Peter,Logged In,12-20-2017  10:20:00 AM,12-20-2017  10:25:00 AM,premium_customer,US,architect
Bob,Logged In,12-20-2017  10:20:00 AM,12-21-2017  10:25:00 AM,premium_customer,US,architect
Peter,Browsed Catalog,12-20-2017  10:35:00 AM,12-20-2017  10:40:00 AM,premium_customer,US,architect
Alice,Browsed Catalog,12-21-2017  10:35:00 AM,12-21-2017  10:40:00 AM,premium_customer,US,architect
Bob,Browsed Catalog,12-21-2017  10:35:00 AM,12-21-2017  10:40:00 AM,premium_customer,US,architect

The above CSV data can now to be fed into a process mining tool to generate the "user/customer's" interaction process, as he navigates and uses the website. This can give many insights into how customers are using our websites and what we can do to make things more convenient for them.

Statistics like what time duration is spent by each user in any of the activities can also be readily obtained as a report from the tool. The major deviations in the main process flow can also be denoted. Animations of the process flow, over a period of months can be played out within minutes to get insight into the activities that are bottle-necking the overall process.

Instead of discovering processes for individual users we can generalize processes for say "platinum customers" by filtering the data within the process mining tool itself.

Process Mining deals with Petrinet and BPMN notations of process definitions and some popular algorithms for process mining are alpha miner, inductive visual miner and fuzzy miner.

Notes on Enterprise Application Development Strategy

At the end of the day, technology is an enabler for extracting business value. Some sure fire ways to burn money needlessly, during enterprise application development are noted below. Use it as a checklist to guard against, in your own enterprise development. As always, use your own context, while weighing-in, any generic advice.

• Don't care why and what the end user actually wants. Its more important to impress top management, with a good-looking demo
• Using a particular tech because it is popular at google, netflix or facebook, irrespective of its applicability within your own context, especially NFR context
• No clarity about the WHY, obsessed with only the HOW
• Not able to contain scope creep and needless escalation of complexity
• First thought that comes up becomes the definitive design, no choices, no alternatives considered
• Resume driven or Ego driven, development by tech leads
• Why use a suitable tech, when a tougher, riskier, newer tech is at hand, no matter how irrelevant
• Underestimating integration and dependencies
• Committing to technology choices without understanding the full scope of functionality and the non-functional requirements(NFR) in detail
• Forgetting that no application is an island. Its important that an application can "fit into" the enterprise landscape in terms of security standards, IDAM (identity and access management), audit standards, deployment, reliability and business continuity standards
• No thoughts given for migration path from existing application/s
• Disproportionate emphasis on UI look and feel rather than UX and information models
• Re-invent the wheel, under the guise of tech bravado
• Dev Teaming mistakes
• Thinking that employing 10 mediocre developers, instead of 3 crack devs, will be better
• Believe the myth that application devs, still write framework level code
• Never acknowledge, that app devs now, require to use proper frameworks and APIs, rather than belt out algorithms and complex programs

Blockchain - Usecases in supply chains

Supply Chains, are the lifeline of businesses globally. Most likely, your business will find itself between upstream suppliers and downstream sales channels/distributors/customers. The figure below indicates this topology succinctly.

Generic Supply Chain connected via a Blockchain

Your organizations ability to fulfillment of its sales orders, totally depends on how efficient, agile your supply chain is. Some of the areas where blockchain technology can add great business value are:

Optimizing Inventory Costs
Though organizations have fairly automated internal processes, most still rely on antiquated ways of interacting with their suppliers and business partners, Orders can be placed over voice calls, email, logging into each other's portals, etc. Tracking of these orders status is never, in real time, mostly left to voice calls, email reminders, etc. In fact, it would not be uncommon for your organization to get notified of a supplier's delay in order fulfillment, close to your own, downstream, fulfillment commitments. By having, deep, blockchain-based integrations, with your supplier's ERP, you can ensure almost real-time notifications of any potential delays, in your order fulfillment. You can automate balancing of orders across your other suppliers as well.

If your distributors/sales partners are also on the blockchain, any weakness in demand, can also be notified to your organization in real-time, so as to re-balance supplier-side orders. This will lead to highly optimized inventories and substantial reduction in inventory costs. De-risking from supply side outages due to a specific supplier is another collateral benefit.

By having most of your suppliers on the chain, you will get unprecedented clarity about the big picture on the supply side, across all your suppliers. This can help you in coordinating receipt of goods from different suppliers, with a consolidated scheduling system based on blockchain data. This can further optimize the timing and logistics aspects of the inventory.

Minimizing Quality Costs using the blockchain
With the blockchain in place, your organization can have selective access to your supplier's ERP Data, including their quality control data. Having this level of accurate data, you can choose to minimize redundant quality checks, at your end. This is the good example of using a blockchain to enable trust and optimize costs.

Suppliers of a Supplier can also be brought onto the blockchain, to enable deeper and far reaching integration via the blockchain.

A blockchain can be used very effectively to "track the state" of a logical unit of delivery across supplier's parts. For example, if your organization was making refrigerators, every part, being supplied, tracked with the various suppliers, can be co-related with the logical entity "refrigerator 123" on the blockchain, hence, at any given point in time, you could not only know that 30 compressors, 26 coolant tubes are being supplied to me, but also that, I will be able to get out 26 complete refrigerators to my customers.

Bidding based supplier order management
The same trust based blockchain technology can be used to operate a bidding platform, to manage orders being issued out to the suppliers

Why traditional integration technologies like API, will just not work?
Point to point integrations with multiple suppliers not are just not scalable, in the supply chain ecosystem. imagine a supplychain with 5 nodes, each talking to other via point to API calls. Blockchains, along with providing a decentralized datastore, to track "shared state", also provide a standard stack for integration, at all peers, not to mention "consistent versions" of same distributed smart contracts for processing the data.

The suppliers that you bring onto the blockchain, may be direct competitors, inherently having a lack of trust about using processes, business logic, databases and codebases, belonging to each other, but the decentralized blockchain, will make it possible, to bring all such parties onto a common platform, sharing data/state, business processes and code.

Countering, Counterfeiting

While researching into the practical applications of blockchain tech, in provenance, tracking and traceability, I came across claims that blockchain can help to tackle counterfeiting, which is a serious issue in the fashion, luxury and branded products industry.

The general opinion seemed to be that its tough to fight counterfeiting because establishing authenticity of an original product item, over a cleverly counterfeited one, was a tough nut to crack (at affordable technologies).

Lets get into the mind of counterfeiter then...Typically counterfeiters will reproduce a near-duplicate of the original, at a fraction of the cost, then package it exactly like an original and send it out to be sold. It would be next to impossible to catch a cleverly made and packaged counterfeit item.

The barcodes and qrcodes on the item package can be copied to the T, as well and then pasted onto the duplicates package. Standard barcodes (EAN-13) contain only the manufacturer code and product category information, which is popularly recognized as a GS1 standard GTIN, like is typically used to scan retail products as POS terminals. But the real problem like I said is that, the entire barcode can be copied and put onto a duplicate item, making it indistinguishable from the original. Authenticity is sadly not guaranteed even if additional attributes about the product item were embedded into extended barcode versions like the GS1 Databar or GS1-128.

For authenticity checking, the trick is to use 2 factor authentication. Imagine a qr code encoding a random number, printed on the outside packaging of a luxury product. Now another QR code with yet another encoded number present inside the product itself (say on the product lining or inside of the cap of the bottle). Now we have 2 numbers, one on the outside and one on the inside(can be accessed, only after breaking the seal of the product).

A simple web page/mobile app can be developed to validate the compatibility of the inside number and the outside number. As long as the compatibility checking algorithm is secret (maybe using a magic number/s), the end user can just scan the outside and inside QR codes/numbers, to let the website/mobile app, determine if the product is authentic or counterfeit.

Even if the counterfeiter manages to exactly copy, the outer packaging, he can never "quess" what the inside number should be, since he does not know the exact algorithm, that is used to derive/correlate to the inside number.

The last problem to solve, is the possibility that the counterfeiter may get hold of a few good samples of "valid" number pairs and use them with all the counterfeits repeatedly ! This can be easily tracked, since the website/app is centralized and can keep track how many times the query for checking authenticity is being received and from what sources, mobiles, geo-locations etc

Let know in the comments, any loopholes, I may have missed or any improvements you can think of.

Integrating existing enterprise apps with blockchain

Lowering the barrier to entry, is key for increased adoption of blockchain based technologies into enterprises. Irrespective of the business value, that blockchains bring, there is a need to decrease the cost, pain and risk of integrating blockchains with existing applications in the enterprise.

Most large enterprises today, have established ERPs, CRMs, or cloud native custom applications, running 24 x 7. It makes sense to interface the new blockchain tech, harmoniously into the enterprise architecture landscape.

Its quite practical to assume that, most organizations, while they would want to participate in a blockchain and get all the benefits, will most likely be reluctant to do any major modifications to their existing enterprise apps. The need of the hour is, to ease ingestion of data onto the blockchain, without requiring expensive UI based / operational alternatives.

Presented below are a few ideas, for ingesting data onto blockchains, in an automated and batched manner, with minimal integration development and risk. Refer my earlier article about, wrapping up blockchain clients inside REST webservices, the below ideas can be used in conjunction with the REST adapters.

ERP Application Logs >> Logstash >> Http Post to Blockchain REST adapters >> Blockchain

Existing Application JDBC DB >> Database Polling >> Http Post to Blockchain REST adapters >> Blockchain

Existing Application publish message >> Topics (Message Provider/Kafka) >> Logstash >> Http Post to Blockchain REST adapters >> Blockchain

When the need for capturing data onto the blockchain is critical, more reliable and resilient, pipeline with Kafka based topics can be used.

Blockchain alternative to API integration across enterprises

What is the go-to solution when 2 enterprises wish to communicate with each other. Right! Its via APIs, that too using REST APIs, with appropriate security in place.

Typically this involves one of the companies developing webservices and exposing those services, to be consumed by another enterprise. There are several constraining factors for this kind enterprise integration:

The webservices paradigm is uni-directional, that is, if org A, invokes a webservice hosted by org B, for org B, to confirm that desired state changes have occured, org B must call another webservice hosted by org A. This is essentially the issue, that the webservice calls are stateless. There is no state sharing at all between the organizations.

For true leverage and reuse of the webservices, there needs to be orchestration at the consumer side. This orchestration is not transparent to the provider of webservices.

The communication is primarily point to point. So the interactions, cannot be scaled easily if newer participants join in. Imagine 4 organizations, each one trying to call services into others and trying to keep track of some shared state, using and exposing status calls

All the above issues are elegantly solved using the "shared state" paradigm of blockchain. All participants talk to the blockchain almost as if it were a logical messaging bus, but more importantly, the bus is not just a physical connection, but maintains state in distributed ledger and also has smart contracts that can house, common and transparent pieces of code, that can be shared across the un-trusting organizations aka partners