Service Delivery Platform on Oracle SOA Suite
It’s been quite a while since the first article, Oracle SOA Suite 12c Sandbox,  where we discussed creation of the VM Box for SOA development on Oracle platform. It was promised that this development will be pattern-based and, therefore, separated into distinctive technology-related and problem-solving segments, specific for each type of typical tasks.
This segments, or implementation areas are commonly known as a Frameworks, accumulation similar approaches, methods and patterns, addressed to solving common SOA problems. This objective is pretty close to what I have presented in the first chapter of the book, dedicated to SOA patterns on oracle platform .
Experience from the previous implementations
As any good intention this one also had to be handled with care in order to not make another tile in the pavement to the certain place. Several things happened after first publication – I have had several interesting SOA projects, based on Oracle 11g\12c and IBM platforms, which could lay the basis for the following pattern based development discussion (one realistic usecase discussed below). Among them the most fascinating and inspiring to me was invitation from my old Employer and Customer to migrate the Service Broker components to the more up to date Oracle SOA platform. Service Brokering solution, designed and implemented before 2006 with some elements of Adapter Framework implemented as oc4j java components had been in production till these days, running smoothly as a black boxes. The reason of replacement was to have it on a modern platform as oc4j are rather old-fashioned nowadays, technically there were no complaints regarding functionality and performance. This fact just demonstrates that principles of Frameworks implementation, described in the book  were well balanced and healthy.
Common traps of mixing SOA and Classic Integration approaches
The idea of implementing complete set of distinctive frameworks on prepared VM is based of clear realisation of frameworks boundaries derived from set of patterns, implemented on distinctive technical platforms (ESB, Orchestration, ERS, etc.). As a design patterns are the response to concrete reoccurring problems , it would be useful to start from most common problems and risks. Regretfully, not all our problems, which makes frameworks boundaries and solutions, based on this frameworks fussy and inconsistent have a technical nature. As a result, this risks could have very profound effect on SOA adoption and implementation, if materialised.
About one year ago as an SOA Architect I had a meeting with IT tops of one global payment service provider company in Netherlands. After quick chat with CIO the one of the immediate problems were recognised as a lack of MEP/protocols standardisation – the company used file based processing In/Out where e-Commerce clients required synchronous processing. After that I had a meeting with Chief Architect where one of my first questions was: what is the main risk for SOA adoption can he sees for his company. From seasoned veteran you can expect something like
- Big Bang implementation approach
- Lack of sponsorship from CEO, leading to budget cuts and “refrigerator” approach
- Misunderstanding of SOA principles, lack of service standardisation
- Positioning the SOA patterns before the SOA implementation problems
Actually, the answer was – “The biggest problem in SOA implementation is a reading of Thomas Erl books”. And why is that? – I asked, completely stunned. They are too long, too difficult to read, full of obvious and well-known concepts, and full of self-repeating, was the answer.
Efforts, time and budget spent on most known to me SOA projects usually indicate that Architects should have more attention on basic SOA concepts and drivers.
Understanding of the role of Service
So, it would be beneficial to start from the “most obvious concept” – Service. Understanding of SOA principles and their relations for the balancing them in individual frameworks begins from understanding of what the service is. Service architecture and boundaries are well defined in  and other publications as atomic units of decoupled business logic equipped with standardised and seamlessly discoverable contract. SOA is declared as architectural approach, focusing of using and constant reusing services as a main building block of business logic (although it is important to stress that services are not only one atomic unit in SOA Infrastructure).
Misunderstanding of the service role and its architecture results in declaration of various hypes like recent “Microservice” or well-matured “SDP” (Service Delivery Platform). As it’s clear from its name, the Service Delivery Platform should be responsible for establishing reusable building blocks, so we obliged to focus on it at first hand. SDP’s “Top-Down” design approach and its real-life adjusted variant, “Meet-in the-Middle” put “Contract- first” as commonly accepted preeminent method for designing SOA services.
To demonstrate it in action close to real life usecase will be selected, the one which will require service boundaries identification and several alternatives of implementation. I followed this practice in every chapter of my book and I will do the same here.
Corporate HR has several applications, handling different parts of Employee data. One application (A1) is used for storing and presenting personal data, another (A2) is directly linked to accounting and salary calculation. Personal data (A1) has some critical elements for A2, required for proper tax calculation (like marital status, personal tax code). Different actors have different access levels to data, stored in A1 and A2, see figure below. Employee can modify personal data in A1, this data is also linked to municipal register (residential address), bank account information (A2) is connected to the local bank. Data in A2 is accessible by HR and Accounting, not Employee. Thus, data from A1 in first interaction must be presented (become accessible) to A2.
Technically these two application (A1 and A2) are absolutely independent, has no direct connections and provided by different vendors. A2 is capable to import CSV files, A1 has no inbound/outbound interfaces at all and all what vendor can offer is full data dump in regular intervals (yes, I know, in year 2015). No information about actual changes (events) can be provided.
Let’s do the quick analysis of the existing situation. It is Client’s desire is to practice SOA, not a classic integration. Therefore, contract standardisation must be applied first. Question – to what extent. Where is the atomic Master Employee service? It must be established for MDM purposes and act as a “single source of truth”. It must be updated regularly and be accessed at any time. Solution must be capable to identify any changes in A1 and propagate events data to A2. Thus, provided solution should combine the following:
- MDM service with access for selecting Employee data
- Event detection subsystem (CDC), as a part of MDM service, capable to identify changes (events) in predefined Employee object elements and populate MDM store
- Event distribution subsystem, capable to access MDM service, extract data related to event and propagate to the events subscriber. This subsystem will require repository for events definitions and event subscribers. For this purpose we will use ESR DB Schema, presented and explained in details in Chapter 5 .
Solution building blocks
Largely, the solution will established around one service with Entity model, surrounded by adapter frameworks components for a) accepting(ingesting) the inbound data at North and b) delivering relevant message to the Subscriber on South, transformed from EBO, acquired from Entity service. At the first stage (a) CDC is implemented for capturing basic events. EDN, connected to the southbound adapters is based on ESR and responsible for converting basic events into enterprise business events and subscribers identification. Solution block-diagram with alternative channels (DB/FSO) in adapter framework elements is presented below.
Establishing Data Model (as a part of Contract)
That’s in theory and here all is clear, let’s assess implementation alternatives, mentioned above, plus “Bottom-Up” approach. It is clear that in any case we have to establish MDM Entity service, and XSD as the essential part of the contract shall be presented in advance, see object structure below with message header on top.
From figure above you can see that real-life Employee object is a bit more complex than the HR schema we get with installation of basic Oracle DB, including:
- Personal data (with reference to addresses)
- Company data (with reference to business address)
- Payroll data (with reference to bank accounts)
- Contract data (with reference to payroll)
Object associations and references can be mined from figure below (from actual DB adapter implementation).
Actual xml elements of every associated object are omitted for brevity, you can implement them as a part of your homework, according to your actual needs. Nevertheless, DB service resource has successfully been established in full accordance with presented XSD (as a part of Service contract). Can it be considered as a “Contract-first” approach? Partly, yes. Partly - because we did not implemented the service logic based on this contract, only DB storage. In fact, we even do not have a complete contract, only data part of it (although we declared, that for simplicity primary operation shall be SELECT or get_* in EJB terms).
So, the first alternative which will tempt any Oracle developer is to use Oracle technology adapter (DB) from BPEL or OSB. As you can see from figure above this option has been explored.
NB!: This article is dedicated to SDP framework as a framework of establishing healthy services. We will not focus on CDC as a part of Adapter framework or EDN which is a part of ESB. They will be discussed in details is separate articles, here I just briefly mention them for completeness of the solution.
Alternative I (Oracle technology adapter’s implementation)
From the previous note is clear that adapter framework elements (like CDC) are inevitable in this solution, so why wouldn’t we implement just one more? The first reason we already mentioned before – we will inevitably break the Loose Coupling SOA principle by tying contract to logic. Secondly, this adapter framework is already thick enough:
- FSO adapter for transferring inbound records to interface tables (i*), or its DB alternative for data ingestion.
- CDC logic comparing content of I* tables with MDM store records, triggered by (possibly) another adapter, calling PL/SQL CDC procedure, if status of the ingestion is positive.
- Converting extracted MDM object into CVS output format (FSO adapter with transformation)
To illustrate the thickness of adapter frameworks footprint with FSO delivery alternative the block diagram is presented below with all adapters calls. The burden to control all these parts and calls will be placed on BPEL as a static composition controller, quite massive. Some can say, that BPEL was invented for this.
Risks of "Hybrid" implementation
Going this way we most probably implement something what “Microservices” apologists call “Monolithic services”. Regretfully I even have registered this approach as valid in some recent handbooks, dedicated to Oracle 12c SOA Suite. The correct term in my opinion (according to ) for this type of service, mixing task orchestrated and entity models is “hybrid”, and their implementation is a result of improper service layering, coming from misunderstanding of the service models and «contract-first» design principle.
So, in practice SDP should prevent from establishing such “hybrid” services, which hamper Reusability, Loose Coupling and, ultimately – Composability principles.
NB!: It would be incorrect to say that “hybrid” services are totally inacceptable. In some cases their implementation can be adjusted, when service layers can be combined. The rule is to avoid combining business agnostic and non-agnostic, stateful and stateless services.
If these design considerations above are not convincing enough, I would like to show, what practical consequences (or disasters) could wait us after implementation of this alternative (with some luck and Oracle help). The ways of mitigation of this problem will also be demonstrated, of course.
This alternative has been implemented with all adapters, mentioned on preceding figure, around MDM Employee storage in two SCAs. Custom CDC worked flawlessly, detecting changes in inbound Employee objects, provided by inbound adapter and stored in interface tables. Changes were propagated to the MDM Employee DB (as inserts or updates) and these events recorded as a new entries in event registration table. In second SCA DB Poller pulled events information (Employee and subscriber identification) and passed it to MSM BPEL service, which extracting Employee object pass it to outbound FSO adapter.
All was implemented as described, but during testing almost immediately random errors were registered practically in all DB adapters, firstly with SELECT operations, like below
… CEST ERROR:1 oracle.soa.adapter JCABinding=> MDMEmployeeDelivery:dbEBOEmployeeReference [ dbEBOEmployeeReference_ptt::selectByPrimaryKey(selectByPrimaryKey_inputParameters,PersonCollection) ] Could not invoke operation 'selectByPrimaryKey' against the 'db' due to: BINDING.JCA-11614
DBReadInteractionSpec Execute Failed Exception.
Query name: [readObject], Descriptor name: [dbEBOEmployeeReference.Person].
Caused by java.lang.NullPointerException.
See root exception for the specific exception. This exception is considered not retriable, likely due to a modelling mistake.
But not only SELECT is susceptible to JCA BINDING error. Quite soon DB invocation operation produced the similar error
dbTransferReference_ptt::dbTransferReference(InputParameters,OutputParameters) ] Could not invoke operation 'dbTransferReference'
against the 'db' due to: BINDING.JCA11812
Interaction processing error.
Error while processing the execution of the MDMOBJECTS.EBO_PERSON.STARTTRANSFER API interaction.
An error occurred while processing the interaction for invoking the MDMOBJECTS.EBO_PERSON.STARTTRANSFER API. Cause: java.lang.NullPointerException
Check to ensure that the XML containing parameter data matches the parameter definitions in the XSD. This exception is considered not
retriable, likely due to a modelling mistake.
What is especially discouraging and misleading here is this suggestion “likely due to a modelling mistake” after java.lang.NullPointerException. Misleading because rigorous rebuilding and remodelling of adapters and XSD produce no results. All possible types of SELECT adapters were tried, including selectByPrimaryKey and ReferenceSelect. Insert operations into interface tables (I*) failed as well.
insert failed. Descriptor name: [MDMDbReference.Iperson].
Naturally, for testing purposes XE DB has been used which is not recommended for SOA Suite 12c. Thus, PROCESSES and SESSIONS DB parameters have been increased significantly, like
alter system set processes=300 scope=spfile;
alter system set sessions=300 scope=spfile;
Oracle also recommends to configure listener local access as follows (/as sysdba)
Result was quite predictable, this time I got an “oracle.fabric.common.FabricInvocationException: java.lang.IllegalStateException: Stepping an inexisting frame” error in file adapter as demonstrated below
… CEST ERROR:1 oracle.soa.mediator.serviceEngine Post to mesh failed oracle.fabric.common.FabricInvocationException: java.lang.IllegalStateException: Stepping an inexisting frame.
So, this is not just DB Adapter error, it seems much wider JCA-stack error in entire Oracle adapter framework (by the way, common for SCA and OSB). Here I have to admit, that I did not patch my environment (dev VM) regularly, so it was a good time to apply some. Apparently, these issues are quite well known and Oracle recommended two patches as a cure: (1) Patch 19707784 and (2) Patch 20649529. The first one is generic patch and must be applied first. The second is addressing JCA adapters particularly. Patches were applied, with no positive effect. I got another patch to apply – 20900599 and so I did, applying it to VM and local environment. Just take a look at list of the bugs, allegedly addressed by this patch:
NB!: Just in case if you decide to go this way, please do not forget to clean-up the following directories
after pathing and restart your WLS.
Alas, problem was not solved. After intensive investigative googling I got a good tips - the culprit is OSB debugger, which is constantly active in your SOA domain, even if you do not use OSB. So, go to your domains setDomainEnv.cmd, (which locally in Windows can be found in C:\Users\<user>\AppData\Roaming\JDeveloper\system220.127.116.11.41.140521.1008\DefaultDomain\bin\setDomainEnv.cmd) and add following:
Here is another confession I have to make – these two parameters didn’t work for me either, maybe I did something wrong. But one thing I know – production mode supress debugging, thus the last resort for me was to put WLS into production mode, as shown below
Just think, how nice is it – instead of fixing NullPointerException in a common module, affecting the entire framework, which is generally is a sign of utter negligence, we were advised developing in Production mode. Please be prepared to be stabbed in the back by any adapter from Oracle SOA Suite and from that moment you will be on your own, as you cannot rely on any patch provided for fixing your issue. I would like to conclude this post with the first phrase from Chapter 6 - ‘The best adapter is the one you do not have to implement’.
Generally, problem here is not with Oracle adapters (quite bugged anyway, and they are not getting better), but with overall architectural approach. From pattern based development the following issues can be highlighted:
- Service Layering pattern is not applied
- Agnostic Context and Capability patterns were not applied
- Entity Abstraction pattern ignored
- Composition Autonomy is not applied
- Consequently, Reusability, Loose Coupling and Composability principles were affected negatively
- From NFR standpoint reliability have been hampered considerably.
Next steps (actions): In the next article we will discuss what we shall do in SDP following “Contract-First” approach in order to mitigate the issues, highlighted in observation above.
- SOA Principles of Service Design, by Thomas Erl, Prentice Hall; 1st edition (July 28, 2007)
- SOA. Concepts, Technology and Design, Thomas Erl, 2005, Prentice Hall
- Applied SOA Patterns on the Oracle Platform, Packt Publishing ( August 12, 2014)
- Oracle SOA Suite 12c Sandbox
- SOA Design Patterns (The Prentice Hall Service-Oriented Computing Series from Thomas Erl) 1st Edition
- Web Services Choreography Description Language Version 1.0