Microservices in action: Building reliable Adapter Framework (Part 1)

Microservices in action: Building reliable Adapter Framework (Part 1)

Some might say that Microservice Architecture (MCSA) is a "new SOA" and therefore very popular, as the old one is dead. In fact, it is not (I will examine the actual statistic a bit further), but as it usually goes, only 20 % of people make 80 % the noise, so in next three parts I would like to present lessons learned from the practical implementation, involving Java-based MCSA and containers in the following order:

  1. Java-based Adapter framework (AF), suitable for so-called "conternisation" and what it takes to make it scalable, reliable and resilient. Due to the amount of code, the part will be further broken down into two subsections. 
  2. Building containers around elements of Adapter framework using various techniques, exploring benefits of this approach, analysing what SOA principles we improved and how many of them we left broken.
  3. The last part will be dedicated to comparing different "conternisation" approaches in order to find which one may really improve the SOA characteristics. For the conclusion, we will discuss the MCSA pros and cons from a practical perspective.

The intention is to compose the aforementioned parts from a practicing coding architect's standpoint without extensive empty theorizing. Furthermore, I would like to initially express my deepest respect to Developers, working hard on developing Open Source products like Docker, Vagrant and the entire Apache foundation. Any criticisms expressed henceforth is not an existential criticism of the products, but merely to the inappropriate way usage.

So, we will start from the Java AF first.

Overall Solution Description

As it was initially requested, a solution will be server less, lightweight and modular. The analytical DB (ElasticSearch) and engine (Joomla with report visualisation plug-ins) are dependent on various data sources, at least three - MySQL DB, XML Files and Flat-fixed format(FF). List of sources can be extended, thus individual adapters shall be standardised and unified for better pluggability and extension.

Ultimately, every adapter shall be atomic, but as the object, based on XML data input requires some enhancement from MySQL data (some historical records), the decision was made to make a shortcut and call MySQL from XML parser. Still, MySQL data fetcher is considered as an independent adapter project and used as a dependant in XML Exporter project. At any moment this module can be packed as an independent adapter. Data assembly is done on the ES DB side, without any wrappers.

The analytical engine will extract and aggregate all necessary information. ES indexes is based on JSON Object representation and defined in AF configuration *. property file. XML and FF file location can be anywhere on Client's side and must be configurable, the same for the concurrent execution. Canonical communication protocol between all components is HTTP. ES DB can be in container or on physical host. Solution should be OS independent and run on Win or Linux/Unix with minimal alterations through the unified config file.

Although config file is unified, it will be individually stored in every container(opt/config/) for the highest configurability. Modularity will be enforced by Docker implementation. Solution's Infrastructure and required tools are presented on the preceding figure.

Structure of unified adapter (XML module)

XML file adapter is used as an example as it is the most complex and complete.

FF Module is almost identical, except that we do not need to enhance the object with MySQL data and the processed files are not archived, and the property file is created with the last successfully processed file, which is used in the next session as a starting point for the file list gathering. The package's structure is depicted in the header of the preceding sequence diagram. PersistencyHelper is from linked DB project, presented in yellow.

Building elements of Java Adapter Framework

Parsers implementation strategy

Problem: We will have a substantial amount of considerably large XML files (several Mb, sometimes up to several dozen Mb). Each of them have to be converted into JSON before posting to ES with minimal memory overhead. Implementation must be very lightweight, suitable for conternisation, minimal external libraries, and high modularity.

Approach: Two things are obvious - we need some kind of SAX parser (no in-memory DOM tree) and as JSON is involved, GSON library (from Google) will be necessary. As GSON is involved, optionally some libraries based on it can be employed, like [8] stanfy/gson-xml, for XML nodes parsing very elegantly. Due to the size limitation, we will be dealing with FileInputStreams, not XML strings, so I decided to use XMLPullParser API (XPP)[9].

Ensuring Anti-Fragile implementation of the components: Resiliency can be derived (but not in every instance) from Simplicity (KISS principle). The simpler and clearer the approach you use for the core problem is (parse an array in our case), the easier it can be peer-reviewed and better covered by a unit test. However, test Driven Development and regular, thorough code quality check is not enough in some instances in regard to MCSA. Certain magic pills in form of patterns shall be applied to make them truly resilient. Patterns are the commonly approved solutions to general and reoccurring problems (sometime caused by severe architectural technical debt). So what patterns makes components a real microservice?

Recommended

Using notes

Event Listeners

Originally Observer Pattern (http://www.oodesign.com/observer-pattern.html). Informs object about events that have occurred in other objects without tight coupling.

Circuit Breaker

Pattern is based on the one above. The way of preventing system overload or getting into rolling disaster state.  Used when interacting with remote systems and its response (or lack thereof) could trigger a chain reaction on the caller's system. More often than not used for controlling a multi-conditional loops, especially poorly implemented ones or maintaining data windows in complex selects.     

Fallback

Further extension of patterns on above. Allows for alternative execution in the case of erroneous execution. Implemented in a single component, which is not a composition controller that can have a devastating effect as an antipattern.

Retries

Policy and Rule-based execution. Opposite to SOA Policy Centralization pattern, which is quite risky in complex compositions. In support of the preceding pattern some basic actions can be predefined depending on the execution outcome. Suitable for connectors and Db operations.

FSO Adapter for XML data

Parsing strategy is to build object from XML, so an XML schema gives the perfect opportunity for building an object, and JAXB is one of the best API tool for that. Using Eclipse, it can be done in few clicks. The ObjectFactory, constructed along all the associated objects from complex schema will be kept as an alternative to direct assignments. This ObjectFactory allows us programmatically construct new instances of the Java representation for XML content. I do not want to use JAXBElement type directly as shown below, because deserialization of XML data in this way will consume a lot of memory.

@XmlElementDecl(namespace = "http://www.statdatacomp.no", name = "seriousstatistics")
public JAXBElement<TopStatObject> createTopStatObjectistikk(TopStatObject value) {
return new JAXBElement<TopStatObject>(_Seriousstatistics_QNAME, TopStatObject.class, null, value);
}

Main parser's function starts as below, where we create parser factory, input stream from file(filename passed as a parameter). It is highly important to set right encoding for XPP input (taken from the property configuration file), as file contains Nordic character set.

public static int XMLExporter (String filename) throws XmlPullParserException, IOException {
       int status = 1;

// 1. get instance of XMLPULL facotry

       XmlPullParserFactory factory = XmlPullParserFactory.newInstance();

//2. factory should produce namespace aware parser; by default it is not

       factory.setNamespaceAware(true);

//3. create instance of the parser

       xpp = factory.newPullParser();

//4. get the input as a file stream

       File file = new File(filename);
       FileInputStream fis = new FileInputStream(file);

//5. set the input encoding, defined in config property file

       xpp.setInput(fis, Config.XMLENCODING);

//6. Initiate the Objects, connections and supporting varaibles

       TopStatObject statstat = new TopStatObject();
       ESPostHttpClient eshc = new ESPostHttpClient();
       JSONBuilder jb = new JSONBuilder();
       PersistencyManager pm = new PersistencyManager();
       ConnectionManager cm = new ConnectionManager();
       Connection connection = cm.getConnection();

Main parsing loop starts as shown below, with an example of one extracted value of simple type, "versjon", and a calling setter of TopStatObject, declared in the preceding block.

       while (eventType != XmlPullParser.END_DOCUMENT) {
              if(eventType == XmlPullParser.START_DOCUMENT) {
                     System.out.println("Start TopStatObject document");
              }

          // simple type (String) parsing and tracing

                     if (xpp.getName().equalsIgnoreCase("versjon")) {
                            if (insideTopStatObjec){
                                   objdata =xpp.nextText();
                                   TopStatObject.setVersjon(objdata);
                                   System.out.println("TopStatObject versjon: "+objdata);
                            }
                     }

XPP is based on the concept of events, occurred in a stream, like START_DOCUMENT, END_DOCUMENT, START_TAG, END_TAG, TEXT (only five), see more in [9]. We may proceed using the method next(), from one event to another until the end of the document. When a tag detected with a certain name, the nextText() method returns the string value of the content. If the object field is not text(String), then some implemented utility functions can be used to cast it to the desired type:

TopStatObject.setStopptid(rdp.RESTTimestamp(objdata, Config.DATE_FORMAT));

The Function, implemented as shown below (do not forget to set the correct value for Locale, it could be tricky if the Nordic characters in the document mixed with the English (or any other) timestamp format, as runtime error could be difficult to comprehend).

public Timestamp RESTTimestamp( String dateTimeStr, String dateFormatStr ) throws WebApplicationException {
       SimpleDateFormat df = new SimpleDateFormat( dateFormatStr, Locale.ENGLISH );
       java.util.Date date;
       java.sql.Timestamp tmsdate;
   try {
       tmsdate = new java.sql.Timestamp( df.parse( dateTimeStr ).getTime() );
   } catch ( final ParseException ex ) {
     throw new WebApplicationException( ex );
   }
   return tmsdate;
  }

Implementation of other converters in Utility package is skipped for brevity. For better looping control (inside or outside the main/nested complex type loop) a set of boolean variables like inside<ObjectName>* has been implemented and used as demonstrated in the snippet of the main loop. In addition, getDepth() method[9] can be used for fine control of the sibling and nesting (bearing in mind that the root is 1, not 0).

Every nested complex type is parsed as presented below:

//------------------ nested complex type/ Associated object NestedObjectLevel1
   else if (xpp.getName().equalsIgnoreCase("NestedObjectLevel1") && eventType == XmlPullParser.START_TAG) {
                  if (insideTopStatObjec){
                       insideNestedObject1 = true;
                       NestedObjectLevel1 nestedObjectLevel1 = new NestedObjectLevel1();
// calling complex type parsing function
                       nestedObjectLevel1 = parseNestedObjectLevel1(insideNestedObject1);
//assigning associated object to the main object
                       nestedObjectLevel1.getNestedObjectLevel1().add(nestedObjectLevel1);
           }
  }

Every associated type NestedObjectLevel1, 2, 3, and so on have the individual parsing function parse<NestedObjectLevelN>, which is identical to the main parser loop with all setters and cast function calls, where needed.

We use ObjectFactory functionality and mutators provided by JAXB, so it could be quite confusing to use get*().add() method for setting lists, constructed during the XML sequence parsing as

topObjects.getNesteddata().add(nesteddata);

This is just the way it is.
The loop closing with tracing functionality is implemented as shown below

                         } else if(eventType == XmlPullParser.END_TAG) {
                                 System.out.println("End tag "+xpp.getName());
                         } else if(eventType == XmlPullParser.TEXT) {
                                 System.out.println("Text "+xpp.getText());
                         }
                         eventType = xpp.next();
              } //end while

closing (breaking) loop for the nested/associated object

else if (eventType == XmlPullParser.END_TAG && xpp.getName().equalsIgnoreCase("NestedObjectLevelN")) {
                        System.out.println("breaking NestedObjectLevelN loop ");
                        insideNestedObjectLevelN = false;
                        break;
              }
              //move to the next element
              eventType = xpp.next();
           }
           return NestedObjectLevelN;

The presented approach provides the highest modularity possible (with each associated object having its own parsing function), an extremely low memory footprint (technically it's a SAX on FileStream) and will work on any platform. Using this technique, it is possible to parse even corrupted/malformed XML files by means of supporting iside<*> variables for controlling current position in the XML tree without building it. This requirement is important when dealing with constantly altered "live" files, or when there is no guarantee that the embedded software will always provide consistent well-formed files.

That concludes the main parsing procedure for XML files. It is time to see how the same may be done for fat-fixed format (FFF).

FSO Adapter for Flat Fixed Format

The requirements are the same as for the preceding task - ff-files are large and the memory footprint must be low. Luckily for us, all the rows in ff-files are equal, so we will only need to know the element's position in a single string, without headers, footers or special string numbers. It can be done by one XML configuration file, as is discussed further.
As long as the problem is similar to the preceding parser core, the solution is identical too (or almost so).

We will use FileInputStream as an input and two loops - the outer for the sequential line picking, and the inner for individual line parsing. For the second loop we will construct an Array from XML config file, containing the beginning and ending of the element's position, the element's names and datatypes.
The outer loop is quite standard.

       File file = new File(controlfolder);
       FileReader fileReader = new FileReader(file);
       BufferedReader bufferedReader = new BufferedReader(fileReader);
       Long i = (long) 0;
       Long j = (long) 0;
       String next, line = bufferedReader.readLine();
       for (boolean first = true, last = (line == null); !last; first = false, line = next) {
            last = ((next = bufferedReader.readLine()) == null);
            //process the line
            status = parse(line);
            i=i+1;
            if (status !=0){
               j=j+1;
            }
            }
       System.out.println("Completed. Lines processed:"+i +"; Failed = " + j);

Every fetched line is fetched and passed to parse function. This function is built around the NodeList array loop 

FFXMLConfig ffcong = new FFXMLConfig();
   ESPostHttpClient eshc = new ESPostHttpClient();
   NodeList nList = ffcong.getConfigArray();

   Statlog slog = new Statlog();
   Class lClass = slog.getClass();
   JSONBuilder jb = new JSONBuilder();
   try {

       for (int temp = 0; temp < nList.getLength(); temp++) {
         Node nNode = nList.item(temp);
         if (nNode.getNodeType() == Node.ELEMENT_NODE) {

           Element eElement = (Element) nNode;
           int start = Integer.parseInt(eElement.getAttribute("start"));
           int end = Integer.parseInt(eElement.getAttribute("end"));
           String datatype = getSubstring(eElement.getAttribute("type"),0,1);
           fname = eElement.getTextContent();
           element = getSubstring(str,start-1,end);
           Field field = lClass.getField(fname);

           // Set field using Java reflection to slog obj

       }
     }
   }

As you can see, this loop is all about substrings, first for extracting elements text, using "start" and "end" parameters from config Nodelist. Here it is important to observe that when defining the start and end indexes for substring, the start is inclusive and the end is exclusive in Java. The second substring is for extracting the datatype parameter (which is used further by casting in reflection). 

Now a demonstration of XML config file would be appropriate, and three lines will suffice:

  <?xml version="1.0" encoding="UTF-8"?>
  <adapter-config name="FF2JSONConverter" adapter="file">
    <mapping-spec className="no.acando.stat.entity.FileActivationSpec">
     <property start="1" end="3" type="I3">Field1</property>
     <property start="4" end="5" type="I2">Field3</property>
     <property start="6" end="6" type="I1">Field4</property>

Thus, the inner loop is all about traversing the Nodelist array for one single string, passed as a parameter from outer loop. The nodelist construction can be done by DOM, and its memory footprint is insignificant. 

          import javax.xml.parsers.DocumentBuilderFactory;
          import javax.xml.parsers.DocumentBuilder;
          import org.w3c.dom.Document;
          import org.w3c.dom.NodeList;
          import org.w3c.dom.Node;
          import org.w3c.dom.Element;
          import java.io.File;

          ......
          File fXmlFile = new File(Config.FFMAPPER);
          DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
          DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
          Document doc = dBuilder.parse(fXmlFile);
          doc.getDocumentElement().normalize();
          NodeList nList = doc.getElementsByTagName("property");

Now the most interesting question: how are the Object fields assigned? Straight usage of object modifiers would be tedious and expansive. Nevertheless, we have the Object name, field name, field type, so Java Reflection API can help with implementing a very elegant and compact code, as shown below:

  if(datatype.equalsIgnoreCase("I")){
        if (element.trim().length()==0){
                                element ="0";
        }
     field.set(slog, Integer.parseInt(element.trim()));
    }else if (datatype.equalsIgnoreCase("B")){
  if (element.trim().length()==0){
       element ="0";
    }
     field.set(slog, new BigInteger(element.trim().getBytes()));
    }else if (datatype.equalsIgnoreCase("F")){
  if (element.trim().length()==0){
    element ="0.0";
  }
    //format cleansing: replacing commas with dots
    element = element.replace(",", ".");
    field.set(slog, Float.parseFloat(element.trim()));
  } else{
  if (element.trim().length()==0){
     field.set(slog, " ");
  }else{
  element = RESTStringParam.getEncodedText(element);
  field.set(slog, element.trim());
  }
}

The code above can be quite compact if you only have one datatype to assign, otherwise a cast or conversion would be required, with some data cleansing, similar to routines for the XML parser. It can be quite more unseemly, if the data file is messy. Lots of "trial and error" would be necessary for many instances.

This concludes core parsers functionality for XML and FF data exporters. Other common functionality of these MCS candidates will be demonstrated in the following blog post.

References

  1. http://blog.zenika.com/2014/10/07/Setting-up-a-development-environment-using-Docker-and-Vagrant/
  2. Microservice Design pattern: http://soapatterns.org/design_patterns/microservice 
  3. http://zeroturnaround.com/rebellabs/java-tools-and-technologies-landscape-2016/
  4. http://blog.osgi.org/2014/08/is-docker-eating-javas-lunch.html
  5. Principles of Service Design, Thomas Erl, 2007, Prentice Hall
  6. SOA. Concepts, Technology and Design, Thomas Erl, 2005, Prentice Hall
  7. Web Services Choreography Description Language Version 1.0 http://www.w3.org/TR/ws-cdl-10/#Purpose-of-WS-CDL
  8. XML to JSON https://github.com/stanfy/gson-xml
  9. https://developer.android.com/reference/org/xmlpull/v1/XmlPullParser.html
  10. http://code.nomad-labs.com/2011/12/09/mother-fk-the-scheduledexecutorservice/
  11. http://customerthink.com/the-what-and-where-of-big-data-a-data-definition-framework/
  12. https://sites.google.com/a/mammatustech.com/mammatusmain/reactive-microservices

 

Om bloggeren:
Sergey has a Master of Science degree in Engineering and is a Certified Trainer in SOA Architecture, Governance and Security. He has several publications in this field, including comprehensive practical guide “Applied SOA Patterns on the Oracle Platform”, Packt Publishing, 2014.

comments powered by Disqus