Microservices in action: Deploying Java Adapter Framework’s containers using Vagrant and Docker

Microservices in action: Deploying Java Adapter Framework’s containers using Vagrant and Docker

After establishing working elements of Java Adapter Framework around Data Lake we obliged to deploy them in isolated autonomous containers as it’s a mandatory prerequisite of Microservice implementation. This article will demonstrate how to wrap previously assembled Java modules into portable Docker containers using Vagrant framework. This is complete working example in two parts which you can use in your projects, from samples of Java  code down to the deployment and maintenance scripts.

1. Preface

In the two previous posts we were introduced to the elements of Java Adapter Framework. We strived to achieve and maintain all four main MCS characteristics, prioritizing Reliability. This characteristic, plus a highly isolated deployment model, “Containerisation”, form the Microservice trend. Even SOA Pattern catalogue has included Microservice pattern [1], distinguishing it from other service types and models by the differences in their respective deployment isolation and packaging. Simply put, if you put a service into an individual isolated container, it becomes a Microservice. It is important to note that the SOE Pattern catalogue also distinguishes between between “SOA services” and “Non-SOA services”, just to add to the confusion. Only services which are part of the “SOA environment” can be perceived as decent candidates for packaging into containers and thus becoming Microservices. The Microservice pattern is often declared as a means of solving “monolithic deployment bundles” which allegedly are unscalable and unmanageable, in this case the “monolithic” is vaguely defined, but more on that later. Ironically the same Patters catalogue contains in itself Functional decomposition, service modern abstraction, service layering and so on, but these were apparently insufficient to decompose this “monolith” and turn it into SOA asset. So, conternistation it is! In this post we will see how put our Java code into containers.

1.1 Project’s Reference Model (PRM)

After pulling from Git the current AF bundle, Server build should have the following structure, explained below

1.2 Project folder structure (PFS) and Vagrant&Docker config organisation

Sequenced labels, assigned to every folder in PRM are explained in the table below

No

Purpose

Usage Description

1

Server's project root

AF Server’s root. Can be also used as the main Vagrant assembly point.

2

Assembly Maven project for assembling and linking HistDataExported(MySQL) and XMLExporter(XML parser and loader)

Assembly project for building and packing DB and XML parsing projects (blue upstream arrows). Has no source code. Host for the main stat.property file, configuration for every individual exporter project. Each Exporter project has individual Config class for parsing and loading config values.

3

Maven and Vagrant project for building ES container

Can be used for individual ES container provisioning

4

Flat-Fixed File Exporter project

This is Flat-Fixed file exporter root. One of three AF adapters for this solution

5

MySQL DB Adapter project

DB adapter for extracting historical data and enhancing object deserialised from XML.  Used as dependant project

6

XML+DB data Exporter project root

Primary XML file adapter, using DB functionality from DB adapter.

7,8

Message Box folder with test files, mapped to Docker host and containers as /msgbox/. Subject to change.

MSGBox/TP/ is mapped as /msgbox/. This should be reflected in adatpter's config.property file for fetching and parsing files from the right location, and archiving them.

9

Main Vagrant and Docker assembly project

Maven ant task (target: clean package) will copy all required binaries from all adapters (red downstream arrows) before manual Vagrant up.  This is the root project for deploying containers. 

10

Dockerfile for Elsaticsearch container with ES DB config script

Dockerfile folder for ES container. In Vagrant’s root, the config folder contains description of ES data folder structures, please check it first.

Use Java image as a template, ES is installed as apt-get, check the latest release.  

11

Dockerfile for Flat-Fixed file exporter(FFExporter), binaries and exporter property file config for container.

Dockerfile for FFExporter container, in bin folder stored all binaries(*.jar) and individual config file.

12

Vagrantfile for Vagrant host (Docker host-hypervisor). Vagrant storage for provisioned Docker images (after vagrant init).

VM Docker Host Vagrant file location. Docker provisioning will refer to this file from every container construction session.

13

Dockerfile for XML+DB data exporter (XMLExporter) binaries and exporter property file config for container.

Dockerfile for XMLExporter container, in the bin folder all binaries(*.jar) and config file are stored.

1.3 Overall Server projects deployment model (PDM) based on Docker/Vagrant

Presented deployment model (PDN) is derived from the PRM and explained in PFS table. This container infrastructure can be deployed in two simple steps. After pulling the latest version from Git repository, new binaries should be compiled and packed using Maven. Maven's Antrun plug-in [2], will copy freshly packed jars and main config.property file into relevant Docker subfolders according to the preceding map, but individual adapter's config property files should be verified according to the folder mappings for inboxes and archive folders, pulling interval, etc. The second phase as simple as the building - Vagrant is employed for deploying containers for adapter framework and one Elasticsearch DB. Configuration can be easily changed to have ES DB on the host instead of the container. Technically in current port mapping configuration it will be localhost:9200.

Dockerhost will be created, acting as a container's hypervisor, interfacing host OS and deployed containers. Host OS' shared(synced) folders will be mapped to the stat-host and from there - to the relevant containers in adapter framework. XML exporter will have direct access to the MySQL DB, check the availability before deployment (this check is part of unit test). ES DB container will have port 9200 mapped through the stat-host to the host OS and all containers. Make sure that property files in adapter framework are configured properly. ES DB will be equally accessible from all elements of Docker infrastructure, trough the Docker link using esdocker synonym, including host OS. Adapter framework containers will populate ES FB right from the start, so ES container will be prepared by vagrant first in no parallel mode.
After this short architectural recap, we can proceed to installation steps.

2. Deployment Phase

We assume that Maven, Git/GitBash, VirtualBox, Docker and Vagrant are already installed on the host system. Please use the latest relevant build and updated deployment instructions and bear in mind that Vagrant has not been distributed as a RubyGem for quite a while.

sudo apt-get install dpkg-dev virtualbox-dkms

//get latest vagrant build for Linux (Debian 64-bit in this case)
wget https://releases.hashicorp.com/vagrant/1.8.5/vagrant_1.8.5_x86_64.deb

sudo dpkg --install vagrant_1.8.5_x86_64.deb
sudo apt-get autoremove

Note: Make sure that you are installing the right package for your host OS, as above …x86_64.deb, and not … i86_64.deb. Also these installation steps will be necessary if you get the infamous “provider 'docker' could not be found” error. It can be cured by complete vagrant’s reinstallation.

2.1 Building and packaging Java Adapter Framework and ES

Pull latest release from GitLab as git pull (or initially git clone <project_url.git>) from the project's designated folder.

2.1.1 Get into the root of Server (9) assembler project.

This is maven project for assembling Adapter Framework java components (FF,DB and XML data fetching and parsing) and main Vagrant/Docker root. Execute maven command:

mvn clean package

The result should be as presented on preceding figure, all Java AF components (4,5,6) are successfully packaged. We will need these jars for Docker deployment, so maven AntRun task will copy fresh jars from XMLExporter/target (6) to xmldb-docker/bin(13) in Vagrant assembler project (9). POM plugin’s configuration will be as follows, you may find it useful for your projects assembly.

           <plugin>
              <artifactId>maven-antrun-plugin</artifactId>
              <version>1.4</version>
                <executions>
                  <execution>
                    <id>copy</id>
                    <phase>compile</phase>
                    <configuration>
                     <tasks>
                       <copy file="../XMLExporter/target/XMLExporter-0.0.1-jar-with-dependencies.jar"
tofile="./xmldb-docker/bin/XMLExporter-0.0.1-jar-with-dependencies.jar"/>

                       <copy file="../FFExporter/target/FFExporter-0.0.1-SNAPSHOT-jar-with-dependencies.jar"
tofile="./ff-docker/bin/FFExporter-0.0.1-SNAPSHOT-jar-with-dependencies.jar"
/>

                       <copy file="../FFExporter/src/af/mcsa/stat/utils/ffadapter-config.xml" tofile="./ff-docker/bin/ffadapter-config.xml"/>
                     </tasks>
                     </configuration>
                   <goals>
                   <goal>run</goal>
                 </goals>
               </execution>
             </executions>
           </plugin>

In addition to AntRun plugin, demonstrated above, other useful POM’s elements are listed here, first is the compiler plugin [3]:    

<plugin>
           <artifactId>maven-compiler-plugin</artifactId>
           <version>3.1</version>
           <configuration>
              <source>1.8</source>
              <target>1.8</target>
           </configuration>
       </plugin>

Modules for assembly and packaging type are described as follows:

   <packaging>pom</packaging>
   <modules>
           <module>../DBExporter</module>
           <module>../XMLExporter</module>
           <module>../FFExporter</module>
   </modules>

In Utils (\JavaAFComponents\DBXMLAdapter\src\no\stat\utils) package (2) you can find the main common stat.property file. The same Antrun plugin’s task will copy this file into adapter's <stat-adapter>-docker/bin (11,13 in this case), please modify parameters accordingly if needed. For combined XML-DB adapter the following parameters are critical (three groups): 

Database connectivity and ES index (use IP instead of DB name in connection URL, if needed)

db_name=dlake
username=<my_user_name>
password=<my_password>
es_localhttp_url=http\://localhost\:9200/
project_db_url= jdbc\:mysql\://localhost:3306/
jdbc_driver=com.mysql.jdbc.Driver
es_statindex= statindex/statkey

Message Ingest and Archive folders (mapped in Vagrant config, usually /msgbox/In and Archive), XMLExporter

xmldir= D\:/Sergey/Projects/Common/MCSA/JavaAFComponents/MSGBox/TP/In/
xmlarcdir= D\:/Sergey/Projects/Common/MCSA/JavaAFComponents/MSGBox/TP/Archive/

Runtime log name and main ES index

xmllog=xml-parser.log
es_malingindex=malingindex/maling

Intake folder (mapped in Vagrant config, usually /msgbox/In and Archive), FFExporter

ffdir= D\:/Sergey/Projects/Common/MCSA/JavaAFComponents/MSGBox/TP/In/

Runtime log name and main ES index, FFExporter

fflog=ff-parser.log
fftrace=ff_parsetrace.properties
es_statlogindex=kontroll/kontrollkey

No MySQL connection is required for FF adapter, as depicted in PDM

2.1.2. Preparing ES config for Docker deployment.

There is no compilation or packaging here, ES container is configured by Vagrant's provider script. ES folders structure is defined by yml script in es-docker/config folder (10). Please amend it if needed.

path:
   data: /data/data
   logs: /data/log
   plugins: /data/plugins
   work: /data/work

This step concludes Java/ES component’s preparation for Vagrant deployment. Actual Docker deployment will be discussed in the second part of this post.

References

1. Miscroservice Pattern, SOA Patterns catalogue
2. Apache Maven AntRun Plugin
3. Apache Maven Compiler Plugin

 

Om bloggeren:
Sergey has a Master of Science degree in Engineering and is a Certified Trainer in SOA Architecture, Governance and Security. He has several publications in this field, including comprehensive practical guide “Applied SOA Patterns on the Oracle Platform”, Packt Publishing, 2014.

comments powered by Disqus