Building Applications with Deep Learning: Expectations vs. Reality

Abstract

Nowadays, building applications involves many technologies. There are technologies to render user interfaces, to retrieve and store data, to serve many users, to distribute computing, etc. Increasingly, certain requirements imply the usage of neural networks. So what is the reality of building enterprise applications with the available state-of-the-art neural network technology?

Expectations

I am not a scientist. I am curious, like to understand how things work, have analytical aptitude and love math, but… this is not what I am being paid for. During working hours, my task is to develop real world applications in timely and cost-efficient manner. And thankfully there is plenty of available technology aka tools aka frameworks that allow me to do exactly that. Without understanding of magnetism, I am still able to store information. Without understanding query optimization principles, I am still able to write efficient data queries or, without knowing how to fill memory of graphic card, I am still able to render user interface on the monitor. There is even this funny quotation^[1] from definition of technology on Wikipedia:

“(Technology)… can be embedded in machines which can be operated without detailed knowledge of their workings.”

I expected that using Neural Network Technology is not different. That I could, merely by obeying the constraints of the design of a framework, applying patterns, avoiding anti-patterns and gluing it with all the other relevant technologies, develop real world application without detailed knowledge of every technology I use. If I still haven’t convinced you, read on here…

Reality

The reality is very different. Someone willing to employ neural network technologies at the moment^[2] is forced to do scientific work or at least have an in-depth understanding of neural network methods, despite a number of publicly available technologies created by the brightest and most resourceful minds of our age.

Sure, for a plethora of reasons such as – the technology is not mature enough, some fundamental issues are still unsolved, there is too little pressure from industry^[3], etc. However, there are some reasons that are focus-related. I will address those which became obvious to me during work on a real-world application. In particular:

Kicking-off with the Technology
Tools to move from experiment to the real application
Development Language
Design of Deep Learning Framework

Coming to Terms with Deep Learning

Background Story

Our project started 2014 with the development of recommendation engine for finding solutions in a text-corpora based on documented customer contacts with an after-sales support organisation. After successfully implementing a number of use cases based on statistical approaches with token frequency, collaborative filtering and number of data quality improvements involving, among others, advanced natural language processing techniques we have rolled out a number of productive systems.

In 2016 we turned our attention to Neural Networks and Deep Learning. Having great success with Java (actually, Scala), Spark and the available Java-based machine learning frameworks our first choice was deeplearning4j^[4] Version dl4j-0.4-rc3.9 (aka dl4j).

It was spring 2016, and we got annoyed with dl4j. In retrospect, I see the main drivers of our annoyance were less the framework itself and more our expectations. What we expected, was a yet another robust enterprise framework. Ok the “0.4” and “rc” in the version number should have given us a hint on the maturity of the framework, but we were ignorant. At that time, making dl4j to work for us was complicated, we did not manage to make it run on GPU backend, and even to make the CPU backend work, we had to compile the framework ourselves, which felt like too much additional work that kept us from fulfilling our main task – implementing a neural network that would learn stuff for our use case. After two months of trial & error and parallel experiments with a well-known Python-based framework we decided to switch to that Python framework and suspend work on the dl4j-based network implementation. Oh, the configuration of Python framework was as complicated as that of dl4j, we just got more lucky with it, that’s all.

By the end of November 2016, seven months later, we still hadn’t managed to build a network configuration that would converge with data from our domain. After the initial success with building toy models (MNIST, seq2seq and some other models) we had decided that the Python framework was the most promising, but boy did we err. There were plenty of assumptions about what we could have gotten wrong, but there was no visible direction that would enable us to succeed.

At that time, a colleague of mine, Wolfgang Buchner^[5], mentioned that he has recently saw that dl4j had undergone a major re-vamp. We immediately attempted to build an experimental model with dl4j Version 0.7.2 and within two weeks, we actually succeeded. Within the next two weeks, our model converged to a satisfactory level with our actual data. Four weeks.

Of course no one was very optimistic at the beginning, thus the result surprised us. Reflecting on this surprise, I attempted to analyze the main factors that, in my opinion, helped us succeed, and I came to the conclusion that there were four.

Kicking-off with the Technology

There are times when it’s OK to skip the documentation and move straight to the code. I personally don’t often need to read documentation to understand a framework providing the MVC pattern or ORM framework, because these are well established patterns that are provided by well established frameworks.

In case of neural networks I do have to read the documentation, if they have it at all at all to kick off a project. Sure there are plenty of papers on arXiv, great lectures on YouTube from renowned MIT professors on KLD, Entropy, Regressions, Backprop and whatnot. But theoretical explanations of principle, and the capability to write the code that implements those principles, are two very different animals.

Dl4j has two strengths when it comes to helping someone at the start:

Documentation. Framework documentation is not my first choice to understand principles, but definitely the first choice if I want to be able start writing code really fast. Reason being – it focuses on making things work instead of explaining algorithm working principles in-depth and it focuses on end-to-end use cases including pre-processing data and giving advice on the “dark art” of hyper-parameter tuning. This I have never seen in the documentation of other deep learning frameworks;
Community. I have been hanging around in every deep learning community I could find. Dl4j has the most vibrant, active, patient and open community I have experienced. Yes, most of the answers still come from Skymind people but there is always someone on dl4j gitter channel^[6] who has a couple of good hints up their sleeve.

In general, I have the feeling that the intention of dl4j community is to build real applications. Based on my experience with other deep learning communities I feel that their intention is to discuss topics of their ph.d. thesis or prove this or that theorem.

Tools to move from experiments to real applications

Dl4j is an ecosystem. And as an ecosystem, it provides number of tools to pre-process data or read it from different formats, to integrate the framework with other (e.g. consumer) technologies and to semi-automatically tune the hyperparameters of my model.

There is one single tool provided by dl4j above all others that had a massive impact on the success of our project so far. It is so called the dl4j User Interface, or UI. It is a web page automatically created by the framework (with minimal configuration, literally five lines of code) that shows graphs of some parameters during network training:

val uiServer = UIServer.getInstance()
val statsStorage = new InMemoryStatsStorage()
uiServer.attach(statsStorage)
listeners +:= new StatsListener(statsStorage)
model.setListeners(listeners: _*)

By itself, that would be fine, if you can “read” this analysis data (which, by the way, does not happen by default). So dl4j goes step further and provides extensive and very concrete documentation^[7] on how to interpret and analyse the readings, even providing very particular advice on tuning my network configuration. That really made difference for our project. I am posting the picture of the UI below, but seriously, just navigate to the visualization documentation page of dl4j, and you can read about it in a way more detail.

Development Language

To my astonishment, most of the deep learning frameworks are implemented in dynamic (type) languages. Dynamic typing is a good thing in many cases, but I believe it is the worst possible choice when developing deep learning software.

If you have already worked with some deep learning framework(s), haven’t you wondered why each and every framework provides a number of classes that download, pre-process and feed the data into neural network? I haven’t seen such a thing in any other class of frameworks but I have a guess as to why it is so: namely, because the data has to be quantified and formated in rather complex structure and the transformation of data in a form readable (and learnable) by network is damn difficult.

And then we have this dynamic language, that is so implicit that I literally NEVER know what the hell method A or method B is returning. And when I look at method A, I see it is calling at some point the method A’ which in turn calls A’’ and so on and so forth until I reach the bottom of the stack where the data array is instantiated. By the time I reach the bottom, I have already forgotten what I wanted to accomplish and am trying to figure out the implementation of some utility method of the framework.

In the domain that is so data-centric, where data structure is so important and models’ ability to learn is so dependant on correctness of data, how for heaven’s sake can someone select dynamic language as the development language?

Fun fact: when you create matrix(10, 255, 64) for training a Recurrent Neural Network in a well known framework, you get 10 sequences of 255 elements of size 64; in dl4j, instead, you get 10 sequences of 64 elements of size 255. How is that not important to know in advance what data structure what method would return?

Dl4j is developed in Java. And although Java itself is not the most innovative language out there, it offers two things extremely important to me and my teammates: type safety and its youngest “cousin” Scala, one of the languages best adapted for machine learning out there.

The Design of the Deep Learning Framework

What is available out-of-the-box versus what has to be built by ourselves is an important issue. My observation is that many frameworks are built with only a limited number of use cases in mind, and all the deep learning frameworks I have encountered have mainly research in mind.

One major design advantage of the dl4j from version 0.7.2 is its ability to switch backends without re-compiling the code. The class-path is being scanned for the backend libraries and the available backend is loaded automatically. The obvious advantage is, to be able to run the code on CPU while testing locally and, to run the same code on GPU when deploying on a GPU-Rig. Another advantage is to be able to do backend specific stuff. Consider this code:

import org.nd4j.linalg.factory.Nd4jBackend
import org.nd4j.linalg.jcublas.JCublasBackend

/**
 * We need the Nd4JBackend just for determining if we are using 
 * CUDA or CPU, Dl4j uses its own instance.
 */
private object Nd4jBackendHolder {
  val backend: Nd4jBackend = Nd4jBackend.load()
}

/**
 * Trait supporting Nd4jBackend dependent configurations.
 * E.g.:
 * {{{
 *    BackendType match {
        case CUDA =>
          import org.nd4j.jita.conf.CudaEnvironment
          CudaEnvironment.getInstance().getConfiguration
            .setMaximumGridSize(512)
            .setMaximumBlockSize(512)
        case CPU => // do some CPU specific stuff
      }
 * }}}
 */
trait Nd4jBackendUtils {
  import Nd4jBackendHolder._
  
  trait BackendType
  
  case object CPU extends BackendType
  case object CUDA extends BackendType
  
  val BackendType: BackendType = backend match {
      case _: JCublasBackend => CUDA
      case _ => CPU
  }
}

With this simple trait you are able to e.g. configure your model differently based on available backend. We set the batch size based on available BackendType because e.g. GPU is able to process larger batches more efficiently:

object TrainingJob extends Nd4jBackendUtils {
   val dataIter = new AsyncMultiDataSetIterator(
      new SegmentedThoughtCorpusIterator(inputFiles, ... train = true),
      batchSize = BackendType match {  256
         case _ => 10
      }
   )
   ...
}

The well-known Python framework broke our back when, in an attempt to improve the convergence of our network, we tried to implement number of custom layers (e.g. Stochastic Depth). Because of the design of that network, there is literally no possibility to debug a layer, the usage of backends (the ones who do the heavy lifting) was so unintuitive that we literally were guessing what the design should be, and since everything we wrote was compiled without problems and failed only in runtime, this attempt turned into a nightmare.

Up until now, we haven’t written own layers in dl4j. That work is commencing now and in a short time I will be able to evaluate this aspect more objectively.

Outlook

Currently I see a lot of discussion, experiments and effort being invested into networks to analyze image information. I believe that there is huge potential in the manufacturing industry, and until now I have heard very little about efforts to build solutions for manufacturing.

I think manufacturers or producers of manufacturing equipment will require networks of choreographed and aligned networks. Often subsystems or aggregates have to work in a intelligent and semi-autonomous mode while contributing to the coordination or analysis of the whole manufacturing process.

So ability to train networks for specific tasks and then join them in larger networks will be important. Also the ability to train a network and then replicate it with slight modifications (such as those required because the next production line has a slightly different configuration) will be extremely important to build products using deep learning efficiently. Last but not least, even with state-of-the-art technology, the hyperparameter tuning of a neural network is a painstaking and elaborate process which, to my opinion, could be one of the main hindrances in bringing deep learning applications to market in a timely manner.

In respect to dl4j I strongly feel that that this framework will overtake the current top dogs of deep learning simply by providing the industry tools to build actual products/applications using deep learning. This feeling is motivated by the current condition and the focus of the framework developers. For instance:

Dl4j team is working on the solution for hyper-parameter tuning, called Arbiter^[8];
the community is very active, just check-out the liveliness of the dl4j gitter channel^[6];
github statistics^[9] look very healthy to me and last but not least;
from the involvement of Skymind employees both in support of community and evolving dl4j code-base, it seems that dl4j is very central for the business model of this company. And, according to my experience, when commercial enterprize is backing open source project, it gives a huge boost to that project.

The work on the described project will continue throughout 2017 and, likely, 2018. Our current plan is to stick with dl4j and use it in production. I would love to hear about your experience with deep learning and currently available deep learning frameworks, so comment on!

by Reinis Vicups
reinis(at)orbit-x(dot)de

I am freelance software developer specializing on machine learning. I am not affiliated with deeplearning4j or skymind.

The described project is being developed for Samhammer AG^[10] and continues as of 2017.

Special thanks to Wolfgang Buchner^[5] and other guys for your excellent criticism and corrections.

Angry rantings

Curse of Technology on achieving Mastery

When I was younger I sought for mastery in every technology I used. If I recall correctly, I have achieved mastery in Borland Delphi Pascal, shortly after it died^[11] (at least in the part of the world visible to me). I attempted to gain mastery on several other technologies every since and concluded this: If I work for a commercial software development company that is successful, there will be a ongoing change of employed technologies because of the reasons of

constantly evolving software development technologies,
constantly changing market demands leading to employment of different technologies,
constantly evolving business model of the software company itself again, leading to changes in employed technology.

So basically, unless your business is to develop frameworks, it is next to impossible to achieve mastery in most of technologies. With 41 I have used more than thousand frameworks and several dozen development languages for at least one project (~6 month to ~3 years). Except couple of languages I haven’t kept much, I sometimes don’t even recall the names anymore.

tl;dr

Working for commercial Enterprizes makes it very hard to achieve mastery in used technologies due to fast evolution of technologies themselves, evolving markets and business models of the Enterprises themselves.

[1]https://en.wikipedia.org/wiki/Technology
[2]as of January 2017
[3]these are my assumptions
[4]https://deeplearning4j.org
[5]https://github.com/wobu
[6]https://gitter.im/deeplearning4j/deeplearning4j
[7]https://deeplearning4j.org/visualization
[8]https://github.com/deeplearning4j/Arbiter
[9]https://github.com/deeplearning4j/deeplearning4j/graphs/contributors
[10]https://www.samhammer.de
[11]http://softwareengineering.stackexchange.com/questions/83009/what-happened-to-borland-delphi

OptaPlanner (a.k.a. Drools Planner) – the brand new version 6

Since the beginning of the year (2013) Redhat(RH) / JBoss is working on the most robust, scalable and easy-to-use version of OptaPlanner formerly called Drools Planner. This new release marks a major milestone in both organizational and technical terms.

This article is intended for those considering to migrate from earlier versions of Drools Planner to OptaPlanner and for those interested in the development of this framework. Those of you unfamiliar with OptaPlanner, read about it here (click).

OptaPlanner to become a top-level JBoss Community project

In mid-April OptaPlanner has graduated from the Drools project to become a top-level JBoss Community project as described here. This has reasonable impact on the amount of attention and resources RH is making available to this project and improves offerings for professional and support services.

Major technical improvements compared to the previous version 5.x

I would like to characterize technical improvements in comparison with the previous version as huge! Indeed the list of major architectural, performance, API and usability improvements is rather long. In this article I am listing those that impressed me the most:

Performance improvements

Subjectively – much faster. Objectively – hard to tell. During the migration from version 5.x to 6.0 I did a number of improvements in both my Production Rules and my Moves. I don’t know if metaheuristic algorithm of OptaPlanner undergo any improvements. What I know is that Drools Expert in version 6.0 is using the PHREAK algorithm (there are some performance comparisons on the internet) which is supposed to be much faster than the “old” RETE algorithm. To be able to compare version 5.x and 6.0 objectively, I would have to use exactly the same Production Rules and Moves(Factories). Alas, up to now I haven’t found those spare hours to implement that.
Adding up all the changes and improvements I just mentioned improved my project performance from about 1500-3000 average counts per second (ACS) in version 5.x to a stunning 6000-22000 ACS in version 6.0! That surely is an impressive performance gain.

Reduction of framework boilerplate

A number of methods required by framework to work are not required anymore in the version 6.0. For instance

Solution.cloneSolution() is not required;
Solution.equals() is not required;
Solution.hashCode() is not required;

These methods were subject to frequent but boiler-platy change if ones domain model was changing/evolving and were often prone to errors and side-effects (at least in my case).

ScoreHolder replaces *ConstraintOccurrence

This one is on the Production-side (Drools Expert) and my favorite one. In previous versions of OptaPlanner one would have to logically insert constraint occurrences, count them and pass them to ScoreHolder. ScoreHolder then is used by OptaPlanner during the search to determine the overall goodness of the current working solution. The code for this looks e.g. like this:

/**
* Before with Drools Planner 5.x
*/
rule "no assignments before kick-off"
	when
		$project : Project(kickOff != null)
		$assignment : Assignment(project == $project, interval != null, $project.kickOff.isAfter(interval.start))
	then
		insertLogical( new IntConstraintOccurrence("assignment before kick off", ConstraintType.NEGATIVE_HARD, 1, $assignment) );
end

/**
* and at the end of rule-set
*/
//Accumulate hard constraints
rule "hardConstraintsBroken"
		salience -1 //Do the other rules first (optional, for performance)
	when
		$hardTotal : Number() from accumulate(
			IntConstraintOccurrence(constraintType == ConstraintType.NEGATIVE_HARD, $weight : weight),
			sum($weight) //Vote for http://jira.jboss.com/jira/browse/JBRULES-1075
		)
	then
		scoreCalculator.setHardConstraintsBroken($hardTotal.intValue());
end

On the one side this is partially a bothersome boilerplate code and on the other side the parametrization of the *ConstraintOccurrence() has caused me headaches a couple of times.

When looking at JavaDoc one can see the optional parameter Object… causes enumerating domain information that has caused the constraint occurrence. If one looks at the rule example above, one sees that only $assignment was passed as a cause for the IntConstraintOccurrence. One could now ask “Why wasn’t e.g. $project passed as a cause as well?” My answer to that question is: “Heck, I have no idea!” (and yes, I wrote this rule myself).
In the version 6.0 everything looks nice, like this:

/**
* Now with OptaPlanner 6.0
*/
rule "no assignments before kick-off"
    when
        $project : Project(kickOff != null)
        $assignment : Assignment(project == $project, interval != null, $project.kickOff > interval)
    then
        scoreHolder.addHardConstraintMatch(kcontext, -1);
end

/* AND THERE IS NO CONSTRAINT ACCUMULATION RULE NEEDED ANYMORE! */

More transparent planning result reporting

This is a very important step towards user friendly and transparent planning-result reporting. Most of my customers are asking me: “What does solution score -2hard/-3soft mean? What constraints were violated, what constraints could be met?” With the OptaPlanner v6.0 the possibilities to report this are somewhat improving. With the code example below one can generate a pretty detailed constraint match report:

@SuppressWarnings("unchecked")
private ArrayList generateSchedulingConstraintList(MySolution mySolution) {
    ArrayList myConstraintList = new ArrayList();
    KieBase kieBase = ((DroolsScoreDirectorFactory) solver.getScoreDirectorFactory()).getKieBase();

    ScoreDirector scoreDirector = solver.getScoreDirectorFactory().buildScoreDirector();
    if (!(scoreDirector instanceof DroolsScoreDirector)) {
        return;
    }

    scoreDirector.setWorkingSolution(mySolution);
    scoreDirector.calculateScore();

    for (ConstraintMatchTotal cmt : scoreDirector.getConstraintMatchTotals()) {
        Rule rule = kieBase.getRule(cmt.getConstraintPackage(), cmt.getConstraintName());

        for (ConstraintMatch cm : cmt.getConstraintMatchSet()) {
            List justificationList = new ArrayList();
            for (Object justification : cm.getJustificationList()) {
                // workaround for nested collections of justifications
                if (justification instanceof Collection) {
                    for (MyFact singleJustification : ((Collection) justification)) {
                        justificationList.add((MyFact) singleJustification);
                    }
                } else {
                    justificationList.add((MyFact) justification);
                }
            }

            if (justificationList.size() > 0) {
                MyConstraint myConstraint = new MyConstraint(cmt.getConstraintPackage(), cmt.getConstraintName(), cmt.getScoreLevel(), justificationList);
                myConstraintList.add(myConstraint);

                myConstraint.setMetaData(new ArrayList());
                for (Entry meta : rule.getMetaData().entrySet()) {
                    myConstraint.getMetaData().add(new MyConstraintMetaData(myConstraint, meta.getKey(), meta.getValue().toString()));
                }
            }
        }
    }
    return myConstraintList;
}

This code, although it “does the job”, is still far from a sufficient solution. I have many arguments why so:

Code is too big with too many boilerplate
Too much of framework internals have to be used to get the result
No direct support for Rule Metadata (it has to be extracted manually with the mechanics provided by Drools Expert API)
I personally dislike how the Justifications are made available (copying back and forth, nested collections – all this adds to complexity and reduces intuitiveness)
Own data structures have to be provided for storing constraint information that, to my opinion, should be natively supported by the framework

Configurable Selectors

One of the difficult tasks to be accomplished during OptaPlanner-based application development is to ensure an appropriate selection. I’ll define an appropriate selection as a selection of planning facts, planning entities, planning values that allow OptaPlanner to search towards the best possible solution and as a selection that at the same time is small enough to fit into (Working)Memory. In version 6.0 a great effort was done to move selection generation from application java code into configuration. At the moment I use solely the features allowing to probabilistically distribute the generation of different types of selections, like this:


    ...
    
        
            
            4.0
            MySwapMoveFactory
        
        
            
            6.0
            MySimpleMoveFactory
        
    ...

The new selection mechanism offers many other powerful features to improve selection performance and scalability e.g.

just in time planning value generation helps to save RAM
entity and value selectors/filters allow to constrain selection to particular entity/entities, particular variable/variables or even filter for particular values
caching allows to control when Moves are being generated thus reducing required RAM and CPU time necessary for the generation (the mimic selection has similar effect)
selection order allows to control selection distribution of moves, planning entities, planning values,…

Multi-level (aka bendable) scoring

This one is important to actually find a solution in a number of real-life planning problems. In older versions of OptaPlanner a limited number of scoring was available (such as either simple or hard and soft). Of course one could always implement own (java) scoring mechanism but it is quite hard to do it right and in case of java scoring one can not take advantage of the production system (Drools Expert). With the version 6.0 a new configurable scoring mechanism was added – bendable score allowing for multiple levels of hard and soft score (a special case of that is the hard, medium and soft score – medium and soft being two levels of soft score).

First step towards the separation of API and Implementation

This one, to my opinion, is also a very important step towards establishing a robust and elegant API design – the challenge OptaPlanner-Team has apparently taken up to. The application programming interface and the implementation of OptaPlanner Framework are being separated! In the version 6.0 org.optaplanner.core has now the following package structure:

--- org.optaplanner.core
 +- api
 +- config
 +- impl

This may not seem as such a big deal, but it is! Good Framework means – an architecture and a design that are intuitive and easy to use by hundreds if not thousands of application developers. And designing this is far more than just moving classes to *.api or to *.impl package respectively.

@ValueRangeProvider

Another nice improvement on the way to a truly intuitive and elegant framework design are the ValueRangeProviders. Although the previous solution was not bad, the new solution offers looser coupling and more explicit demarcation of elements involved in interactions with OptaPlanner API. Here’s what I mean:

/**
* Before with Drools Planner 5.x
*/
@PlanningEntity
public class MyPlanningEntity
    @PlanningVariable(strengthWeightFactoryClass = IntervalStrengthWeightFactory.class)
    // the annotation below means - values are comming from Solution and are returned by getIntervalList()
    @ValueRangeFromSolutionProperty(propertyName = "intervalList")
    public Interval getInterval() {
        return interval;
    }
...

public class MySolution implements Solution {
    // no annotation here!
    public List getIntervalList() {
        return intervalList;
    }
...

/**
* Now with OptaPlanner 6.0
*/
@PlanningEntity
public class MyPlanningEntity
    // the annotation below means - values are comming from ValueRangeProvider with the id "intervalList"
    @PlanningVariable(valueRangeProviderRefs = { "intervalList" }, strengthWeightFactoryClass = IntervalStrengthWeightFactory.class)
    public Interval getInterval() {
        return interval;
    }
...

public class MySolution implements Solution {
    @ValueRangeProvider(id = "intervalList")
    public List getIntervalList() {
        return intervalList;
    }
...

As one can see, the declaration of PlanningVariable in the version 6.0 is consolidated, the specific annotations such as @ValueRangeFromSolutionProperty are dropped and more general @ValueRangeProvider introduced (this annotation also improves code readability since now it is apparent what properties provide planning values).

Introduction of Generics

I was wondering for quite some time as to when the OptaPlanner team will introduce support for generics. This time is now, featuring two elegant improvements:

Generic *WeightFactory

Especially while learning concepts of OptaPlanner, and in addition to rather detailed and well-structured documentation, usage of generics helps an inexperienced user a lot. God knows how many hours I initially invested to understand and implement my first WeightFactory. Below I added two code fragments that demonstrate this:

/**
* Before with Drools Planner 5.x
*/
public class MyWeightFactory implements PlanningValueStrengthWeightFactory {
	@Override
	public Comparable createStrengthWeight(Solution solution, Object planningEntity) {
        MySolution mySolution = (MySolution) solution;
        MyPlanningEntity myEntity = (MyPlanningEntity) planningEntity;

        return new MyEntityStrengthWeight(mySolution.getWeight(myEntity), myEntity.getId());
	}

/**
* Now with OptaPlanner 6.0
*/
public class MyWeightFactory implements SelectionSorterWeightFactory {
    @Override
    public Comparable createSorterWeight(MySolution mySolution, MyPlanningEntity myEntity) {
        return new MyWeight(myEntity, mySolution.getWeight(myEntity));
    }

Generic Move*Factory

MoveFactories got generics-pimped as well, here a short comparison of the code:

/**
* Before with Drools Planner 5.x
*/
public class MyMoveFactory implements MoveListFactory {
    @SuppressWarnings("unchecked")
    @Override
    public List createMoveList(@SuppressWarnings("rawtypes") Solution solution) {
        MySolution mySolution = (MySolution) solution;

/**
* Now with OptaPlanner 6.0
*/
public class MyMoveFactory implements MoveListFactory {
    public List createMoveList(MySolution mySolution) {

What’s missing

I personally miss two things in OptaPlanner:

1. The constraint reporting is for me personally the most lacking feature in OptaPlanner. I can only speculate why it was “neglected” up to now and my guess is that OptaPlanner started as a Research & Development or Proof-of-Concept project and as such it was important to prove that it performs well. I draw this speculation from the available reporting/benchmarking mechanisms in OptaPlanner. The scalar scores are good to compare different planning runs or quantitatively compare the solution of two different problems by OptaPlanner. For a qualitative analysis a detailed constraint report is a must, imho! Also, the benchmarking support is quite good in OptaPlanner (you can even visualize it graphically and generate a html based report that is just beautiful) – this too, as of my guess, comes from the intention to prove that the framework performs well.

2. The age of big data and machine learning is almost there and since OptaPlanner or, for that matter, the whole Drools Ecosystem is actually predestined to solve voluminous problems with highly complex correlations and rules I wonder when that will come to OptaPlanner and consorts. I personally think that multi-threading or another form of parallelism along with the ability for OptaPlanner to learn (some ideas such as hyper-heuristics are already in the air) shall come quickly.

Summary

By all means use OptaPlanner v6.0! It is better, more robust and with a better performance than all its predecessors. And even if you don’t care for predecessors, still try it since it is a beautiful and easy-to-use framework for you to build solutions for planning problems.

reinis.

End-to-end setting up TomEE on a linux server

In this post I describe an end-to-end setup for TomEE+ and my application on a vanilla linux (Debian) server.

This is just one of the many possible configurations. Be advised that changes done to system(s) or configuration(s) might be useful in some cases while in other cases not. Although I have put an effort in explaining why I perform these or those changes, errors and omissions are likely. That’s why I cannot take any responsibility for the loss of data or damage to your systems. Always use your own brain and question stuff you read here at all times.

One more disclaimer – information in this post is mainly credited by other people in numerous publications in internet. I just aggregated, structured and adapted it to my needs. If some of you recognize own material (or that of your friend) please let me know and I will gladly add the credit. I am not doing this right away since this article is a result of days and days of research and I just can’t remember the sources I got this information from.

Enjoy!

Preconditions

64bit Debian Server
LAMP (Linux Apache MySql PHP)

1. Install Java JDK 1.7

I am installing oracle jdk and yes it has to be jdk (as in no – jre is NOT sufficient). OpenJDK had some issue (with either TomEE or, more likely, my own web application) I unfortunately cannot remember.

get Java 1.7 here: http://www.oracle.com/technetwork/java/javase/downloads/index.html

move Java to the “right” location

mkdir /usr/lib64/jvm
mv [java download dir] /usr/lib64/jvm/

set symlink (so that later java updates get propagated)

ln -s /usr/lib64/jvm/jre1.7.0_09/ /usr/lib64/jvm/java-7-oracle

activate java

update-alternatives --install "/usr/bin/java" "java" "/usr/lib64/jvm/jdk-7-oracle/bin/java" 1
update-alternatives --install "/usr/bin/jar" "jar" "/usr/lib64/jvm/jdk-7-oracle/bin/jar" 1

open profile file, set JAVA_HOME, save it and exit

nano /etc/profile

JAVA_HOME="/usr/lib64/jvm/java-7-oracle"
export JAVA_HOME

refresh environment
```
source /etc/profile
```

2. Configure apache2

So why do we need apache2 at all? I had two reasons, the reason one being this article on stackoverflow and the sconed reason being apache2 already pre-installed by my server provider as part of LAMP.

get mod_jk
```
apt-get install libapache2-mod-jk
```

change two lines in /etc/libapache2-mod-jk/workers.properties

workers.tomcat_home=/usr/local/tomee
workers.java_home=/usr/lib64/jvm/java-7-oracle

create and fill /etc/apache2/conf.d/mod-jk.conf

JkWorkersFile /etc/libapache2-mod-jk/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel error
JkShmFile /var/log/apache2/jk-runtime-status

create your virtual Host in /etc/apache2/sites-available/ic.remasoft.com


    JkMount /* ajp13_worker
    ServerName ic.remasoft.com
    DocumentRoot /usr/local/tomee/webapps
    ErrorLog /usr/local/tomee/logs/error.log
    CustomLog /usr/local/tomee/logs/access.log common
    
        Options -Indexes

activate vHost and restart apache2

a2ensite ic.remasoft.com
/etc/init.d/apache2 restart

3. Install TomEE

If at this point you still don’t know what TomEE is please leave a comment explaining why the heck did you read this article up to this point! 🙂 Seriously tho, here’s good starting point.

get TomEE Plus here: http://tomee.apache.org/downloads.html

move TomEE to the “right” location

mkdir /usr/local/tomee
mv [TomEE download dir] /usr/local/tomee

add tomee user

groupadd tomee
useradd -g tomee -d /usr/local/tomee tomee
usermod -G www-data tomee

chmod +x /usr/local/tomee/bin/*.sh

create init file for tomee
```
nano /etc/init.d/tomee
```

add following text to tomee’s init file and save it

#!/bin/bash
### BEGIN INIT INFO
# Provides:          tomee
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $local_fs $remote_fs $network
# Should-Start:      $named
# Should-Stop:       $named
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: TomEE_Plus_ic
# Description:       start TomEE with iC
### END INIT INFO

#TomEE auto-start
#description: Auto-starts TomEE
#processname: tomee
#pidfile: /var/run/tomee.pid

export JAVA_HOME=/usr/lib64/jvm/java-7-oracle
case $1 in

start)
  /bin/su tomee -c /usr/local/tomee/bin/startup.sh
  ;;

stop)  
  /bin/su tomee -c /usr/local/tomee/bin/shutdown.sh
  ;;

restart)
  /bin/su tomee -c /usr/local/tomee/bin/shutdown.sh
  /bin/su tomee -c /usr/local/tomee/bin/startup.sh
  ;;
esac

exit 0

set rights for the init file
```
chmod 755 /etc/init.d/tomee
```
set autostart
```
update-rc.d tomcat defaults
```

4. Configure TomEE

Now TomEE runs out-of-the-box, so this part is required only if you have explicit configuration needs related to your specific application. I have an application that uses MySql and have couple of special needs regarding logging and application deployment.

configure lib dir of TomEE
- get mysql connector here : http://www.mysql.com/downloads/connector/j/mysql-connector-java-x.x.xx-bin.jar and copy it to tomee’s lib dir
- get log4j-1.2.17.jar and copy it to tomee’s lib dir
- get slf4j-log4j12-1.7.1.jar and copy it tom tomee’s lib dir
- remove slf4j-jdk14-1.7.2.jar from lib dir to avoid slf4j init conflicts
```
mv mysql-connector-java-5.1.22-bin.jar /usr/local/tomee/lib
mv log4j-1.2.17.jar  /usr/local/tomee/lib
mv slf4j-log4j12-1.7.1.jar /usr/local/tomee/lib
rm /usr/local/tomee/lib/slf4j-jdk14-1.7.2.jar
```

add log4j config directly in lib folder and add configuration

nano /usr/local/tomee/lib/log4j.properties

log4j.debug=true
log4j.rootLogger=INFO, CATALINA, CONSOLE

# Define all the appenders
log4j.appender.CATALINA=org.apache.log4j.FileAppender
log4j.appender.CATALINA.file=${catalina.base}/logs/catalina.log
log4j.appender.CATALINA.encoding=UTF-8
log4j.appender.CATALINA.layout=org.apache.log4j.PatternLayout
log4j.appender.CATALINA.layout.conversionPattern = %d [%t] %-5p %c - %m%n
log4j.appender.CATALINA.append=true

log4j.appender.LOCALHOST=org.apache.log4j.FileAppender
log4j.appender.LOCALHOST.file=${catalina.base}/logs/localhost.log
log4j.appender.LOCALHOST.encoding=UTF-8
log4j.appender.LOCALHOST.layout=org.apache.log4j.PatternLayout
log4j.appender.LOCALHOST.layout.conversionPattern = %d [%t] %-5p %c - %m%n
log4j.appender.LOCALHOST.append=true

log4j.appender.MANAGER=org.apache.log4j.FileAppender
log4j.appender.MANAGER.file=${catalina.base}/logs/manager.log
log4j.appender.MANAGER.encoding=UTF-8
log4j.appender.MANAGER.layout=org.apache.log4j.PatternLayout
log4j.appender.MANAGER.layout.conversionPattern = %d [%t] %-5p %c - %m%n
log4j.appender.MANAGER.append=true

log4j.appender.HOST-MANAGER=org.apache.log4j.FileAppender
log4j.appender.HOST-MANAGER.file=${catalina.base}/logs/host-manager.log
log4j.appender.HOST-MANAGER.encoding=UTF-8
log4j.appender.HOST-MANAGER.layout=org.apache.log4j.PatternLayout
log4j.appender.HOST-MANAGER.layout.conversionPattern = %d [%t] %-5p %c - %m%n
log4j.appender.HOST-MANAGER.append=true

log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.encoding=UTF-8
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.conversionPattern=%d [%t] %-5p %c - %m%n

# Configure which loggers log to which appenders
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost]=INFO, LOCALHOST
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager]=\
  INFO, MANAGER
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager]=\
  INFO, HOST-MANAGER

remove standard log

mv /usr/local/tomee/conf/logging.properties /usr/local/tomee/conf/logging.properties.jul

adjust Engine and Host tag in /usr/local/tomee/conf/server.xml to
```
...

  
  
```

replace all Resource tags in /usr/local/tomee/conf/tomee.xml with this


        JdbcDriver com.mysql.jdbc.Driver
        JdbcUrl jdbc:mysql://localhost:3306/ic?autoReconnect=true
        UserName root
        Password 
        JtaManaged true



        JdbcDriver com.mysql.jdbc.Driver
        JdbcUrl jdbc:mysql://localhost:3306/ic?autoReconnect=true
        UserName root
        Password 
        JtaManaged false

change welcome-file-list in /usr/local/tomee/conf/web.xml to

    
     
        index.xhtml
        index.html
        index.htm
        index.jsp

adjust properties in /usr/local/tomee/conf/system.properties

tomee.jaxws.subcontext = /
openejb.deploymentId.format = {appId}/{ejbName}
openejb.wsAddress.format = /{wsdlService}

remove the default ROOT webapp
```
rm -R /usr/local/tomee/webapps/ROOT
```

5. Install iC

get iC *.war files and move them to TomEEs webapps dir

mv ROOT.war /usr/local/tomee/webapps
mv some-services.war /usr/local/tomee/webapps

6. Configure MySql

create iC database

mysql -u root -p
mysql> create database ic character set utf8;
mysql> grant all privileges on ic.* to root@localhost;

add ic tables

SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0;
SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0;
SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='TRADITIONAL';

CREATE SCHEMA IF NOT EXISTS `ic` DEFAULT CHARACTER SET utf8 ;
USE `ic` ;

-- -----------------------------------------------------
-- Table `ic`.`Permission`
-- -----------------------------------------------------
CREATE  TABLE IF NOT EXISTS `ic`.`Permission` (
  `id` INT(11) NOT NULL AUTO_INCREMENT ,
  `permission` VARCHAR(128) NOT NULL ,
  PRIMARY KEY (`id`) ,
  UNIQUE INDEX `permission_UNIQUE` (`permission` ASC) )
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;

-- -----------------------------------------------------
-- Table `ic`.`Role`
-- -----------------------------------------------------
CREATE  TABLE IF NOT EXISTS `ic`.`Role` (
  `id` INT(11) NOT NULL AUTO_INCREMENT ,
  `name` VARCHAR(45) NOT NULL ,
  `description` TEXT NULL DEFAULT NULL ,
  PRIMARY KEY (`id`) ,
  UNIQUE INDEX `name_UNIQUE` (`name` ASC) )
ENGINE = InnoDB
AUTO_INCREMENT = 2
DEFAULT CHARACTER SET = utf8;

-- -----------------------------------------------------
-- Table `ic`.`Role_Permission`
-- -----------------------------------------------------
CREATE  TABLE IF NOT EXISTS `ic`.`Role_Permission` (
  `role_id` INT(11) NOT NULL ,
  `permission_id` INT(11) NOT NULL ,
  PRIMARY KEY (`role_id`, `permission_id`) ,
  INDEX `fk_role_permission_1` (`role_id` ASC) ,
  INDEX `fk_permission_role_1` (`permission_id` ASC) ,
  CONSTRAINT `fk_permission_role_1`
    FOREIGN KEY (`permission_id` )
    REFERENCES `ic`.`Permission` (`id` )
    ON DELETE NO ACTION
    ON UPDATE NO ACTION,
  CONSTRAINT `fk_role_permission_1`
    FOREIGN KEY (`role_id` )
    REFERENCES `ic`.`Role` (`id` )
    ON DELETE NO ACTION
    ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;

-- -----------------------------------------------------
-- Table `ic`.`User`
-- -----------------------------------------------------
CREATE  TABLE IF NOT EXISTS `ic`.`User` (
  `id` INT(11) NOT NULL AUTO_INCREMENT ,
  `username` VARCHAR(20) NOT NULL ,
  `password` VARCHAR(256) NOT NULL ,
  `salt` TINYBLOB NULL DEFAULT NULL ,
  `version` INT(11) NULL DEFAULT NULL ,
  PRIMARY KEY (`id`) ,
  UNIQUE INDEX `username_UNIQUE` (`username` ASC) ,
  INDEX `version_NONUNIQUE` (`version` ASC) )
ENGINE = InnoDB
AUTO_INCREMENT = 2
DEFAULT CHARACTER SET = utf8;

-- -----------------------------------------------------
-- Table `ic`.`User_Role`
-- -----------------------------------------------------
CREATE  TABLE IF NOT EXISTS `ic`.`User_Role` (
  `role_id` INT(11) NOT NULL ,
  `user_id` INT(11) NOT NULL ,
  PRIMARY KEY (`role_id`, `user_id`) ,
  INDEX `fk_role_user_1` (`role_id` ASC) ,
  INDEX `fk_user_role_1` (`user_id` ASC) ,
  CONSTRAINT `fk_role_user_1`
    FOREIGN KEY (`role_id` )
    REFERENCES `ic`.`Role` (`id` )
    ON DELETE NO ACTION
    ON UPDATE NO ACTION,
  CONSTRAINT `fk_user_role_1`
    FOREIGN KEY (`user_id` )
    REFERENCES `ic`.`User` (`id` )
    ON DELETE NO ACTION
    ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;

SET SQL_MODE=@OLD_SQL_MODE;
SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS;
SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS;
}}}

add data

INSERT INTO `Role` (`id`,`name`,`description`) VALUES (1,'admin',NULL);
INSERT INTO `User` (`id`,`username`,`password`,`salt`,`version`) VALUES (1,'ic','somepassword',NULL,1);
INSERT INTO `User_Role` (`user_id`,`role_id`) VALUES (1,1);

7. Run

set tomee as owner of tomee dir
```
chown tomee:tomee /usr/local/tomee -R
```
execute as super user
```
/etc/init.d/tomee start
```

8. Test

Finally, call your application (I did it by calling http://ic.remasoft.com).

You have set up a TomEE instance on a linux system, congratulations!

maven-ear-plugin and application.xml

If you are building a modern JavaEE 6 application you might need to package it in EAR.

Supposingly, specification says if your EAR contains META-INF/application.xml file, you must provide configuration of modules so that the application server knows what to load.

Now, specification also supposingly says that, if you want your application server to AUTO DISCOVER your modules (EJBs, CDI beans and so on) you MUST omit application.xml file alltogether.

Well I didn’t know that and wasted serious amount of time making the EAR the “right way”.

So how do I create EAR easily if I live in a maven world?

Luckily there is maven-ear-plugin that supposingly allows for easy EAR creation. Modern versions of this plugin has an configuration option named generateApplicationXml that makes maven-ear-plugin to stop generating application.xml… supposingly.

If you simply add the generateApplicationXml to your plugin configuration, your build will fail with the message:

Failed to execute goal org.apache.maven.plugins:maven-ear-plugin:2.8:ear (default-ear)
on project my-ear: Deployment descriptor:
my-ear\target\retena-worker-ear\META-INF\application.xml does not exist. -> [Help 1]

Ye okay, another couple of hours searching internet until I came up to solution. maven-ear-plugin has another configuration option called version this indicates somehow the java version the descriptors are to be generated for. And only combo of generateApplicationXml and version makes the plugin to stop generating application.xml and not failing to build.

Stupid me or stupid plugin?

Here’s a complete example:


 my-ear
 
  
   org.apache.maven.plugins
   maven-ear-plugin
   2.8
   
    6 
    false 
    lib

Quirks and twists of testing CODI @Bundle with OpenEJB ApplicationComposer

Using JavaEE means also testing JavaEE with all the implications!

I personally use an ApplicationComposer from OpenEJB when writing unit tests not requiring all containers (as in web, cdi, ejb, …) up and running but just enough to have injection and deploy ejbs. I am not happy with ApplicationComposer because I think it has number of limitations but that’s another discussion. If you care, you can read up on ApplicationComposer in this nice post from Romain Manni Bucau.

Just recently I encountered an issue when ApplicationComposer-testing application using MyFaces CODI extensions. The error shows as an exception (I marked the interesting parts red):

SEVERE - CDI Beans module deployment failed
javax.enterprise.inject.UnsatisfiedResolutionException: Api type [org.apache.myfaces.extensions.cdi.core.api.resource.bundle.ResourceBundle]
is not found with the qualifiers Qualifiers: [@javax.enterprise.inject.Default()] for injection into Field Injection Point, field name : excelTemplate
, Bean Owner : [ScheduleWorkbookController, Name:null, WebBeans Type:MANAGED, API Types: my.app.ScheduleWorkbookController,java.lang.Object,java.io.Serializable]
, Qualifiers:[javax.enterprise.inject.Any,javax.enterprise.inject.Default]]
at org.apache.webbeans.util.InjectionExceptionUtil.throwUnsatisfiedResolutionException(InjectionExceptionUtil.java:77)
at org.apache.webbeans.container.InjectionResolver.checkInjectionPoints(InjectionResolver.java:178)
at org.apache.webbeans.container.BeanManagerImpl.validate(BeanManagerImpl.java:914)
at org.apache.webbeans.config.BeansDeployer.validate(BeansDeployer.java:440)
at org.apache.webbeans.config.BeansDeployer.validateInjectionPoints(BeansDeployer.java:390)
at org.apache.webbeans.config.BeansDeployer.deploy(BeansDeployer.java:194)
at org.apache.openejb.cdi.OpenEJBLifecycle.startApplication(OpenEJBLifecycle.java:182)
at org.apache.openejb.cdi.ThreadSingletonServiceImpl.initialize(ThreadSingletonServiceImpl.java:158)
at org.apache.openejb.cdi.CdiBuilder.build(CdiBuilder.java:43)
...

Now what we see here is that

CDI container does not find some bean class (UnsatisfiedResolutionException)
in particular it cannot find ResourceBundle class
injected as a variable excelTemplate
into ScheduleWorkbookController

Ok so according to ApplicationComposer configuration rules “all” we have to is to add all the CDI relevant classes to a Class array returned by the @Module method:

@RunWith(ApplicationComposer.class)
public MyTest {
    @Module
    public Class[] classes() {
        return new Class[] { ScheduleWorkbookController.class, ExcelTemplate.class, ResourceBundle.class };
    }
}

That’s it right? Wrong, the exception will still occur!

The reason is the fact that the org.apache.myfaces.extensions.cdi.core.api.resource.bundle.ResourceBundle is just an interface. You still need the implementing class so that during CDI container initialization the real instances can be injected.

So let us find out if there are classes provided in CODI implementation that implement ResourceBundle. In fact there is exactly one default class org.apache.myfaces.extensions.cdi.core.impl.resource.bundle.DefaultResourceBundle.

Alas, adding it to the @Module class array will NOT work since the DefaultResourceBundle has package visibility!

Now, the solution is logical at the end but believe me, based on error messages and semi-chaotic attempts to somehow make it work, it did not came to me the easy way.

Unless you have already guessed – the solution is to add org.apache.myfaces.extensions.cdi.core.impl.resource.bundle.ResourceBundleProducer to the @Module class list.

That’s right – the class producing the actual instances of ResourceBundle!

JMX and JPA with Hibernate

Firstly, I cheated. Actually this post should be named “JMX and application-specific resources”. But, since I found this architectural property of JMX while attempting to use the JPA within JMX managed bean, the title is what it is.

Secondly, this post is neither on what JMX is nor what JPA is. If you are unfamiliar with those – read basics someplace else. Be advised though that while there is plethora of excellent material on JPA (just google for it and you’ll find everything you could’ve dreamed about), there is next to nothing on JMX good enough for ME to understand it! Well yeah, there are number of docs and articles from Sun and independent authors available out there. Me, I am dissatisfied with all of them. For instance, how the heck I work with composite or tabular data types of the open beans or, in fact, how do I work with application specific resources, huh?

So, as I said, there was this use case I was working on – “Show me in a JMX managed bean some application configuration data out of persistence”. My application is a web-app running on the tomcat. I initialized MBean in her (bean’s) constructor and registered it with the default MBeanServer of the JVM with something like this:
MBeanServer server = ManagementFactory.getPlatformMBeanServer(); server.registerMBean(this, "name");
Within MBean I was using a DAO which loaded for me some config data over the JPA with Hibernate. To my great pleasure – the MBean worked at the first try until… I changed something. I did some minor optimization and my MBean broke down. For whole two days I and my colleague (wink Christian) were sweating hard to fix the bug.

The symptom was that the JPA couldn’t instantiate EntityManager anymore with the exception of something like “There are no providers for the persistence unit MyPersistenceUnit”.

After two day trial and error, endless reading and attempting to understand scarce JMX docs the issue turned out to be initialization of that DAO I mentioned earlier. Inside the DAO I was creating an entity manager like this:
emf = Persistence.createEntityManagerFactory("MyPersistenceUnit"); em = emf.createEntityManager();
In a web-app (EJB app to be precise) the JPA (or in fact Hibernate) expects a persistence.xml to be located in /META-INF/persistence.xml of the web-app. The persistence.xml is loaded then by Hibernate as a resource using context class loader of the current thread.

This is where things get ugly when using JMX. How I understand it, registering MBean with the platform MBean server causes the MBean methods be called from the system class loader of the JVM when the MBean methods are invoked by the call-backs(?) from the MBeanServer. Yes, the system class loader of the JVM (as in Java Virtual Machine). Not the class loader of the web-app, not even class loader of the tomcat, but the class loader of the JVM itself. Why? Because MBeanServer runs directly in JVM. This in turn means that if attempts are made to load resources located on the class path of the web-app in the MBean methods called back by the MBeanServer, they will fail since the class loader of the MBeanServer (the system class loader of the JVM) knows nothing about web-app or, in fact, any class path that lies below that of the JVM.

Workaround

Yeah, I can’t call it a solution since it’s not. But the workaround that worked for me was to initialize the DAO and store it (the DAO) as a class attribute of the MBean implementation before I hand the control over to MBeanServer. This way, the methods of MBean can work with the initialized instance of the DAO. This workaround sucks for number of reasons but I can’t seem to find a better one.

The final point is – any resource loading done in the methods of MBean invoked by call-backs from MBeanServer will fail since those invocations are done within thread of the JVM itself.

Composite UI Application Block: Modules explained

I hope my article will at least add to common understanding on this otherwise so poorly covered theme.

Last week I was crunching module loading for a product of ours requiring dynamically replaceable customer specific (business) module. There are dozens of poor variations on GPSModule Quick Start from original Microsoft Composite UI Application Block package all poorly constructed and even poorer explained. The answers I was looking for I found in this excellent blog of Rich Newman where I found out how exactly does module initialization work.

Generally you do three things

1. Move relevant code into separate project (that will later be compiled into separate assembly);
2. Implement class specializing Microsoft.Practices.CompositeUI.ModuleInit. This class will provide entry point for module loading where you will do initialization, loading and binding events for example;
3. Add to shell application ProfileCatalog.xml that will define what modules to load

Simple huh? Well not really. There are number of dodgy things about loading modules which are not related to module loading itself but rather to how you properly initialize business related code within module. In particular I’d like to touch two themes…

I will use this code as an example:
public class CustomerListModuleInit : ModuleInit { private WorkItem parentWorkItem;
[ServiceDependency]
public WorkItem ParentWorkItem
{
set { parentWorkItem = value; }
}
[EventSubscription(“event://MyApplication/Administration/DisplayCustomerList”)]
public void DisplayCustomerList(object sender, EventArgs e)
{
if (parentWorkItem.RootWorkItem is MyApplicationWorkItem)
{
WorkItem customerListWorkITem = parentWorkItem.RootWorkItem.WorkItems
.AddNew<CustomerListWorkItem>();
List<IPresenter> presenters = customerListWorkItem.GetPresenters();
presenters.ForEach(delegate(IPresenter presenter)
{
customerListWorkItem.Services.Get<IWorkspaceLocatorService>();
IWorkspace wks = locator.FindContainingWorkspace(customerListWorkITem, presenter.View);
if (wks != null) wks.Show(presenter.View);
});
}
}
}

Access to Workitem during module initialization

Because property ParentWorkItem has a [ServiceDependency] attribute it is clear that it will be injected by means of service locator. But the true question is – what ParentWorkItem? The answer is that the WorkItem that is passed is the WorkItem that caused the module to be loaded. We can clearly see it in LoadModules() method of CabApplication that in turn is is called directlly in Run() method of CabApplication.
private void LoadModules() { IModuleLoaderService loader = rootWorkItem.Services.Get<IModuleLoaderService>(true); IModuleEnumerator modEnumerator = rootWorkItem.Services.Get<IModuleEnumerator>(true); if (modEnumerator != null) loader.Load(rootWorkItem, modEnumerator.EnumerateModules()); // ROOTWORKITEM !!! }
As you can see first parameter of Load() method of loader is rootWorkItem in default application module loading algorithm. The reason why property is still called ParentWorkItem and not RootWorkItem is that in some cases if you do hocus pocus you could delegate loading of particular modules to any workitem. So to keep code error free in the class specializing ModuleInit I used parentWorkItem.RootWorkItem notation. This allows me to ensure that I always have reference to the RootWorkItem even in the cases where ParentWorkItem injected by service locator actually is one of child work items.

Event subscriptions

Another interesting thing about module loading is the way one can wire-up events. I mean, in some typical application of yours you might have shell application with some basic (or if you like cross-cutting) stuff in it such as user administration, master layouts, main menu or main page, etc. The actual business logic of your application might be implemented in different modules such as customer management, product catalogue and invoicing. Now, let us assume you would like to move your customer management into a separate module. You do that by performing three steps I described above. After you’ve done this it is time to wire-up your freshly created module into main application. Typically, you’d have for example main menu item for showing customer list. When user clicks on this main menu an event is fired. For example like this:
[EventPublication("event://MyApplication/Administration/DisplayCustomerList")] public event EventHandler<EventArgs> DisplayCustomerList;
Now, you would like to handle this event in your customer management module and upon firing of event, create customer list work item that would do the job of showing application user that list of customers.

It took me some time to understand where to actually place event handler (or in terms of CAB – EventSubscription) that will handle published event, create customer list work item and run it. The way I did it is shown in code of CustomerListModuleInit class above:
public class CustomerListModuleInit : ModuleInit { ... [EventSubscription("event://MyApplication/Administration/DisplayCustomerList")] public void DisplayCustomerList(object sender, EventArgs e) ... }
Why like this? Well, for starters EventSubscription method must be in a class that is instantiated when event is fired. RootWorkItem itself can not have this method since it has no reference to CustomerListWorkItem which is located in a module. To instantiate CustomerListWorkItem upon module loading makes no sense as well, since there is no guarantee that application user will use it at all in current work session. Thus the only logical place that remains is – CustomerListModuleInit that is instantiated unconditionally and per definition should contain code that ensures module wire-up with main application. As soon as I realized this, the implementation became obvious 🙂

FormsAuthentication HashPasswordForStoringInConfigFile in Windows.Forms

Lord Jesus in the heaven! I refuse to reference System.Web in my Windows.Forms application… period! So i went to look for an alternative to HashPasswordForStoringInConfigFile. You need this if for example:

– in your win forms client you want to use same DB you are using in your web application;
– you use secure web-services which should authenticate user of your win forms client against her identity in DB of your web application;

Pretty much common scenarios, ey? I’ve still got sweaty armpits and bleeding nose from all the crunching through bazillions of forums and blogs. With no success.

Now, after two hours of putting bits and bytes together here’s WORKING version of the method in c# that will generate identical hash with that of HashPasswordForStoringInConfigFile from plain password stored in string:

string EncryptPassword(string password)
{
  Byte[] passwordBytes = (new ASCIIEncoding()).GetBytes(password);
  SHA1Managed hashProvider = new SHA1Managed();
  hashProvider.Initialize();
  passwordBytes = hashProvider.ComputeHash(passwordBytes);
  hashProvider.Clear();
  string encryptedPassword = string.Empty;

  foreach (byte b in passwordBytes)
  {
    encryptedPassword += b.ToString("X2");
  }

  return encryptedPassword;
}

Don’t get fooled by simplicity of this method, because it wasn’t simple at all to make it work. Here are pitfalls I falled into:

– (new ASCIIEncoding()).GetBytes(password); <- do NOT use unicode encoding here no matter that your password contains unicode characters cause that’s how FormsAuthentication method does it;
– encryptedPassword += b.ToString(“X2”); <- forget “2” in that format string and resulting hash will be ALMOST identical but not EXACTLY since FormsAuthentication generates with leading zero!
– no, the loop does not the same thing Convert.ToBase64String does, don’t even ask!