Modelling Message-Oriented-Middleware Brokers using Autoregressive Models for Bottleneck Prediction.

Abstract

Message brokers are the backbone of modern publish subscribe messaging systems. These brokers can degrade or fail for a variety of reasons. This research specifically looks at the detection, prediction and mitigation of bottlenecks in brokers. The message-oriented-middleware framework here uses either a cluster of brokers on a Local Area Network (LAN) or a federation of brokers on a Wide Area Network (WAN) to route messages, facilitate multicasting and ameliorate demand surges and geographically related faults. Sensors have been constructed to monitor brokers and controllers to run the bottleneck detection algorithms. An overlay manager controls broker and topic pairing. Each topic is assigned a primary and secondary broker. When a failure is predicted, the overlay manager routes messages from the failing broker by switching topics to its secondary broker(s). The application for bottleneck forecasting is to allow us to pre-empt a broker failure and hence reroute messages to other brokers to increase resilience and reliability. The key contributions of this research are an abstract model of message-oriented-middleware broker based on the Apache Qpid message broker coupled with the use of analytical autoregressive exogenous (ARX) models that describes the broker behaviour during bottleneck conditions. The Apache Qpid message broker is a message broker that implements the Advanced Message Queuing Protocol (AMQP) for publish-subscribe messaging. ARX models are autoregressive models where the output depends on the previous output as well as external stimuli. These components are integrated to produce a generalised technique for calibrating broker performance and detection of bottlenecks in the broker. This research show how models were initially constructed using a complete range of input data. As bottlenecks occur only when the broker is heavily loaded, input data during idle periods can cause corruption to the model fit. Models were constructed with segmented input data, with each segment covering the range of one peak period. The segmented input allows the modelling of the broker behaviour only when it is experiencing a bottleneck. The result of this is a much-improved fit of the predictive models. The work here is compared against previous work using Markov-chains for creating predictive models. The results of both approaches are compared and reported.