Runtime Monitoring

In general, runtime monitoring has 2 main approaches, online and offline monitoring. In online monitoring, a system is monitored during its runtime for execution for any violation(s) of specified properties. A runtime monitor will continuously receive information in terms of system events which are generated by the monitored system. Based on the events received so far, the runtime monitor has to determine if any violations have occurred up till that point in time.

Due to its capability to monitor a system as it executes, online monitoring also provides the ability to make early detections which are crucial for properties which may require immediate action upon violation. These properties include safety and critical properties. On the other hand, this capability also introduces a runtime overhead on the system. A spectrum of runtime monitoring online approaches will be discussed later in this report.

In offline monitoring, a monitor takes as input a complete execution trace from a system which has already finished executing and analyses the complete trace for any possible violation of specified properties. This means that no monitoring is performed during a system’s execution phase.

Offline monitoring, while not as intrusive as online monitoring, suffers from late detection as it is not able to monitor a system’s execution trace during a system’s runtime execution and is only able to perform analysis once the systems finishes its execution and generates a complete execution trace for the monitor. The table below shows a comparison between offline and online monitoring.

	Offline Monitoring	Online Monitoring
Monitoring execution	After monitored system terminates (trace is forwarded to offline monitor)	While executing the system
Relation with system	Independent from the system	Execute alongside the system
Monitoring overhead	Little runtime overhead	Inevitable runtime overhead
Violation detection	Late detection (detection happens after system stops executing)	Early detection
Suitable verification property	Properties that can be verified by globally analyzing the execution trace	Security and critical properties
Suitable to use	Double checking the behavior of systems	Power plant controller

 

Online Runtime Monitoring Spectrum

Cassar et al. (2017) introduced a spectrum for the different online runtime monitoring approaches (in terms of component-based systems) as seen in the figure above. Briefly, component-based systems are made up of entities such as actors, nodes, threads, and etc that can be in a running or blocked state.

The spectrum includes Completely Synchronous (CS) monitoring instrumentation, Synchronous Monitoring with Synchronous Instrumentation (SMSI), Asynchronous Monitoring with Checkpoints (AMC), Asynchronous Monitoring with Synchronous Detections (AMSD) and Completely Asynchronous (CA) monitoring approach.

The spectrum is (as seen in the above figure) is ranked from left to right in order of tightly coupled systems to loosely coupled systems. A tightly coupled system blocks the system every time an event occurs thus allowing for more control over the system at the cost of introducing overhead. On the other end, a loosely coupled system does not perform any system blocking when an event occurs resulting in more efficiency in terms of lesser overhead at the cost of sacrificing the degree of control over the system.

For SMSI, AMC, and AMSD, these approaches fall in between the extreme ends of the spectrum which shows the compromise between the overhead and degree of control over the system as a whole.

• CS monitoring

On the left end of the spectrum, the CS monitoring instrumentation approach refers to the tight coupling between the runtime monitor and the system. When an event occurs in any one particular component of the system, all system components are blocked and remain blocked until the required monitoring checks are performed after which the system components are then unblocked.

This approach allows the runtime property to be immediately detected as all system components are blocked when a violation of the said property detected. Conversely, this tightly coupled behavior is a highly intrusive approach due to the required amount of synchronization between the system, system events and the monitor which results in introducing higher performance overhead on the monitored system.

• SMSI

The SMSI approach also falls close to the left of the spectrum suggesting that it is a tightly coupled approach but with some caveats. Similar to the CS approach, whenever a specified event is executed, blocking occurs. The difference is that instead of blocking the entire system, only the component responsible for the execution of the event is blocked. The components remains blocked until the necessary monitoring steps are completed and are then reset to its running state.

Other components in the system which are not responsible for causing the (monitored) event to occur are allowed to continue executing. However, as all events generated by the system still has to be inspected by the monitor synchronously, it causes this approach to still be highly intrusive (albeit not as intrusive as the CS approach) which in turn introduces performance overhead.

• AMC

AMC provides a balanced approach of allowing the user to specify the components and its generated events which require synchronization with the runtime monitor. A specified component which requires synchronization with the monitor has to temporarily block its execution and await for synchronization while non-specified components may continue with its execution uninterrupted.

This approach allows uses to manually determine the level of control and intrusiveness the monitor has over the system such as specifying checkpoints for critical events, whereby the system has to block its execution and remain blocked until the runtime monitor has completed the necessary checks. This also means that other general non critical events do not cause the system to be blocked.

• AMSD

The AMSD approach lies closer to the right side of the spectrum which means that the runtime monitor and the system is much more loosely coupled than the previous approaches. In particular, this approach only performs synchronization between system events which may cause a violation and the runtime monitor.

For example, if a particular component generates a specified event which may cause a violation, only that component is blocked and other components which do not contribute to the event may continue its execution. Therefore, this approach still allows for runtime detections (for specified system events) while minimizing its intrusiveness to the system.

• CA Monitoring

On the far right end of the spectrum, the CA monitoring approach is completely asynchronous and loosely coupled. This means that the runtime monitor does not perform any form of blocking for all events generated by the system and its components. The handling and checking of events are performed independently by the runtime monitor from the system.

Oftentimes, this approach may be mistaken for a form of offline monitoring due to its loosely coupled and asynchronous characteristics, similar to the offline monitoring approach. The caveat is that offline monitoring requires complete execution traces which are only obtainable once a system finish its execution whereas CA monitoring analyses partial incremental traces during the runtime of a system.

CA monitoring, although extremely non-intrusive, suffers from late detection. Even if violation causing events were to be detected by the monitor, as no form of component blocking is carried out, the violation would not be flagged in time for any sort of mitigating response to be carried out.

 

Runtime Monitoring Approaches implemented in tool

Based on the above discussed approaches, the below table shows a categorization of existing tools and the approach type it belongs to.

Tool	Online					Offline
	CS	SMSI	AMC	AMSD	CA
JEAGLE		●
Java-MOP		●			●	●
LARVA toolkit		●			●	●
detectEr toolkit		●	●	●	●

JEAGLE is a Java implementation of Eagle, which is a runtime verification tool supporting future and past time logics, interval logics, state machines, extended regular expression and real-time and ata constraints. JEAGLE implements SMSI by converting specified runtime monitoring properties in Java code and uses AOP to inline the necessary monitoring code into the monitored application. Inlining is the method of replacing functions call with the actual body of the function itself.

Java-MOP is a client-server Java tool for runtime monitoring and verification using the monitoring oriented programming (MOP) concept. MOP is a paradigm which combines specification and implementation to form a system. During runtime, the specification is checked against the execution trace created by the implementation. Interaction between specification and implementation occurs in the form of an example of providing recovery code for execution during violation of a safety specification. (ref: Chen, Feng & Roşu, Grigore. (2003). Towards Monitoring-Oriented Programming. Electronic Notes in Theoretical Computer Science – ENTCS. 89. 108-127. 10.1016/S1571-0661(04)81045-4. )

Java-MOP supports specification languages such as past and future-time LTL, extended regular expressions and other. It supports offline monitoring, CA approach and SMSI approach by inlining verification checks. Automated code instrumentation is done by Java-MOP to inline the monitoring verification code within the application.

LARVA-toolkit is a Java runtime verification tool for object oriented systems. Using an automata-based specification language called Dynamic Automata with Timers and Events (DATE), it allows users to specify real-time and discrete-time properties in terms of automata. The DATE specification is converted into an offline monitor or an online monitor; an online monitor using the SMSI approach. Verification checks are inserted into the monitored application and its specified components using AOP instrumentation.

The detecEr toolkit is a framework for runtime monitoring in Erlang systems. It converts properties specified in Hennessey Milner logic with recursion into monitors. Its capabilities include allowing for CA, SMSI, AMC and AMSD approaches. Code instrumentation is achieved using an AOP framework for Erlang called eAOP. Th eAOP framework reports events to the runtime monitor as asynchronous messages and is able to force some system components to block while waiting for feedback from monitor thus giving it synchronous capability.

RV-Monitor is a re-implementation of Java-MOP by separating its monitoring from its code instrumentation capabilities and improved its efficiency in terms of monitoring multiple properties at the same time and the merging of index trees used in the original Java-MOP’s indexing algorithm. Other improvements include removing the behavior for multiple weak references to a single object caused by monitoring a single specification by using a global weak reference table (GWRT) that uses only one weak reference for each distinct object.

ReMINDS is a runtime monitoring framework designed for monitoring System of Systems (SoS) and its constituent systems. It uses an event model for analyzing and managing events that occur at runtime. Generated events are sent to the ReMINDS framework during runtime, which is made possible by probes instrumenting the monitored system. At its core, ReMINDS is made up of four layers responsible for specific parts in the entire monitoring process, and another layer which cross cuts into each layers for managing variability.