health checking

Resin Professional includes a powerful and configurable system for monitoring application server health. The system is intentionally similar to the Resin's "URL Rewrite" rules, based on a configurable set of checks, conditions, and actions. The health checking system runs internal to Resin on a periodic basis. Checks are generally always performed on the local Resin node, and if actions are to be taken, they are performed against the local Resin node as well.

Configuration

health.xml

health: namespace

Health check naming

ee: namespace

Default names

Duplicate names

Default health configuration

Standard health checks

Default actions

<health:HealthSystem>

Health checks

Health status

System checks

<health:ConnectionPoolHealthCheck>

<health:CpuHealthCheck>

<health:HealthSystemHealthCheck>

<health:HeartbeatHealthCheck>

<health:JvmDeadlockHealthCheck>

<health:MemoryPermGenHealthCheck>

<health:MemoryTenuredHealthCheck>

<health:TransactionHealthCheck>

User checks

<health:HttpStatusHealthCheck>

<health:ExprHealthCheck>

Health actions

<health:DumpHeap>

<health:DumpThreads>

<health:ExecCommand>

<health:Restart>

<health:SendMail>

Health conditions

Basic conditions

<health:IfCron>

<health:IfExpr>

<health:IfRechecked>

<health:IfUptime>

Combining conditions

<health:And>

<health:Nand>

<health:Nor>

<health:Not>

<health:Or>

Health check conditions

<health:IfHealthOk>

<health:IfHealthWarning>

<health:IfHealthCritical>

<health:IfHealthFatal>

<health:IfHealthUnknown>

<health:IfMessage>

<health:IfRecovered>

Lifecycle conditions

<health:OnStart>

<health:OnStop>

<health:OnRestart>

Configuration

Health configuration is an extension of the standard Resin configuration file resin.xml. Because Resin uses CanDI to create and update Java objects, each XML tag exactly matches either a Java class or a Java property. As a result, the HealthSystem JavaDoc and the JavaDoc of the various checks, actions, and predicates help to supplement the documentation as much as this reference.

health.xml

Resin version 4.0.16 and later includes health.xml as a standard Resin configuration file alongside resin.xml and app-default.xml.

health.xml is imported into resin.xml as a child of <cluster> or <cluster-default>.

Example: importing health.xml into resin.xml

<resin xmlns="http://caucho.com/ns/resin"
       xmlns:resin="urn:java:com.caucho.resin">
  <cluster-default>  
    ...
    <!--
       - Admin services
      -->
    <resin:DeployService/>
    
    <resin:if test="${resin.professional}">
      <resin:AdminServices/>
    </resin:if>

    <!--
       - Configuration for the health monitoring system
      -->
    <resin:if test="${resin.professional}">
      <resin:import path="${__DIR__}/health.xml" optional="true"/>
    </resin:if>
    ...
  </cluster-default>
</resin>

Example: simple health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthFatal/>
  </health:Restart>
  
</cluster>

health: namespace

health.xml introduces a new XML namespace, health:, defined by xmlns:health="urn:java:com.caucho.health". health: separates health objects from standard resin: elements for clarity and performance. The packages references by health: are:

Health check naming

ee: namespace

The ee: namespace is used for naming objects, for example ee:Named="someName", so that they may be referred to by name later in the configuration. This is sometimes necessary as some health conditions permit referencing a specific health check, as demonstrated in the following example.

Example: referencing named objects

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HttpStatusHealthCheck ee:Named="pingJspCheck">
    <url>http://localhost:8080/test-ping.jsp</url>
  </health:HttpStatusHealthCheck>
  
  <health:Restart>
    <health:IfHealthCritical healthCheck="${pingJspCheck}"/>
    <health:IfRechecked/>
  </health:Restart>
  
</cluster>

In this example, an instance of HttpStatusHealthCheck is named 'pingJspCheck' and referred to by name in the IfHealthCritical criteria using an EL expression. The Restart action will only trigger if the health status is CRITICAL for this specific health check and no others.

Default names

All health checks classes are annotated with @Named, and therefore have a default name that corresponds to their bean name. For example <health:CpuHealthCheck/> can be referred to by ${cpuHealthCheck} without the use of ee:Named.

Example: default health check name

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:CpuHealthCheck>
    <warning-threshold>95</warning-threshold>
  </health:CpuHealthCheck>
  
  <health:DumpThreads>
    <health:IfHealthWarning healthCheck="${cpuHealthCheck}"/>
  </health:DumpThreads>  
  
</cluster>

Duplicate names

Duplicate health check names are not permitted. Resin will fail to startup due to invalid configuration in this case. This can be caused by configuring duplicate checks without using ee:Named, or by configuring more than one check with the same name. The following examples demonstrate both illegal cases.

Example: illegal unnamed duplicate checks

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HttpStatusHealthCheck">
    <url>http://localhost:8080/test1.jsp</url>
  </health:HttpStatusHealthCheck>
  
  <health:HttpStatusHealthCheck">
    <url>http://localhost:8080/test2.jsp</url>
  </health:HttpStatusHealthCheck>
  
</cluster>

In the preceding example, use of ee:Named is required.

Example: illegal duplicate names

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HttpStatusHealthCheck" ee:Named="healthCheck">
    <url>http://localhost:8080/test1.jsp</url>
  </health:HttpStatusHealthCheck>
  
  <health:CpuHealthCheck ee:Named="healthCheck">
    <warning-threshold>95</warning-threshold>
  </health:CpuHealthCheck>
  
</cluster>

In the preceding example, the health check names must be different, regardless of the type of check.

Default health configuration

If for any reason you are missing health.xml, for example you are upgrading from an older version of Resin and don't have the health.xml import in resin.xml, there's no need to worry. Resin creates some checks by default regardless of the presence of health.xml. Furthermore, Resin will detect if no checks are configured and setup default actions and conditions.

Standard health checks

The following health checks are considered critical to standard operation and thus will be created by Resin regardless of the presence of health.xml. If you wish to disabled any of these standard health checks, configure the check in health.xml and set the attribute enabled="false".

<health:JvmDeadlockHealthCheck>
<health:MemoryTenuredHealthCheck>
<health:MemoryPermGenHealthCheck>
<health:CpuHealthCheck>
<health:TransactionHealthCheck>
<health:HealthSystemHealthCheck>

Default actions

If any health checks are configured besides the standard checks mentioned above, Resin will assume the user is using health.xml and will not setup any health actions. If however health.xml is missing or empty, the following basic actions will be created.

  <health:Restart>
    <health:IfHealthFatal/>
  </health:Restart>

<health:HealthSystem>

child of cluster

javadoc <health:HealthSystem>

Configures overall health checking frequency and recheck rules. This element is present in health.xml for clarity, but is not strictly required since it will be created upon startup with default values.

<health:HealthSystem> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	All health checking enabled or disabled	boolean	true
startup-delay	The time after startup before actions are triggered (checks still execute).	Period	15m
period	The time between checks	Period	5m
recheck-period	The time between rechecks	Period	30s
recheck-max	The number of rechecks before returning to the normal checking period	int	10

Example: <health:HealthSystem> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:HealthSystem>
    <enabled>true</enabled>
    <startup-delay>15m</startup-delay>
    <period>5m</period>
    <recheck-period>30s</recheck-period>
    <recheck-max>10</recheck-max>
  </health:HealthSystem>

</cluster>

Health checks

Health checks are status monitors which are executed on a periodic basis by the health system to determine an individual health status. Health checks are designed to be simple; repeatedly evaluating the same data. The health system determines an overall Resin health status by aggregating the results of all the configured health checks.

Health status

Every time a health check executes it produces a HealthStatus and a message. The following is a list of all health statuses and their generally implied meaning.

HealthStatus
NAME	ORDINAL VALUE	DESCRIPTION
UNKNOWN	0	Health check has not yet executed or failed to execute properly; status is inconclusive.
OK	1	Health check reported healthy status. This does not imply recovery.
WARNING	2	Health check reported warning threshold reached or critical is possible.
CRITICAL	3	Health check reported critical status; action should be taken.
FATAL	4	Health check reported fatal; restart expected.

The descriptions above should be understood to be entirely dependent on health action and predicate configuration. For example, a FATAL status does not imply a restart will occur unless health:Restart is configured with the health:IfHealthFatal predicate, as it is in the default health.xml.

System checks

System checks are health checks that can only exist once per JVM due to the nature of the data they sample. Most system checks are pre-configured in the default health.xml.

Note: System checks are singletons. Configuring duplicate system checks with different names will not result in the creation of duplicate system checks. The following is technically valid configuration, but results in configuring the same system check twice.

Example: duplicate system checks

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:CpuHealthCheck ee:Named="cpuCheck1">
    <warning-threshold>95</warning-threshold>
  </health:CpuHealthCheck>
  
  <health:CpuHealthCheck ee:Named="cpuCheck2">
    <warning-threshold>99</warning-threshold>
  </health:CpuHealthCheck>

</cluster>

In this example, warning-threshold will be set to 95 and then overrided to 99.

<health:ConnectionPoolHealthCheck>

child of cluster

javadoc <health:ConnectionPoolHealthCheck>

Monitors the health of Resin database connection pools. See <database> for additional information.

<health:ConnectionPoolHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true

<health:ConnectionPoolHealthCheck> Conditions
HEALTHSTATUS	CONDITION
WARNING	Upon exceeding max-connections.
CRITICAL	Upon exceeding max-overflow-connections.

Example: <health:ConnectionPoolHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:ConnectionPoolHealthCheck/>

</cluster>

<health:CpuHealthCheck>

child of cluster

javadoc <health:CpuHealthCheck>

Monitors CPU usage. On multi-core machines, each CPU is checked individually.

<health:CpuHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true
warning-threshold	CPU usage warning threshold	int (percentage 0-100)	95
critical-threshold	CPU usage critical threshold	int (percentage 0-100)	200 (disabled)

<health:CpuHealthCheck> Conditions
HEALTHSTATUS	CONDITION
WARNING	Upon exceeding warning-threshold on any CPU.
CRITICAL	Upon exceeding critical-threshold on any CPU.

Example: <health:CpuHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:CpuHealthCheck>
    <warning-threshold>95</warning-threshold>
    <critical-threshold>99</critical-threshold>
  </health:CpuHealthCheck>
  
</cluster>

<health:HealthSystemHealthCheck>

child of cluster

javadoc <health:HealthSystemHealthCheck>

Monitors the health system itself by using a separate thread to detect if health checking is frozen or taking too long.

<health:HealthSystemHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true
thread-check-period	The polling frequency of the independent thread.	Period	1m
freeze-timeout	The max time for no health checks to occur to declare the health system frozen	Period	15m

<health:HealthSystemHealthCheck> Conditions
HEALTHSTATUS	CONDITION
FATAL	If health checking has not occurred within freeze-timeout.
FATAL	If health checking has not completed within an acceptable time, calculated using <health:HealthSystem>startup-delay, period, and recheck-period.

Example: <health:HealthSystemHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HealthSystemHealthCheck>
    <thread-check-period>1m</thread-check-period>
    <freeze-timeout>15m</freeze-timeout>
  </health:HealthSystemHealthCheck>
  
</cluster>

<health:HeartbeatHealthCheck>

child of cluster

javadoc <health:HeartbeatHealthCheck>

Monitors for heartbeats from other members of the cluster.

<health:HeartbeatHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true

<health:HeartbeatHealthCheck> Conditions
HEALTHSTATUS	CONDITION
WARNING	If no heartbeat has been received from a know member of the cluster.
WARNING	If a heartbeat has not been received in the last 180 seconds from a known member of the cluster.

Example: <health:HeartbeatHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HeartbeatHealthCheck/>
  
</cluster>

<health:JvmDeadlockHealthCheck>

child of cluster

javadoc <health:JvmDeadlockHealthCheck>

Monitors for deadlocked threads, as determined by the JVM.

<health:JvmDeadlockHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true

<health:JvmDeadlockHealthCheck> Conditions
HEALTHSTATUS	CONDITION
FATAL	If deadlocked threads are detected.

Example: <health:JvmDeadlockHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:JvmDeadlockHealthCheck/>

</cluster>

<health:MemoryPermGenHealthCheck>

child of cluster

javadoc <health:MemoryPermGenHealthCheck>

Monitors the amount of free memory in the JVM PermGen memory pool. Requests a garbage collection if memory falls too low.

Note: This check does not apply to all JVM vendor implementations, and will report UNKNOWN if there is no PermGen pool.

<health:MemoryPermGenHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true
memory-free-min	The critical minimum amount of free memory.	Bytes	1m
free-warning	The warning threshold percentage.	double (percentage 0.0 - 100.0)	0.01
objectName	Explicitly set the MBean name to query for memory stats. When left unset the health check will search available MBeans for the appropriate memory stats MBean.	javax.management.ObjectName	N/A

<health:MemoryPermGenHealthCheck> Conditions
HEALTHSTATUS	CONDITION
UNKNOWN	If there is no PermGen pool (JVM vendor dependent) or the appropriate MBean could not be determined.
WARNING	If free memory is below free-warning percentage after a GC.
CRITICAL	If free memory is below memory-free-min after a GC.

Example: <health:MemoryPermGenHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:MemoryPermGenHealthCheck>
    <memory-free-min>1m</memory-free-min>
    <free-warning>0.01</free-warning>
  </health:MemoryPermGenHealthCheck>

</cluster>

<health:MemoryTenuredHealthCheck>

child of cluster

javadoc <health:MemoryTenuredHealthCheck>

Monitors the amount of free memory in the JVM Tenured memory pool. Requests a garbage collection if memory falls too low. This check will monitor heap memory on JVMs where there is no tenured pool.

Note: memory-free-min will default to the value of <server> <memory-free-min> if present in resin.xml.

<health:MemoryTenuredHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true
memory-free-min	The critical minimum amount of free memory.	Bytes	1m
free-warning	The warning threshold percentage.	double (percentage 0.0 - 100.0)	0.01
objectName	Explicitly set the MBean name to query for memory stats. When left unset the health check will search available MBeans for the appropriate memory stats MBean.	javax.management.ObjectName	N/A

<health:MemoryTenuredHealthCheck> Conditions
HEALTHSTATUS	CONDITION
UNKNOWN	If there is no PermGen pool (JVM vendor dependent) or the appropriate MBean could not be determined.
WARNING	If free memory is below free-warning percentage after a GC.
CRITICAL	If free memory is below memory-free-min after a GC.

Example: <health:MemoryTenuredHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:MemoryTenuredHealthCheck>
    <memory-free-min>1m</memory-free-min>
    <free-warning>0.01</free-warning>
  </health:MemoryTenuredHealthCheck>

</cluster>

<health:TransactionHealthCheck>

child of cluster

javadoc <health:TransactionHealthCheck>

Monitors the Resin transaction manager for commit failures.

<health:TransactionHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true

<health:TransactionHealthCheck> Conditions
HEALTHSTATUS	CONDITION
WARNING	If there were commit failures since the last check.

Example: <health:TransactionHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:TransactionHealthCheck/>

</cluster>

User checks

User checks are not pre-defined in health.xml; an administrator must configure them in health.xml as appropriate for an application. User checks are not singletons; the same check type can be configured in health.xml more than once provided they have different names.

Example: duplicate user checks

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <!-- Http status check 1 for database with email to database admin -->
  
  <health:HttpStatusHealthCheck ee:Named="databaseCheck">
    <url>http://localhost:8080/databaseCheck.jsp</url>
  </health:HttpStatusHealthCheck>
  
  <health:SendMail>
    <to>database_team@yourdomain.com</to>
    <health:IfHealthCritical healthCheck="${databaseCheck}"/>
    <health:IfRechecked/>
  </health:SendMail>
  
  <!-- Http status check 2 for application with email to application admin -->

  <health:HttpStatusHealthCheck" ee:Named="appCheck">
    <url>http://localhost:8080/applicationTest.jsp</url>
  </health:HttpStatusHealthCheck>
  
  <health:SendMail>
    <to>app_team@yourdomain.com</to>
    <health:IfHealthCritical healthCheck="${appCheck}"/>
    <health:IfRechecked/>
  </health:SendMail>

</cluster>

<health:HttpStatusHealthCheck>

child of cluster

javadoc <health:HttpStatusHealthCheck>

Monitors one or more URLs on the current Resin instance by making an HTTP GET request and comparing the returned HTTP status code to a pattern.

<health:HttpStatusHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true
ping-host	The server's ping host (for use where url is a URI)	String	N/A
ping-port	The server's ping port (for use where url is a URI)	int	80
url	A URL or URI to be tested	String	N/A
socket-timeout	The socket connection timeout	Period	10s
regexp	The HTTP status regular expression	java.util.regex.Pattern	200

<health:HttpStatusHealthCheck> Conditions
HEALTHSTATUS	CONDITION
CRITICAL	If the HTTP GET request failed to connect or the status code does not match the regexp.

Example: <health:HttpStatusHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HttpStatusHealthCheck>
    <ping-host>localhost</ping-host>
    <ping-port>8080</ping-port>
    <url>/custom-test-1.jsp</url>
    <url>/custom-test-2.jsp</url>
    <socket-timeout>2s</socket-timeout>
    <regexp>^2|^3</regexp>
  </health:HttpStatusHealthCheck>

</cluster>

In some clustered configurations it may be simpler to use the <server-default> <ping-url> element rather than specifying the host and port in the health check itself. <ping-url> will dynamically construct a URL using the current server host. To enabled this, configure HttpStatusHealthCheck with no <url> elements and add <ping-url>to <server> or <server-default>.

Example: <health:HttpStatusHealthCheck> using ping-url

<!-- resin.xml -->
<resin xmlns="http://caucho.com/ns/resin">

  <cluster id="web-tier">
    <server-default>
      <ping-url>/ping-test.jsp<ping-url>
    </server-default>
    ...
  </cluster>
</resin>

<!-- health.xml -->
<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:HttpStatusHealthCheck ee:Named="serverHttpPingCheck">
    <socket-timeout>5s</socket-timeout>
    <regexp>200</regexp>
  </health:HttpStatusHealthCheck>

</cluster>

<health:ExprHealthCheck>

child of cluster

javadoc <health:ExprHealthCheck>

Evaluates user supplied EL expressions to a boolean.

<health:ExprHealthCheck> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enabled	Check is enabled or disabled	boolean	true
fatal-test	EL expression that will trigger FATAL if it evaluates to true	Expr	N/A
critical-test	EL expression that will trigger CRITICAL if it evaluates to true	Expr	N/A
warning-test	EL expression that will trigger WARNING if it evaluates to true	Expr	N/A

<health:ExprHealthCheck> Conditions
HEALTHSTATUS	CONDITION
FATAL	If any fatal-test expression evaluates to true
CRITICAL	If any critical-test expression evaluates to true
WARNING	If any warning-test expression evaluates to true

Example: <health:ExprHealthCheck> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:ExprHealthCheck>
    <critical-test>${mbean('java.lang:type=Threading').ThreadCount > 100}</critical-test>
  </health:ExprHealthCheck>

</cluster>

Health actions

Health actions perform a task, usually in response to specific conditions, or as remediation for a health check status. Like health checks, health actions are configured in health.xml and executed by the health system on a periodic basis. Health actions are usually accompanied by one or more conditions, or predicates, but this is not required. All actions have the potential to be executed once per period, determined by evaluation of associated conditions. A health action with no conditions will execute once per period.

<health:DumpHeap>

child of cluster

javadoc <health:DumpHeap>

Create a memory heap dump. The heap dump will be logged to the resin log file using com.caucho.health.action.DumpHeap as the class, at the info level.

<health:DumpHeap> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:DumpHeap> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:DumpHeap/>

</cluster>

<health:DumpThreads>

child of cluster

javadoc <health:DumpThreads>

Create a thread dump. The thread dump will be logged to the resin log file using com.caucho.server.admin.AdminThreadDump as the class, at the info level.

<health:DumpThreads> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
only-active	Output only currently active threads (RUNNABLE state)	boolean	false

Example: <health:DumpThreads> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:DumpThreads>
    <only-active>false</only-active>
  </health:DumpThreads>

</cluster>

<health:ExecCommand>

child of cluster

javadoc <health:ExecCommand>

Execute an operating system shell command.

<health:ExecCommand> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
command	The command to execute, relative to dir if set	String	N/A
dir	The directory to execute from	java.io.File	N/A
timeout	Timeout on execution of the command, after which it will be killed if not complete	Period	2s
env	A custom env variable, available to the command.	Name/Value Pair	N/A (System variables are available by default)

Example: <health:ExecCommand> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:ExecCommand>
    <dir>/tmp</dir>
    <command>remediation.sh</command>
    <timeout>2s</timeout>
    <env>
      <name>resin_home</name>
      <value>${resin.home}</value>
    </env>
    <env>
      <name>password</name>
      <value>foo</value>
    </env>
  </health:ExecCommand>

</cluster>

<health:Restart>

child of cluster

javadoc <health:Restart>

Restart Resin. Resin will exit with the reason HEALTH.

<health:Restart> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:Restart> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:Restart/>

</cluster>

<health:SendMail>

child of cluster

javadoc <health:SendMail>

Send an email containing a summary of the current Resin health status.

<health:SendMail> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
to	A "TO:" address; a mail recipient	String	N/A
from	The "FROM:" address	String	resin@localhost
mail	A <mail> resource to use, see example	javax.mail.Session	N/A

Without the mail parameter, the default behaviour for sending mail is to contact an SMTP server at host 127.0.0.1 (the localhost) on port 25. System properties can be used to configure a different SMTP server.

resin.xml - smtp server configuration

  <system-property mail.smtp.host="127.0.0.1"/>
  <system-property mail.smtp.port="25"/>

Example: <health:SendMail> in health.xml using system properties

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:SendMail>
    <to>admin@yourdomain.com</to>
    <to>another_admin@yourdomain.com</to>
    <from>resin@yourdomain.com</from>
  <health:SendMail>

</cluster>

Much more advanced SMTP options can be set by configuring a <mail> resource to use for sending health alerts.

Example: <health:SendMail> in health.xml referencing a <mail> resource

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <mail name="healthMailer">
    <smtp-host>mail.yourdomain.com</smtp-host>
    <smtp-port>25</smtp-port>
    <from>resin@yourdomain.com</from>
  </mail>
  
  <health:SendMail mail="${healthMailer}">
    <to>admin@yourdomain.com</to>
  <health:SendMail>

</cluster>

Health conditions

Health condition, or predicates, qualify an action to execute based on a set of criteria. The action/condition pattern is intentionally similar to Resin's rewrite dispatch/condition pattern, so it should be familiar to some users. Health actions are evaluated every period. Conditions prevent the execution of an action unless all condition evaluate to true. A health action with no conditions will execute once per period. When more than one condition is present for an action, the default combining condion is <health;And>.

Basic conditions

Basic conditions evaluate some general criteria and return true if the condition matches. Basic conditions do not evaluate the status of a health check. Instead they evaluate some general criteria like the time of day.

<health:IfCron>

child of cluster

javadoc <health:IfCron>

Qualify an action to execute if the current time is in an active range configured by cron-style times. This can be used both to schedule regular actions or to prevent restarts or other actions during critical times.

<health:IfCron> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
enable-at	The cron enable times	CronType	N/A
disable-at-at	The cron diable times	CronType	N/A

Example: <health:IfCron> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfCron>
      <enable-at>0 0 * * *</enable-at>
      <disable-at>5 0 * * *</disable-at>
    </health:IfCron>
  </health:Restart> 

</cluster>

<health:IfExpr>

child of cluster

javadoc <health:IfExpr>

Qualifies an action to execute based on the evaluation of an JSP EL expression. Expression can include references to system properties, config properties, and JMX mbean attributes.

<health:IfExpr> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
test	the JSP-EL expression value	Expr	N/A

Example: <health:IfExpr> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfExpr>
      <test>${mbean('java.lang:type=Threading').ThreadCount > 100}</test>
    </health:IfExpr>
  </health:Restart> 

</cluster>

<health:IfRechecked>

child of cluster

javadoc <health:IfRechecked>

Qualifies an action to match only after the required number of rechecks have been performed. Since rechecking is not a health check specific condition, this predicate simply matches when recheck cycle count matches the HealthSystem parameter recheck-max.

<health:IfRechecked> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:IfRechecked> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthFatal/>
    <health:IfRechecked/>
  </health:Restart> 

</cluster>

<health:IfUptime>

child of cluster

javadoc <health:IfUptime>

Qualifies an action to match an amount of time after startup.

<health:IfUptime> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
limit	The time after startup (at least)	Period	N/A

Example: <health:IfUptime> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfUptime limit="12h"/>
  </health:Restart> 

</cluster>

Combining conditions

General condition or health check conditions can be combined or negated using these conditions.

<health:And>

child of cluster

javadoc <health:And>

Qualifies an action to match if all of the child predicates match.

Note: <health:And> is implied and thus not strictly necessary except in when used in conjunction with more complex combining conditions.

<health:And> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:And> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:And>
      <health:IfHealthCritical health-check="${memoryTenuredHealthCheck}"/>
      <health:IfHealthCritical health-check="${memoryPermGenHealthCheck}"/>      
    </health:And>
  </health:Restart> 

</cluster>

<health:Nand>

child of cluster

javadoc <health:Nand>

Qualifies an action to match if all of the child predicates fail.

<health:Nand> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:Nand> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:Nand>
      <health:IfHealthCritical health-check="${memoryTenuredHealthCheck}"/>
      <health:IfHealthCritical health-check="${memoryPermGenHealthCheck}"/>      
    </health:Nand>
  </health:Restart> 

</cluster>

<health:Nor>

child of cluster

javadoc <health:Nor>

Qualifies an action to match if none of the child predicates match.

<health:Nor> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:Nor> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:Nor>
      <health:IfHealthCritical health-check="${memoryTenuredHealthCheck}"/>
      <health:IfHealthCritical health-check="${memoryPermGenHealthCheck}"/>      
    </health:Nor>
  </health:Restart> 

</cluster>

<health:Not>

child of cluster

javadoc <health:Not>

Qualifies an action to match if the child predicate is false.

<health:Not> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:Not> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthCritical health-check="${memoryTenuredHealthCheck}"/>
    <health:Not>
      <health:IfCron>
        <enable-at>0 7 * * *</enable-at>
        <disable-at>0 11 * * *</disable-at>
      </health:IfCron>
    </health:Not>
  </health:Restart> 

</cluster>

<health:Or>

child of cluster

javadoc <health:Or>

Qualifies an action to match if any of the child predicates match.

<health:Or> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:Or> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:Or>
      <health:IfHealthCritical health-check="${memoryTenuredHealthCheck}"/>
      <health:IfHealthCritical health-check="${memoryPermGenHealthCheck}"/>      
    </health:Or>
  </health:Restart> 

</cluster>

Health check conditions

All health check conditions evaluate some aspect of the results of a health check. All optionally accept the parameter health-check, which can reference a specific named health check. In absence of this parameter, overall aggregated Resin health will be used.

<health:IfHealthOk>

child of cluster

javadoc <health:IfHealthOk>

Qualifies an action to match if health status is OK.

<health:IfHealthOk> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)
time	The minimum amount of time since the status started	Period	N/A

Example: <health:IfHealthOk> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:Not>
      <health:IfHealthOk health-check="${memoryTenuredHealthCheck}"/>
    </health:Not>
  </health:Restart> 

</cluster>

<health:IfHealthWarning>

child of cluster

javadoc <health:IfHealthWarning>

Qualifies an action to match if health status is WARNING.

<health:IfHealthWarning> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)
time	The minimum amount of time since the status started	Period	N/A

Example: <health:IfHealthWarning> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthWarning health-check="${memoryTenuredHealthCheck}"/>
  </health:Restart> 

</cluster>

<health:IfHealthCritical>

child of cluster

javadoc <health:IfHealthCritical>

Qualifies an action to match if health status is CRITICAL.

<health:IfHealthCritical> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)
time	The minimum amount of time since the status started	Period	N/A

Example: <health:IfHealthCritical> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthCritical health-check="${memoryTenuredHealthCheck}"/>
  </health:Restart> 

</cluster>

<health:IfHealthFatal>

child of cluster

javadoc <health:IfHealthFatal>

Qualifies an action to match if health status is FATAL.

<health:IfHealthFatal> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)
time	The minimum amount of time since the status started	Period	N/A

Example: <health:IfHealthFatal> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthFatal health-check="${memoryTenuredHealthCheck}"/>
  </health:Restart> 

</cluster>

<health:IfHealthUnknown>

child of cluster

javadoc <health:IfHealthUnknown>

Qualifies an action to match if health status is UNKNOWN.

<health:IfHealthUnknown> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)
time	The minimum amount of time since the status started	Period	N/A

Example: <health:IfHealthUnknown> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthUnknown health-check="${memoryTenuredHealthCheck}"/>
  </health:Restart> 

</cluster>

<health:IfMessage>

child of cluster

javadoc <health:IfMessage>

Qualifies an action to match the health result message to a regular expression.

<health:IfMessage> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)
regexp	The health message match regular expression	java.util.regex.Pattern	N/A

Example: <health:IfMessage> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:Restart>
    <health:IfHealthCritical/>
    <health:IfMessage health-check="${httpStatusCheck}" regexp="Not Found"/>
  </health:Restart> 

</cluster>

<health:IfRecovered>

child of cluster

javadoc <health:IfRecovered>

Qualifies an action to match upon recovery. Recovery is defined as the state change from FATAL, CRITICAL, or WARNING to OK.

<health:IfRecovered> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
health-check	The target health check	HealthCheck	N/A (Overall Resin health will be used if absent)

Example: <health:IfRecovered> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:SendMail>
    <to>admin@yourdomain</to>
    <health:IfRecovered health-check="${cpuHealthCheck}"/>
  </health:SendMail> 

</cluster>

Lifecycle conditions

Lifecycle conditions evaluate the current state of Resin, qualifying actions to execute only during a Resin lifecycle state change.

<health:OnStart>

child of cluster

javadoc <health:OnStart>

Qualifies an action to match only when Resin is starting.

<health:OnStart> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:OnStart> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:SendMail>
    <to>admin@yourdomain.com</to>
    <health:OnStart/>
  </health:SendMail> 

</cluster>

<health:OnStop>

child of cluster

javadoc <health:OnStop>

Qualifies an action to match only when Resin is stopping.

<health:OnStop> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:OnStop> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:SendMail>
    <to>admin@yourdomain.com</to>
    <health:OnStop/>
  </health:SendMail> 

</cluster>

<health:OnRestart>

child of cluster

javadoc <health:OnRestart>

Qualifies an action to match only when Resin is restarted by the watchdog. This generally only occurs during an error condition. OnStart will fire during this event also.

<health:OnRestart> Attributes
ATTRIBUTE	DESCRIPTION	TYPE	DEFAULT
None

Example: <health:OnRestart> in health.xml

<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:SendMail>
    <to>admin@yourdomain.com</to>
    <health:OnRestart/>
  </health:SendMail> 

</cluster>