How to Monitor IBM MQ with SPLUNK and LAMAXU

What is SPLUNK

Splunk (the product) captures, indexes and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards and visualizations.
SPLUNK Web Site

Sample SPLUNK Dashboard

Using SPLUNK

Once you have added the LAMAXU logs data to SPLUNK the MQ metrics can be queried using SPLUNK search syntax.

Graphing the Current Queue Depth

source=”statusData.log” sourcetype=”_json_lamaxuStatus” “identifier.dataSource”=DEMO “attributes.CURDEPTH”=”*” “attributes.QUEUE”!=”AMQ.*” “attributes.QUEUE”!=”SYSTEM.*”| timechart avg(“attributes.CURDEPTH”) by attributes.QUEUE

Graphing Message Volume Statistics

source=”statData.log” sourcetype=_json_lamaxuEvents “identifier.dataType”=STATISTICS  identifier.dataSource”=DEMO ” attributes.QUEUE”=”QM_TEST_REPLY1*” | eval pfields=split(‘attributes.MQIAMO_PUTS’, ” “)  | eval PUTSNP=mvindex(pfields,0)| eval PUTSP=mvindex(pfields,1) |eval TPUTS=PUTSNP+PUTSP| eval gfields=split(‘attributes.MQIAMO_GETS’, ” “) | eval GETSNP=mvindex(gfields,0)| eval GETSP=mvindex(gfields,1) |eval TGETS=GETSNP+GETSP | timechart avg(attributes.MQIAMO_Q_MAX_DEPTH)  as “Hwm” avg(TPUTS) as “Puts” avg(TGETS) as Gets

Graphing Message Time on Queue

source=”statusData.log” “attributes.QTIME”=”*” “attributes.QUEUE”!=”SYSTEM.*” “attributes.QUEUE”!=”AMQ.*” sourcetype=”_json_lamaxuStatus” “identifier.dataSource”=DEMO| timechart avg(“attributes.QTIME”) by attributes.QUEUE

LAMAXU/SPLUNK Source Types

The following source type need to be added to your SPLUNK props.conf file in order to correctly recognise the time-stamp fields in the log file. Alternatively you could manually create the source types using the red highlight values as a guide to defining the time-stamps.

Source type for LAMAXU Status logs

[_json_lamaxuStatus]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = createDate
TIME_FORMAT = %d/%m/%Y %OI:%M:%S %p
TZ = Australia/Sydney
category = Structured
description = JavaScript Object Notation format. For more information, visit
http://json.org/
disabled = false
pulldown_type = true

Source type for LAMAXU Event logs

[_json_lamaxuEvents]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = attributes.MQCAMO_END_DATE,attributes.MQCAMO_END_TIME
TIME_FORMAT = %Y-%m-%d %H.%M.%S
TZ = Australia/Sydney
category = Structured
description = JavaScript Object Notation format. For more information, visit
http://json.org/
disabled = false
pulldown_type = true

Log File Format for statusData.log

Log file attributes are modeled on the IBM MQ runmqsc attribute names.

QMGR_STATUS
{
    "identifier": {
        "dataSource": "DEMO",
        "dataType": "QMGR_STATUS",
        "sui": "DEMO"
    },
    "createDate": "10/10/2015 9:47:45 AM",
    "attributes": {
        "STARTTI": "13.44.07",
        "INSTDESC": "",
        "CONNS": "24",
        "STANDBY": "NOPERMIT",
        "INSTNAME": "Installation1",
        "CHINIT": "RUNNING",
        "STATUS": "RUNNING",
        "STARTDA": "2015-09-30",
        "CMDSERV": "RUNNING",
        "LDAPCONN": "INACTIVE",
        "QMNAME": "DEMO"
    },
    "category": "STATUS"
}

QUEUE_STATUS
{
    "identifier": {
        "dataSource": "DEMO",
        "dataType": "QUEUE_STATUS",
        "sui": "LOCALQNAME"
    },
    "createDate": "10/10/2015 11:47:42 AM",
    "attributes": {
        "IPPROCS": "2",
        "LGETDATE": "",
        "QTIME": "-1 -1",
        "QUEUE": "LOCALQNAME",
        "MONQ": "OFF",
        "OPPROCS": "2",
        "CURDEPTH": "0",
        "LGETTIME": "",
        "MSGAGE": "-1",
        "LPUTDATE": "",
        "LPUTTIME": "",
        "TYPE": "QSTATUS",
        "UNCOM": "NO"
    },
    "category": "STATUS"
}

SUBSCRIPTION_STATUS
{
    "identifier": {
        "dataSource": "DEMO",
        "dataType": "SUBSCRIPTION_STATUS",
        "sui": "TEST.SUBSCRIBER "
    },
    "createDate": "10/10/2015 9:47:45 AM",
    "attributes": {
        "MCASTREL": "-1 -1",
        "DURABLE": "NO",
        "LMSGTIME": "",
        "RESMDATE": "2015-10-10",
        "QMNAME": "DEMO",
        "CURDEPTH": "4",
        "MSGAGE": "0",
        "TOPICSTR": "TEST.TOPIC",
        "SUB": "TEST.SUBSCRIBER",
        "LMSGDATE": "",
        "SUBID": "414d512044454d4f2020202020202020885a0b56049a9c20",
        "NUMMSGS": "0"
    },
    "category": "STATUS"
} 

PUBSUB_STATUS
{
    "identifier": {
        "dataSource": "DEMO",
        "dataType": "PUBSUB_STATUS",
        "sui": "DEMO"
    },
    "createDate": "10/10/2015 9:47:45 AM",
    "attributes": {
        "TPCOUNT": "6",
        "STATUS": "ACTIVE",
        "TYPE": "LOCAL",
        "SUBCOUNT": "3",
        "QMNAME": "DEMO"
    },
    "category": "STATUS"
}

LISTENER_STATUS
{
    "identifier": {
        "dataSource": "DEMO",
        "dataType": "LISTENER_STATUS",
        "sui": "DEMO.1414"
    },
    "createDate": "10/10/2015 9:47:26 AM",
    "attributes": {
        "STARTTI": "00.31.37",
        "PORT": "1414",
        "PID": "38076556",
        "TRPTYPE": "TCP",
        "BACKLOG": "100",
        "IPADDR": "::ffff:10.30.40.74",
        "STATUS": "RUNNING",
        "LISTENER": "DEMO.1414",
        "STARTDA": "2015-10-10",
        "DESCR": "DEMO default Listener"
    },
    "category": "STATUS"