In-doubt channels

An in-doubt channel is a channel that is in doubt with a remote channel about which messages have been sent and received. Note the distinction between this and a queue manager being in doubt about which messages should be committed to a queue.

You can reduce the opportunity for a channel to be placed in doubt by using the Batch Heartbeat channel parameter (BATCHHB). When a value for this parameter is specified, a sender channel checks that the remote channel is still active before taking any further action. If no response is received the receiver channel is considered to be no longer active. The messages can be rolled-back, and re-routed, and the sender-channel is not put in doubt. This reduces the time when the channel could be placed in doubt to the period between the sender channel verifying that the receiver channel is still active, and verifying that the receiver channel has received the sent messages. See Channel attributes for more information on the batch heartbeat parameter.

In-doubt channel problems are usually resolved automatically. Even when communication is lost, and a channel is placed in doubt with a message batch at the sender whose receipt status is unknown, the situation is resolved when communication is re-established. Sequence number and LUWID records are kept for this purpose. The channel is in doubt until LUWID information has been exchanged, and only one batch of messages can be in doubt for the channel.

You can, when necessary, resynchronize the channel manually. The term manual includes use of operators or programs that contain WebSphere® MQ system management commands. The manual resynchronization process works as follows. This description uses MQSC commands, but you can also use the PCF equivalents.

1. Use the DISPLAY CHSTATUS command to find the last-committed logical unit of work ID (LUWID) for each side of the channel. Do this using the following commands:
   • For the in-doubt side of the channel:

     DISPLAY CHSTATUS(name) SAVED CURLUWID

     You can use the CONNAME and XMITQ parameters to further identify the channel.
   • For the receiving side of the channel:

     DISPLAY CHSTATUS(name) SAVED LSTLUWID

     You can use the CONNAME parameter to further identify the channel.
   The commands are different because only the sending side of the channel can be in doubt. The receiving side is never in doubt.
   On WebSphere MQ for i5/OS®, the DISPLAY CHSTATUS command can be executed from a file using the STRMQMMQSC command or the Work with MQM Channel Status CL command, WRKMQMCHST
2. If the two LUWIDs are the same, the receiving side has committed the unit of work that the sender considers to be in doubt. The sending side can now remove the in-doubt messages from the transmission queue and re-enable it. This is done with the following channel RESOLVE command:

   RESOLVE CHANNEL(name) ACTION(COMMIT)

2537 (09E9) (RC2537): MQRC_CHANNEL_NOT_AVAILABLE

An MQCONN call was issued from a client to connect to a queue manager but the channel is not currently available. 
Common causes of this reason code are:
The channel is currently in stopped state.
The channel has been stopped by a channel exit.
The queue manager has reached its maximum allowable limit for this channel from this client.
The queue manager has reached its maximum allowable limit for this channel.
The queue manager has reached its maximum allowable limit for all channels.

Maximum instances (MAXINST)
Specifies the maximum number of simultaneous instances of a server-connection channel that can be started.

This attribute can be set from zero through 999 999 999. A value of zero indicates that no client connections are allowed on this channel. The default value is 999 999 999.

If the value is reduced below the number of instances of the server-connection channel that are currently running, then the running channels are not affected. However, new instances are not able to start until sufficient existing ones have ceased to run.

This attribute is valid for server-connection channels only.


Maximum instances per client (MAXINSTC)
Specifies the maximum number of simultaneous instances of a server-connection channel that can be started from a single client.

This attribute can be set from zero through 999 999 999. A value of zero indicates that no client connections are allowed on this channel. The default value is 999 999 999.

If the value is reduced below the number of instances of the server-connection channel that are currently running from individual clients, then the running channels are not affected. However, new instances from those clients are not able to start until sufficient existing ones have ceased to run.


This attribute is valid for server-connection channels only.

How to create a linear logging qmgr in MQ

-bash-4.1$ crtmqm -ll -lp 20 -ls 10 -lf 1024 LLQM
There are 81 days left in the trial period for this copy of WebSphere MQ.
WebSphere MQ queue manager created.
Directory '/var/mqm/qmgrs/LLQM' created.
The queue manager is associated with installation 'Installation1'.
Creating or replacing default objects for queue manager 'LLQM'.
Default objects statistics : 74 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.

-ll  : linear logging 
-lp : number of primary logs
-ls : number of secondary logs
-lf : log-file size


How to findout particular qmgr is in Circular/Linear logging ????

Please, Find the difference between queue manager status's using "DIS QMSTATUS ALL" for LLQM and QM1:

dis qmstatus all  (LLQM is a linear logging qmgr)
     3 : dis qmstatus all
AMQ8705: Display Queue Manager Status Details.
   QMNAME(LLQM)                            STATUS(RUNNING)
   CONNS(21)                               CMDSERV(RUNNING)
   CHINIT(RUNNING)                         INSTNAME(Installation1)
   INSTPATH(/opt/mqm)                      INSTDESC( )
   STANDBY(NOPERMIT)                       CURRLOG(S0000000.LOG)
   RECLOG(S0000000.LOG)                    MEDIALOG(S0000000.LOG)
   STARTDA(2014-02-09)                     STARTTI(04.45.40)

dis qmstatus all (QM1 is a circular logging qmgr)
     1 : dis qmstatus all
AMQ8705: Display Queue Manager Status Details.
   QMNAME(QM1)                             STATUS(RUNNING)
   CONNS(21)                               CMDSERV(RUNNING)
   CHINIT(RUNNING)                         INSTNAME(Installation1)
   INSTPATH(/opt/mqm)                      INSTDESC( )
   STANDBY(NOPERMIT)                       STARTDA(2014-02-09)

   STARTTI(04.39.40)                    

How to find the channel initiator is running or not..??    CHINIT(RUNNING)
How to find command server is running or not..??   CMDSERV(RUNNING)

DSPMQVER and DSPMQ Command in Unix Box's

[root@nagi reddy]# dspmqver
Name:        WebSphere MQ
Version:     7.5.0.2
Level:       p750-002-130704.TRIAL
BuildType:   IKAP - (Production)
Platform:    WebSphere MQ for Linux (x86 platform)
Mode:        32-bit
O/S:         Linux 2.6.32-71.el6.i686
InstName:    Installation1
InstDesc:    
Primary:     Yes
InstPath:    /opt/mqm
DataPath:    /var/mqm
MaxCmdLevel: 750

[root@nagi reddy]# dspmq
QMNAME(QM1)                                               STATUS(Running)
QMNAME(QM2)                                               STATUS(Running)
QMNAME(CLUS1)                                             STATUS(Running)
QMNAME(CLUS2)                                             STATUS(Running)
QMNAME(CLUS3)                                             STATUS(Running)
QMNAME(CLUS4)                                             STATUS(Running)

bash-4.1$ crtmqm QMQ1

There are 84 days left in the trial period for this copy of WebSphere MQ.
WebSphere MQ queue manager created.
Directory '/var/mqm/qmgrs/QMQ1' created.
The queue manager is associated with installation 'Installation1'.
Creating or replacing default objects for queue manager 'QMQ1'.
Default objects statistics : 74 created. 0 replaced. 0 failed.
Completing setup.

Setup completed.

How do you tell if there are uncommitted messages on a queue??

Display the curdepth of the queue. If it is non-zero, but you can not browse or get a message, the MQPUT has not been committed.
* The sample browse program is amqsbcg.
iSeries platform
iSeries platform
OR
OR
* The sample browse program is panel WRKMQMMSG 
* The sample browse program is panel WRKMQMMSG 
z/OS platform
where "cpf" is the Command Prefix for the queue manager's subsystem and "qname" is the name of the queue to display.

As an alternative to browsing the queue, you can display the UNCOM attribute using DISPLAY QSTATUS command.
You can display the curdepth and uncommitted (uncom) status of a queue in the following ways:
Distributed platforms
In runmqsc, issue
DIS QS(qname) CURDEPTH UNCOM

DSPMQMQ QUEUE(qname)Curdepth is at the bottom of the last panel.

WRKMQMQ QNAME(*ALL) QTYPE(*ALL) MQMNAME(QMGRNAME)The depth column is on the resulting panel.

From WRKMQM panel, select option 26=MQSC for the appropriate qmgr to access the runmqsc panel. Enter
DIS QS(qname) CURDEPTH UNCOM


1) Using the CSQOREXX panels, do a display of object type QUEUE. On the resulting screen, hit F11 for status. The resulting screen has columns for "Depth" and "Uncommitted messages"
OR
2) Through the command line in the joblog. Issue
/cpf DIS QS(qname) CURDEPTH UNCOM


==> Now display the queue status "type(handle) all" if there are uncommitted messages on the queue to determine which application has the queue open for output:

DISPLAY QSTATUS(TEST.QUEUE) TYPE(HANDLE) ALL
This displays an output with the PID and this:
AMQ8450: Display queue status details.                                
   QUEUE(TEST.QUEUE)    TYPE(HANDLE)               
   APPLDESC( )                             APPLTAG(myput.sol)         
   APPLTYPE(USER)                          BROWSE(NO)                 
   CHANNEL( )                           CONNAME( )                 
   ASTATE(NONE)                            HSTATE(INACTIVE)           
   INPUT(NO)                               INQUIRE(NO)                
   OUTPUT(YES)                             PID(26066)                 
   QMURID(0.6007)                          SET(NO)                    
   TID(1)                                                             
   URID(XA_FORMATID[00000000] XA_GTRID[] XA_BQUAL[]) URTYPE(QMGR)
USERID(userID2)             
                                                                     
The above example shows that the application "myput.sol" (attributr APPLTAG) has the queue open for output. If it were a client, the application name shown in APPLTAG would be "amqrmppa".
                                                                     
==> Using the value for the attribute PID from the previous command (in this case is 26066), then find the connection details by displaying the connections that match the PID.

  DISPLAY CONN(*) ALL WHERE(PID EQ 26066)
AMQ8276: Display Connection details.                                  
   CONN(515C36C720002C01)                                             
   EXTCONN(414D5143514958312020202020202020)                          
   TYPE(CONN)                                                         
   PID(26066)                              TID(1)         
APPLDESC( )                             APPLTAG(myput.sol)
   APPLTYPE(USER)                          ASTATE(NONE)               
   CHANNEL( )                          CONNAME( )                 
   CONNOPTS(MQCNO_SHARED_BINDING)       
USERID(userID2)             
   UOWLOG( )                               UOWSTDA(2013-04-04)        
   UOWSTTI(15.26.48)                   UOWLOGDA( )                
   UOWLOGTI( )                             URTYPE(QMGR)               
   EXTURID(XA_FORMATID[00000000] XA_GTRID[] XA_BQUAL[])               
   QMURID(0.6007)                          UOWSTATE(ACTIVE)           
                                                                     

This shows the start date and time of the Unit of Work (UoW). If it were a client connection, the CHANNEL attribute would contain the channel name, and the CONNAME attribute would show the IP address.

WebSphere MQ for Linux Trace

Follow the instructions listed below to start, stop and format WebSphere MQ for Linux trace. Trace files are written to the directory /var/mqm/trace, so delete or relocate old trace files before beginning a new trace.

Start trace for every WebSphere MQ process:

strmqtrc -e

Or start trace only for one queue manager:

strmqtrc -m MY.QMGR

Or start a high detail trace for one queue manager:

strmqtrc -t all -t detail -m MY.QMGR

Or start a high detail wrapping trace and limit the file size to ~5MB:

strmqtrc -l 5 -t all -t detail -m MY.QMGR

End all tracing:

endmqtrc -a

Format the trace files:

dspmqtrc *.TRC

Or format wrapping trace files:

dspmqtrc *.TRC *.TRS

The trace formatter program converts binary files named AMQppppp.TRC (where ppppp is the process identifier or pid which created the file) into readable files named AMQppppp.FMT.

If you used a wrapping trace, then each time a .TRC reaches the size limit MQ renames it to a .TRS extension and starts a new .TRC file. The trace formatter can convert both files to a single formatted file, but only if you format the .TRC and .TRS files at the same time, as shown above.

Send formatted trace files to IBM support unless binary traces are specifically requested. To save space, compress the formatted trace files with compress, zip, gzip, or bzip2.

http://www-01.ibm.com/support/docview.wss?uid=swg21174924#UNIXlt

MQ Triggering

WebSphere MQ Triggering: WebSphere MQ provides a feature that enables an application or channel to be started automatically when there are messages available to retrieve from a queue.

Trigger messages generated by the queue manager are not persistent. This has the effect of reducing logging (thereby improving performance), and minimizing duplicates during restart, so improving restart time.

Initiation queue:
A local queue on which the queue manager puts trigger messages.

Trigger event: 
An event (such as a message arriving on a queue) that causes a queue manager to create a trigger message on an initiation queue.

Trigger message: 
A message containing information about the program that a trigger monitor is to start.

Trigger monitor: 
A continuously-running application serving one or more initiation queues. When a trigger message arrives on an initiation queue, the trigger monitor retrieves the message. It uses the information in the trigger message to start a process that serves the queue on which a trigger event occurred.

Trigger monitor interface (TMI):
The MQSeries interface to which customer- or vendor-written trigger monitor programs must conform. A part of the MQSeries Framework.

Types of triggering:
1. Channel Triggering 
2. Application Triggering
3. Client Triggering

Before your application can take advantage of triggering, follow the steps below:

1. Create an initiation queue for your application queue. For example:
 DEFINE QLOCAL (initiation.queue) REPLACE +
 LIKE (SYSTEM.DEFAULT.LOCAL.QUEUE) +
 DESCR ('initiation queue description')

2. Associate the initiation queue with the application queue. A queue manager can own more than one initiation queue. You may want some of your application queues to be served by different programs, in which case you could use one initiation queue for each serving program, although you do not have to. Here is an example of how to create an application queue:
 DEFINE QLOCAL (application.queue) REPLACE +
 LIKE (SYSTEM.DEFAULT.LOCAL.QUEUE) +
 DESCR ('appl queue description') +
 INITQ ('initiation.queue') +
 PROCESS ('process.name') +
 TRIGGER +
 TRIGTYPE (FIRST)

3. If you are triggering an application, create a process definition object to contain information relating to the application that is to serve your application queue. For example, to trigger-start a CICS payroll transaction called PAYR:
 DEFINE PROCESS (process.name) +
 REPLACE +
 DESCR ('process description') +
 APPLTYPE ('CICS') +
 APPLICID ('PAYR') +
 USERDATA ('Payroll data')
When the queue manager creates a trigger message, it copies information from the attributes of the process definition object into the trigger message.

4. You need to create a process definition if you want to trigger a channel and you are using a Version 2 MQSeries product. Otherwise, you can create a transmission queue definition and use blanks for the ProcessName attribute. The TrigData attribute can contain the name of the channel to be triggered or it can be left blank. Except on WebSphere MQ for z/OS, if it is left blank, the channel initiator searches the channel definition files until it finds a channel that is associated with the named transmission queue. When the queue manager creates a trigger message, it copies information from the TrigData attribute of the transmission queue definition into the trigger message.

5. If you have created a process definition object to specify properties of the application that is to serve your application queue, associate the process object with your application queue by naming it in the ProcessName attribute of the queue.

6. Start instances of the trigger monitors (or trigger servers in WebSphere MQ for iSeries) that are to serve the initiation queues you have defined.

==>DEFINE QLOCAL(QM2.XMITQ) TRIGGER INITQ(SYSTEM.CHANNEL.INITQ) PROCESS(P1) USAGE (XMITQ)
==>DEFINE PROCESS(P1) USERDATA(QM2.XMITQ)









Troubleshooting:

1. Triggering must be enabled, and of the correct type:
CHECK:
Display the TRIGGER TRIGTYPE and TRIGDPTH attributes of the queue to determine the type of triggering that is enabled:
DIS QL(TRIGGERED_QUEUE) TRIGGER TRIGTYPE TRIGDPTH
Sample output:
AMQ8409: Display Queue details. 
   QUEUE(TRIGGERED_QUEUE)                  TYPE(QLOCAL) 
   TRIGGER                                 TRIGDPTH(1) 
   TRIGTYPE(FIRST)

ACTION:
If NOTRIGGER is seen in the output, then triggering is disabled. Enable it as follows:
ALTER QL(TRIGGERED_QUEUE) TRIGGER
In most cases you can expect the type of triggering (TRIGTYPE) to be FIRST.
A summary of the three types is as follows:
EVERY
Trigger events occur every time a message is delivered to the queue. This occurs even if applications already have the queue open to retrieve messages.
FIRST
Trigger event occur when the depth of the queue transitions from zero to one.
DEPTH
Trigger events occur when the depth of the queue transitions from
(TRIGDPTH-1) to TRIGDPTH.
Triggering is automatically disabled on the queue, and must be re-enabled by the application (using MQSET).

2. No application can already have the queue open to get messages:
This only applies to FIRST or DEPTH triggering.
For EVERY triggering, a trigger event occurs every time a message arrives, regardless of how many applications are already actively retrieving messages from the queue.

CHECK:
Check the IPPROCS attribute in the QSTATUS for the initiation queue:
DIS QS(TRIGGERED_QUEUE) IPPROCS

Sample Output:
AMQ8450: Display queue status details. 
   QUEUE(TRIGGERED_QUEUE) 
   TYPE(QUEUE)                             CURDEPTH(0) 
   IPPROCS(0)

You can find which application has the queue open (IPPROCS>0) as follows:
DIS QS(TRIGGERED_QUEUE) TYPE(HANDLE) APPLTAG PID
Sample Output:
AMQ8450: Display queue status details. 
   QUEUE(TRIGGERED_QUEUE) 
   TYPE(HANDLE) 
   APPLTAG(/home/myuser/bin/mycmd) PID(1234)

ACTION:
For FIRST or DEPTH triggering, if the IPPROCS number is any number greater than zero, then an application has the queue open to retrieve messages and triggering will not occur.
If the application is not the one expected to run against this queue, then it must end before the intended application can be triggered.
The name and PID of the application is shown in the QSTATUS output. If the name contains ‘amqcrsta’ or ‘amqrmppa’, then a remote client application has the queue open (this could be an administrator on another machine).
If an ‘amqrmppa’ process is terminated (kill -9 PID), this will shut down any other channels running within that process.

If the application connected to the queue is the triggered application for this queue, then the application team should be involved to investigate why the application is failing to process messages from the queue.

3. The initiation queue must be valid:
CHECK:
Display the INITQ attribute of the triggered queue:
DIS QL(TRIGGERED_QUEUE) INITQ
Sample Output:
AMQ8409: Display Queue details. 
  QUEUE(TRIGGERED_QUEUE)                    TYPE(QLOCAL) 
  INITQ(SYSTEM.DEFAULT.INITIATION.QUEUE)
Display the initiation queue to check it exists, is not an XMITQ, and is not inhibited for put or get:
DIS QL(SYSTEM.DEFAULT.INITIATION.QUEUE) USAGE PUT GET
Sample Output:
AMQ8409: Display Queue details. 
   QUEUE(SYSTEM.DEFAULT.INITIATION.QUEUE) 
   TYPE(QLOCAL)                      GET(ENABLED) 
   PUT(ENABLED)                      USAGE(NORMAL)
Remember to use single ticks for queues with lower case names:
DIS QL('inititaion.queue') USAGE PUT GET

ACTION:
If the INITQ attribute is blank, or incorrect, then change it to the name of the initiation queue against which the trigger monitor will run.
If the initiation queue is inhibited for put, then enable it as follows:
ALTER QL(SYSTEM.DEFAULT.INITIATION.QUEUE) PUT(ENABLED)

4. A trigger monitor must be active against the initiation queue:
CHECK:
Check the IPPROCS attribute in the QSTATUS for the initiation queue:
DIS QS(SYSTEM.DEFAULT.INITIATION.QUEUE) IPPROCS
Sample Output:
AMQ8450: Display queue status details. 
   QUEUE(SYSTEM.DEFAULT.INITIATION.QUEUE) 
   TYPE(QUEUE)                             CURDEPTH(0) 
   IPPROCS(1)
IPPROCS is the number of applications that have the queue open for input, or browse. This should be exactly 1.

Check the app which has the queue open is the Trigger Monitor:
DIS QS(SYSTEM.DEFAULT.INITIATION.QUEUE) TYPE(HANDLE) APPLTAG
Sample Output:
AMQ8450: Display queue status details. 
   QUEUE(SYSTEM.DEFAULT.INITIATION.QUEUE) 
   TYPE(HANDLE) 
   APPLTAG(/opt/mqm/bin/runmqtrm)
ACTION:
The trigger monitor can be started in the background as follows on UNIX:
nohup runmqtrm -m QMGR_NAME -q SYSTEM.DEFAULT.INITIATION.QUEUE >/dev/null 2>&1 &

5. The process definition must be valid
CHECK:
Display the PROCESS attribute of the triggered queue:
DIS QL(TRIGGERED_QUEUE) PROCESS
Sample Output:
AMQ8409: Display Queue details. 
   QUEUE(TRIGGERED_QUEUE)                  TYPE(QLOCAL) 
   PROCESS(TRIGGERED_PROCESS)
Ensure the process object exists, and validate the command which will be run by the trigger monitor when the trigger event occurs:
DIS PROCESS(TRIGGERED_PROCESS) APPLICID
Sample Output:
AMQ8407: Display Process details. 
   PROCESS(TRIGGERED_PROCESS)     APPLICID(/home/myuser/bin/mycmd)
ACTION:

If the PROCESS attribute is blank, or incorrect, then change it to the name of a valid process object.
If the APPLICID attribute of the process object is incorrect, then alter it to correctly match the command which should be run.
Remember that on UNIX machines commands are case-sensitive, so must be entered in single ticks when defining/altering a process object in MQSC:
DEF PROCESS(TRIGGERED_PROCESS) APPLICID('/home/myusr/bin/mycmd')

6. The trigger monitor must be able to start the application
CHECK:
If the application fails to start an application, it places the trigger message on the dead-letter queue, with a feedback code in the ‘Reason’ field.
The Application Programming Reference manual describes the possible MQFB_* feedback codes

You can use the WebSphere MQ Explorer (or MO71) to view the message:
Connect to the queue manager using the WebSphere MQ Explorer.
Browse the contents of the dead-letter queue for the queue manager.
The WMQ Explorer understands the format of the DLH header that is added to a message when it is placed on the DLQ.
Look at the ‘Reason’ field in the dead-letter header
If it is displayed in numeric form, the following link to the Constants manual should help to translate it to a MQFB_* code:

ACTION:
Investigate and resolve the problem starting the application.

7. The application must connect to the queue after it is started

CHECK:
From a WMQ administrator’s point of view, you can check whether the application is being started by the trigger monitor. Further investigation of the application’s operation would be performed by inspecting the application’s own logging/diagnostic info.

If you need to absolutely confirm that an application is being started by a trigger monitor, then the following process can be performed to restart the trigger monitor in the foreground (so you can see the output on your screen):

1. The best way to stop a trigger monitor is to inhibit the initiation queue for get: ALTER QL(SYSTEM.DEFAULT.INITIATION.QUEUE) GET(DISABLED)
2. You can then immediately un-inhibit get on the queue:                                    ALTER QL(SYSTEM.DEFAULT.INITIATION.QUEUE) GET(ENABLED)
3. You can now run the trigger monitor as a foreground process, so you can monitor the output:runmqtrm -m QMGR_NAME -q SYSTEM.DEFAULT.INITIATION.QUEUE
4. You will now need to open a new terminal (telnet/ssh session)
5. Disable and re-enable triggering on the queue to see the output from the trigger monitor when a trigger message is processed:                                                 ALTER QL(TRIGGERED_QUEUE) NOTRIGGER                                                   ALTER QL(TRIGGERED_QUEUE) TRIGGER