UNICOS CMWServer User Manual  9.0.2
 All Files Pages
Diagnostics of running CMW server

CMWServer manager has extended debugging/diagnostics features, controlled by single panel. The panel is opened when clicking Diagnostics button on specific manager configuration panel.

In order to interpret correctly available diagnostics, especially alarm conditions and statistics, some minimum understanding of data processing by CMWServer manager is needed.

Internal queues

CMWServer manager is written in C++ language using WinCC OA API. There is one very basic restriction of WinCC OA framework for custom manager development: one thread only may perform communications of this manager with WinCC OA core, first of all - with Event Manager. It was decided that this 'main manager thread' shall only deal with manager communication, it shall not spend his time for CMW clients notifications and similar tasks: the communication with Even Manager shall run uninterrupted at maximum possible speed.

By this reason, CMWServer manager internally contains a number of queues, where one thread can place a task to do and another thread can take that task out of queue and execute it. One of the worst things that could happen while CMWServer manager runs is queue overflow; if this happens the 'task' will be just lost: CMW clients will not be notified about new value of DPE, or SET request from CMW client will 'silently' die inside CMWServer manager.

Let's consider briefly these internal queues.

New DPE values

The CMWServer manager connects to all DPEs corresponding to Acquisition (direction = 'out') and Setting (direction = 'in/out') properties when CMW server starts. New DPE values from Event Manager are processed as following:

new_value_queue.png
Processing of new DPE values
  • callback for new DPE value is processed by main Manager thread, the processing by this thread is really minimal: the new value is added to the queue for further processing
  • another thread takes the value from queue and
    • first updates cached value; this cached value will be used for subsequent GET requests from CMW clients
    • next notifies the CMW clients if any

In case of this queue overflow new DPE value is just lost and CMW clients will not know about changes in running WinCC OA system.

In addition to 'fatal' event (queue overflow), such processing introduces delay between the moment when DPE value was changed and the moment when CMW clients got notified. These delays are shown on pitcure above:

  • t1 = waiting time, the time interval from arrival of new DPE value ro the manager till the moment when the value is taken by queue processing thread for further processing. Clearly, this time depends on number of items currently waiting in the queue
  • t2 = cache update time, the time interval from the moment value is taken out of queue to the moment when the update of cached value is finished. This time interval is expected to be negligible, but it can significantly increase in case of many GET or SUBSCRIBE requests who access the same cached value in parallel with queue processing
  • t3 = client notification time, the time interval from the end of cached value update till all clients are notified. This depends on number of clients connected, speed of transport etc.

Of course, there is also time delay from actual moment of DPE change to the moment when message from Event Manager will reach our CMWServer manager, but this delay is out of our control.

SET request processing

Processing of SET requests from CMW client is illustrated on the following picture:

set_queues.png
Processing of SET request from CMW client

Processing of SET requests uses 2 queues; the logic of SET request processing is the following:

  • SET request is received by CMW thread, it is placed to the SET request queue
  • After time interval t1 the Main Manager thread takes the request out of queue and send dpSet() request to WinCC OA core (to Event Manager).
  • After time interval t2 response from Event Manager is received for dpSet() request (i.e. in fact CMWServer manager processes SET requests of CMW clients according to the logic of dpSetWait() function in CTRL scripts). The response is received by Main Manager thread, this thread just places responce (OK/error) to the SET response queue
  • After time interval t3 another thread takes the response waiting in the queue for sending response to CMW client who sent SET request
  • And finally, after another time interval t4, response was sent to CMW client, and processing of this SET request is completed

RBAC is disabled

Normally, especially in production systems, CMW server shall run with RBAC support. However, this requires RBAC infrastructure to be available for CMW server when it starts. This may be not the case for small development/test projects, where host computer to run WinCC OA project is not configured in the same way as 'standard' computer for production systems. This can lead to the problem: CMW server is trying to start, but it can't access some standard files, required for RBAC. As a result, CMW server does not work.
Typically in such case the following message appears in WinCC OA log:

WCCOACMWServer(8), 2019.02.08 07:49:30.319, IMPL, INFO, 0, , create(): CMW Server created, 1 DPEs, 1 CMW devices, CMW server name: CMWServerName_Test2
WCCOACMWServer(8), 2019.02.08 07:49:30.321, IMPL, INFO, 0, , run(): CMW device server was started successfully: CMWServerName_Test2
WCCOACMWServer8:Exception creating or running RDAServer: Error opening file with public key: /user/rbac/pkey/rba-pub.txt

Note that according to the sequence of messages, CMW server was created and 'succeessfully' started, but shortly after another message comes saying that RBAC initialization failed and server, in fact, was not started.

Because of implementation of DS by BE-CO, the CMW server itself does not know that it failed to start, there are just two consequences which can be observed:

  • Despite the fact that CMW server thinks it was started successfully, CMW clients can't connect to it
  • The above error message appears in WinCC OA log

In order to be able to run CMW server on development computers, which do not fully conform to requirement for production system, a possibility to disable RBAC for this server was added. RBAC can be disabled using special command line argument:

disable_rbac_arg.png
Command line option to disable RBAC
Warning
This option shall not be used in production systems!!!

In order to prevent running CMW server on production with disabled RBAC, the fact that RBAC was disabled is considered to be serious error that is shown as error in different diagnostics panels. In addition, running with CMW server with disabled RBAC always produces system integrity alarm, of course, provided that system integrity was configured for CMWServer manager.

Diagnostics panel

After short introduction of internal data processing, let's consider what diagnostics is available and how is it presented in the panel. The panel looks as following:

diag_stop.png
Diagnostics panel, manager is stopped

The main elements of this panel are:

  1. The field displaying name of CMWServer manager and its current state. Such field is used in many panels, its functionality was described here
  2. The state of 6 alarm conditions for CMW server, this includes:
    • The state of 3 internal queues, described above
    • The state of 'DPE connection lost' alarm condition. This condition can occur, for example, if one (or more) of DPs to which CMWServer manager is connected, were deleted while manager is running
    • The validity of configuration file for running CMW server. Note that modifying the configuration file does not affect already started CMW server, but this can result in problems when trying to restart it
    • The indication if RBAC is disabled for this manager.
    As one can guess, green color means normal operation, red color means error. Note that any error in one of these 4 fields results in alarm for system integrity
  3. A tab with 4 pages, each page displaying different aspects of diagnostics, they are described in more details below

The picture above shows the state of panel when CMWServer manager is not running, that's why everything is gray, even though some colors were promised above. In the following sections we will see how does this panel look with running CMWServer manager and CMW server.

CMW Server state

The first page (State) displays static information about running CMW server. This information is 'static' because it is set only once, when CMW server is started, and can't change until CMW server will stop. Example of such information is shown on the following picture:

diag_state.png
State of running CMW server

In this case very small configuration file is used, containing just 3 DPEs/properties. In fact, the information shown here is just a 'short summary' of configuration file used for CMW server, plus capacities of all internal queues.

A brief reminder on terminology borrowed from Device Server model by BE-CO:

  • Acquisition property is a property with direction 'out'
  • Command property is a property with direction 'in'
  • Setting property is a property with direction 'in/out'

Note that if connection to DPE(s) is lost while server is running - the numbers will be shown differently as illustrated on the following picture:

diag_state_dpe_lost.png
DPE lost for running CMW server

In this example one DP was deleted from project while server is running. First, we see the red 'DPE connect' alarm condition. Next, the number of Acquisition properties, as well as number of DPEs, are shown using different format:

M of N

where:

  • N is total number of corresponding items, read configuration file
  • M is number of items which are still active after DPE was lost

Note that on picture above the 'Config file' alarm indicator is also red, the reason is simple: DP was deleted -> the config file became invalid. Please keep in mind, however, that check for validity of configuration file is only performed if system integrity was configured for this manager.

Debug messages control

CMWServer manager can write to WinCC OA log text messages with short description of important events. Some messages are really important, and they always appear in WinCC OA log (for example, start/stop of server, some fatal errors...).

But there are many other messages, whose appearance in WinCC OA log is optional because they are only needed for debugging, not in normal operation. The 2nd tab of diagnostics panel allows to control the appearance in WinCC OA log for such optional messages:

diag_debug.png
Control of debug messages

The functionality seems evident. A number of checboxes are used to enable/disable producing messages for particular kinds of events. Two texts fields are used to specify filters for device/property names for which debug messages shall be produced. Two types of name filters can be used: wildcards or regular expressions.

Note
Empty filter is considered as 'none', i.e. no any device or property name can match empty filter. As a consequence, some messages will only appear in WinCC OA log if both filter strings are not empty and match at least one CMW device and property name.

Statistics for new DPE value processing

The following statistics can be collected/displayed for processing of new DPE values:

diag_new_value.png
New DPE value processing
  1. Collecting and reporting statistics requires extra processing by CMWServer manager. By this reason, statistics collection shall be explicitly requested by user. Furthermore, statistics collection can only be enabled for limited duration (just in order to be sure statistics will not run forever). After pressing Apply button, statistics collection will be enabled for given number of minutes. In order to stop statistics collection immediately, you can set Enable for to 0 and press Apply
  2. Current state of statistics collection: either 'enabled till <time>', or 'disabled since <time>'
  3. Requested period for reporting collected statistics, can be changed at any time. The value 0 is interpreted as '1 second'
  4. Indication on when statistics report was produced, and what was the time interval to collect this statistics. The time interval displayed is not very precise. If '5 sec' is displayed (like on the picture above) - the interval was defintely not 4 or 6 seconds, but interval could be 5.1 sec. The interval is shown here not for precise measurements!
  5. Number of new DPE values successfully processed, and number of new DPE values lost because of queue overflow. Total amount of 'new DPE value' events from Event Manager equals to sum of these two values
  6. Average and maximum number of events waiting in the queue (see description above). 'Average' here is acalculated as following:
    Imagine we collect statistics for 1 sec, during this time interval we had 1 item in queue for 1 msec, and then for the remaining 999 msec queue was empty. The average is calculated as (1 * 1 + 0 * 999) / (1 + 999) = 0.001.
  7. Average and maximum waiting time in the queue for one event, this corresponds to t1 in description above
  8. Average and maximum time spent for cached value update, this corresponds to t2 in description above
  9. Average and maximum time spent for CMW client(s) notification, this corresponds to t3 in description above. In this example no clients were connected, that's why the time to notify clients is less than the time to update cached value.

Statistics for SET request processing

The following statistics can be collected/displayed for processing of SET requests from CMW clients:

diag_set_stats.png
SET requests processing
  1. Staistics activation, parametrization and display of timing parameters - the same as points 1...4 for previuos statistics
  2. Number of SET request passed through the queue of SET requests, and number of SET requests lost because of this queue overflow (see detailed description here). Total number of SET requests from CMW clients if a sum of these two values
  3. Number of responses to SET requests passed through the queue of SET responses, and number of responses to SET requests lost because of this queue overflow (see detailed description here)
  4. Average and maximum number of SET requests waiting in SET request queue. See comments on average value calcualtion here
  5. Average and maximum time SET requests from CMW clients were waiting in SET requests queue, this corresponds to t1 in description above
  6. Average and maximum time the CMWServer manager was waiting for response from Event Manager to dpSet() requests, this corresponds to t2 in description above
  7. Average and maximum number of responses to SET requests waiting in SET response queue. See comments on average value calcualtion here
  8. Average and maximum time the responses for CMW clients were waiting in SET response queue, this corresponds to t3 in description above
  9. Average and maximum time the CMWServer manager spent for sending responses to CMW clients, this corresponds to t4 in description above