Notification Logs

The rXg incorporates a comprehensive system and network monitoring mechanism. Every critical aspect of the rXg system health is continuously monitored for potential problems.

Standalone rXgs maintain their own individual health notices logs and engage in independent reporting. Clustered rXg nodes report all health notices back to the cluster controller for centralized logging and reporting. The network devices may be also be monitored by the health notices mechanism through the creation of ping targets.

Notification Events

When a Notification Action is triggered a Notification Event will be logged by the rXg. It will list the ID of the event, the timestamp when it was Created , the Event Type , along with the Related object (Account, LoginSession, etc). The Resolved checkbox allows the operator to resolve any health notices generated from the event, editing the event and checking the resolved box will cure any health notices generated from the event. The Processed box indecates that the any responses to the event were completed.

Capacity Planning

The primary purpose of the rXg health notice mechanism is to allow the operator to perform capacity planning using true data. To meet this need, the rXg traps and reports the following subsystems for near limit and over-utilization conditions:

DHCP pools
DHCP shared networks
Filesystem
Identities
IP addresses
Load average
Login sessions
Memory
Connections Warnings are issued when the subsystem begins to exceed 80% of the maximum capacity. Critical failure messages are issued when the limits are reached.

When any of these health notices are issued by the rXg, the operator should take immediate action to correct the issue. An rXg that is operating in a condition where these health notices are triggered will not perform properly. Perceived and actual end-user performance will eventually degrade until traffic cannot be passed unless corrective action is taken.

Configuration Errors

The rXg administrative console integrates validation at every step of data entry. When individual records are created, updated and/or deleted, the change is validated before being committed to the database. However, the broad spectrum of features contained within the rXg means that it sometimes is possible for the operator to create a configuration that is partially or completely illegal despite the fact that the individual configuration records are valid.

The rXg creates warning health notices when the following configuration errors are encountered.

the data in one or more WAN targets is invalid (e.g., bad DNS entry)
failure to resolve a domain (e.g., a WAN target)
one or more bandwidth queue definitions are overlapping
the sum of all configured real-time guarantees exceeds the configured available bandwidth
configured upload real-time guarantee exceeds capability to deliver
configured download real-time guarantee exceeds capability to deliver

The rXg creates error health notices when the following configuration errors are encountered.

missing physical interface for defined Ethernet interface
email transmission failure
remote content filter list extraction failure
PMS server connection failure
PMS server invalid request
PMS server socket exception
PPPoE DNS configuration error
recurring biller malfunction
session management malfunction

The rXg creates fatal error health notices when the following configuration errors are encountered.

invalid time zone
unable to ping
primary uplink undefined
unable to set default route
failure during PMS demultiplexor initialization

Fatal errors are more severe than errors which are more severe than warnings. However, notification of any of the conditions discussed above requires immediately attention.

System and Network Element Monitoring

The rXg monitors itself for a broad spectrum of error conditions including but not limited to:

DHCP server failure
IP address conflict between the rXg and a neighbor node
IP address conflict between two neighboring nodes
ARP problems pertaining to critical hosts
License failure
Packet filter failure
Ping target (operator specified) not responding When any of these events occur, a critical failure message is issued. ## DHCP Server Failure

The rXg issues a DHCP server failure heath notice when the DHCP server cannot be started. This failure is usually the result of an error in the operator specified configuration found in the Services :: DHCP view.

Due to the broad spectrum of possible DHCP server configurations, an operator may inadvertently configure the DHCP server into an invalid state. It is strongly recommended that at least two administrators review a complete understanding of a new DHCP configuration before any changes are made.

DHCP server changes should made in single steps. A map of all proposed DHCP server changes should be created and each step executed with a several second delay between steps. By following this simple protocol, the operator is enabling the rXg to restart the DHCP server between steps. If the DHCP server restart fails, then the rXg will issue a DHCP server failed health message. By staging DHCP configuration changes and waiting between steps, the cause of a failed DHCP server restart is isolated to the last change.

Additional information may be found by looking at the DHCP server log. The DHCP server log may be accessed by navigating to the Archives :: Logs view and then clicking on the DHCP sub menu option.

IP address conflicts

The IP address and MAC address of each and every device that communicates with the rXg is stored temporarily. This mechanism enables the rXg to detect when two or more nodes with different MAC addresses attempt to communicate with the rXg using the same IP. It also allows the rXg to detect when any node tries to use an IP configured on the rXg to communicate with the rXg itself.

It is extremely important for the rectify the situation that is causing the production of IP address conflict health notices. An IP address conflict between two end-user nodes often indicates that there is a loop in a network adjacent to the rXg. Furthermore, the rXg will not be able to communicate with all nodes on any adjacent network when a conflict between the IP address of the rXg and a node exists.

Packet Filter Failure

The rXg issues a packet filter failure health notice when the rXg cannot load the packet filter rule set that is generated from the policy enforcement configuration. This situation usually occurs due to operator error when configuring the policy to be enforced.

Policy enforcement changes should made in single steps. A map of all proposed policy enforcement changes should be created and each step executed with a several second delay between steps. By following this simple protocol, the operator is enabling the rXg to reload the packet filter rules between steps. If the packet filter rules fail to load, then the rXg will issue a packet filter failure health message. By staging policy enforcement configuration changes and waiting between steps, the cause of a packet filter failure is isolated to the last change.

Ping Target Not Responding

The rXg issues a ping target not responding health notice when an operator specified ping target does not respond to an ICMP ping message. Operator specified ping targets are configured on the Instruments :: Pings view.

The issuance of a ping target not responding health notice generally does not in any way reflect the health of the rXg itself. Instead, it can reflect the status of the node being targeted or the transit network.

When a ping target not responding message is issued, the operator should check on the status of the target node. If the target node is operational, then the failure notice is most likely due to a problem on the transit network. If the ping target is on the WAN (for example, if the ping target are public DNS servers), then the issuance of a ping target not responding message is an indication of a faulty uplink. Similarly, if the ping target is on the LAN, then the issuance of a ping target not responding health notice is an indication that there is a problem with the LAN distribution network.

Uplink Next Hop Router Not Responding

The rXg constantly monitors the status of the next hop router on all designated uplinks by sending ICMP ping requests. When the next hop router does not respond, the rXg assumes that the link is down. In a single uplink deployment scenario, this would mean that the entire network is down. In a multiple uplink scenario, the uplink associated with the router that does not respond will be removed from the pool of viable uplinks. The precise action that will be taken by the rXg is defined by the multiple uplink control records that have been configured.

The issuance of a uplink next hop router not responding health notice is an extremely critical issue that needs to be addressed by the operator immediately. Even if the uplink appears to come back on its own, a thorough investigation is warranted. Uplinks that periodically drop packets are faulty and will cause performance and reliability problems.