Resolution:
Description:
===========
Message being lost after EMS FT pair failover when using the GFS2 file system as shared state when using an older Linux kernel.

Environment:
===========
EMS server FT pair is configured with the shared state on a GFS2 file system on an old Linux kernel.

Symptom:
========

This message loss happened in April 2010 when testing the EMS failover feature using the GFS2 file system as a shared state on an old Linux Kernel.  The EMS client sending application using the FT connection is sending PERSISTENT messages to a queue in the primary EMS server. After stopping the EMS server by simulating the system kernel panic which causes the machine to go down, the EMS standby server running on another machine becomes the primary server and starts. After examing the pending messages being sent across to the queue during failover, some messages are lost. The EMS client sending application doesn't receive any exception during the failover.

Cause:
======

The reason for the dataloss is due to disk caching.  GFS2 caches in RAM and commits to disk just like any other file system under Linux. The changes aren't written directly to disk as the performance impact would render the file system useless.

There is a "sync" mount option for the GFS2 file systems that can be used to disable GFS2 disk caching. This option is undocumented in RHEL but is available via mount.gfs2. Though it is not mentioned in the man page it does work. You would simply pass sync as a mount option in your mount directive either in cluster.conf, fstab, or on the command line.

Example:

Command line:

$ mount /dev/mapper/gfs-file-system /mountpoint -o sync

fstab:
/dev/mapper/gfs-file-system       /mountpoint    gfs2     sync     0 0

cluster.conf:
&ltclusterfs device="/dev/mapper/gfs-file-system" mountpoint="/mountpoint" options="sync"/>

You can use the gfs2_tool to force GFS2 to sync to disk. With this feature turned on, the message loss issue will be addressed but performance will be poor.

The Red Hat kernel and the GFS2 filesystem have gone through a number of enhancements since then. The message loss issue incurred in April 2010 can no longer be reproduced in current versions of Red Hat Enterprise Linux. It is not necessary to use the "sync" mount option in order to avoid message loss. Red Hat has confirmed that this is not the best approach.  Instead they suggest that where synchronous disk writes are required (failsafe mode) the files to be written be opened with the O_SYNC or O_DSYNC flag.  EMS opens the sync.db (or other store files with mode=sync) using the O_DSYNC option, so this will ensure that disk writes are synchronous.  Using the "sync" mount option has the additional often unintended consequence of causing all disk writes to any file in the filesystem with the "sync" mount option enabled to be synchronous. This means that if the async.db or meta.db resides in that same filesystem writes to those files will be synchronous rather than asynchronous.
So we don't need to use "sync" mount option" for GFS2  on recent version of RHEL 5 or 6.

TIBCO PSG has worked with Red Hat and gone through our set of tests with them trying to break GFS2.  GFS2 has passed all of our tests without any issues. GFS2 on a recent version of RHEL 5 or 6 should be an acceptable and supportable configuration that meets all EMS shared state criteria.

Message loss when using the GFS2 file system as a shared state when using an older Linux kernel.