TIBCO LogLogic LMI - Error some processes are still using /loglogic/data/* directory when enabling Data Vault

TIBCO LogLogic LMI - Error some processes are still using /loglogic/data/* directory when enabling Data Vault

book

Article ID: KB0077066

calendar_today

Updated On:

Products Versions
TIBCO LogLogic Enterprise Virtual Appliance 6.3.0 and higher

Description

When enabling Data Vault, you may encounter an error similar to the one below:

> system data_vault enable
This will turn on the Data Vault. All the local, and newly archived remote log data would be encrypted.
The data encryption process would take a while, and all services would be suspended until the process is completed.
Do not reboot or power off the machine while the process is still ongoing, or data loss could occur.
Continue? (y/N) y
Save the password to automatically decrypt the data on boot time? (y/N)  N
Please enter the password:
Repeat:

printing logs from background daemon...
Checking if any non-LogLogic process is still using external data volume.

ERROR! Some processes are still using /loglogic/data/* directory.
Please stop the process before enabling the Data Vault feature.
PIDS: 1070,1073,1074
Exit.


The PID numbers you see will be different but these PIDs refer to processes accessing /loglogic/data or one of its sub-directories or files.

Resolution

To resolve this, we first need to determine which processes are causing the problem by searching for the PIDs provided above in a process listing using the ps command:

Logapp root:/var/logapp 2$ ps -ef | grep '1070\|1073\|1074'
root      1070 18099  0 16:22 pts/0    00:00:00 -bash
root      1073  1070  0 16:22 pts/0    00:00:00 tee -a /root/.bash_history
root      1074  1070  0 16:22 pts/0    00:00:00 logger -p 14 -t -bash: HISTORY: PID=18099 UID=0


From this output we can see that our bash prompt is actually at fault.  A closer inspection:

Logapp root:/loglogic/data/vol1 2$ lsof -p 1070
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
bash    1070 root  cwd    DIR  253,3     4096   7012353 /loglogic/data/vol1
bash    1070 root  rtd    DIR    8,1     4096         2 /
bash    1070 root  txt    REG    8,1   964608      3089 /usr/bin/bash
bash    1070 root  mem    REG    8,1    61632      3799 /usr/lib64/libnss_files-2.17.so
bash    1070 root  mem    REG    8,1  2151712      3781 /usr/lib64/libc-2.17.so
bash    1070 root  mem    REG    8,1    19296      3787 /usr/lib64/libdl-2.17.so
bash    1070 root  mem    REG    8,1   174576      4140 /usr/lib64/libtinfo.so.5.9
bash    1070 root  mem    REG    8,1   163408      3774 /usr/lib64/ld-2.17.so
bash    1070 root    0r  FIFO    0,9      0t0 109903968 pipe
bash    1070 root    1u   CHR  136,0      0t0         3 /dev/pts/0
bash    1070 root    2u   CHR  136,0      0t0         3 /dev/pts/0
bash    1070 root  255u   CHR  136,0      0t0         3 /dev/pts/0


indicates the issue. The highlighted line references /loglogic/data/vol1.  When configuring Data Vault in this scenario the user was already logged in using the toor account, so su was used to switch to the root prompt. However, the su command was executed from inside /loglogic/data/vol1, which is what caused the configuration to fail.  So in this instance the fix is to simply change to a directory outside of the /loglogic/data path before using su, or even connect to the appliance using the root account rather than the toor account.  If the cause was not obvious then we are left with a number of options as to how to proceed.  Firstly, we could try manually killing the PIDs, e.g.

kill -9 <PID>

If processes will not die then potentially the next step is to try an application restart, bearing in mind that will stop log collection for the duration and would also cause a failover in an HA environment:

mtask stop; mtask start

Finally, if the PIDs in question are still causing problems then we may need to restart the appliance. This is exceptionally rare but can happen.

Before we can try enabling Data Vault again, we need to clear up the files that the first attempt has created, namely the cryptographic hashes of the password we chose, the boot password file (if you selected Y to this option) and the GoCrypt enable flag:

Logapp root:/loglogic/data/vol1 0$ cd /var/logapp/

Logapp root:/var/logapp 2$ ls -la
total 16
drwx------  2 root root 4096 Aug 28 16:24 .
drwxr-xr-x 24 root root 4096 Aug 28 16:23 ..
-rw-------  1 root root   32 Aug 28 16:23 .pass
-rw-------  1 root root  128 Aug 28 16:23 .pass_hash
-rw-------  1 root root    0 Aug 28 16:24 gocryptfs_enabled

Logapp root:/var/logapp 0$ rm -rf * .*
rm: refusing to remove '.' or '..' directory: skipping '.'
rm: refusing to remove '.' or '..' directory: skipping '..'

Logapp root:/var/logapp 1$ ls -la
total 8
drwx------  2 root root 4096 Aug 28 17:29 .
drwxr-xr-x 24 root root 4096 Aug 28 16:23 ..


You should now be able to enable Data Vault without issue. If you encounter any further problems then contact TIBCO LogLogic support before proceeding.

Issue/Introduction

This article shows how to resolve an error relating to the data directory being in use when enabling Data Vault.