How to delete un-used persistence files when using persistence type "SharedNothing" and why all additional files exist in the first place
Article ID: KB0077340
Updated On:
| Products | Versions |
|---|---|
| TIBCO BusinessEvents Enterprise Edition | 5.x |
Description
TIBCO ActiveSpaces controls the persistence storage (file system) when persistence type is set to “SharedNothing” in project CDD file.
Once the number of required CacheServers are started to achieve quorum (controlled by quorum -> CDD Cluster configuration parameter) space recovery is triggered.
This will recover the data from persistence store for all connected CacheServers. While recovery is in progress a new data file is created as a copy in order to avoid losing data and corrupting the actual store file due to unforeseen issues. The CacheServer reads the contents of the latest data file, adds the data to the memory and creates a new data file (records marked as deleted not added to the new file). Once recovery finished old data file is removed.
Once the number of required CacheServers are started to achieve quorum (controlled by quorum -> CDD Cluster configuration parameter) space recovery is triggered.
This will recover the data from persistence store for all connected CacheServers. While recovery is in progress a new data file is created as a copy in order to avoid losing data and corrupting the actual store file due to unforeseen issues. The CacheServer reads the contents of the latest data file, adds the data to the memory and creates a new data file (records marked as deleted not added to the new file). Once recovery finished old data file is removed.
Issue/Introduction
How to delete un-used persistence files when using persistence type "SharedNothing" and why all additional files exist in the first place
Environment
All Operating Systems
Resolution
TIBCO ActiveSpaces ignores existing data files, if the CacheServer was not part of the cluster on shutdown (different incarnation number).
A new data file is created too, when the CacheServer joins the cluster after recovery has already started. In this scenario data redistribution starts after recovery finished and data stored in a new data file.
In both cases, TIBCO ActiveSpaces will not remove the old data file for each Space of that Seeder (CacheServer) from persistence store. This can cause disk space issues due to all the redundant files left in the store.
We strongly recommend monitoring available disk space in the persistence store and have alerting mechanisms to catch high-disk usage. For instance alert when disk usage reaches 70% or 80%.
You can delete all un-used persistence files as only a single file stores the data per Space in Seeder, all other files are redundant as explained above.
To delete un-used persistence files connect to the the cluster with TIBCO ActiveSpaces tool “as-admin” (cluster connection details, see CDD file or *-as.log files) and execute below command:
execute on peer members cleanup datastore
See:
https://docs.tibco.com/pub/activespaces/2.3.0/doc/html/GUID-AED44327-6509-4647-AF44-6232D780B07C.html
Note:
Recovery will fail, if the number of connected CacheAgents is too small to recover all data (backup count too small).
To avoid that recovery starts before all CacheAgents joined the cluster we recommend to set quorum count in CDD file to the number of available CacheServers. You can define the value as a GlobalVariable to be able to modify the settings if one or more CacheAgents cannot be started (machine unavailable). The configured quorum count is validated on start-up only, recovery starts when number of connected CacheServers reached configured quorum count.
A new data file is created too, when the CacheServer joins the cluster after recovery has already started. In this scenario data redistribution starts after recovery finished and data stored in a new data file.
In both cases, TIBCO ActiveSpaces will not remove the old data file for each Space of that Seeder (CacheServer) from persistence store. This can cause disk space issues due to all the redundant files left in the store.
We strongly recommend monitoring available disk space in the persistence store and have alerting mechanisms to catch high-disk usage. For instance alert when disk usage reaches 70% or 80%.
You can delete all un-used persistence files as only a single file stores the data per Space in Seeder, all other files are redundant as explained above.
To delete un-used persistence files connect to the the cluster with TIBCO ActiveSpaces tool “as-admin” (cluster connection details, see CDD file or *-as.log files) and execute below command:
execute on peer members cleanup datastore
See:
https://docs.tibco.com/pub/activespaces/2.3.0/doc/html/GUID-AED44327-6509-4647-AF44-6232D780B07C.html
Note:
Recovery will fail, if the number of connected CacheAgents is too small to recover all data (backup count too small).
To avoid that recovery starts before all CacheAgents joined the cluster we recommend to set quorum count in CDD file to the number of available CacheServers. You can define the value as a GlobalVariable to be able to modify the settings if one or more CacheAgents cannot be started (machine unavailable). The configured quorum count is validated on start-up only, recovery starts when number of connected CacheServers reached configured quorum count.
Additional Information
SharedNothing, persistence store
Feedback
Yes
No
Powered by
