OPTION FORCE_DISK, Cursor Temp Files

OPTION FORCE_DISK, Cursor Temp Files

book

Article ID: KB0079937

calendar_today

Updated On:

Products Versions
TIBCO Data Virtualization 7.0.6, 7.0.7, 7.0.8, 8.0, ALL

Description

Why TDV creates Cursor Temp Files which will consumes more space in TDV installation directory.

Issue/Introduction

Why TDV creates Cursor Temp Files which will consumes more space in TDV installation directory.

Environment

All

Resolution

Below are the main reasons, suggestion and solution of the issue.

Temp files will be created in CIS for following cases:

1. Cursor failing over to disk        
2. CIS Email stored procedure
3. Dataship feature (currently supported only for Netezza)
4. file cache
5. CIS cluster log

In the above list #1, #2, #3 will create intermediary/tmp files.
Temporary files created will be cleaned up automatically in the following cases:
#1 cursor
#3 dataship

Following are the names of files/directories that will get created.

– Cursor Temp Files –

hashcursors
distinctcursors
pipecursors
resetcursors
seekcursors
sortedcursor
dynamicmap
dynamicmapext
preparedmap
streamtransform/index
streamtransform/value

When a request is served from disk, the memory in use decreases and the query engine creates files under <CIS>/tmp/ subfolders ( for example: <CIS>/tmp/resetcursors folders) . 
These folders contain the in-memory federated join swaps to disk for that request. This will also happen if you use: 

{ OPTION FORCE_DISK } 
or 
{ OPTION FORCE_DISK2} 
in the query.

These tmp tables are created when we "spill to disk" when the results of a query are too large to hold in memory and the results from 2 data sources are still being processed with a hash sort.

If the timestamp of the files in<CIS>/tmp/  sub-folders was from today, it's possible the query was still running, especially if the files  couldn't be deleted - CIS may still have had them open.  

Please make sure no queries are running before you try to remove the files under the <CIS>/tmp/ subfolders. Otherwise  you can stop the CIS  server and  these files  will be deleted automatically during a CIS restart.
=======================================================================================================================================
The <CIS_INSTALL>/tmp/resetcursors directory stores the temporary files used for CIS query that requires extra storage in disk during an execution of a SQL request.

Usually the files will be deleted once a request completes, also if the server crashes while executing requests that generate tmp files in the folder, those files will be cleaned up once CIS server starts up next time.  The reason why there are extra temporary storage needed in disk could be:

-	Some query in the CIS might have used the ‘FORCE_DISK’ option for the clause like SELECT, ORDER BY/GROUPBY, JOIN etc, so that query data will be stored in disk to free up memory for other server operation. 
-	If FORCE_DISK is not specified, the query engine uses memory rather than disk whenever possible to maximize performance.

•	So all in all the problem is caused by some certain SQL request in CIS that processes a large set of data so that disk storage is being used supplementing memory usage. So you will need to find out and isolate any particular requests that cause the high memory and disk space.

•	Next time when you see the disk space full issue, you can run a SELECT against the CIS system table /services/databases/system/SYS_REQUESTS from Studio to find out what requests are running that causes this situation.
=======================================================================================================================================