How to Automatically Introspect Newly Added Files and Tables from a Data Source Using a Spotfire Data Virtualization Procedure?

book

Article ID: KB0138612

calendar_today

Updated On:

Products Versions
TIBCO Data Virtualization 8.5.5, 8.8.XX

Description

Spotfire Data Virtualization provides the following Resource Administration Web Service operations for resource introspection:

  • introspectResourcesTask – Starts an introspection request for the specified resource. The operation returns a task ID that uniquely identifies the background introspection task.
  • introspectResourcesResult – Retrieves the status of the introspection task by using the task ID returned from introspectResourcesTask. 

These two web service operations can be used together to automate the introspection process. In the example provided in this article, both operations are combined into a single TDV procedure that starts the introspection, waits for completion, and returns the task ID and final status.

The example uses the following resource path:

/shared/d/LocalFileStorage
Replace this path with the desired introspectable resource in your environment. When the procedure is executed, any newly added supported resources (such as tables or CSV files) under the specified path are automatically discovered and added according.

Note:

  • The procedure in this article is provided as an example implementation.
  • Depending on your requirements, you can combine both web service operations into a single procedure or separate them into multiple procedures.
  • After the procedure completes successfully, refresh the Studio repository to view the newly introspected resources.

Environment

All supported environments.

Resolution

Steps to Create the Procedure in TDV Studio

  1. Open TDV Studio.
  2. Navigate to Studio → Folder.
  3. Right-click the folder where you want to create the procedure.
  4. Select New → SQL Script.
  5. Enter the script name:
    IntrospectAndWait
  6. Paste the provided procedure.
PROCEDURE IntrospectAndWait(
OUT o_taskId VARCHAR(255),
OUT o_status VARCHAR(255)
)
BEGIN
DECLARE triggerReq XML;
DECLARE triggerRes XML;
DECLARE triggerFault XML;

DECLARE resultReq XML;
DECLARE resultRes XML;
DECLARE resultFault XML;

-- Build request
SET triggerReq = CAST(
'<resource:introspectResourcesTask
xmlns:resource="http://www.compositesw.com/services/system/admin/resource">

<resource:path>/shared/d/LocalFileStorage</resource:path>

<resource:plan>
<resource:scanForNewResourcesToAutoAdd>true</resource:scanForNewResourcesToAutoAdd>
<resource:updateAllIntrospectedResources>true</resource:updateAllIntrospectedResources>
<resource:failFast>false</resource:failFast>
</resource:plan>

</resource:introspectResourcesTask>'
AS XML);

-- Start introspection
CALL /services/webservices/system/admin/resource/operations/introspectResourcesTask(
triggerReq,
triggerRes,
triggerFault
);

-- Get Task ID
SET o_taskId = TRIM(
CAST(
EXTRACTVALUE(triggerRes,'//*[local-name()="taskId"]/text()')
AS VARCHAR(255)
)
);

-- Build result request
SET resultReq = CAST(
'<resource:introspectResourcesResult
xmlns:resource="http://www.compositesw.com/services/system/admin/resource"
xmlns:common="http://www.compositesw.com/services/system/util/common">

<common:taskId>' || o_taskId || '</common:taskId>
<common:block>true</common:block>
<resource:detail>FULL</resource:detail>

</resource:introspectResourcesResult>'
AS XML);

-- Wait for completion
CALL /services/webservices/system/admin/resource/operations/introspectResourcesResult(
resultReq,
resultRes,
resultFault
);

-- Get final status
SET o_status = TRIM(
CAST(
EXTRACTVALUE(
resultRes,
'//*[local-name()="status"]/*[local-name()="status"]/text()'
)
AS VARCHAR(255)
)
);

END

Issue/Introduction

This article demonstrates how to use the introspectResourcesTask and introspectResourcesResult Web Service operations to automate resource introspection in Spotfire Data Virtualization (SDV).

Attachments

MasterWorkflow.car get_app