what to look for if the DV server won’t start up or is “hanging”?

what to look for if the DV server won’t start up or is “hanging”?

book

Article ID: KB0082319

calendar_today

Updated On:

Products Versions
TIBCO Data Virtualization 7.0.6

Description

The first thing to look for is if a cache is blocking.
In this case: Database connections could be hanging, and depending on the connection settings on the datasource, that connection might hang forever  or at least longer than the expected time to start the server.

Diagnostics:
Capture a Thread Dump on the Server process id using this command: kill -3 <pid> or use JStack utility for windows servers.
Review CS_Server.log

Analysis:

In the thread dump, look at the “main” thread, which is usually the very last thread in the stack. 

When the server is starting up, it starts all the modules in a specific order, and the CacheModule is one of them. 

The CacheModule tries to clean up any failed caches in the cache_status tables at startup time since this is a good opportunity because there are no caches running at the time

It’s unusual for a connection to hang – usually it either succeeds or fails. But if that does happen, there’s generally something wrong in the target database that CIS can’t fix – it has to be handled at the database end. 

Sometimes Cs_server.log might also reveal some evidence related to Caching. It could also be another problem in a different module, but this is the most common. The stack trace will show you where the startup is getting stuck.

For example, in this case Oracle Connection was hanging preventing the Server to startup. Killing the Oracle session resolved the issue:

 "main" prio=10 tid=0x0000000008624000 nid=0x75b5 runnable [0x00002b926f480000]

   java.lang.Thread.State: RUNNABLE

                at java.net.SocketInputStream.socketRead0(Native Method)

                at java.net.SocketInputStream.read(Unknown Source)

                at java.net.SocketInputStream.read(Unknown Source)

                at oracle.net.ns.Packet.receive(Packet.java:300)

                at oracle.net.ns.DataPacket.receive(DataPacket.java:106)

                at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)

                at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)

                at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)

                at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)

                at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)

                at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)

                at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)

                at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)

                at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)

                at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)

                at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:193)

                at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:1033)

                at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)

                at oracle.jdbc.driver.OracleStatement.executeUpdateInternal(OracleStatement.java:1838)

                at oracle.jdbc.driver.OracleStatement.executeUpdate(OracleStatement.java:1803)

                - locked <0x00002b93e1e4b498> (a oracle.jdbc.driver.T4CConnection)

                at oracle.jdbc.driver.OracleStatementWrapper.executeUpdate(OracleStatementWrapper.java:294)

                at com.compositesw.cdms.ds.oracle.OracleConnection.execute(OracleConnection.java:132)

                at com.compositesw.server.qe.physical.operator.TableDelete.a(SourceFile:132)

                at com.compositesw.server.qe.physical.operator.TableDelete.execute(SourceFile:82)

                at com.compositesw.server.qe.topology.DeletePTree.execute(SourceFile:37)

                at com.compositesw.server.qe.QueryEngine.initializePlan(SourceFile:1060)

                at com.compositesw.server.qe.QueryEngine.a(SourceFile:790)

                at com.compositesw.server.qe.QueryEngine.a(SourceFile:273)

                at com.compositesw.server.qe.QueryEngine.execute(SourceFile:154)

                at com.compositesw.server.qe.QESqlRequest.execute(SourceFile:147)

                at com.compositesw.server.cache.CacheSql.executeSqlTransaction(CacheSql.java:3436)

                at com.compositesw.server.cache.CacheSql$CacheDataSql$TargetSql.clearOldCacheData(CacheSql.java:3119)

                at com.compositesw.server.cache.CacheSql$CacheMetadata$StatusSql.clearJunkEntriesOnCacheManagerLoad(CacheSql.java:2082)

                at com.compositesw.server.cache.DefaultCacheManager.loadAllCacheEntries(DefaultCacheManager.java:2525)

                at com.compositesw.server.cache.DefaultCacheManager.start(DefaultCacheManager.java:265)

                at com.compositesw.server.module.CacheModule.upToAdminMode(CacheModule.java:100)

                at com.compositesw.server.module.CacheModule.moveUpToLevel(CacheModule.java:41)

                at com.compositesw.server.module.CompositeServerModuleManager.moveAllToLevel(CompositeServerModuleManager.java:549)

                at com.compositesw.server.module.CompositeServerModuleManager.moveToLevel(CompositeServerModuleManager.java:331)

                at com.compositesw.server.module.CompositeServerModuleManager.serverStartup(CompositeServerModuleManager.java:252)

                at com.compositesw.server.services.CompositeServer.startServer(CompositeServer.java:1197)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

                at java.lang.reflect.Method.invoke(Unknown Source)

                at com.compositesw.base.boot.ServerBoot.main(ServerBoot.java:86)

Resolution

Killing the hung Oracle Session on the database resolved the issue

Issue/Introduction

what to look for if the DV server won’t start up or is “hanging”?