TIBCO Spotfire Statistics Services server fails to start with error "Database may be already in use: "Locked by another process"" in SplusServer.log

TIBCO Spotfire Statistics Services server fails to start with error "Database may be already in use: "Locked by another process"" in SplusServer.log

book

Article ID: KB0076721

calendar_today

Updated On:

Products Versions
Spotfire Statistics Services All Versions

Description

TIBCO Spotfire Statistics Services (TSSS) server with the default H2 database installed on Linux OS may fail to start with the following error in SplusServer.log:
org.h2.jdbc.JdbcSQLException: Database may be already in use: "Locked by another process"

This error can occur on both standalone as well as clustered TSSS deployments. The SplusServer.log will have the following log entries when the service fails to start:
2019-11-13 10:17:36,940 | ERROR |  | ContextLoader | Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.insightful.splusserver.SplusStateInitializer#0' defined in class path resource [applicationContext-job-execution.xml]: Invocation of init method failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Database may be already in use: "Locked by another process". Possible solutions: close all other connection(s); use the server mode; SQL statement:
null/161b3f6d8a44ce64ebc5fb175c35d46142bc679b47e [90020-129])
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1514)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:636)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:938)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Database may be already in use: "Locked by another process". Possible solutions: close all other connection(s); use the server mode; SQL statement:
null/161b3f6d8a44ce64ebc5fb175c35d46142bc679b47e [90020-129])
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:243)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy5.processUnfinishedJobs(Unknown Source)
at com.insightful.splusserver.SplusStateInitializer.init(SplusStateInitializer.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1640)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1581)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
... 25 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Database may be already in use: "Locked by another process". Possible solutions: close all other connection(s); use the server mode; SQL statement:
null/161b3f6d8a44ce64ebc5fb175c35d46142bc679b47e [90020-129])
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1225)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:202)
... 40 more
Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in use: "Locked by another process". Possible solutions: close all other connection(s); use the server mode; SQL statement:
null/161b3f6d8a44ce64ebc5fb175c35d46142bc679b47e [90020-129]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:317)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.get(DbException.java:145)
at org.h2.store.FileLock.getExceptionAlreadyInUse(FileLock.java:433)
at org.h2.store.FileLock.lockFile(FileLock.java:329)
at org.h2.store.FileLock.lock(FileLock.java:129)
at org.h2.engine.Database.open(Database.java:513)
at org.h2.engine.Database.openDatabase(Database.java:200)
at org.h2.engine.Database.<init>(Database.java:195)
at org.h2.engine.Engine.openSession(Engine.java:56)
at org.h2.engine.Engine.openSession(Engine.java:140)
at org.h2.engine.Engine.getSession(Engine.java:120)
at org.h2.engine.Session.createSession(Session.java:119)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:236)
at org.h2.engine.SessionRemote.createSession(SessionRemote.java:214)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:111)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:95)
at org.h2.Driver.connect(Driver.java:58)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1247)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1221)

Issue/Introduction

This article provides the resolution for the "Database may be already in use: "Locked by another process"" error which may occur when starting TIBCO Spotfire Statistics Services (TSSS) server on Linux operating systems.

Environment

OS: Linux

Resolution

This can be caused due to stale java processes holding the H2 database files and having an explicit lock on it. Normally stopping the TSSS service should release the lock on the H2 database files but there may be certain scenarios where the stale java processes still have a hold on those files. 

To resolve, follow these steps:
  1. Stop the TSSS service. For clustered installations, stop all Manager nodes and Worker nodes.
  2. Check if there is still ‘db.lock.db’ file inside the H2 folder. 
    • For standalone installations, check inside <TSSS installation>\h2 folder. 
    • For clustered installations, check inside <Cluster Shared path>\h2 folder.
  3. If the ‘db.lock.db’ file exists in the above folder location, check if there are any stale Java processes which are still holding a lock on the H2 database. Run the following command to find any such processes and terminate them.
ps -ef | grep spserver   
    4.   Delete the ‘db.lock.db’ file from the H2 folder.
    5.   Start the TSSS service again. For clustered installations, start all Manager nodes and worker nodes again.    

 

Additional Information

External : "H2 database error: Database may be already in use: “Locked by another process”"