book
Article ID: KB0072485
calendar_today
Updated On:
Description
The error in the engine log is:
Object not found on remote node C.X with transaction 'SyncThreadRunnable for module:
default '[read committed:136419912617833795:540:6163221:8884139048490421, tid 17732]
for object 3305907272:1331177160:3353210168581527:25, operation default.operations.worker.aggregations::destroy
Although the application keeps running, what is the cause and impact of this error?
Issue/Introduction
The error may be benign, may indicate a mis-deployment of different versions of the application, or be a side-effect of an earlier runtime error.
Resolution
The error, "Object not found on remote node", refers to some element in DTM Shared Memory (Transactional Memory) which should have been common across all nodes, but for one or more nodes is not present.
The error means that node "C" cannot preserve state for operator "default.operations.worker.aggregations", so if the other nodes are stopped this operator's state will not be preserved in the cluster. For example, if this is part of a three node cluster with nodes "A", "B", and "C", then currently nodes "A" and "B" are preserving this operator's state and operator "C" is not.
This situation can happen if:
a) one node experiences a previous error that either prevents this operator from initializing, or stops this operator later invalidating its object in Transactional Memory
b) the nodes have a slightly different version of the application archive where the set of operators which use Transactional Memory are different.
c) an entirely different application was started in the "X" cluster which does not match the application started in the other nodes of the cluster.
Confirm the same application archive was installed into all nodes by checking the "epadmin install node" command options used to create each node. If a different application archive was specified, and this is part of a rolling upgrade of the application where some nodes have the previous version of the application and some have the new version, then this error may be expected and can be ignored.
Since the error message identifies the operator by name, "default.operations.worker.aggregations" that is the best information to use to troubleshoot the issue. Check all the logs in the node "logs/" folder for errors during startup or related to this operator or its parent modules. See if the long "id" numbers in the error are repeated elsewhere in the logs. If there is a prior error, correct that and see if this error is eliminated from future sessions.