Resolution:
Description:
===========
1). How persister API behaves when the database is down and what happens to the tuples stored in the space during this period?

2). Once the database is brought back up, does persister automatically update the tuples into the database or will it update only those tuples stored after the database is brought up?

Environment:
===========
AS 2.1.*
All operating systems

Resolution:
===========
ActiveSpaces will be unaware in the event the database goes down. ActiveSpaces is also unaware of what kind of system of record is used for persistence implementation.

The callbacks onWrite/onRead will be coming as necessary, and the user is responsible for handling exceptions and any database down scenarios. Users can store tuples in a temporary cache. Once the database is back up the user can start processing again from the temporary cache.


Database (DB) down scenario with Sync persistence:

The persistence API onWrite operation will block the space.put/putAll operation if there is a problem with access to the DB or persisting information. A user needs to check if the problem is due to the DB being down and try to reconnect. Some DBs have idle timeouts which can cause the open connection to be terminated without the user's knowledge. We suggest checking for these options if such a  DB is used. Usually the inside persistence API onOpen method database connection is made and never checked later. If a reconnect goes through then re-execute the DB operation. Otherwise, there is no point to wait and rethrow the exception as RuntimeException. This exception will be propagated to the space.put/putAll operation and it will throw ASException/SpaceResult with the exception in it.

Database down scenario with  Async persistence:

Persistence API  onWrite will not block space.put operation (due to async persistence). If your are testing with DB failover, we don't recommend using async persistence. Since write is not blocked on onWrite and immediately returns success, even inside the onWrite user throws exception. So, async should be used in expense of this possibility. To avoid this, sync persistence needs to be selected.


Below is a code snippet example. While processing PutOp, if there is an exception we throw a runtime exception. This causes space.put/space.putAll operation to return an exception.

The exception will be available if the user is using sync persistence.  If you look at process(PutOp putOp) you will see that there is a try-catch around the whole block.  If anything goes wrong inside the try-catch, the exception is rethrown.  If the exception is handled but not rethrown is the user's choice.

    public ActionResult onWrite (WriteAction writeAction)
    {
        System.out.println("onWrite for " + writeAction.getSpaceName());
        for (WriteOp operation : writeAction.getOps())
        {
            switch (operation.getType())
            {
                case PUT:
                    db.process((PutOp) operation);
                    break;
                case TAKE:
                    db.process((TakeOp) operation);
                    break;
            }
        }
        return ActionResult.create();
    }


    public void process (PutOp putOp)
    {
        try
        {
            // access DB to persist write
        }
        catch (Exception ex)
        {
            throw new RuntimeException(ex);
        }
    }

    public void process (PutOp putOp)
    {
        try
        {
            // access DB to persist write
        }
        catch (Exception ex)
        {
            throw new RuntimeException(ex);
        }
    }


Some hints about put/putAll.

-  Using space.putAll, make sure SpaceResultList is checked if there are any errors.
-  Using space.put, make sure the exception is handled if there is any.

--------------------------------------------------------------------------------------------