Resolution:
A rule of thumb which works surprising well is to make the number of workers
equal to twice the number of CPUs plus 1 (2*n + 1).  The Dispatcher, Notifier, and Event Action
Daemon are not counted in the number of workers.

Each worker process can handle multiple client connections and will queue up requests if the current API takes a long time coming back from the database for TIBCO InConcert versions at or below 6.2.1.  Each worker process has one connection to the database.  It is important to understand that each worker process take up system resources (Oracle connections, RAM, and CPU switching time).  

The best approach is to look at the type and frequency of API calls from the various type of programs connecting to the InConcert server.  With InConcert 6.0 and above you can segregate that traffic using Secondary Worker processes.  If you have end users working on a task list their traffic is going to be relative light and you can probably have a worker for five logged in users.  Agent programs which are constantly polling for work should probably have their own group of Secondary Workers.  They typically place a heavy load on the workers.

You may also want to segregate admin users of the PC GUI if thousands of jobs are owned by a single user.  When the Process Manager is opened in the PC GUI the entire job set that the user owns is retrieve and this one API can take a long time to complete.

The number of worker processes which work best for you will have to be discovered by actual tests which can take into account your actual use and your environment.
Note the more worker processes you have the longer it is going to take for the InConcert server to start.

Note the following information is applicable to InConcert 6.2.1 and below.  From InConcert 7.1 and above work is no longer queued to individual workers.  Instead each worker pulls work off of a central queue when they are ready for more work.

To get measures of queuing use the apianalyze perl script which when run with a -w command line option over the debug log will give a
report of each worker with three measures of utilization for each.  The first one is
active time divided by the sum of active and idle time.  If this percentage is
very high 80 or 90 percent for most of the workers then the customer needs to
add more workers.  You can get a false negative with this indicator if the
customer has periods of low activity in the debug log.  You might have 90 or 100 per cent
utilization during business hours but then it falls to nothing during the
evening.  

The '% queued' is the percentage of times a worker finished an API and had one
or more APIs queued.  This has the advantage of collapsing idle time since the
queue is only looked at when an API is finished.

'Avg(depth)' reports what the average queue depth was for that worker.  Note
you can get some surprising high values.  I have seen examples where the queue depth averaged 8.  
The the maximum depth of the queue
for each worker is also given.

If only one worker has a high percentage queued adding additional workers will
not do much good.  What you have is one agent program which is extremely busy.  Separating
out that client to a secondary server may provide some performance improvement.

Note, although we have the actual service time for each API in the debug log
we do not have the wait time in the queue.  The API time accounts only for the
time the worker spent working on the API.  You can get an estimate of the
average wait time by taking the average time for an API to complete whether it
failed or succeeded and multiplying that average times the average queue
depth.

What is the trade off in increasing the number of worker processes in Inconcert Server or how many worker processes are enough.

Product: TIBCO InConcert Version: All OS: All --------------------