As per the documentation the default statusMode (if not explicitly specified) is "ACTIVE". i.e. it will return all active process instances (PI). However, it returns "FAILED" instances too. Here is an example : 

REQUEST : 
<soapenv:Envelope xmlns:proc="http://www.tibco.com/bx/2009/management/processManagerType" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <proc:queryProcessInstancesInput>
          <proc:query>SELECT INSTANCE.ID,INSTANCE.STATUS FROM process WHERE INSTANCE.NAME = 'TestFailProcess' ORDER BY INSTANCE.START_DATE ASC</proc:query>
          <proc:pageSize>30</proc:pageSize>
          <proc:attributeMap/>
      </proc:queryProcessInstancesInput>
   </soapenv:Body>
</soapenv:Envelope>

RESPONSE (contains FAILED PIs too):
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <SOAP-ENV:Body>
      <queryProcessInstancesOutput xmlns="http://www.tibco.com/bx/2009/management/processManagerType">
         <processInstances>
            <processInstance>
               <id>pvm:0a126g</id>
               <state>FAILED</state>
            </processInstance>
            <processInstance>
               <id>pvm:0a126h</id>
               <state>ACTIVE</state>
            </processInstance>
            <processInstance>
               <id>pvm:0a126i</id>
               <state>ACTIVE</state>
            </processInstance>
         </processInstances>
         <totalCount>3</totalCount>
         <pageNumber>1</pageNumber>
         <isLastPage>true</isLastPage>
         <pagingID>5308438</pagingID>
      </queryProcessInstancesOutput>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

While if statusMode is specified as ACTIVE then it ONLY returns the ACTIVE PIs. Here is an example : 

REQUEST with statusMode as ACTIVE:

<soapenv:Envelope xmlns:proc="http://www.tibco.com/bx/2009/management/processManagerType" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <proc:queryProcessInstancesInput>
                 <proc:query>SELECT INSTANCE.ID,INSTANCE.STATUS FROM process WHERE INSTANCE.NAME = 'TestFailProcess' ORDER BY INSTANCE.START_DATE ASC</proc:query>
        <proc:statusMode>ACTIVE</proc:statusMode>
         <proc:pageSize>30</proc:pageSize>
         <proc:attributeMap/>
      </proc:queryProcessInstancesInput>
   </soapenv:Body>
</soapenv:Envelope>

RESPONSE (doesn't have FAILED PIs):

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <SOAP-ENV:Body>
      <queryProcessInstancesOutput xmlns="http://www.tibco.com/bx/2009/management/processManagerType">
         <processInstances>
            <processInstance>
               <id>pvm:0a126h</id>
               <state>ACTIVE</state>
            </processInstance>
            <processInstance>
               <id>pvm:0a126i</id>
               <state>ACTIVE</state>
            </processInstance>
         </processInstances>
         <totalCount>2</totalCount>
         <pageNumber>1</pageNumber>
         <isLastPage>true</isLastPage>
         <pagingID>5308439</pagingID>
      </queryProcessInstancesOutput>
   </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

queryProcessInstances request has statusMode as an optional parameter. This article clarifies the default behavior when statusMode is not specified.

ALL

If the "statusMode" is NOT specified it returns ALL the PIs matching the query and which are NOT automatically deleted from the database.

The JVM properties com.tibco.bx.autoDelete and com.tibco.bx.autoDeleteFailedProcesses controls how process instances are automatically deleted from the database. See documentation for details :
https://docs.tibco.com/pub/amx-bpm/4.2.0/doc/html/bpmhelp/GUID-A2880508-D608-4A66-B31D-775A466EE5C6.html

The documentation will reflect this clarification in the next major release of BPM.

queryProcessInstances - https://docs.tibco.com/pub/amx-bpm/4.2.0/doc/html/bpmhelp/GUID-2C1A159D-C442-41AA-A36F-3FD55CBFAF9A.html
BPM JVM Properties reference - https://docs.tibco.com/pub/amx-bpm/4.2.0/doc/html/bpmhelp/GUID-A2880508-D608-4A66-B31D-775A466EE5C6.html
Internal document change number - WRM-11304