Administrator won't start after installing TRA 5.11.3, Administrator 5.11.3 on top of TRA 5.11.2, Admin 5.11.2 or lower versions. 
TIBCO Administrator may run into below errors:
Error 1:
<---------
Startup error: java.lang.ClassNotFoundException: org.apache.catalina.startup.Bootstrap
----->

Error 2: 
<-----------
Exception in thread "Thread-40" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
    at java.lang.Class.getConstructor0(Class.java:3075)
    at java.lang.Class.newInstance(Class.java:412)
    at com.tibco.ps.webapp.deployment.ServletEngineWebServerFactory.createServletEngineWithWebServer(ServletEngineWebServerFactory.java:48)
    at com.tibco.administrator.AdministratorPlugInManager.a(AdministratorPlugInManager.java:109)
    at com.tibco.administrator.AdministratorPlugInManager.deployPlugIn(AdministratorPlugInManager.java:76)
    at com.tibco.administrator.AdministratorServlet$7.run(AdministratorServlet.java:744)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: org.apache.tools.ant.BuildException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 9 more
------>

Error 3:
When running tibcoadmin_domainname, it exit quickly and below error can be seen in $TIBCO_ADMIN_DOMAIN_HOME/domainname/tomcat/logs/tomcat.log
<------------
ERROR main org.apache.tomcat.util.digester.Digester - Begin event threw exception
java.lang.ClassNotFoundException: org.apache.catalina.core.JasperListener
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_331]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_331]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_331]
        at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:102) ~[tomcat-util-scan.jar:9.0.62]
        at org.apache.catalina.startup.ListenerCreateRule.begin(ListenerCreateRule.java:68) ~[catalina.jar:9.0.62]
        at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1277) [tomcat-util-scan.jar:9.0.62]
        at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) [xercesImpl.jar:?]
        at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) [xercesImpl.jar:?]
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) [xercesImpl.jar:2.12.2]
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) [xercesImpl.jar:2.12.2]
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) [xercesImpl.jar:2.12.2]
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl.jar:?]
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) [xercesImpl.jar:?]
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) [xercesImpl.jar:?]
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) [xercesImpl.jar:?]
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) [xercesImpl.jar:?]
        at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1535) [tomcat-util-scan.jar:9.0.62]
        at org.apache.catalina.startup.Catalina.parseServerXml(Catalina.java:617) [catalina.jar:9.0.62]
        at org.apache.catalina.startup.Catalina.load(Catalina.java:709) [catalina.jar:9.0.62]
        at org.apache.catalina.startup.Catalina.load(Catalina.java:746) [catalina.jar:9.0.62]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_331]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_331]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_331]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_331]
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305) [catalina.jar:9.0.62]
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475) [catalina.jar:9.0.62]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_331]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_331]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_331]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_331]
        at com.tibco.administrator.startup.AdminWrapper.<init>(AdminWrapper.java:208) [administrator.jar:?]
        at com.tibco.administrator.startup.AdminWrapper.main(AdminWrapper.java:75) [administrator.jar:?]
FATAL main org.apache.catalina.startup.Catalina - Cannot start server, server instance is not configured
------->

(Please note similar ant error also may happen when running deployed applications.)

These error happens if the domain uses non-default TIBCO_TRA_DOMAIN_HOME, TIBCO_ADMIN_DOMAIN_HOME, i.e. the domain files are in folders other than $TIBCO_HOME/tra/domain/domainname/, $TIBCO_HOME/administrator/domain/domainname/. 
It is because TRA 5.11.3 upgrades tomcat 7 to tomcat 9, and ant from 1.6 to 1.10. Old libraries are moved, but Administrator and applications still refer to old libraries. 
(Please note change request TRA-5658 already is created for this.)

This issue is fixed in TRA 5.11.3 HF01 (TRA-5658, see hotfix01 release info). Users may download and apply the latest hotfix for TRA 5.11.3.

Alternatively users can do below to fix the issue without applying the hotfix:
1) change "tomcat/7" to "tomcat/9" in tibcoadmin_domainname.tra file. 

2) change "ant/1.6" to "ant/1.10"  in tibcoadmin_domainname.tra

If it is Windows machine, please also run command prompt as Administrator and run below command to update TIBCO Administrator service:

cd %TIBCO_ADMIN_DOMAIN_HOME%\domainname\bin
tibcoadmin_domainname.exe --update

3) Comment below line in server.xml file under "TIBCO_Home/Administrator/domain/domain_name/tomcat/conf":

<Listener className="org.apache.catalina.core.JasperListener"/>

then restart TIBCO Administrator. 

Please some application may also use tomcat or ant libraries. Please update the application's .tra file too in case similar error is observed. 

Please contact TIBCO support in case you still have questions.