How can I ensure I have all the libraries in my local Maven repository I need for TIBCO StreamBase® application development while I do not have network connectivity?

Configuration steps

Sections:
  A. BACKGROUND
  B. START ONLINE
  C. RESPONDING TO ERRORS
  D. ERRORS WHEN ONLINE
  E. ERRORS WHEN OFFLINE
  F. STUDIO STARTUP REPOSITORY CHECK
  G. INSTALLING A LIBRARY MANUALLY
 

A. BACKGROUND

It is difficult to anticipate all the libraries a developer may need for their projects. Maven is designed to provide libraries on-demand when first referenced from a project. When adding a library dependency to a project, it is best to have that library in your local machine's Maven repository first so it can be found by simple lookup without reference to external repositories. If a dependency is added without lookup, then Maven first checks the local repository, then the available remote (internet) repositories including Maven Central (https://repo.maven.apache.org/maven2). 

When working offline, it becomes the responsibility of the developer to populate the machine's local repository. TIBCO Streaming includes the 'epdev offline maven --for-studio' command while on-line to populate the local repository. It works by copying the product libraries from the product install, and then copying additional libraries from Maven Central. It performs this task by building over 100 different example Streaming projects which comprehensively represent the dependencies of all the available adapters and use of all platform features. Normally these would be downloaded incrementally as needed by your specific project from the internet, but before working offline these must be acquired first.

The TIBCO Streaming product places over 500 MB (for version 10.6) of proprietary product-specific libraries to the local repository. 
The 'epdev offline maven' process adds an additional 500 MB of public libraries which are not shipped in the product installer. If the '--for-studio' option is included, the command instead adds an additional 1100 MB of public libraries, sources, and javadoc, which is useful if you are writing custom Java operators and adapters.

Note that the libraries in the repository are not used to run TIBCO Streaming, but only for building your applications. Similarly, your deployment systems (test and production) do not need a local Maven repository because the compiled application archive will contain all the libraries identified in its minimal list of dependencies. If you deploy and run directly and not in a container or Cloud environment, your deployment system will also need the TIBCO Streaming runtime installed separately from your application. If deploying into a container or Cloud environment, then the TIBCO Streaming runtime is included in the deployment image automatically.

Here are some items about how Maven and Streaming work together:

• The development system will have a local repository at locations (Linux) /home/user/.m2, or (Windows) \Users\user\.m2
• Maven's preferred internet repository is Maven Central (https://repo.maven.apache.org/maven2).
• Your company may also have other repositories or an internal repository identified in your .m2/settings.xml file.
• When a project has a dependency on a library configured in the project pom.xml file, in order to build that project it needs access to at least one repository which provides that library.
• Local and business repositories are checked first.
• Because over half of the product is based on public libraries delivered over the internet from Maven Central, internet connectivity is mandatory at first use after a new install of TIBCO Streaming.
• If the local Maven repository is fully populated with required libraries, then nothing is needed from Maven Central. This means you may copy an .m2/ folder by any means from one user to another and one system to another and avoid needing internet connectivity from the second account or system.

The 'epdev offline maven' process attempts as much as possible to load the local repository with as many typically required libraries as we can guess you might need for typical development tasks in TIBCO Streaming. However, we cannot anticipate all future needs, so you may find the occasional dependency which will need a separate manual step to load into the repository.

The 'epdev' command was introduced in StreamBase 10.3 and is documented in the product documentation here:
  TIBCO Streaming > StreamBase References > Command Reference > epdev
 

B. START ONLINE

1. Confirm you have good connectivity to Maven Central (https://repo.maven.apache.org/maven2) or your preferred mirror by opening the top URL in a browser.

If you want to use another mirror such as Maven UK Central (http://uk.maven.org/maven2), place in your .m2/ directory this settings.xml file (example):

<settings>  
  <mirrors>
    <mirror>
      <id>UK</id>
      <name>UK Central</name>
      <url>http://uk.maven.org/maven2</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
  </mirrors>
</settings>

2. Open the StreamBase Command Prompt

3. Run:   epdev offline maven --for-studio

This can take twenty minutes or more when run for the first time.

Typical console output includes:

Finding product Maven artifacts to install into C:\Users\gbuhtz\.m2\repository...
Local repository is up to date.
Getting Maven artifacts for future offline operation.
Exercising simple project creation...
Exercising Streaming project creation...
Getting artifacts for simple-eventflow-fragment-0 in str-10.6.1-product-artifacts-repo-offline-projects.zip.
  and their javadoc and sources.....

4. When completed, you may take the system off-line and work without internet connectivity.
 

C. RESPONDING TO ERRORS 

For most users, the correct action when the 'epdev offline maven' command reports errors for specific libraries is to simply ignore the missing dependencies. Most will be as for artifacts for specialty adapters that only a few customers use.

However, initial and repeated errors related to an inability to get foundation maven feature libraries (such as 'wagon') indicate there is a connectivity problem of some kind, such as a miss-configured URL in a custom settings.xml file. There would be hundreds of ERRORS in this case if the operation runs to completion. But if the whole script runs to completion and you see fewer than 20 ERRORs, it's probably OK.
 

D. ERRORS WHEN ONLINE

Some errors may be ignored if they relate to a feature you do not intend to use. The projects used by 'epdev' are expected to be comprehensive and some have dependencies on proprietary 3rd-party libraries which can only be obtained from their manufacturer's proprietary installer or website.

Unfortunately, the 'epdev' command cannot tell the difference between a library which has been made publicly available for download and a proprietary library. This can result in a relatively long list of errors.

To interpret whether an error involves a technology important to your project, look at the list of JAR files reported in the error. 

The 'epdev' errors look like this:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.1.2:go-offline (default-cli) on project 
simple-eventflow-fragment-23: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact 
com.tibco.ep.sb.thirdparty.com.exegy:XCAPI:jar:3.5.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
Maven exit 1 for -T2 org.apache.maven.plugins:maven-dependency-plugin:3.1.2:go-offline -f C:\Users\gbuhtz\AppData\Local\Temp
\offline4458790619253204377\0\simple-eventflow-fragment-23\pom.xml -Dcom.tibco.ep.ep-maven.product=C:\Users\gbuhtz\AppData
\Local\Temp\offline4458790619253204377\product-install
Problem with offline project simple-eventflow-fragment-23 in C:\tibco\sb-cep\10.6\sdk\maven\product-artifact\
str-10.6.1-product-artifacts-repo-offline-projects.zip
Will continue.

For example, this tells you that the com.exegy libraries were unavailable from any available repository. In this case, these are only available from a separate installer and not online.

For any feature or adapter which you recognize as something you expect to use, see the TIBCO Streaming product documentation for the feature and look for how the middleware dependencies can be satisfied.

Note that if the needed library is already in the repository due to some prior action, these errors will not occur, since the library is found.

The typical errors involve libraries from:

com.epam (High-Performance FIX Trading adapters)
com.exegy (Trading adapter)
com.wombat (Wombat Market Data adapter)
javax.resource (IBM Websphere)
com.oracle (Change-Data-Capture)
com.microsoft (Change-Data-Capture)
com.db (Deutsche Bank AutobahnFX)
com.tibco.ftl (TIBCO FTL)

If you are not using a listed feature or technology, then these errors may be ignored. If you are using the feature, satisfy the middleware dependency and re-run the 'epdev' command to confirm the error is resolved. 
 

E. ERRORS WHEN OFFLINE

If you have not used Maven before and are currently offline, you will see this error in the StreamBase Studio console when you attempt a build:

[ERROR] No plugin found for prefix 'wagon' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] 
available from the repositories [local (C:\Users\user\.m2\repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException

For initial use of any Maven capability in Studio or from the command-line, you MUST have access to Maven Central or a well-populated local Maven repository (by using a settings.xml configuration). After a build has completed once successfully (such as when using 'epdev offline maven') then you should rarely need additional libraries from the internet and can work offline.
 

F. STUDIO STARTUP REPOSITORY CHECK

When SB Studio starts up, it performs a local repository check to make sure that the minimal set of TIBCO Streaming libraries are available. If there are no Streaming libraries in the local repository then it runs the equivalent of 'epdev install maven' which takes a few seconds. If the repository is already populated, then this takes no noticeable time.
 

G. INSTALLING A LIBRARY MANUALLY

If you find while working offline that you need a library which is not currently in the local repository, you may add it to the repository manually after loading the library onto your system however you choose. The Maven command to do that is (line-breaks inserted for clarity):

mvn install:install-file 
  -Dfile={path-to-file/filename.jar} 
  -DgroupId={group-id}
  -DartifactId={artifact-id}
  -Dversion={version}
  -Dpackaging=jar 
  -DgeneratePom=true

The pom.xml dependency would then look like:

<dependency>
  <groupId>{group-id}</groupId>
  <artifactId>{artifact-id}</artifactId>
  <version>{version}</version>
</dependency>

Note that most libraries have a manufacturer preferred Group ID, Artifact ID, and Version representation. For consistency across projects it is best to research what these values should be. Maven will let you make up your own and this will work fine for you, but will limit portability when re-using the project.