The maven build command "mvn compile" hangs when trying to build the EAR for BE project. 
The pom.xml file was created successfully and external libraries added to the dependencies section correctly.
"%BE_HOME%\studio\bin\Studio-tools -core buildEar ... " creates the ear file with parameters displayed on console output.


Console Output (command hang, required to close/kill it manually):
D:\work\BE\testproject>mvn compile
[INFO] Scanning for projects...
...
[INFO] --- be-maven-plugin:5.6.1:becompile (default-becompile-1) @ TestProject ---
Build project EAR file TestProject-1.0.0.ear in progress..
org.eclipse.m2e.logback.configuration: The org.eclipse.m2e.logback.configuration bundle was activated before the state location was initialized.  Will retry after the state location is initialized.
org.eclipse.m2e.logback.configuration: Logback config file: D:\tibco_BE56test\be\5.6\studio\workspace\.metadata\.plugins\org.eclipse.m2e.logback.configuration\logback.1.12.0.20190529-1916.xml
org.eclipse.m2e.logback.configuration: Initializing logback

****************************************************************************
TIBCO BusinessEvents 5.6.1.106 (2020-04-02)
Using arguments :-core buildEar -p D:\work\BE\testproject -o c:/temp/testProject-1.0.0.ear -x -cp C:\Users\testuser\.m2\repository\com\tibco\be\jms-2.0\1.0.0\jms-2.0-1.0.0.jar;C:\Users\testuser\.m2\repository\com\tibco\be\tibjms\1.0.0\tibjms-1.0.0.jar;
Copyright(c) 2004-2019 TIBCO Software Inc. All rights reserved.

... 
 

This is a product issue (CR: BE-28984, raised for BE 5.6.1HF5 and BE 6.0.0) on Windows systems. 

As a workaround two options are available:

1. Build the ear on a non-Windows environment.

2. Move in-built custom function to a project library and import the project library to current project with below steps.
2.1.
Create a new project and add the custom function (javaSrc folder) and the EMS libraries to the Java build path

2.2.
Remove your project from workspace and delete below folders in file explorer:
<projectHome>/JavaBin
<projectHome>/JavaSrc

2.3.
Import your project into Studio workspace

2.4.
Remove references to external libraries in Java build path

2.5.
Add the new project library with in-built custom functions (Project -> Properties -> BuildPath -> Tab Project Libraries)

2.6.
Rebuild ear in Studio (test and confirm project has no build errors)

2.7.
Add the project library and the new external library to maven repository

e.g.:
set BE_HOME=D:\tibco_BE56test\be\5.6
set PROJLIB=D:\work\BE\projlibs\projLibCustfunc.projlib
set EMS_LIBDIR=D:\tibco_BE56test\ems\8.5\lib

cd %BE_HOME%\maven\bin
install-projlib -p "%PROJLIB%\ProjCustFunctions.projlib" -g com.tibco.be -a ProjCustFunctions_V1.0 -v 1.0.0
mvn install:install-file -Dfile="%EMS_LIBDIR%\jms-2.0.jar" -DgroupId=com.tibco.be -DartifactId=jms-2.0 -Dversion=1.0.0 -Dpackaging=jar
mvn install:install-file -Dfile="%EMS_LIBDIR%\tibjms.jar" -DgroupId=com.tibco.be -DartifactId=tibjms -Dversion=1.0.0 -Dpackaging=jar


2.8.) 
Update dependencies in pom.xml file

...
<dependencies>
    <dependency>
      <groupId>com.tibco.be</groupId>
      <artifactId>ProjCustFunctions_V1.0</artifactId>
      <version>1.0.0</version>
      <type>projlib</type>
    </dependency>
    <dependency>
      <groupId>com.tibco.be</groupId>
      <artifactId>jms-2.0</artifactId>
      <version>1.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.tibco.be</groupId>
      <artifactId>tibjms</artifactId>
      <version>1.0.0</version>
    </dependency>
  </dependencies>
...  
 
2.9.)
Build ear (mvn compile)