This article provides the essential steps needed to run a TIBCO ModelOps 1.2 server on Amazon Elastic Kubernetes Service (EKS), using Azure Active Directory (AD) for authentication (oauth2). The commands shown below are assumed to be run from a Linux terminal. 

This procedure may not be appropriate for production use, but should help to quickly spin up a working ModelOps server that SSO users may access for testing.

Install the following prerequisites on your Linux machine:

• ModelOps 1.2 installer files (typically found under /opt/tibco/modelops/1.2)
• Amazon Web Services Command Line Interface (aws cli)
• Azure CLI
• eksctl
• kubectl
• helm
• tkn
• jq

Several actions (detailed in the steps below) will need to be accomplished by your AWS administrator. Please have them review this procedure before beginning.

This installation procedure assumes that the person tasked with installing ModelOps (herein referred to as "the SSO user" or "you") has SSO access to a shared AWS team account, which has the following AWS SSO roles available:

• A role with create-access to the AWS Route 53 service - This role is named 'use'. This role is needed to create the DNS hosted zone for the ModelOps web server. 
• A role with read-access to the AWS IAM (Identity & Access Management) service - This role is named 'view'. Although this role cannot perform the ModelOps installation, the SSO user may use this read-access role to temporarily assume a seperate custom role. That custom role ('TIBCOAppsInstaller') has a different set of AWS policies attached, which allows the SSO user to then install ModelOps. For details, refer to steps 3 and 4.
• A role with administrative-access to the AWS IAM service - This role is named 'manage'. Although this role cannot perform the installation of ModelOps, it is still needed to create the custom role discussed in step 3.

1.) Create a DNS hosted zone for the ModelOps web server. Use the AWS web Management Console, which may be accessed via your organization's AWS SSO start URL (e.g. https://companycloud.awsapps.com/start). In this example, the 'SSO start URL' page presents the AWS account number to the SSO user. This account has 3 available SSO roles: 'view', 'use' and 'manage'. Select 'Management Console' for the 'use' role. Then navigate to the Route 53 service and select Hosted Zones > Create hosted zone. Provide the following configuration values:

• Domain name: eksapps.companycloud.com
• Description: Sub-domain for the EKS ingress that manages external access to cloud applications such as ModelOps.
• Type: Public hosted zone

After the hosted zone is created, its name servers will be displayed under Route 53 > Hosted zones > eksapps.companycloud.com > Hosted zone details. For example:

• ns-1234.awsdns-56.org
• ns-789.awsdns-10.com
• ns-1112.awsdns-13.net
• ns-1415.awsdns-16.co.uk

In this case, the hosted zone eksapps.companycloud.com is a sub-domain of the parent domain companycloud.com. This means you need to request that your hosted zone's name servers be added to the parent. Enter this request to your AWS administrator. Do this first, as it may take some time for your request to be fulfilled and for the DNS propagation to complete.

2.) Configure the AWS Command Line Interface (AWS CLI) to interact with your Amazon Web Services. Your AWS administrator should provide you with the values for the 'SSO start URL' and 'SSO Region' as shown below. Like you saw in the AWS Management Console previously, one account with three roles is available to the SSO user. Select the 'manage' role. 

Before choosing a value for the 'CLI default client Region', first login to the AWS Management Console (by navigating to the SSO start URL in your web browser), and review each region's VPC service. The default per-region limit for VPCs and Elastic IPs is 5. So if the 'CLI default client Region' you selected already has 4 or 5 active VPCs and Elastic IPs, select a different region, or request that the AWS Account Administrator make additional VPCs and Elastic IPs available in your desired region.
 

aws sso logout
aws configure sso

SSO start URL [None]: https://companycloud.awsapps.com/start
SSO Region [None]: us-west-2
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.us-west-2.amazonaws.com/

Then enter the code:

DTFW-GZZF
The only AWS account available to you is: 012345678910
Using the account ID 012345678910
There are 3 roles available to you.
  *view
  *use 
  *manage <---
Using the role name "manage"
CLI default client Region [None]: us-west-1
CLI default output format [None]: json
CLI profile name [manage-012345678910]: manage

To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile manage

Confirm the Amazon Resource Name (ARN) for your role:
 

aws sts get-caller-identity --profile manage

{
    "UserId": "AROAQGCIOINQYSW7ZGODO:ssouser@company.com",
    "Account": "012345678910",
    "Arn": "arn:aws:sts::012345678910:assumed-role/AWSReservedSSO_manage_95e0eb61e8ieb6f8/ssouser@company.com"
}

Here, the SSO user 'ssouser@company.com' has assumed the 'manage' role (whose full name is 'AWSReservedSSO_manage_95e0eb61e8ieb6f8').

Note: The selected 'CLI default client Region' is different from the 'SSO Region'. However, the 'CLI default client Region' is irrelevant for this installation, as the example commands shown below explicitly specify the region for the EKS cluster (using the --region option).

Configure the 'view' profile as well. The 'use' role is not needed for the remainder of the installation. After running 'aws configure sso' for your 'view' role, confirm its ARN:
 

aws sts get-caller-identity --profile view

{
    "UserId": "AROAQGCIOINQYSW7ZGODO:ssouser@company.com",
    "Account": "012345678910",
    "Arn": "arn:aws:sts::012345678910:assumed-role/AWSReservedSSO_view_58e0eb8mh1ienu39/ssouser@company.com"
}

3.) Create a custom Role named 'TIBCOAppsInstaller', and attach the following AWS-managed policies:

• PowerUserAccess
• AdministratorAccess

Note: If the EKS cluster, nodegroup and OIDC (OpenID Connect) provider have already been created, AdministratorAccess is not required and you can skip steps 5 and 6.

Also, add a trust relationship with your 'view' role, which allows you to assume the custom role in step 4:
 

aws sso login --profile manage
aws sso login --profile view
export AWS_SSO_READ_ACCESS_ROLE_ARN=$(aws sts get-caller-identity \
  --profile view | jq '.Arn' | tr -d '"')
echo $AWS_SSO_READ_ACCESS_ROLE_ARN
rm -rf trust-relationship.json
echo -e \
    "{ \
        \"Version\": \"2012-10-17\", \
        \"Statement\": \
        {\
            \"Effect\": \"Allow\", \
            \"Principal\": { \
                \"AWS\": \"$AWS_SSO_READ_ACCESS_ROLE_ARN\" \
            },\
            \"Action\": \"sts:AssumeRole\" \
        } \
    }" | tr -d " \t" > trust-relationship.json
export AWS_CUSTOM_ROLE_NAME=TIBCOAppsInstaller
aws iam create-role \
    --role-name $AWS_CUSTOM_ROLE_NAME \
    --assume-role-policy-document file://trust-relationship.json \
    --profile manage
aws iam attach-role-policy \
    --role-name $AWS_CUSTOM_ROLE_NAME \
    --policy-arn arn:aws:iam::aws:policy/AdministratorAccess \
    --profile manage
aws iam attach-role-policy \
    --role-name $AWS_CUSTOM_ROLE_NAME \
    --policy-arn arn:aws:iam::aws:policy/PowerUserAccess \
    --profile manage

4.) Assume the custom 'TIBCOAppsInstaller' role using your SSO 'view' role, and set environment variables to enable the custom role to run aws and eksctl commands:
 

export AWS_ACCOUNT_ID=012345678910
export ASSUMED_ROLE_SESSION_NAME=ssouser-assuming-tibcoappsinstaller
export ASSUMED_AWS_ROLE_JSON=$(aws sts assume-role \
    --role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/$AWS_CUSTOM_ROLE_NAME \
    --role-session-name $ASSUMED_ROLE_SESSION_NAME --profile view \
    --duration-seconds 3600 | jq .)
export AWS_ACCESS_KEY_ID=$(echo $ASSUMED_AWS_ROLE_JSON | \
    jq .Credentials.AccessKeyId | tr -d '"')
export AWS_SECRET_ACCESS_KEY=$(echo $ASSUMED_AWS_ROLE_JSON | \
    jq .Credentials.SecretAccessKey | tr -d '"')
export AWS_SESSION_TOKEN=$(echo $ASSUMED_AWS_ROLE_JSON | \
    jq .Credentials.SessionToken | tr -d '"')

# The HELM_* variables shown below will be used later to pass the necessary AWS credentials to the
# 'helm upgrade' command. However, the 'helm upgrade' command will fail if the AWS_* variables are still
# set in your environment. The AWS_* variables will be unset before the 'helm upgrade' command is run.

export HELM_AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
export HELM_AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
export HELM_AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN
aws sts get-caller-identity

{
    "UserId": "ABOAOKNIUSEQQPKBFAC63:ssouser-assuming-tibcoappsinstaller",
    "Account": "012345678910",
    "Arn": "arn:aws:sts::012345678910:assumed-role/TIBCOAppsInstaller/ssouser-assuming-tibcoappsinstaller"
}

Here, the SSO user has created a temporary (60-minute) session named 'ssouser-assuming-tibcoappsinstaller' that will enable all actions allowed in the 'TIBCOAppsInstaller' role's attached 'PowerUserAccess' policy (and 'AdministratorAccess' if you need to complete steps 5 and 6). 

5.) Create the EKS cluster and nodegroup. If these resources already exist, you may skip this step. 
 

export EKS_CLUSTER_NAME=epapps
export EKS_CLUSTER_REGION=us-east-2
eksctl create cluster \
    --name $EKS_CLUSTER_NAME \
    --version 1.21 \
    --region $EKS_CLUSTER_REGION \
    --without-nodegroup \
    --max-pods-per-node 100
eksctl create nodegroup \
        --name $EKS_CLUSTER_NAME-nodes \
        --nodes-min 2 \
        --nodes-max 4 \
        --node-volume-size 200 \
        --cluster $EKS_CLUSTER_NAME \
        --node-type t3.2xlarge \
        --region $EKS_CLUSTER_REGION

6.) Associate an AWS IAM OIDC provider with your EKS cluster. This is needed to coordinate with the external identity provider (which, in this example, is Azure AD). This enables users (managed by your external identity provider) to login to the ModelOps server. If your EKS cluster already has an associated OIDC provider, you may skip this step. The 'associate-iam-oidc-provider' command will create an OIDC provider if one does not already exist.
 

eksctl utils associate-iam-oidc-provider \
    --cluster $EKS_CLUSTER_NAME \
    --approve \
    --region $EKS_CLUSTER_REGION
export OIDC_PROVIDER_ID=$(aws eks describe-cluster \
    --name $EKS_CLUSTER_NAME \
    --region $EKS_CLUSTER_REGION \
    --query "cluster.identity.oidc.issuer" \
    --output json | tr -d '"' | grep -o "id/.*")

At this time, you can detach the 'AdministratorAccess' policy from the 'TIBCOAppsInstaller' role (as it is not needed for the remainder of the installation):
 

# Unset AWS_* variables for the assumed custom role session.
# This allows the 'manage' role be assumed again.
# Then detach the AdministratorAccess policy.

unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
aws iam detach-role-policy \
    --role-name $AWS_CUSTOM_ROLE_NAME \
    --policy-arn arn:aws:iam::aws:policy/AdministratorAccess \
    --profile manage

7.) Add a trust relationship to your OIDC provider. To do this, recall the OIDC provider's ID you obtained in step 6, and update the trust relationships for your custom role.

rm -rf trust-relationship.json
echo -e \
    "{ \
        \"Version\": \"2012-10-17\", \
        \"Statement\": [ \
        { \
            \"Sid\": \"Statement1\", \
            \"Effect\": \"Allow\", \
            \"Principal\": { \
                \"AWS\": \"$AWS_SSO_READ_ACCESS_ROLE_ARN\" \
            }, \
            \"Action\": \"sts:AssumeRole\" \
        }, \
        { \
            \"Sid\": \"Statement2\", \
            \"Effect\": \"Allow\", \
            \"Principal\": { \
                \"Federated\": \"arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/\
                oidc.eks.$EKS_CLUSTER_REGION.amazonaws.com/$OIDC_PROVIDER_ID\" \
            }, \
            \"Action\": \"sts:AssumeRoleWithWebIdentity\" \
        } \
        ] \
    }" | tr -d " \t" > trust-relationship.json
aws iam update-assume-role-policy \
    --role-name $AWS_CUSTOM_ROLE_NAME \
    --policy-document file://trust-relationship.json \
    --profile manage

8.) Configure a redirect URL in your Azure Application to allow Azure AD to authenticate users for your ModelOps server. If you don't currently have an Azure Application for ModelOps, you may create one in the Azure portal under Home > App Registrations > + New registration. 

Once your Azure app is created, go to (again in the Azure portal) Home > App Registrations > [Your Azure Application's Display Name] > Authentication > Platform Configurations > + Add a platform (type 'Web'). The redirect URL has the format: https://modelops-server.$AWS_HOSTED_ZONE/oauth2/callback. For this example, the redirect URL is: https://modelops-server.eksapps.companycloud.com/oauth2/callback

In the Azure portal, also note the values for your Azure Application's Service Principal ID and Tenant ID. Set environment variables for these, as shown below. You can also get these values using the Azure Command-Lind Interface (az cli):
 

az login
export AZ_APP_DISPLAY_NAME=ModelOpsAzApp
export AZ_TENANT_ID=$(az ad sp list \
    --display-name $AZ_APP_DISPLAY_NAME | jq '.[].appOwnerTenantId' | tr -d '"')
export AZ_SERVICE_PRINCIPAL_ID=$(az ad sp list \
    --display-name $AZ_APP_DISPLAY_NAME | jq '.[].appId' | tr -d '"')

Also, set an environment variable for a valid secret configured in your Azure Application (under 'Certificates and Secrets' in the portal). You can create a new secret if you lost the value for your existing secret.
 

export AZ_APP_SECRET=*******

9.) Create the AWS ECR (Elastic Container Registry) repositories:
 

# Switch back to your custom role using the AWS_* variables
export AWS_ACCESS_KEY_ID=$HELM_AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$HELM_AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN=$HELM_AWS_SESSION_TOKEN

aws ecr create-repository --repository-name install-pipeline --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name tools --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name data-channel-registry --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name file-datasink --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name file-datasource --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name git-server --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name kafka-datasink --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name kafka-datasource --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name modelops-metrics --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name modelops-server --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name pmml --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name python --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name sbrt-base --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name scheduling-server --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name scoring-flow --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name tensorflow --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name test-datasink --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name test-datasource --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name rest-datasink --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name rest-datasource --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name statistica --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name jdbc-datasource --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name spark --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name jdbc-datasink --region $EKS_CLUSTER_REGION
aws ecr create-repository --repository-name rest-request-response-datachannel --region $EKS_CLUSTER_REGION

10.) Determine the NodeInstanceRole name for your nodegroup, and attach the AWS-managed 'AmazonEC2ContainerRegistryFullAccess', 'AmazonElasticContainerRegistryPublicFullAccess' and 'PowerUserAccess' policies to the NodeInstanceRole:
 

unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
export NODE_INSTANCE_ROLE_NAME=$(aws iam list-roles --profile manage | \
    jq -r '.Roles| .[] | .RoleName' | grep $EKS_CLUSTER_NAME | \
    grep NodeInstanceRole)
aws iam attach-role-policy \
    --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess \
    --role-name $NODE_INSTANCE_ROLE_NAME \
    --profile manage
aws iam attach-role-policy \
    --policy-arn arn:aws:iam::aws:policy/AmazonElasticContainerRegistryPublicFullAccess \
    --role-name $NODE_INSTANCE_ROLE_NAME \
    --profile manage
aws iam attach-role-policy \
    --policy-arn arn:aws:iam::aws:policy/PowerUserAccess \
    --role-name $NODE_INSTANCE_ROLE_NAME \
    --profile manage

11.) Update your kubeconfig using the 'view' role, create the K8s cluster, and create the secrets needed to authorize ModelOps services:
 

aws eks update-kubeconfig \
    --name $EKS_CLUSTER_NAME \
    --region $EKS_CLUSTER_REGION \
    --role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/$AWS_CUSTOM_ROLE_NAME \
    --profile view
export MODELOPS_NAMESPACE=modelops12
export MODELOPS_ROOT_PASSWORD=tibco123
export EKS_ES_PASSWORD=ElAsticPW123
export EKS_GIT_PASSWORD=G1tPW123
export EKS_NEXUS_PASSWORD=NxtPW123
export EKS_SCORING_PASSWORD=Sc0rePW123
export ARTIFACT_MGMT_PASSWORD=Art1f@ct123

kubectl create namespace $MODELOPS_NAMESPACE
kubectl create secret generic elasticsearch-es-elastic-user \
        --from-literal=elastic=$EKS_ES_PASSWORD \
        --namespace $MODELOPS_NAMESPACE
kubectl create secret generic git-server \
        --from-literal=modelops=$EKS_GIT_PASSWORD \
        --namespace $MODELOPS_NAMESPACE
kubectl create secret generic nexus-server \
        --from-literal=admin=$EKS_NEXUS_PASSWORD \
        --namespace $MODELOPS_NAMESPACE
kubectl create secret generic modelops-server \
        --from-literal=admin=$MODELOPS_ROOT_PASSWORD \
        --namespace $MODELOPS_NAMESPACE
kubectl create secret generic scoring-admin \
        --from-literal=admin=$EKS_SCORING_PASSWORD \
        --namespace $MODELOPS_NAMESPACE
kubectl create secret generic artifact-management \
        --from-literal=admin=$ARTIFACT_MGMT_PASSWORD \
        --namespace $MODELOPS_NAMESPACE
kubectl create secret generic oauth2 \
        --from-literal=TENANT_ID=$AZ_TENANT_ID \
        --from-literal=CLIENT_ID=$AZ_SERVICE_PRINCIPAL_ID \
        --from-literal=CLIENT_SECRET=$AZ_APP_SECRET \
        --namespace $MODELOPS_NAMESPACE

12.) Run the helm command to install ModelOps:
 

export MODELOPS_HOME=/opt/tibco/modelops/1.2
export AWS_HOSTED_ZONE=eksapps.companycloud.com
# NOTE: the externalDNS.aws.eksRoleArn option shown below is intentionally
# set to an empty value. Since you added the PowerUserAccess policy to the
# NodeInstanceRole earlier, you do not need to specify a value for the
# eksRoleArn (i.e. the NodeInstanceRole will be the default value).
# However, the option must still be set in the 'helm upgrade' command,
# otherwise the external-dns pod will not start properly.
helm upgrade \
    --install modelops12 $MODELOPS_HOME/helm-charts/kubernetes-installer-1.0.2.tgz \
    --atomic \
    --set cloud=eks \
    --set eks.externalDNS=aws \
    --set externalDNS.aws.eksRoleArn= \
    --set eks.containerRegistry=$AWS_ACCOUNT_ID.dkr.ecr.$EKS_CLUSTER_REGION.amazonaws.com \
    --namespace $MODELOPS_NAMESPACE \
    --set eks.networkExposure=ingress \
    --set eks.ingressDomain=$AWS_HOSTED_ZONE \
    --set eks.oauth2=azure \
    --set eks.awsAccessKeyID=$HELM_AWS_ACCESS_KEY_ID \
    --set eks.awsSecretAccessKey=$HELM_AWS_SECRET_ACCESS_KEY \
    --set eks.awsTokenId=$HELM_AWS_SESSION_TOKEN \
    --set medium.nexus.memory=3Gi \
    --timeout 10m0s

13.) When the helm upgrade command completes, it will instruct you to copy the maven artifacts. To do that, run this command:
 

cd $MODELOPS_HOME/maven-repository-artifacts/
kubectl cp modelops-repo-1.2.0-mavenrepo.zip mavenrepo-0:/tmp/ -n $MODELOPS_NAMESPACE

This copy will take a few minutes to complete.

14.) Then continue to monitor the progress:
 

tkn pipelinerun logs bootstrap --follow --namespace $MODELOPS_NAMESPACE

15.) Once the bootstrapping is done, you may follow the progress for the modelops-server:
 

tkn pipelinerun logs modelops-server --follow --namespace $MODELOPS_NAMESPACE

16.) Since the initialization of the ModelOps server will take some time, you can now add the 'Admin' App Role for your Azure Application (recall step 8). In the Azure Portal, go to 'Home > App registrations > ModelOpsAzApp > App roles > +Create app role' and set the following values:

• Display Name: ModelOpsAdminAppRole
• Allowed member types: Users/Groups
• Value: Admin
• Description: This App Role enables Azure AD users to become administrators of the ModelOps server.

You may choose the 'Display name', but the 'Value' must be set to exactly 'Admin', because 'Admin' is the default administrative role configured in the ModelOps server pod.

After adding the role, then assign it to the appropriate Azure AD user (e.g. ssouser@company.com). Do this in the Azure Portal under 'Home > Enterprise applications > ModelOpsAzApp > Users and groups > +Add user/group'. On the 'Add Assignment' page, select your Azure AD user, and also select the 'Admin' App Role you created earlier. Then click 'Assign'. Your Azure AD user will now have administrative access to the ModelOps server running in AWS.

17.) Navigate to the server in a web browser:

https://modelops-server.eksapps.companycloud.com

Provides the essential steps needed to run a TIBCO ModelOps (TMO) 1.2 server on Amazon Elastic Kubernetes Service (EKS), using Azure Active Directory (AD) for authentication (oauth2).