Root Cause Analysis: Azure Service Bus Relay HTTPS Migration — Cloud ↔ Premise Connectivity Impact

book

Article ID: KB0138720

calendar_today

Updated On:

Products Versions
TIBCO Scribe Online All

Description

Customer Impact

Impact was limited to operations that required a live, synchronous round-trip to an on-premise Agent over the relay, most notably the Test Connection action, and other real-time commands (for example disable, decommission, or run-now requests) issued during the affected window. Customers attempting these operations during the impact window may have experienced delays or failures.

Scheduled and already-running work was not affected. Any work a customer had scheduled on an Agent continued to execute as normal throughout the incident, and execution history continued to synchronize to the cloud without interruption at any point.

For customers who were affected, we applied a configuration-level workaround on their runtime Agents that restored command functionality via an alternate path while the permanent fix was developed, so that normal operation could resume without waiting for the full fix to roll out.

Timeline

Date / Stage

Event

Aug 9, 2026

Microsoft begins a phased, region-by-region operating-system-level removal of TLS 1.0/1.1 support on Azure Relay gateway nodes, beginning in Germany West Central.

Late Mon, Aug 10

First reports of EU relay connectivity issues are received.

Tue, Aug 11

Our team opens a Critical-priority investigation. Initial troubleshooting focuses on the on-premise Agent side, consistent with early symptoms.

Aug 11–12

Investigation pivots to the API tier after the Agent-side theory is ruled out. An interim configuration workaround is applied for affected customers, restoring command functionality via an alternate path, and database-tier monitoring is increased to support the added load.

Aug 12-13

Root cause isolated and code changes made to the recommended configuration 

Aug 14–16

Testing, packaging and pre-release deployment work. 

Aug 16

Fix deployed to Customer Sandbox.

Aug 17

Fix deployed to AWS US East 2 and Sydney (AU) regions.

Aug 18

Fix deployed to AWS US/EU regions.

Aug 19

Fix deployed to Azure West, completing rollout to all regions. Monitoring continues across all environments.

Root Cause

What happened

To reference Microsoft's Root Cause Analysis for this event (Support Case 2608110040003474), Microsoft had announced the retirement of TLS 1.0/1.1 across Azure services in 2023, with a service-level retirement completed in December 2025. Microsoft's RCA identifies an oversight in that implementation: the specific WCF Relay rendezvous path TIBCO Scribe Online used did not carry that enforcement, so connections on that path kept succeeding for several months after the broader retirement. Starting the week of August 10, 2026, Microsoft rolled out a phased, region-by-region operating-system-level change on the Relay gateway nodes to close that gap. This caused our cloud-to-premise command traffic to stop flowing.Microsoft's RCA confirms this was an unannounced enforcement change because Microsoft believed service-level TLS 1.2 enforcement was already complete. They did not treat the operating-system-level change as a customer-visible event requiring advance notice.

On our side, this compounded with two factors that extended our time to root-cause: Microsoft's available advance communication about the broader retirement centered on TLS 1.2 compliance, a bar our platform had already met for years, so the actual breaking change (loss of the transport itself), due to a configuration setting change we were unaware of, wasn't the first suspect during troubleshooting; and the affected code was foundational relay-handling logic that had been in place, largely unmodified, since the platform's original design, so there was limited recent operational experience with how a failure there would present. Confirming the true nature of the change required direct engagement with Microsoft support and low-level network tracing, since Microsoft's own documentation did not clearly indicate the legacy transport path would stop functioning.

Resolution

How we're preventing recurrence

The permanent fix removes our dependency on the deprecated transport entirely by migrating to an HTTPS-based relay binding, so this specific class of failure cannot recur. Separately, we have been reconfiguring the agents and api components away from this relay technology entirely as part of a longer-term architectural initiative that was underway before this incident; groundwork for that migration (isolating relay usage behind a swappable implementation layer) had shipped ahead of this event. We are also improving our deployment tooling to close a configuration-restoration issue identified during this rollout, and streamlining our deployment checklist to reduce friction in future full-platform releases.

Remediation

The following actions were completed to resolve this incident:

  • Deployed an interim configuration workaround for affected customers, restoring command functionality via an alternate path while the permanent fix was developed.
  • Increased monitoring of the affected database tier to support the additional load introduced by the workaround.
  • Designed, built, and deployed a permanent fix migrating the relay transport to an HTTPS-based binding across all affected components (Agent, Agent Manager, API, Worker).
  • Completed standard regression and upgrade testing across every affected component, plus additional targeted testing given the scope and severity of the change.
  • Completed an independent security review of all changes, confirming no regressions and a reduced attack surface.
  • Rolled out the fix to all regions (Customer Sandbox, AWS US East 2, Sydney AU, AWS US/EU, Azure West) between August 16 and August 19, 2026, with monitoring in place throughout.

Ongoing Monitoring & Prevention

  • Continuing our longer-term migration away from this relay technology entirely, building on groundwork already in place ahead of this incident.
  • Continuing to monitor all environments following this rollout to confirm long-term stability.

Our Commitment to You

We know that connectivity issues affecting real-time operations cause major disruptions to your workflows, and we apologize for the impact this had on your business. This issue originated from an unannounced, unilateral transport change on Microsoft's side, but we recognize that doesn't change the impact you experienced. We moved quickly to restore functionality through an interim workaround and to deliver a permanent, thoroughly tested fix across every environment. We are committed to the longer-term work described above so that our platform can continue to provide the best performance possible. 

If you have any questions about this incident or its impact on your environment, please raise a support case.

Issue/Introduction

Root Cause Analysis

Azure Service Bus Relay HTTPS Migration — Cloud ↔ Premise Connectivity Impact

Incident Reference

SCCORE-12664 / SCCORE-12666

Severity

Critical

Impact Window

August 10 – August 16, 2026

Affected Environments

EU datacenter connectivity; global rollout of the permanent fix completed across Customer Sandbox, AWS US East 2, Sydney (AU), AWS US/EU, and Azure West by August 19, 2026

Status

Resolved — permanent fix deployed to all regions; monitoring ongoing

Related Microsoft Case

Microsoft Support Case / Tracking ID 2608110040003474 (Azure Relay, Severity A)

Executive Summary

Beginning August 10, 2026, customers using TIBCO Scribe Online platform began experiencing degraded cloud-to-premise command connectivity — most notably affecting real-time operations such as creating or testing an on-premise connection. The underlying cause was a transport-level change on the Azure Service Bus side that prevented our components from being able to connect to their respective Service Buses.

While root cause was being isolated, customers were provided with an interim workaround to have their agents leverage the fallback polling path, instead of leveraging the Azure Service Bus. Our team corrected the transport-level configuration on the platform and the Scribe agent components and rolled the fix out to all regions starting on August 16th and completing the final environment on August 19, 2026