The web-site to read uses cookies for session state management and authentication. How can we use the StreamBase Web Reader adapter with this site?

This is an example of using the Web Reader adapter with cookies.

1. In the adapter properties, HTTP Headers (headers that will never change) property, add name "User-Agent" and value (MS Edge example): "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"

This allows the Web Reader adapter to mimic a browser compatible with the site.

2. Set the adapter to do a POST, not a GET, so that we get a "Set-Cookie" response. No cookie is provided to a simple GET request.

This picture shows an execution of the modified webreader.sbapp app where we asked it to get the page " https://community.tibco.com/" which uses cookies.

In this case, the field address of the "Cookies" in the response is: “ Headers.Header[6].Values”.

To extract just the Cookies value in a way that can be used in the following requests, add a Map operator with this expression on the Raw Output port tuple:
   Add | Cookies | join(";",Headers[indexof(unzip(Headers).Header,"Set-Cookie",)].Values)
This expression finds the index of the “Set-Cookie” value and returns the Values list as a single string with individual items separated by a semi-colon (;).

The result is highlighted in blue in the console output.


For following web requests, pass the combined “Cookies” string as an element of the HTTPHeaders array field on the Control stream input tuple. The “Cookies” value can be saved in a Dynamic Variable for future use.

Some typical user agent strings are:
  "Apache-HttpClient/4.3.6 (java 1.5)"
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
  "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36"
  "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
  "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36"
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"

Adapter configuration example