Add support for SOAP #4618

madhukaw · 2023-06-27T05:47:18Z

Description:

Need to improve the soap module to facilitate new requirements.

shafreenAnfar · 2023-06-27T06:03:49Z

Following meeting notes from release planning meeting.

Support for consuming SOAP services in Ballerina

Is there a WSDL? If yes
- We can write bal wsdl –mode=client foo.wsdl
- WSDL can contain refs to use of WS-Addressing, SOAPAction, WS-Sec basic auth and some basic token profile
We don’t need to support other WS-* specs
If not, we can write a simple SOAPClient remote client similar to HTTPClient for SOAP/HTTP
- WSDL generated client one can use this
- We can do SOAP/JMS the same way
Support WSDL 1.1, SOAP 1.1 and 1.2, WS-Addressing xx, WS-Security yy

shafreenAnfar · 2023-07-18T03:01:43Z

This is what we need to target to support.

shafreenAnfar · 2023-07-18T03:02:18Z

Sample SOAP message.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://www.example.com/webservice">
    <soapenv:Header>
        <!-- Optional SOAP header elements go here -->
    </soapenv:Header>
    <soapenv:Body>
        <web:HelloRequest>
            <web:Name>John Doe</web:Name>
        </web:HelloRequest>
    </soapenv:Body>
</soapenv:Envelope>

shafreenAnfar · 2023-07-18T03:03:31Z

Sample MTOM message.

POST /webservice HTTP/1.1
Host: www.example.com
Content-Type: multipart/related; type="application/xop+xml"; boundary="uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1"; start="<soap@envelope>"; start-info="text/xml"

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
Content-ID: <soap@envelope>

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                  xmlns:web="http://www.example.com/webservice"
                  xmlns:xop="http://www.w3.org/2004/08/xop/include">
    <soapenv:Header>
        <!-- Optional SOAP header elements go here -->
    </soapenv:Header>
    <soapenv:Body>
        <web:UploadAttachments>
            <!-- Image Attachment -->
            <web:Attachment>
                <xop:Include href="cid:image1"/>
            </web:Attachment>
            <!-- Additional Attachment -->
            <web:Attachment>
                <xop:Include href="cid:file1"/>
            </web:Attachment>
        </web:UploadAttachments>
    </soapenv:Body>
</soapenv:Envelope>

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1
Content-Type: image/jpeg
Content-Transfer-Encoding: binary
Content-ID: <image1>

... Binary Image Data ...

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-ID: <file1>

... Binary File Data ...

--uuid:1ab9c480-8f78-482a-92f8-7df4f1e8e2c1--

shafreenAnfar · 2023-07-18T03:06:25Z

Sample WSDL.

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
                  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
                  targetNamespace="http://www.example.com/weather">
    <wsdl:types>
        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.example.com/weather">
            <xs:element name="City" type="xs:string"/>
            <xs:element name="WeatherForecast" type="xs:string"/>
        </xs:schema>
    </wsdl:types>
    <wsdl:message name="GetWeatherRequest">
        <wsdl:part name="City" element="tns:City"/>
    </wsdl:message>
    <wsdl:message name="GetWeatherResponse">
        <wsdl:part name="WeatherForecast" element="tns:WeatherForecast"/>
    </wsdl:message>
    <wsdl:portType name="WeatherServicePortType">
        <wsdl:operation name="GetWeather">
            <wsdl:input message="tns:GetWeatherRequest"/>
            <wsdl:output message="tns:GetWeatherResponse"/>
        </wsdl:operation>
    </wsdl:portType>
    <wsdl:binding name="WeatherServiceBinding" type="tns:WeatherServicePortType">
        <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
        <wsdl:operation name="GetWeather">
            <soap:operation soapAction="http://www.example.com/weather/GetWeather"/>
            <wsdl:input>
                <soap:body use="literal"/>
            </wsdl:input>
            <wsdl:output>
                <soap:body use="literal"/>
            </wsdl:output>
        </wsdl:operation>
    </wsdl:binding>
    <wsdl:service name="WeatherService">
        <wsdl:port name="WeatherServicePort" binding="tns:WeatherServiceBinding">
            <soap:address location="http://www.example.com/weather"/>
        </wsdl:port>
    </wsdl:service>
</wsdl:definitions>

shafreenAnfar · 2023-07-18T03:13:11Z

Above SOAP payload converted to JSON.

{
   "soapenv:Envelope":{
      "soapenv:Header":{
         
      },
      "soapenv:Body":{
         "web:HelloRequest":{
            "web:Name":"John Doe",
            "@xmlns:web":"http://www.example.com/webservice"
         }
      },
      "@xmlns:soapenv":"http://schemas.xmlsoap.org/soap/envelope/",
      "@xmlns:web":"http://www.example.com/webservice"
   }
}

shafreenAnfar · 2023-07-18T16:16:16Z

A possible client.

public client class SoapClient {
    
    public function init(string url, http:ClientConfiguration config) {}

    remote function sendReceive(xml|mime:Entity[] soap) returns xml|mime:Entity[]|error {}

    remote function sendOnly(xml|mime:Entity[] soap) returns error? {}

    remote function sendReceiveEnve(SoapEnv soapenv, boolean simpleJsonMapping = true) returns SoapEnv|error {}
    
    remote function sendOnlyEnve(SoapEnv soapenv, boolean simpleJsonMapping = true) returns error? {}
}

public type SoapEnv record {|
    xml|record {} header?;
    xml|record {} body;
|};

// Is for the MTOM suff
public type Attachment record {|
    string name;
    byte[]|stream<byte[]>|string content;
|};

shafreenAnfar · 2023-07-18T16:23:00Z

Just like we have VSCode command to go from JSON to record, we need one to go from XML to record.

manuranga · 2023-07-19T08:47:44Z

We can use xml|[xml, mime:Entity...] soap instead xml|mime:Entity[] soap as a better type.

manuranga · 2023-07-19T08:54:00Z

s/simpleJsonMapping/ingoreNamespace/g

manuranga · 2023-07-19T09:11:25Z

We are providing two levels of functionality here. The low-level api (sendReceive, sendOnly) and high-level opinionated api (sendReceiveEnve, sendOnlyEnve). Better to give these as separate abstractions. Users for former should not concern themselves with the latter.

shafreenAnfar · 2023-07-20T09:32:00Z

I think we can start with the below two clients.

public client class BasicClient {
    
    public function init(string url, http:ClientConfiguration config) {}

    remote function sendReceive(xml|mime:Entity[] soap) returns xml|mime:Entity[]|error {}

    remote function sendOnly(xml|mime:Entity[] soap) returns error? {}
}

public client class AdvancedClient {
    
    public function init(string url, http:ClientConfiguration config) {}

    remote function sendReceive(SoapEnv soapenv, boolean simpleJsonMapping = true) returns SoapEnv|error {}
    
    remote function sendOnly(SoapEnv soapenv, boolean simpleJsonMapping = true) returns error? {}
}

public type SoapEnv record {|
    xml|record {} header?;
    xml|record {} body;
|};

// Is for the MTOM suff
public type Attachment record {|
    string name;
    byte[]|stream<byte[]>|string content;
|};

sameerajayasoma · 2023-07-21T06:24:43Z

Had a quick chat with @shafreenAnfar on this API. We agreed to go ahead with the BasicClient first. We can review the AdvancedClient in the meantime.

Here are some of the things that we discussed:

Can we say soap:Client instead of soap:BasicClient?
[Optional] I prefer fireAndForgot instead of sendOnly as it is commonly used in this context. See: https://axis.apache.org/axis2/java/core/apidocs/org/apache/axis2/client/ServiceClient.html#fireAndForget(org.apache.axiom.om.OMElement)
I would use soap:Envelop instead of soap:SoapEnv

shafreenAnfar · 2023-09-15T03:48:55Z

Given the behaviour of the SOAPAction, I think we are better off having two clients which user common code underneath as soap11:Client and soap12:Client.

Nuvindu · 2023-09-26T11:53:06Z

SOAP 1.1 & 1.2 clients offer the capability to configure and apply one or multiple security policies during the initiation of a SOAP client connection. These security policies can be applied as follows:

SOAP 1.1 Client: UsernameToken and TranportBinding Policy

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap:soap11;
import ballerina/soap:wssec;

public function main() returns error? {
    soap11:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL", 
        {
            soapSecurity: [
            {
                username: "username",
                password: "password",
                passwordType: wssec:TEXT
            },
            TRANSPORT_BINDING
            ]
        });

    xml envelope = xml `<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
                            <soap:Body>
                            <quer:Add xmlns:quer="http://tempuri.org/">
                                <quer:intA>2</quer:intA>
                                <quer:intB>3</quer:intB>
                            </quer:Add>
                            </soap:Body>
                        </soap:Envelope>`;
    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

SOAP 1.2 Client: Symmetric Binding Policy

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap:soap12;
import ballerina/soap:wssec;

public function main() returns error? {
    crypto:KeyStore keyStore = {
        path: KEY_STORE_PATH,
        password: KEY_PASSWORD
    };

    crypto:PrivateKey privateKey = check crypto:decodeRsaPrivateKeyFromKeyStore(keyStore, KEY_ALIAS, KEY_PASSWORD);
    crypto:PublicKey publicKey = check crypto:decodeRsaPublicKeyFromTrustStore(keyStore, KEY_ALIAS);

    soap12:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL", 
        {
            soapSecurity: {
                signatureAlgorithm: wssec:RSA_SHA256,
                encryptionAlgorithm: wssec:RSA_ECB,
                bindingKey: privateKey,
                verificationKey: publicKey,
                x509Token: X509_PUBLIC_CERT_PATH_2
            };
        });

    xml envelope = xml `<soap:Envelope
                        xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                        soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
                            <soap:Body>
                            <quer:Add xmlns:quer="http://tempuri.org/">
                                <quer:intA>2</quer:intA>
                                <quer:intB>3</quer:intB>
                            </quer:Add>
                            </soap:Body>
                        </soap:Envelope>`;
    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

Nuvindu · 2023-10-05T04:25:50Z

We decided to use two security configurations for the SOAP client:

inboundSecurity: This configuration is applied to the SOAP envelope when a request is made. It includes various ws security policies such as Username Token, Timestamp Token, X509 Token, Symmetric Binding, Asymmetric Binding, and Transport Binding, either individually or in combination with each other.
outboundSecurity: This configuration is applied to the SOAP envelope when a response is received. Its purpose is to decrypt the data within the envelope and verify the digital signature for security validation.

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap:soap12;
import ballerina/soap:common;

public function main() returns error? {
    crypto:KeyStore clientKeyStore = {
            path: X509_KEY_STORE_PATH_2,
            password: KEY_PASSWORD
    };
    crypto:PrivateKey clientPrivateKey = check crypto:decodeRsaPrivateKeyFromKeyStore(clientKeyStore, KEY_ALIAS, KEY_PASSWORD);

    crypto:PublicKey clientPublicKey = check crypto:decodeRsaPublicKeyFromTrustStore(clientKeyStore, KEY_ALIAS);

    crypto:PublicKey serverPublicKey = ...//

    soap12:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL",
    {
            inboundSecurity: {
                signatureKey: clientPrivateKey,
                signatureAlgorithm: wssec:RSA_SHA256,
                encryptionKey: serverPublicKey,
                encryptionAlgorithm: wssec:RSA_ECB
            },
            outboundSecurity: {
                verificationKey: serverPublicKey,
                signatureAlgorithm: wssec:RSA_SHA256,
                decryptionKey: clientPrivateKey,
                decryptionAlgorithm: wssec:RSA_ECB
            }
    });

    xml envelope = xml `<soap:Envelope
                        xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                        soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
                            <soap:Body>
                            <quer:Add xmlns:quer="http://tempuri.org/">
                                <quer:intA>2</quer:intA>
                                <quer:intB>3</quer:intB>
                            </quer:Add>
                            </soap:Body>
                        </soap:Envelope>`;

    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

Nuvindu · 2023-10-05T08:44:35Z

Following our discussion, we've decided to remove the soap.common sub-module and merge into the main soap module. As a result, we will now expose only three modules: soap, soap.soap11, and soap.soap12.

import ballerina/crypto;
import ballerina/mime;
import ballerina/soap;
import ballerina/soap:soap12;

public function main() returns error? {
    crypto:KeyStore clientKeyStore = {
            path: KEY_STORE_PATH,
            password: KEY_PASSWORD
    };
    crypto:PrivateKey clientPrivateKey = check crypto:decodeRsaPrivateKeyFromKeyStore(clientKeyStore, KEY_ALIAS, KEY_PASSWORD);
    crypto:PublicKey clientPublicKey = check crypto:decodeRsaPublicKeyFromTrustStore(clientKeyStore, KEY_ALIAS);
    crypto:PublicKey serverPublicKey = ...//

    soap12:Client soapClient = check new ("http://www.dneonline.com/calculator.asmx?WSDL",
    {
            inboundSecurity: {
                    signatureAlgorithm: soap:RSA_SHA256,
                    encryptionAlgorithm: soap:RSA_ECB,
                    signatureKey: clientPrivateKey,
                    encryptionKey: serverPublicKey,
            },
            outboundSecurity: {
                    verificationKey: serverPublicKey,
                    signatureAlgorithm: soap:RSA_SHA256,
                    decryptionKey: clientPrivateKey,
                    decryptionAlgorithm: soap:RSA_ECB
            }
    });

    xml body = xml `<soap:Envelope
                    xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                    soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
                    <soap:Body>
                    <quer:Add xmlns:quer="http://tempuri.org/">
                        <quer:intA>2</quer:intA>
                        <quer:intB>3</quer:intB>
                    </quer:Add>
                    </soap:Body>
                </soap:Envelope>`;

    xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
}

Nuvindu · 2023-11-06T05:34:29Z

In the previous SOAP APIs, users had to work with a union type of xml | mime:Entity[] for the responses. In most cases, this required users to add conditional statements to extract the specific type from the union.

xml|mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");
if response is xml {
    //…
} else {
    //…
}

However, with the new update, this additional condition is no longer necessary. Users can now directly infer the response type, whether it's xml or mime:Entity[].

xml response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");

OR

mime:Entity[] response = check soapClient->sendReceive(envelope, "http://tempuri.org/Add");

madhukaw added the Type/Improvement label Jun 27, 2023

madhukaw self-assigned this Jun 27, 2023

madhukaw added Team/PCM Protocol connector packages related issues module/soap labels Jun 27, 2023

madhukaw added this to Ballerina Team Main Board Jun 27, 2023

madhukaw moved this to In Progress in Ballerina Team Main Board Jun 27, 2023

madhukaw changed the title ~~Improve SOAP standard library module~~ Improve SOAP standard library module design Jun 27, 2023

shafreenAnfar changed the title ~~Improve SOAP standard library module design~~ Add support for SOAP Jul 6, 2023

manuranga closed this as completed Jul 19, 2023

github-project-automation bot moved this from In Progress to Done in Ballerina Team Main Board Jul 19, 2023

manuranga reopened this Jul 19, 2023

madhukaw moved this from Done to In Progress in Ballerina Team Main Board Jul 20, 2023

This was referenced Jul 27, 2023

Add support for SOAP basic client ballerina-platform/module-ballerina-soap#81

Closed

Add basic soap client and test case ballerina-platform/module-ballerina-soap#82

Merged

madhukaw moved this from In Progress to PR Sent in Ballerina Team Main Board Aug 4, 2023

madhukaw closed this as completed in ballerina-platform/module-ballerina-soap#82 Aug 10, 2023

github-project-automation bot moved this from PR Sent to Done in Ballerina Team Main Board Aug 10, 2023

shafreenAnfar reopened this Sep 14, 2023

Nuvindu mentioned this issue Oct 3, 2023

Improve the SOAP module to support web service securities ballerina-platform/module-ballerina-soap#103

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for SOAP #4618

Add support for SOAP #4618

madhukaw commented Jun 27, 2023 •

edited

Loading

shafreenAnfar commented Jun 27, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023 •

edited

Loading

shafreenAnfar commented Jul 18, 2023

manuranga commented Jul 19, 2023

manuranga commented Jul 19, 2023

manuranga commented Jul 19, 2023

shafreenAnfar commented Jul 20, 2023 •

edited

Loading

sameerajayasoma commented Jul 21, 2023 •

edited

Loading

shafreenAnfar commented Sep 15, 2023 •

edited

Loading

Nuvindu commented Sep 26, 2023 •

edited

Loading

Nuvindu commented Oct 5, 2023 •

edited

Loading

Nuvindu commented Oct 5, 2023 •

edited

Loading

Nuvindu commented Nov 6, 2023

Add support for SOAP #4618

Add support for SOAP #4618

Comments

madhukaw commented Jun 27, 2023 • edited Loading

shafreenAnfar commented Jun 27, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023

shafreenAnfar commented Jul 18, 2023 • edited Loading

shafreenAnfar commented Jul 18, 2023

manuranga commented Jul 19, 2023

manuranga commented Jul 19, 2023

manuranga commented Jul 19, 2023

shafreenAnfar commented Jul 20, 2023 • edited Loading

sameerajayasoma commented Jul 21, 2023 • edited Loading

shafreenAnfar commented Sep 15, 2023 • edited Loading

Nuvindu commented Sep 26, 2023 • edited Loading

SOAP 1.1 Client: UsernameToken and TranportBinding Policy

SOAP 1.2 Client: Symmetric Binding Policy

Nuvindu commented Oct 5, 2023 • edited Loading

Nuvindu commented Oct 5, 2023 • edited Loading

Nuvindu commented Nov 6, 2023

madhukaw commented Jun 27, 2023 •

edited

Loading

shafreenAnfar commented Jul 18, 2023 •

edited

Loading

shafreenAnfar commented Jul 20, 2023 •

edited

Loading

sameerajayasoma commented Jul 21, 2023 •

edited

Loading

shafreenAnfar commented Sep 15, 2023 •

edited

Loading

Nuvindu commented Sep 26, 2023 •

edited

Loading

Nuvindu commented Oct 5, 2023 •

edited

Loading

Nuvindu commented Oct 5, 2023 •

edited

Loading