Failed Tech Hype: SOAP Web Services
Many used SOAP wrongly. If you still have to work (programming/testing) with SOAP, there is a much simpler way.
Table of Contents:
· What’s Wrong with SOAP?
∘ 1. Every time WSDL changes, a re-generation of Stub classes.
∘ 2. Real-life SOAP Service Request/Response are large and complex
∘ 3. Most programmers don’t know XML Schema well.
· Why was SOAP so hyped?
· Early Wisdom
· How did I work with SOAP?SOAP (Simple Object Access Protocol) is a messaging protocol that allows exchanging of structured information in XML between computer applications. SOAP is often associated with the term “Web Service”. A web service differs from a website in that a web service provides information consumable by software rather than humans.
SOAP was hugely popular between 2004 to 2011, after that it was gradually being replaced by RESTful web service. Nowadays, there are still many SOAP web services are being used. Proof of that is: SmartBear acquired SoapUI, a testing tool mainly for testing SOAP web services, in 2011. Later, SmartBear even released a more expensive product: ReadyAPI. By the way, I don’t use these products, they have made API testing harder and more expensive. I will post an article on how to test SOAP/REST web services efficiently, and totally free. (If you can’t wait, check out my book: API Testing Recipes in Ruby).
What’s Wrong with SOAP?
Despite its name starting with ‘Simple’, SOAP is complex.
Let’s start with the objective: a computer program retrieves data from another remotely, like this: getting a book detail via an ISBN. There must be communication; it works as below:
So far, it is not complex, another form of XML-RPC. However, some software architects think it will be cooler to let the programmer code it just like a normal function.
// soapService defined above
BookDetail detail = soapService.retrieveBook("1505882893")
detail.getPrice(); // => 24.95It seemed ‘natural’ right? Sadly, no.
To make a function call in a compiled language, such as Java and C#, two object types must be defined first: Parameter and Return.
BookDetail retrieveBook(String isbn);The parameter in the above example is simple: String, a primitive type. The return type BookDetail is a new class. This class is from the web service definition, specified in WSDL (Web Services Description Language). Therefore, a new process is introduced in the build process: “Generating stub code from the WSDL File” (this page was on the IBM site, was updated in 2021, indicating many still suffer from this).
Initially, programmers (including me in 2004) were excited, generating hundreds of Java classes, without coding. It was kind of ‘cool’ and had a feeling of ‘automation’. However, not long after, it turned out to be a nightmare.
1. Every time WSDL changes, a re-generation of Stub classes.
One major advantage of XML is backward compatibility. Let me illustrate this with an example.
<!-- v1 -->
<Person>
<FirstName>Homer</FirstName>
<LastName>Simpson</LastName>
<Person>Later, in v2.
<!-- v2 -->
<Person>
<FirstName>Homer</FirstName>
<MiddleName>J</MiddleName>
<LastName>Simpson</LastName>
<Person>The client software developed for v1 can still work with v2 XML properly.
However, with the Stub class generated from WSDL, programmers often find they need to re-generate Stub every single time WSDL changes. This might take a long time.
I remembered, in 2005, it took about 20 minutes to complete schema object generation. The library we used was Castor.
Furthermore, the XML schema structure is not 100% perfectly matching the code. I won’t bother to list the difference here, just remind you that XML schema was defined solely to define XML document, nothing to do with code.
That’s also why you often see published SOAP web services with a version number. Please note, the published ones. Just imagine, during the development, how much time was wasted on generating/compiling stub schema classes. This is sad because XML was designed with backward compatibility.
2. Real-life SOAP Service Request/Response are large and complex
In the above “Retrieve book by ISBN” example, the request and response in the SOAP web service look fine. However, in real life, those are much more complicated.
For example, I worked on one government project. Several data architects spent over a year trying to define the national schema for development application assessment and failed. This schema was huge, over thousands of elements. I remember it even included ApplicantDeathDate !
3. Most programmers don’t know XML Schema well.
WSDL defines SOAP service requests and responses in XML Schema. However, most programmers either don’t know XML Schema well or overestimate their knowledge. I think I am qualified to say that: I worked in a W3C Australia office for 3.5 years. I detected one schema mistake in the W3C RDF specification (for that reason, I was a paper reviewer at the WWW8 conference). RDF is heavily schema-based, this W3C specification listed my co-authored article.
Almost on every project (2005–2010), the team members including data architects came to ask for my help on XML schema. Here I don’t want to bother you with XML schema,
Why was SOAP so hyped?
1. Simpler than the previous
Prior to SOAP, there were CORBA and DCOM (Distributed Component Object Model) for software interaction over a network. CORBA and DCOM were very complex, much more than SOAP, so you no longer hear them anymore (but might still use SOAP).
2. The demand for integration was high
After Y2K, most software developments started towards the web. Therefore, the demand for integration was strong. For example, an eBusiness site needs to talk to the payment gateway.
3. Supposed to work across all programming languages and platforms
DCOM works only on Microsoft platforms. So a standard that works on both Unix and Microsoft was welcomed.
Please beware there might be gotchas, such as dealing with date time in schema objects.
4. Cool to throw jargon words
Architects, who don’t do hands-on coding and testing, like to use new jargon words. For example, many software architects like to talk about “microservices” now.
Early Wisdom
From my memory, I saw one article. Tim O’Reilly once predicted during SOAP’s prime time: “in 10 years, people would say: ‘what is SOAP?’ ”
DHH, the creator of Ruby on Rails and author of the popular ReWork book series, deprecated ActionWebService is deprecated in 2009 (the actual decision was long before that). This is visionary, as you would expect DHH now. However, around 2006, Ruby on Rails (and DHH) was not well-known, DHH’s decision to drop SOAP support was courageous and correct.
DHH also predicated the wastefulness of Gherkin (Cucumber, SpecFlow, ..). For more, check out this article: Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?
How did I work with SOAP?
Whether we like it or not, we still need to work with (programming or testing) SOAP services.
“I don’t mind SOAP, as long as no stub involved. Working directly with XML is much easier. “ — Zhimin Zhan
If we treat SOAP as XML over HTTP, it becomes much easier to work with.
In 2008, the state government wanted all government agencies to use the one shared software service, implemented in SOAP. One day, two consultants came to our project to help with the integration. In the morning of the day, I specifically asked for some automated tests that they had used.
They arrived in the afternoon, I showed them our integration results. They were totally shocked at how quickly we were able to get it working. Because they had been doing these for a few government agencies, taking a few weeks. I told them: “We used SOAP XML, not SOAP objects”. The lead engineer, after a pause, said: “That is fine”.
How did I do it?
Based on the sample SOAP request XML, I constructed one template (in Java, I used Velocity) for it.
Using code to inject data into the template
POST the request XML to the SOAP service
Parse returned XML using the Xerces library to extract the data I wanted.
I replicated the above in every SOAP work, nice and simple.
Now I use Ruby most of the time, Ruby made the above even simpler (and fun): ERB template is a built-in feature, and Nokogiri is an XML-processing gem that is a pleasure to use.
Related reading: