Support On-Demand Test Environments With Service Virtualization
As 2018 comes to a close, the hottest movement in the software industry revolves around “DevOps”. Nearly every software team is looking to speed up development cycles and improve production release times by embracing Continuous Integration & Deployment principles. Most of the effort in CI/CD comes in the early phases of the SDLC:
• How do we manage our code repository and branching strategy?
• What tools are used for the build environment?
• Which platform configures and hosts our virtual machines?
Once we’ve moved from the “Integration” into the “Deployment” aspect of the CI/CD, there is now the additional complexity of how our newly deployed builds interact and communicate with existing systems.
Let’s say, for example, that we are developing an E-Commerce Web site, using SAP as the back-end to manage product data and inventory. Traditionally, the Web server for testing has been a dedicated server, communicating with a dedicated SAP test environment:
In a CI/CD process, that dedicated E-Commerce test server is replaced by a virtual machine, generated by VMWare or Kubernetes or some other available platform. This test VM is available on-demand for any build version, containing its own local data and configuration information. The test VM exists for as long as a given set of tests are to be run, and then it is released.
The problem with this approach is that legacy test environments for large applications such as SAP were not designed to work with temporary servers like the on-demand VMs we’re using. The standalone server is not an option for what we need. Neither is including a version of SAP within each VM; the size of the application and volume of necessary data is prohibitive, plus the licensing model tends not to work for such dynamic demand.
To solve this issue, we need to virtualize the 3rd-party service (in this case, SAP). If your software project is starting from scratch, it is quite easy to build mock API services into your platform and utilize them throughout all layers of testing. But what if we’re looking to change to a DevOps methodology sometime in the middle of the entire lifecycle of the E-Commerce product?
There are a number of solutions on the market which allow automation of the service virtualization process. Some commercial API testing tools, like SmartBear’s SoapUI Pro, include utilities to support virtual Web, REST or SOAP services. Others, like WireMock, are standalone and open-source.
Most available tools are based on a two-step record-and-playback methodology, using JSON and/or XML message structures to put the headers necessary for the tool to manage the original information. For the following examples, WireMock was used, and SOAP is the API service protocol.
Step 1 – Perform the operations to be virtualized in the original test environments and record the request mappings
Using our sample E-Commerce Web portal, let’s search for orders existing for Account 12345 for the last two weeks of November 2018. During the operation, WireMock recorded this JSON-formatted message request to SAP:
{
“id” : “eaf9f657-78aa-31eb-bd8e-c54c6a0dedaa”,
“request” : {
“url” : “/sap/bc/srt/xip/sap/ZgetOrders?sap-client=100”,
“method” : “POST”,
“bodyPatterns” : [ {
“equalToXml” : “<env:Envelope xmlns:env=’http://schemas.xmlsoap.org/soap/envelope/’>
<env:Header></env:Header>
<env:Body>
<ns2:OrderSearchRequestMsg
<FromDate>2018-11-16</FromDate>
<ToDate>2018-11-30</ToDate>
<OrderStatusCats xmlns:xsi=\”http://www.w3.org/2001/XMLSchema-instance\” xmlns:xs=\”http://www.w3.org/2001/XMLSchema\” xsi:type=\”xs:string\”>INPROGRESS</OrderStatusCats>
<OrderTypes xmlns:xsi=\”http://www.w3.org/2001/XMLSchema-instance\” xmlns:xs=\”http://www.w3.org/2001/XMLSchema\” xsi:type=\”xs:string\”>STANDARD</OrderTypes>
<Accounts>
<Account>
<AccountNumber>12345</AccountNumber>
</Account>
</Accounts>
</ns2:OrderSearchRequestMsg></env:Body></env:Envelope>”
} ]
},
“response” : {
“status” : 200,
“bodyFileName” : “body-sap-ZgetOrders-5OCpW.xml”,
“headers” : {
“set-cookie” : “sap-usercontext=sap-client=100; path=/”,
“content-type” : “text/xml; charset=utf-8”,
“accept” : “text/xml”,
“sap-srt_id” : “20181215/213824/v1.00_final_6.40/005056AF37C81EE8BAAA22435E8C7FFA”,
“sap-srt_server_info” : “TST_100,26 ,urn:sap-com:soap:xms:application:xip,ZII_GET_ORDERS,OrderSearchRequestMsg,14”,
“sap-srt_server_info_ext” : “0”,
“sap-perf-fesrec” : “30467.000000”
}
},
“uuid” : “eaf9f657-78aa-31eb-bd8e-c54c6a0dedaa”
}
As mentioned, much of this is SOAP message structure designed for either WireMock to track the messages, or internal to the structure, for SAP to understand which schema is being used. We do, see, however, four important pieces of information:
1. The “equalToXml” tag means when playing back, WireMock will look to match this entire message string exactly.
2. The FromDate and ToDate fields show our specified search range of the last two weeks of November 2018.
3. The AccountNumber is 12345.
4. The bodyFileName specifies the name of the XML file which contains the response data (more on that in Step 2).
Now, there is a problem to solve. Our search range is hardcoded to be the last two weeks of November 2018. To be flexible, this should be able to accept any range of FromDate and ToDate, to work well into the future. If we’re testing our E-Commerce application, we really don’t care all that much which orders were created when in SAP and when. The important thing is to have orders come back and be displayed with a date which fits into the range, even if we run this test years into the future.
Fortunately, the service virtualization tools have thought about this problem already, and they all support standard Regular Expression structures in the request mapping messages. In this case, a date Regular Expression takes the format of “[0-9]{4}-[0-9]{2}-[0-9]{2}”…four digits, a dash, two digits, another dash, and finally two more digits. In WireMock, when you use a Regular Expression in a request mapping, you change the “equalToXml” keyword to “matches”. The body of our request mapping moving forward looks like this:
“bodyPatterns” : [ {
“matches” : “<env:Envelope xmlns:env=’http://schemas.xmlsoap.org/soap/envelope/’>
<env:Header></env:Header>
<env:Body>
<ns2:OrderSearchRequestMsg
<FromDate>([0-9]{4}-[0-9]{2}-[0-9]{2})</FromDate>
<ToDate>>([0-9]{4}-[0-9]{2}-[0-9]{2})</ToDate>
<OrderStatusCats xmlns:xsi=\”http://www.w3.org/2001/XMLSchema-instance\” xmlns:xs=\”http://www.w3.org/2001/XMLSchema\” xsi:type=\”xs:string\”>INPROGRESS</OrderStatusCats>
<OrderTypes xmlns:xsi=\”http://www.w3.org/2001/XMLSchema-instance\” xmlns:xs=\”http://www.w3.org/2001/XMLSchema\” xsi:type=\”xs:string\”>STANDARD</OrderTypes>
<Accounts>
<Account>
<AccountNumber>12345</AccountNumber>
</Account>
</Accounts>
</ns2:OrderSearchRequestMsg></env:Body></env:Envelope>”
} ]
Step 2 – Capture responses from the original test environments and update as necessary
For each JSON request mapping, there is an XML response file generated. In the case of our Get Orders query, the original response message contained in the “5OCpW.xml” file referenced by the request mapping looks like this:
<soap-env:Envelope xmlns:soap-env=”http://schemas.xmlsoap.org/soap/envelope/”>
<soap-env:Header/><soap-env:Body>
<n0:OrderSearchResponseMsg xmlns:prx=”urn:sap.com:proxy:TST:/1SAI/TAS00000000000000000070:740″>
<OrderResult>
<PONumber>Test_1542121016597</PONumber>
<OrderNumber>0703906065</OrderNumber>
<OrderDate>2018-11-30</OrderDate>
<TotalPrice>231.58</TotalPrice>
<SoldToInfo>
<SoldToNumber>12345</SoldToNumber>
<SoldToName>Sample Test</SoldToName>
<Address>
<Name>SPR Consulting</Name>
<Street>233 S. Wacker Dr</Street>
<City>Chicago</City>
<District>Cook</District>
<Region>IL</Region>
<PostalCode>60606</PostalCode>
<Country>US</Country>
</Address>
</SoldToInfo>
</OrderResult>
</n0:OrderSearchResponseMsg>
</soap-env:Body>
</soap-env:Envelope>
Great, we have an order that matches our query! But there is still a change to make. The hardcoded date needs to be updated so that it always falls within the range for our search. The virtualization tools provide ways to do that too, by referencing parameter values from the request in the response. In WireMock, these are marked by “{{ }}”, also called “handlebars”. We replace the “2018-11-30” value with a handlebar reference to the “ToDate” field from the input, so that any value put into the date range in the order request is dynamically put into the order response.
<OrderResult>
<PONumber>Test_1542121016597</PONumber>
<OrderNumber>0703906065</OrderNumber>
<OrderDate>{{soapXPath request.body ‘/OrderSearchRequestMsg/ToDate/text()’}}</OrderDate>
<TotalPrice>231.58</TotalPrice>
<SoldToInfo>
<SoldToNumber>12345</SoldToNumber>
<SoldToName>Sample Test</SoldToName>
<Address>
<Name>SPR Consulting</Name>
<Street>233 S. Wacker Dr</Street>
<City>Chicago</City>
<District>Cook</District>
<Region>IL</Region>
<PostalCode>60606</PostalCode>
<Country>US</Country>
</Address>
</SoldToInfo>
</OrderResult>
Making these modifications for the request mappings and response messages may seem like a lot of work. But if you identify all of the fields which can use Regular Expressions instead of hardcoded values, and identify which sets of response data can be reused, you may be able to support an entire test environment with a relatively small number of virtualized services. The initial overhead will be worth it to have stable on-demand test environments moving forward.