Hitting the Open Road: Test Coverage for IoT Device Integration

Turn the key, start the ignition, and … begin testing device integration? Most IT testing takes place on site or at the developer’s location. But SPR testers were making sure that a variety of IoT devices worked with a mobile app when it was, in fact, mobile. So they took to the open road to perform regression testing for logistics innovator Vypin LLC, a shipping company.

SPR was contracted by Vypin to help develop the next generation of logistics management applications around their Bluetooth-enabled IoT environment sensors. Vypin’s devices were the solution to a wide variety of logistics and regulated shipping issues, but only if they have the mobile, network hub and cloud-based data management applications to organize and report on shipments in real-time and generate alerts if any issues were detected.

The Vypin solution for shipping logistics came with four different IoT devices:

Chill – A special kind of Tag that also records environment data (temperature, humidity, etc.), to ensure that refrigeration-sensitive cargo (produce, eggs, dairy, etc.), meets the Food Safety Management Act federal government regulation.Tags – A long-life low energy Bluetooth-enabled IoT device which stores real-time GPS coordinates, placed in a shipping container, most likely a truck trailer.

MicroZone – An aggregation IoT device which detects nearby Tags via Bluetooth and flags when a Tag comes in/out of range.

Gateway – A network hub for MicroZones, sending aggregated Tag data to the Azure cloud server.

When a Tag and its cargo are en route, a mobile app for iOS or Android phones acts as the MicroZone for that specific Tag, transmitting data to the cloud and generating system alerts if the parameters go out of range (temperature fluctuation, truck deviates from the designated GPS coordinates for the route, etc.).

Since Vypin was building new, leading-edge hardware solutions, each of these IoT devices was on its own firmware version for processing algorithms, messaging formats, etc. The SPR developers writing the applications to communicate with the devices were faced with the daunting challenges of making sure the application software kept in sync. Similarly, the SPR testers needed to track latest versions, versions in our device inventory for backward compatibility support, and connectivity with the mobile application on different iOS and Android phones and operating system versions.

As the testers were performing regression testing at the end of each sprint, IoT device versions were accounted for when planning the regression repository and individual test runs in Jira Zephyr. A new firmware version meant thorough regression test coverage. For backward compatibility, or if an IoT device was on the same firmware as the previous application release, the testers planned for partial regression around key functions and data validation.

Here’s where the real fun began…. Since these physical devices are meant to work in motion, collecting real-world data, SPR testers left the lab as part of regression testing. The team installed the mobile application on their phones and took the Tags out of range of the MicroZones. Pickup times and “routes” were programmed to match a team member’s commute. Chill Tag environment rules were set for temperature ranges, and when a sensor was placed in the sun or the freezer, expected real-time alerts were checked.

The insurmountable challenge the team faced was validating a system that could support thousands of Tags reporting data, but only a handful of the IoT devices were available for SPR’s use. The team circumvented this issue with a mock device data server. This custom application would mimic a specified count of different devices with a data stream to the available Gateway or directly to the Azure server. The mock devices could also be told which firmware version to simulate, meaning that backward-compatible message formats could be included in test scenarios. The mock device server was used not only for regression testing, but to generate a volume of messages sufficient to drive load and performance stress.

When Vypin released their firmware for User Acceptance and beta trials in the field, the techniques utilized by the testers had thoroughly vetted a wide range of different usage scenarios. Very few issues were found when in the hands of potential customers, which is always the end goal of any testing effort.