Setting the Stage: Testing a New Online Marketplace for Entertainers

Need a keynote speaker? How about an emcee? Or maybe a comedian to get an event audience warmed up? Finding experienced, reliable people with these talents is the job of event coordinators and agencies. But how do THEY find the right performers – and keep a flow of fresh new talent coming through the doors?

BookedOut by Shiftgig developed a virtual marketplace to connect performers to the coordinators and agencies putting on events. Much like a job board, agencies post upcoming events and performers can upload profile information and apply to work at the events they are interested in. Agencies can score, evaluate and rate the talent, plus negotiate with the contractor to work at the event.

Since there is no way to know what equipment each contractor would use, it was critical that SPR provide testing services for the BookOut app on a wide array of platforms, devices, and OS versions using physical devices.

Testing of all functions, all devices, and all builds would have been impossible to handle at the desired pace of the project. Instead, the BookedOut project team agreed upon a set of 10 iOS and Android phones and tablets that were a cross-section of the most popular devices on the market. The project team also agreed on which platform functions were in or out of scope for testing.

Since not every feature could be tested on every device, SPR took a risk-based approach to testing, one that was heavy on feature awareness and planning. The team had visibility to all bugs and application additions/changes being made with each new released build. This visibility allowed the team to make informed decisions about which tests to include for each feature and which devices to include in scope, successfully managing the project risk normally associated with not having 100% coverage in each build.

As builds were identified as possible release candidates, SPR made the decision to further mitigate risk by moving from a partial-regression approach to full regression. Running all test cases on all platforms provided the highest confidence level to ensure the viability of a build being ready for production release.

SPR was responsible for testing the back-end administration system as well as the front-end agency and contractor portals. The back-end system had its own functionality and requirements to be derived, tested, and maintained for traceability. Since the data generated by this administration system fed directly into the front-end portals (creating new clients, jobs, events, etc.), SPR’s strategy had to account for these two simultaneous testing efforts.

The testing team took careful planning steps for traceability not only between the requirements and tests for the back-end and front-end systems, but between the back-end and front-end tests as well. Coordinating the test dependencies meant that the outcome of administration tests would feed into the front-end portal tests as a “just in time” data management flow.

The SPR team used Microsoft Team Foundation Server (TFS), with the associated Test Manager plug-in, to coordinate all testing activity. Each device/OS combination was represented by a separate Configuration value within TFS, so tests could be run multiple times on multiple devices. Runs and results filtered and organized by Configuration, allowed for visibility and reporting at the most granular levels to determine: Is there a model of phone or tablet which fails more tests than others? For both the Agency and Contractor portals, is the iOS or Android application version of higher quality? Microsoft TFS was also used as the defect repository, providing similar visibility around bug creation and closure as with execution metrics.

By the end of the BookedOut project, SPR had provided the client with test results and a test repository that reflected the application’s quality. This in turn gave the client a high degree of comfort that the agencies and contractors could find each other, no matter which kind of device was in their pocket.