Building and deploying an Azure application security testing service
Snapshot
The main research team at Microsoft has traditionally been responsible for publishing white papers, and is generally staffed with PhDs, with an atmosphere similar to a college institution. The challenge for Microsoft was how to transfer the knowledge and insights gained by these researchers into products that have life in the marketplace. The problem was addressed through the Microsoft New Experiences and Technologies (NExT) initiative.
To earn a NExT project, a team makes a ‘pitch’ for what they want to create and are then given a certain level of support, much like how a startup might get started. The research team presented their unique fuzz testing technology for finding security critical bugs in software, and proposed to provide it as a SaaS offering.
If the technology was going to move from an internal offering to one that could on-board external customers, the team needed to address several security-related features. Their main objectives were to satisfy security requirements from their security review before going into public preview, and to re-architect their design to allow for more isolation in their fuzzing workloads.
Challenge
Kalles Group was brought in to support the team from a software development perspective. The problems that needed to be solved were clear. Their online Kanban board had Product Backlog Items (PBI) or bugs for all the work that needed to be done. As a development group, the team was tasked with working through the
set of bugs.
One early technical challenge was the need to move ETW events which were being written out to Azure storage tables from the virtual machines, through a DMZ which would then be able to write them out to storage. These events were used for logging and reporting
and so could not be ‘dropped’ on the floor. It was also important to keep the format of the events similar so the tooling around the events could stay the same if possible.
The research team presented their unique fuzz testing technology for finding security critical bugs in software, and proposed to provide it as a SaaS offering.
Approach
There were no KPIs or metrics put in place to measure success other than the set of backlog items needed to be worked through. The Kanban method was used to see how many tickets (PBIs or bugs) a developer resolves over a period of time. For this team, the information is used not as a hard performance measure for the developer, but as an indicator to see if they are going to meet their goals and self-imposed deadlines.
In this way, they can intelligently decide which tickets are higher priority, and which ones might need to be postponed until later. Also with Kanban, there is no restriction on which tickets are assigned to which developer. In practice, though, there are some tickets that are simpler and some that are more difficult and require a deeper understanding of the intricacies of the architecture.
Solutions that were implemented included:
- Moving ETW events via a DMZ
- Making sure all http headers and responses contained the appropriate security settings
- Handling initial login flow
- Annotating REST requests and responses so problems could be correlated if debugging was needed
Results
Microsoft’s fuzz testing service is an ever-evolving and growing project. The research group at Microsoft now has the capability to sell to both internal and external Microsoft enterprise customers. As they validate and integrate this service into their own unique development processes and environments, it will allow them to realize increased quality and security through each release, along with more consistency in results across ongoing efforts. Already, there is significant interest in the broader marketplace for this type of service, and the initial feedback and reception has been exciting.
Furthermore, the service is also now listed as a way for internal projects to satisfy one of their Security Development Lifecycle (SDL) requirements. Microsoft and Kalles Group worked closely to help the team take a major step towards making the service generally available across the wider organization, so other development efforts can take advantage of the scalability and increased security this service provides across a variety of unique and innovative projects.