If developers are unable work with your software, they will never become customers. Developer testing helps provide insight for improving developer experience. It will increase trial rates, and delight developers with your product, and increase sales.

Haxor conducts developer experience studies to audit your developer on-boarding. We evaluate the quality of your documentation, content, and tutorials.

Developer Experience (DX) Testing is similar to User Experience (UX) Testing but is focused specifically on developer products. Our software records developers as they work with your product. It collects qualitative and quantitative data that can be used to improve the experience of developers working with your software.

We gather feedback from real developers in order to provide a method for quantifying the quality of the experience you currently provide. We identify gaps in documentation and identify opportunities for improvement. We then work with you to establish a plan of action.

Developer Testing

Developers are assigned a scenario to complete within a given time limit (typically one hour).

A typical scenario will take the following format:

“Build an app in language that uses features x,y, and z starting from http://yoururl.com”

As developers attempt to complete this task, our software records their behavior. Actions recorded include any activity on their screen, microphone, code changes, and application usage. The software also transcribes the developer’s voice as they narrate their thoughts along the way, providing insight into what they care about and what confuses them.

Developer Surveys

After the task is completed or the specified time runs out, developers upload their video. They then respond to a questionnaire* that asks quantitative and open-ended questions about their experience working with your product.

*We use the Computer System Usability Questionnaire (CSUQ) developed by IBM in 1995 in combination with our own open-ended questions. We can also customize this questionnaire to answer any unique questions you may have.

Scorecard & KPIs

At Haxor, we measure your DX using our Developer Experience Score (DXS). 

Your DXS is made up of 4 metrics that matter most to a successful DX. We use the following TECH metrics.

• Time to Hello World – How long does it take for developers to complete the task?
• Error Rate – How many developers report errors during the task?
• Completion Rate – How many developers succeed at the task within the given timeframe?
• Happiness – A calculated average of responses to the CSUQ.

Each metric contributes 25% to your DXS, with a perfect score being 100.

Developer Experience Audits

Using the information gathered during testing, we create a story about the experience provided to developers. We identify areas where developers became stuck or complained, and create a list of recommendations for improving your DXS.