Experimenting on promising projects
This photo is metaphorical; we are not actually scientists. Photo by Hush Naidoo on Unsplash.
Challenge
In the Office of School Performance at the NYC Department of Education, our team’s traditional approach to producing new data tools for educators was to commit significant team capacity to fully design and develop a new product based on a promising idea. In some cases, this approach worked well, resulting in successful releases; but in other cases, it was unclear whether the end product truly met user needs and advanced our mission of helping educators improve experiences and outcomes for students.
How might we reduce the risk of investing significant resources into products that don’t meet user needs and increase our chances of releasing impactful products?
Process
We decided to identify and test promising project ideas using human-centered design and lean-startup principles. We held workshops where team members reviewed user research (collected from observations and interviews with educators and administrators), identified user needs, generated solution ideas, and assessed their expected impact and effort. We surveyed field staff who work with schools about the expected impact of the solution ideas and identified a short list of promising projects to explore further.
We then established small project teams that articulated the value hypothesis, identified key assumptions, created prototypes, tested assumptions, and summarized their findings. This process allowed the team to gain valuable information about project ideas with modest investments of time and effort rather than the traditional process of fully pursuing promising but untested ideas.
Outcome
We gathered valuable information about our solution ideas. For example, we created and tested a prototype for a web report that showed teachers how their students performed on different academic standards. Based on our findings, this product is a strong candidate for future development. Of equal importance, we identified solution ideas that we should not pursue because key assumptions were not met.
My Role
I helped plan and facilitate the team workshops where team members reviewed user research, identified user needs, generated solution ideas, and evaluated them based on impact and effort. I synthesized the notes from these workshops into actionable summary documents. I led a working group that identified the short list of projects to explore. I established small project teams that articulated the value hypothesis, identified key assumptions, created prototypes, tested assumptions, and summarized their findings.
I also participated as a member of the small project teams that explored the new tool for teachers about student performance on academic standards. On that team, I helped articulate the value hypothesis, identify key assumptions, develop visualizations for the prototype, gather information about the prototype from teachers, synthesize results from the design research, and present a summary of our findings.
Project Artifacts
User needs summary
Summary of user needs identified during team design workshop that involved reviewing and analyzing user-research notes.
Problem statements, solution ideas, and user stories
Synthesis document that captures ideas and insights generated from team design workshops.
Impact and effort of potential projects
Visualization created by a team member to show estimates of the impact and effort of different solution ideas. The vertical axis reflects the expected impact of the product idea on improving student experiences and outcomes. The vertical lines show ranges of impact estimates and reflect uncertainty. The horizontal axis reflects the expected amount of effort needed to implement the solution idea.
Lean Enterprise concepts
Key ideas from Lean Enterprise that we discussed with the team during a workshop.
Business model canvas example
An example of a business model canvas that I presented to the team during a workshop.
Business model canvas
Business model canvas created by a small project team exploring the idea of creating videos to explain our reports to users.
Sharing findings from testing
Slide from presentation by a small project team that explored creating a web report that showed teachers how their students did on different academic standards.