Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths
AI-generated Key Points
⚠The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.
- Researchers conducting systematic reviews and meta-analyses face challenges in making evidence-based recommendations to decision-makers
- Researchers must navigate a garden of forking paths, which refers to a series of analytical decision-points that have the potential to influence findings
- The authors interviewed 11 professional researchers who conduct research synthesis to inform decision-making within three organizations
- Through a qualitative analysis, the authors identified 480 analytical decisions made by researchers throughout the scientific process
- Current practices in applied research synthesis present design challenges such as making it more feasible for researchers to try different analyses and compare their results; shifting researchers' focus from rationales for decisions to impacts on outcomes; and supporting communication techniques that acknowledge decision-makers' reluctance towards uncertainty
- Opportunities exist to design systems that can help researchers explore, reason about, and communicate uncertainty in decision-making about possible analyses in research synthesis. This includes developing tools that allow for easy comparison between different analytical approaches as well as highlighting how different decisions affect results.
- Managing uncertainty in research synthesis is important and designers can support researchers in making informed decisions when navigating complex analytical landscapes.
Authors: Alex Kale, Matthew Kay, Jessica Hullman
Abstract: To make evidence-based recommendations to decision-makers, researchers conducting systematic reviews and meta-analyses must navigate a garden of forking paths: a series of analytical decision-points, each of which has the potential to influence findings. To identify challenges and opportunities related to designing systems to help researchers manage uncertainty around which of multiple analyses is best, we interviewed 11 professional researchers who conduct research synthesis to inform decision-making within three organizations. We conducted a qualitative analysis identifying 480 analytical decisions made by researchers throughout the scientific process. We present descriptions of current practices in applied research synthesis and corresponding design challenges: making it more feasible for researchers to try and compare analyses, shifting researchers' attention from rationales for decisions to impacts on results, and supporting communication techniques that acknowledge decision-makers' aversions to uncertainty. We identify opportunities to design systems which help researchers explore, reason about, and communicate uncertainty in decision-making about possible analyses in research synthesis.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
⚠The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.
Assess the quality of the AI-generated content by voting
Score: 0
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through a
tree representationLook for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.