Clarifying Questions Research

Modern generative AI operates by accepting a prompt from the user, then using that prompt to generate some form of output, such as a continuous chat, a piece of writing, visual art, or code. My dissertation research, the Clarifying Questions research project, observes that when humans are engaging in creative processes such as writing, art, or software development, there is usually a back-and-forth conversation between the writer, artist, or developer and their client prior to work beginning on the desired product. This research project seeks to show that by asking insightful clarifying questions prior to beginning work on the requested output, AI can produce superior results that are more in line with the user’s needs, and can simultaneously create a more engaging experience in which users are encouraged to think deeply about what they need and work in tandem with the AI to produce it, rather than offloading all creative power to the AI.

Proposal

This work will focus specifically in the area of generating short professional documents. The motivations for this limitation are:

1. To keep the initial research sufficiently focused
2. To focus on an area which has immediate potential benefits
3. Readily available AI APIs are well-suited to the task of creating short documents

The original proposal can be found here.

Note that research has expanded somewhat from the original proposal. See the “Full Study” section below for details.

Pilot Study

A pilot study was conducted based on the initial proposal. Key findings for the proposal were used to refine the full study, which is now in progress.

A report on the results of the pilot study can be found here.

Although data is no longer being collected for the pilot study, you can still find the study here: https://bjtix.github.io/ClarifyingQuestions/PilotApp/PilotApp.html

Full Study

Key takeaways from the Pilot study included technical fixes and interface modifications for a smoother user experience, but also led to several key design changes in the full study:

• The full study includes 3 different LLMs, the pilot included just one.
• The full study gives the user the opportunity to refine their results, which was a highly requested feature.
• The full study asks users to indicate a relative preference between two documents, rather than attempting to rate each document on an objective scale. Users in the pilot study expressed that it was difficult to consistently rate documents in an objective way.

The full study can be found here: https://bjtix.github.io/ClarifyingQuestions/FullStudy/FullStudy.html