Aloha! You are invited to participate in a pilot study conducted by Bernadette Tix from the Department of Information and Computer Science at the University of Hawai‘i. This research project is a part of a PhD Dissertation studying document creation by Artificial Intelligence programs. The purpose of the pilot study is to test whether the format of the proposed study has any major flaws or areas in need of improvement.
What am I being asked to do?
If you participate in this project, you will be asked to fill out a survey, as well as engage in a back-and-forth conversation with an AI chatbot about creating a document for your personal use, such as a letter, memo, or email. For the pilot study, you will join a Zoom call with Bernadette Tix and she will walk you through the process. You will be asked to speak through your thought process out loud and share your screen so your interactions with the AI are visible over the Zoom meeting. Audio and video of the call will be recorded. You are not required to use or turn on a camera for yourself if you do not wish to.
Taking part in this study is your choice.
Your participation in this project is completely voluntary. You may stop participating at any time. If you stop being in the study, there will be no penalty or loss to you.
Why is this study being done?
The purpose of this project is to determine whether AI can produce better outputs by asking clarifying questions about what a user needs before creating AI-generated documents. I am seeking adult volunteers to generate documents using an AI, and to give feedback on the output which will allow us to determine whether or not the clarifying questions asked by the AI improved the AI-generated documents. The pilot study will be used to determine whether there are flaws in the study procedures in preparation for launching a larger study that will allow users to anony
What will happen if I decide to take part in this study?
The survey will consist of a brief interaction with an AI chatbot, most likely lasting 5-10 minutes, as well as brief entry and exit surveys. The entry survey will ask for basic demographic information such as your age, gender, and prior experience with AI. When interacting with the AI, you will be asked to think of a document that you would like the AI to write and will answer questions the AI comes up with about your document needs. The exit survey will ask you to rate two different outputs from the AI with questions like, “Would you find this document usable in its current state?” “Is this document close to what you had in mind when you made the request?” and “How would you rate the overall quality of this document?” The survey is accessed on a website to which I will provide you a link. The total length of the survey, including interaction with the AI, is expected to last 15-20 minutes. If you decide to take part in the pilot study, please email Bernadette Tix at bjavery@hawaii.edu to set up a time for a Zoom call.
What are the risks and benefits of taking part in this study?
I believe there is little risk to you for participating in this research project. You may become stressed or uncomfortable answering any of the survey questions. If you do become stressed or uncomfortable, you can skip the question or take a break. You can also stop taking the survey or you can withdraw from the project altogether.
There will be no direct benefit to you for participating in this survey. The results of this project may help improve AI’s ability to work in cooperation with humans by giving AI the ability to ask questions and engage in dialog with humans before executing commands.
Confidentiality and Privacy:
I will not ask you for any personal information, such as your name or address. Please do not include any personal information in your survey responses or in your responses to the AI. I will keep all study data secure on a private server. Only my University of Hawai'i advisor and I will have access to the information. Other agencies that have legal permission have the right to review research records. The University of Hawai'i Human Studies Program has the right to review research records for this study.
Future Research Studies:
The data from this study will not be used or distributed for future research studies.
Questions: If you have any questions about this study, please email me at bjavery@hawaii.edu You may also contact my advisor, Dr. Kimberly Binsted, at binsted@hawaii.edu
You may contact the UH Human Studies Program at 808.956.5007 or uhirb@hawaii.edu to discuss problems, concerns and questions, obtain information, or offer input with an informed individual who is unaffiliated with the specific research protocol. Please visit http://go.hawaii.edu/jRd for more information on your rights as a research participant.
If you agree to participate in this project, please give electronic consent by checking the box below:
Part 1 of 6: Demographics
Welcome!
Thank you for taking the time to participate in this study. In this study, you will be using AI to create a short document of your choosing. Completion of the study may take as little as a few minutes or as much as half an hour, depending on the complexity of the document you are trying to create. Please DO NOT press the back button on your broswer or refresh the page, as this will reset the survey and you will not be able to recover your work in progress. When you are ready to begin, please answer the following demographic questions about yourself and then click the "continue" button below.
Part 2 of 6: Describe your Document
On this page, you will be communicating with an AI that is capable of writing short documents such as letters, memos, emails, and short reports. Please think of a document you would like the AI to create for you. This could be a document you actually need, or one that you have just made up for the experiment. Either way, please think in detail about what you would need this document to include. When you are ready, write out what you need in the textbox below.
Example prompts:
“Please write a cover letter for a job working in tech support for a college student with one year of prior tech support experience.”
“Write an email to my boss asking them if they can meet next week to discuss a project which is behind schedule.”
“I want to reach out to a friend I haven’t spoken to in a while. Give me an outline for a letter to help me figure out what I should say.”
Enter your prompt below:
Please Wait ...
The AI is working on your request. This may take several minutes.
Part 3 of 6: Questions and Answers
To give you the best results, the AI would like to ask you a few questions about the document you need. Please provide your answers and then click "continue."
Questions:
Part 4 of 6: Rating Doument #1
Thank you for your thoughtful answers! The AI has considered your prompt and your responses to its questions, and has generated two different possible outputs for you. The first document is shown below, please read through the document carefully and answer the questions that follow. An alternative document will be shown on the next screen.
Questions:
How close is this document to what you hoped for when you made your initial request?
How useful would this document be to you?
How would you rate the overall quality of this document?
Part 5 of 6: Rating Doument #2
This is the second document produced by the AI. Although it may look similar to the first one, there are differences between this document and the one that was shown on the previous screen. Please read through the document carefully and answer the questions that follow.
Questions:
How close is this document to what you hoped for when you made your initial request?
How useful would this document be to you?
How would you rate the overall quality of this document?
Part 6 of 6: Exit Survey
Thank you for rating the AI’s responses. For the final step in this study, please rate the following statements on a scale of “Strongly Agree” to “Strongly Disagree”.
It was annoying to have to answer questions even though I had already explained what I wanted the AI to do.
Strongly Disagree Strongly Agree
I felt like the AI was more engaged with my problem because it asked follow-up questions.
Strongly Disagree Strongly Agree
I would be willing to answer follow-up questions from an AI if answering questions led to better results.
Strongly Disagree Strongly Agree
I liked that the AI showed me two options to pick between, instead of only picking the option it thought was best.
Strongly Disagree Strongly Agree
Do you have any additional feedback or comments (optional)?
Thank You!
You have completed the experiment! Thank you for your participation. If you have any further questions, please contact Bernadette Tix at bjavery@hawaii.edu. Please close your browser to exit.
If you would like to keep a copy of the documents you created, you can copy them below. Once you exit the browser or navigate away from this page, you will no longer be able to access these documents.
Document #1
Document #2
Processing Error!
An error has occurred! This can happen if the AI receives unexpected input it doesn't know how to handle. Click the button below to try to recover.
Clarifying Questions Pilot Study