Courtesy of Mish.
For an entire semester, Georgia Tech computer science professor Ashok Goel hid the fact that his online teaching assistant “Jill Watson” was a robot.
Goel did not disclose the robot until final grades were handed out. None of the students figured it out.
Each year, students have an option of selecting an assistant and next year Goel will do the same, but names will change. Students will have to guess if they are dealing with a robot or not.
Jill Watson
Please consider Professor reveals to students that his assistant was an AI all along.
To help with his class this year, a Georgia Tech professor hired Jill Watson, a teaching assistant unlike any other in the world. Throughout the semester, she answered questions online for students, relieving the professor’s overworked teaching staff.
Students were amazed. “I feel like I am part of history because of Jill and this class!” wrote one in the class’s online forum. “Just when I wanted to nominate Jill Watson as an outstanding TA in the CIOS survey!” said another.
Goel and his teaching assistants receive more than 10,000 questions a semester from students on the course’s online forum. Sometimes the same questions are asked again and again. Last year he began to wonder if he could automate the burden of answering so many repetitive questions.
As Goel looked for a technology that could help, he settled on IBM Watson, which he had used for several other projects. Watson, an artificial intelligence system, was designed to answer questions, so it seemed like a strong fit.
To train the system to answer questions correctly, Goel fed it forum posts from the class’s previous semesters. This gave Jill an extensive background in common questions and how they should be answered.
Goel tested the system privately for months, having his teaching assistants examine whether Jill’s answers were correct. Initially the system struggled with similar questions such as “Where can I find assignment two?” and “When is assignment two due?” Goel tweaked the software, adding more layers of decision-making to it. Eventually Jill reached the point where its answers were good enough.
“I cannot create chaos in my classroom. Jill had to be almost as perfect as a human TA or I am,” Goel said.
The system is only allowed to answer questions if it calculates that it is 97 percent or more confident in its answer. Goel found that was the threshold at which he could guarantee the system was accurate.
There are many questions Jill can’t handle. Those questions were reserved for human teaching assistants.
Goodbye TAs
Jill will not eliminate the need for all TA’s just 97% of them.
