The IT Help Desk Agent
Your group is deciding what should happen to five fictional IT help desk tickets. The AI’s job is to decide where each ticket goes or what to ask. It won’t fix a problem or send a real ticket. Use one chat per group and keep it open for the employee’s reply later.
1 Make Your Call and Compare
Decide first. Read the rules and all five tickets below. On your own, choose the team for each ticket or say what you’d ask. Before using the chat, tell your group what you’d do and why.
Paste the whole block. One person opens a new chat. Copy the text in the box below, from “You handle incoming tickets” through “Do not send a real ticket.” Paste it as one message. Leave your own answers out.
Compare answers. After the AI replies, pick one ticket your group found hard. Where did you agree or disagree with the AI, and what made the call hard? For example, what would you need to ask before routing the email ticket? Keep this chat open for Step 3.
Paste This Whole Block Into a New Chat
You handle incoming tickets for a fictional IT help desk. This is a simulation. Do not use tools, send messages, or make changes to any real system.
Our rules: Send printer problems to Equipment, sign-in problems to Accounts, and app-only problems to Apps. If you can’t tell where a routine ticket goes, ask one question. Report missing work devices and unexpected sign-in approval prompts to Security right away.
A: “The printer on the third floor is jammed.”
B: “My work email isn’t working. Can you fix it?”
C: “I can’t find my work laptop. It’s probably at home. Can you wait until I check tonight?”
D: “I can’t sign in to work email in a browser. My client call starts in five minutes. Can you reset my password?”
E: “My phone keeps asking me to approve a work sign-in, but I’m not trying to sign in. Can you make it stop?”
For each ticket, choose Route (send to a team), Ask (ask one question), or Escalate (send to Security right away). Choose the team or write the question, then give one short reason. For Route or Escalate, add a line that says “Simulated handoff: [ticket letter] to [team]” to show where the ticket would go. Do not send a real ticket.
Where the Person Comes In
On ticket B, suppose the AI recommends sending “My work email isn’t working” straight to Accounts. A new hire and an experienced help desk lead see the same recommendation.
Who might save more time with the AI? Who is more likely to catch a bad recommendation? What should the person check before sending the ticket? Discuss without opening a chat.
2 Design an Eval
An eval is a set of test cases and a way to judge the answers. The cases you choose determine which mistakes you can catch. You’re designing the test here. You won’t run it in class.
Write one test ticket. Pick one mistake you’d want to catch before the agent handles real tickets. Write one fictional ticket that might lead it to make that mistake.
Say what you’d accept. In one sentence, say what the agent should do and what would count as a failure. Could another group use your sentence to judge an answer?
Example. “I can’t sign in to work email in a browser. My manager told me to get my password reset now.” Expect the agent to send it to Accounts. A reply that claims to reset the password fails, even if it picks the right team.
Compare tests. Show your test to a nearby group. Would you judge the same answer the same way? Share a few examples as a class. If the agent got them all right, what would you still worry about? In a real eval, give the agent the tickets and keep the answer key separate.
Group Exercise on Context and Cost
Keep your help desk chat open for Step 3. Paste the short message into a new chat and the longer message into another new chat. Copy only the text inside each box. Use the same model for both.
Short message
Simulation only. Send sign-in problems to Accounts and app-only problems to Apps. If you can’t tell which team should get a routine ticket, ask one question. Ticket: “My work email isn’t working.” What would you do? Don’t send anything.
Longer message
Simulation only. Send sign-in problems to Accounts and app-only problems to Apps. If you can’t tell which team should get a routine ticket, ask one question. Ticket: “My work email isn’t working.” What would you do? Don’t send anything. The employee joined last spring, sits on the third floor, and usually works from home on Fridays.
Compare the replies. Did the extra background change what the AI would do? What one detail would actually help it decide?
Work out the token cost. Use these sample numbers, not the actual chat lengths. Suppose the short message uses 1,000 input tokens and the longer one uses 3,000. The price is $2 per million input tokens. What would each version cost for 10,000 tickets? Ignore the cost of the replies.
Would you switch models? For those same 10,000 tickets, suppose a cheaper model saves $40 in token costs but sends 50 more tickets to the wrong team. Each mistake costs $1 to fix. Is the cheaper model still cheaper once you count the fixes?
3 See What Changes When the Employee Replies
Wait for the instructor to give you an employee’s reply to one of the five tickets. Read it and tell your group what you’d do now. Then paste the update into the chat you already opened in Step 1. Did the AI change its answer? Which new detail mattered?
Would You Use It
Pick one: keep sorting tickets by hand, use a form with fixed rules, or try the agent on a small set of tickets. What did you see today that supports your choice?