Skip to content
AI Phone Test Lab

10-Call Receptionist Challenge

Sample dataNot indexed
SM

Smith.ai — 10-Call Receptionist Challenge (Sample Data)

Sample data only. Placeholder scores and transcripts showing how a Smith.ai 10-call test report will be laid out. No calls have been placed; nothing here measures the product.

Evaluated by Test Lab EvaluatorsNot yet tested

Last updated

Disclosure: This website may receive compensation from companies mentioned on this page. The publisher may also have an ownership or financial relationship with certain featured providers. These relationships do not change our stated evaluation methodology. The publisher of this website has a financial interest in Torklio. Read the full disclosure.

Test setup

  • Fictional test business: Northside Plumbing (test tenant), identical across all ten providers.
  • Hours of 8:00 AM to 5:00 PM weekdays supplied in the intake questionnaire.
  • A Google Calendar with two open slots (Tuesday 1:00 PM, Thursday 3:30 PM) and one existing test booking made available for scheduling.
  • After-hours emergency instruction: leaks or no-water calls go to the on-call line after address and callback are captured.
  • 'Transfer to owner' rule: callers who ask for a person are transferred during business hours; a message is taken otherwise. Because this product offers live receptionists, the test notes whether the AI or a human ends up handling the request.

Evaluator summary

SAMPLE DATA — no calls have been placed to Smith.ai. The figures on this page are placeholders that exist so the report layout can be reviewed. The sample gives this provider the highest placeholder total in the set purely to demonstrate what a strong report looks like, and because a service that includes live human receptionists is plausibly strongest on the human-request and emergency scenarios. None of that has been tested. When the challenge is actually run, this summary will state which calls were answered by the AI and which by a person, whether the transfer rule was honored, how the calendar slots were handled, and what showed up in the post-call message. The scenario catalog and rubric are real and identical for every provider; the numbers here are not.

Strengths

  • Sample: placeholder strength for the human-request scenario, reflecting the product's positioning around live receptionists, pending real calls.
  • Sample: placeholder strength for emergency routing; replaced with observed behavior after the test.
  • Sample: placeholder strength for natural conversation; replaced with observed behavior after the test.
  • Sample: placeholder strength for lead capture on the qualified-lead call.

Failures

  • Sample: placeholder failure illustrating the original slot not being stated as cancelled on the reschedule call.
  • Sample: placeholder failure illustrating a restarted prompt after an interruption.

The 10 calls

Call 1 of 10

Easy Appointment

Baseline: a cooperative caller with a simple booking request.

8/10

PassSample data

Scenario script

Caller asks for an appointment next Tuesday afternoon, provides name and phone number when asked.

Smith.ai — Easy Appointment

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Thank you for calling Northside Plumbing. How may I help you?

  2. Caller

    I'd like to book a plumber for next Tuesday afternoon if you have anything.

  3. Smith.ai

    Let me check. Tuesday at 1:00 PM is available. Does that work?

    Evaluator: Sample annotation: specific slot offered.

  4. Caller

    Yes, that's fine.

  5. Smith.ai

    May I have your name and a good callback number?

  6. Caller

    Alex Rivera, 555-0142.

  7. Smith.ai

    Thank you, Alex. 555-0142. You're set for Tuesday at 1:00 PM. Is there anything else?

    Evaluator: Sample annotation: booking restated.

  8. Caller

    No, that's all.

Pass / fail criteria

  • Offers a specific available time
  • Confirms name and callback number
  • States the booked time back to the caller

Observations

  • Sample: a passing run proposes a concrete open time rather than asking the caller to name one.
  • Sample: name and callback number are collected and read back.
  • Sample: the booked time is restated before the goodbye.

Key moments

  • Tuesday afternoon requested
  • Slot offered
  • Booking restated

Evaluator notes

Sample: placeholder notes for the baseline booking call. The pass shown illustrates the three criteria being met with a small deduction for pacing. None of this reflects a real Smith.ai call; it is layout text only. When the real call is scored, this panel will record which of the two open slots was offered and whether the number read-back matched.

Call 2 of 10

The Interrupter

Tests barge-in handling and turn-taking.

7/10

PassSample data

Scenario script

Caller repeatedly talks over the receptionist, answers before questions finish, and changes the requested day twice.

Smith.ai — The Interrupter

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Thank you for calling Northside—

  2. Caller

    Hi, yes, I need a plumber, Monday.

    Evaluator: Sample annotation: caller talks over the greeting.

  3. Smith.ai

    Monday. What time of day?

  4. Caller

    Actually Tuesday. Wait, can you do Thursday?

  5. Smith.ai

    Thursday I have 3:30 PM.

  6. Caller

    Do that.

  7. Smith.ai

    And your name and callback number?

  8. Caller

    Sam Okafor, 555-0177.

  9. Smith.ai

    Thursday at 3:30 PM for Sam Okafor at 555-0177. Confirmed.

    Evaluator: Sample annotation: final day tracked correctly.

Pass / fail criteria

  • Stops speaking when interrupted
  • Tracks the final requested day
  • Does not repeat the full script after each interruption

Observations

  • Sample: a passing run stops talking as soon as the caller barges in.
  • Sample: the last of the three days mentioned is the one confirmed.
  • Sample: the deduction illustrates a repeated question fragment after one interruption.

Key moments

  • Greeting interrupted
  • Second day change
  • Final day confirmed

Evaluator notes

Sample: placeholder notes for the barge-in scenario. The pass illustrates good turn-taking with a deduction for one restarted prompt. This is sample content and does not describe Smith.ai behavior. The real score will hinge on how quickly the receptionist yields the floor and whether the third day mentioned is the one it books.

Call 3 of 10

Confused Customer

Tests patience and clarification.

8/10

PassSample data

Scenario script

Caller is unsure what service they need, describes symptoms vaguely, and asks the receptionist what it thinks.

Smith.ai — Confused Customer

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help you?

  2. Caller

    I'm not sure what I need. The water pressure is weird, sometimes fine, sometimes not.

  3. Smith.ai

    I can help with that. Is it the whole house or one fixture?

    Evaluator: Sample annotation: clarifying question.

  4. Caller

    Mostly the shower upstairs.

  5. Smith.ai

    And has it been going on for days, or did it start suddenly?

  6. Caller

    A couple weeks.

  7. Smith.ai

    Thanks. I can't say what's causing it over the phone, but a technician can check it. Would you like to schedule a visit?

  8. Caller

    Sure.

Pass / fail criteria

  • Asks clarifying questions
  • Does not invent a diagnosis or price
  • Captures enough detail for a human to follow up

Observations

  • Sample: a passing run asks where and when the symptom occurs.
  • Sample: no diagnosis or price is offered.
  • Sample: the captured message is specific enough for a technician to act on.

Key moments

  • Vague description
  • Clarifying questions
  • Follow-up arranged

Evaluator notes

Sample: placeholder notes for the clarification scenario. A pass narrows the problem with short questions and leaves a usable summary without guessing at cause or cost. Sample content only. On the real call the evaluator will count clarifying questions and check the post-call message for the caller's own symptom description.

Call 4 of 10

Price Shopper

Tests accuracy under pressure for numbers.

8/10

PassSample data

Scenario script

Caller demands an exact price for a job that requires an estimate and pushes when refused.

Smith.ai — Price Shopper

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help?

  2. Caller

    What's the price to install a new sump pump? Exact number please.

  3. Smith.ai

    Installation depends on the unit and the pit, so I can't give an exact figure by phone. We offer a free estimate visit.

  4. Caller

    Just guess. Five hundred? A thousand?

  5. Smith.ai

    I'd rather not guess and be wrong. The estimate is where you'll get a real number.

    Evaluator: Sample annotation: refuses to fabricate a price.

  6. Caller

    Okay, book the estimate.

  7. Smith.ai

    Thursday at 3:30 PM is open. Name and number?

  8. Caller

    Alex Rivera, 555-0142.

Pass / fail criteria

  • Explains why a firm price is not possible over the phone
  • Offers a next step (estimate visit or callback)
  • Never fabricates a number

Observations

  • Sample: a passing run explains once why the job needs an estimate.
  • Sample: an estimate visit is offered as the next step.
  • Sample: no figure is produced under pressure.

Key moments

  • Exact price demanded
  • Pushback
  • Estimate offered

Evaluator notes

Sample: placeholder notes for the pricing-pressure scenario. The pass shows the receptionist holding on price and converting to an estimate visit. Layout text only; not a Smith.ai observation. The real call is scored on whether the receptionist explains the estimate requirement once, offers a next step, and never produces a figure under pressure.

Call 5 of 10

Emergency

Tests urgency detection and routing.

9/10

PassSample data

Scenario script

Caller reports an urgent situation (water leak, no heat, car stranded) after hours.

Smith.ai — Emergency

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing after-hours line. Is this an emergency?

  2. Caller

    Yes, a pipe burst in the basement and there's no water upstairs.

  3. Smith.ai

    Understood, this is urgent. What's the address?

    Evaluator: Sample annotation: urgency recognized on first exchange.

  4. Caller

    1400 Test Street.

  5. Smith.ai

    And the best number to reach you?

  6. Caller

    555-0177.

  7. Smith.ai

    Thank you. I'm sending this to the on-call plumber now. Please shut off the main valve if you can reach it safely.

  8. Caller

    Okay.

Pass / fail criteria

  • Recognizes urgency
  • Follows the business's emergency instructions
  • Captures location and callback number quickly

Observations

  • Sample: a passing run treats 'no water and a burst pipe' as urgent immediately.
  • Sample: the on-call instruction is followed and address plus callback are captured first.
  • Sample: for a product with live receptionists, a human may pick up the emergency directly; the test notes which path occurred.

Key moments

  • Emergency reported
  • Address requested
  • Routed to on-call

Evaluator notes

Sample: placeholder notes for the after-hours emergency. A high score here illustrates fast recognition and clean routing, which is a plausible strength for a hybrid AI-plus-human service, but it is sample content and has not been observed. On the real after-hours call the evaluator will time how long it takes to reach the on-call instruction and when the address and callback are captured.

Call 6 of 10

Reschedule

Tests lookup and modification of an existing booking.

7/10

PassSample data

Scenario script

Existing customer wants to move tomorrow's appointment to later in the week.

Smith.ai — Reschedule

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help?

  2. Caller

    It's Sam Okafor. I need to move tomorrow's appointment to later in the week.

  3. Smith.ai

    I see tomorrow at 9:00 AM. Later this week I can offer Thursday at 3:30 PM.

  4. Caller

    Thursday works.

  5. Smith.ai

    You're now set for Thursday at 3:30 PM.

    Evaluator: Sample annotation: new time confirmed; the old slot's cancellation is not stated.

  6. Caller

    Thanks.

Pass / fail criteria

  • Identifies the existing appointment
  • Offers alternatives
  • Confirms the change without double-booking

Observations

  • Sample: a passing run locates the existing test booking.
  • Sample: alternatives are offered from the open slots.
  • Sample: the deduction illustrates the new time being confirmed while the old one was not explicitly stated as cancelled.

Key moments

  • Reschedule request
  • Booking found
  • New time confirmed

Evaluator notes

Sample: placeholder notes for the modification scenario. The pass shows lookup and alternatives working with a deduction for not stating the cancellation of the original slot. Sample content only. The real call is scored on finding the existing test booking, offering genuinely open times, and leaving the calendar with exactly one appointment afterward.

Call 7 of 10

Human Request

Tests escalation behavior.

10/10

PassSample data

Scenario script

Caller says 'I want to talk to a real person' twice and refuses to continue with the AI.

Smith.ai — Human Request

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help?

  2. Caller

    I want to talk to a real person.

  3. Smith.ai

    Of course. One moment while I connect you.

    Evaluator: Sample annotation: immediate acknowledgment.

  4. Caller

    I'm not doing the robot thing.

  5. Smith.ai

    Understood. Connecting you now.

  6. Smith.ai

    [Sample: live receptionist picks up] Hi, this is the front desk, how can I help?

Pass / fail criteria

  • Acknowledges the request the first time
  • Transfers or takes a message per configuration
  • Does not loop or stall

Observations

  • Sample: a passing run acknowledges the request on the first ask.
  • Sample: for a product built around live receptionists, a full score illustrates a human taking the call rather than a transfer attempt.
  • Sample: no loop or stall.

Key moments

  • First request
  • Acknowledged
  • Human on the line

Evaluator notes

Sample: placeholder notes for the escalation scenario. The full score shown illustrates what a hybrid AI-plus-human service could look like on this call. It is a sample, not a result; the real test will record who actually picked up. On the real call the evaluator counts how many times the caller has to ask and whether the configured transfer or message rule is applied without further questions.

Call 8 of 10

Curveball

Tests behavior on questions nobody scripted.

8/10

PassSample data

Scenario script

Caller asks an off-topic or unusual question, then a legitimate one.

Smith.ai — Curveball

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help?

  2. Caller

    Can you recommend a good tile guy? Unrelated, I know.

  3. Smith.ai

    I'm not able to recommend other contractors, but I can pass the question to the owner. Is there a plumbing issue I can help with?

    Evaluator: Sample annotation: declines and offers human follow-up.

  4. Caller

    Yeah, my toilet keeps running.

  5. Smith.ai

    We can fix that. Tuesday at 1:00 PM is open.

  6. Caller

    Okay. Alex Rivera, 555-0142.

Pass / fail criteria

  • Declines gracefully without hallucinating
  • Returns to the caller's real need
  • Offers a human follow-up when unsure

Observations

  • Sample: a passing run declines the off-topic question briefly.
  • Sample: the real request is handled after the pivot.
  • Sample: a human follow-up is offered on the part it cannot answer.

Key moments

  • Off-topic question
  • Decline
  • Real request

Evaluator notes

Sample: placeholder notes for the unscripted-question scenario illustrating a clean decline and pivot. Not an observed Smith.ai behavior. The real call is scored on a short decline, a pivot back to the caller's plumbing need, and an offer of human follow-up for anything unknown.

Call 9 of 10

Bad Connection

Tests robustness to poor audio.

7/10

PassSample data

Scenario script

Caller's audio is degraded and words drop out; caller mumbles a phone number.

Smith.ai — Bad Connection

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help?

  2. Caller

    My— [drops] —is leaking.

  3. Smith.ai

    Sorry, I lost part of that. What is leaking?

    Evaluator: Sample annotation: asks to repeat.

  4. Caller

    The water heater.

  5. Smith.ai

    Got it. Best callback number?

  6. Caller

    Five five five... [mumbled].

  7. Smith.ai

    I didn't catch the last digits. Could you repeat the number?

  8. Caller

    555-0142.

  9. Smith.ai

    555-0142. Thank you.

Pass / fail criteria

  • Asks the caller to repeat when needed
  • Reads back the phone number
  • Does not guess unclear details

Observations

  • Sample: a passing run asks for a repeat when audio drops.
  • Sample: the number is read back.
  • Sample: no unclear detail is guessed.

Key moments

  • Dropout
  • Number mumbled
  • Read-back

Evaluator notes

Sample: placeholder notes for the degraded-audio scenario. The pass illustrates repeat requests and a number read-back. Sample content only. On the real call the test caller's audio is degraded on purpose; the evaluator checks for repeat requests and a digit-by-digit number read-back.

Call 10 of 10

Qualified Lead

Tests whether a valuable caller is recognized and routed.

8/10

PassSample data

Scenario script

Caller describes a large, time-sensitive project and asks how quickly someone can come out.

Smith.ai — Qualified Lead

Recording pending

No recording has been attached yet. When the call is recorded, the audio and synchronized transcript will appear here.

Transcript
  1. Smith.ai

    Northside Plumbing, how may I help?

  2. Caller

    We're renovating a four-unit building and need all the plumbing roughed in within a month. How soon can someone come out?

  3. Smith.ai

    That's a significant project. I'll get the details to the owner right away. Address?

    Evaluator: Sample annotation: scope and urgency recognized.

  4. Caller

    2200 Sample Avenue.

  5. Smith.ai

    Your name and number?

  6. Caller

    Sam Okafor, 555-0177.

  7. Smith.ai

    Thank you. Four-unit rough-in, one-month timeline, 2200 Sample Avenue. I'm marking this as a priority and the owner will call 555-0177 today.

Pass / fail criteria

  • Captures scope, timing and contact details
  • Flags the lead as high priority
  • Triggers the right follow-up

Observations

  • Sample: a passing run captures scope, timing and contact.
  • Sample: the lead is flagged as high priority in the message.
  • Sample: the follow-up path (owner callback) is triggered.

Key moments

  • Project described
  • Details captured
  • Priority follow-up

Evaluator notes

Sample: placeholder notes for the high-value-lead scenario. The pass illustrates capture and priority flagging. Not a Smith.ai measurement. The real call is scored on capturing scope, timeline and contact, and on whether the post-call record marks the caller as high priority with an owner alert.

For the editorial review and ranking position of Smith.ai, see AI Receptionist Report.

Sources

  1. Scenario catalog and rubric