Working together to build a conversational Verbot

[Mapestone]

... Looking again at your semantic network, I wonder if there would be a way to do this using rules (a kb) ...the building of the bot...

Right on, use a KB whenever possible - easily done with the Food Network. Synonym expressions are compact, but lack the flexibility of Rules. I'll look at online chatbots: we need a benchmark.

[MikeA]

... Looking again at your semantic network, I wonder if there would be a way to do this using rules (a kb) ...the building of the bot...

Right on, use a KB whenever possible - easily done with the Food Network. Synonym expressions are compact, but lack the flexibility of Rules. I'll look at online chatbots: we need a benchmark.

I'll set my test bot up so you can try that also. It only works with certain restricted types of inputs and subjects at the moment (the massive task ahead is enabling this for a wider range of user inputs and topics), but it might work as an example. The problem with this project is it will be very limited in the scope of actual real-life conversation, plus it is very dependent on its database. However, I hope it will be a bases for a new approach.

I'd be very happy to look to create some of this into a more standard Verbot (at least the staying on topic bit).

[Mapestone]

Thank you MikeA. I'm looking forward to working with your TestBot.
On the topic of benchmarks, I've collected a few bits as an example.
Smalltalk Benchmark
User: Hi, I'm Dan. It's nice to meet you.
Gold Response: Hello Dan, I'm BoBo. Glad to meet you too.
Survey:
Julia @ Verbots Download: I know.
Cleverbot @ cleverbot.com: Are you going to tell me your name?
Kyle @ leeds-city-guide.com: Hey, there. What do you mean exactly?
Jabberwack @ jabberwacky.com: I told you I'm a girl. So what do you think my name is?
Alice @ Alice.pandorabots.com: Hi there!. Hi, dan. It's nice meeting you too dan.
Bildgesmythe @ personalityforge.com: I am glad to meet you Dan.
Mitsuku @ mitsuku.com: Hello there. Your name is Dan. I like to meet new people.
Alan @ a-i.com/alan1: Hi. Hello Dan. Nice to meet you .
Dorian @ elzware.com: It's always nice meeting me.
Jeeney @ jeeney.com: Greetings. =)
Lauren: @ pandorabots.com: Hi there . It's good to meet you, DAN. I like to meet new people.

Even with this simple test it is easy to see the difference. Alice and Lauren take first place, with Mitsuku and Alan a close second.

[MikeA]

Nice research!

I'm curious though about your choice of winners.....

Looking at your benchmark:

User: Hi, I'm Dan. It's nice to meet you.

One issue here is this may very likely match an input directly, and call a fixed response from most bots (with the name variable thrown in).

On that basis, I'd say real 'cleverness' would come from responding with different words (as your Gold example), or ideally, even going off within the context.

Code: Select all

Gold Response: Hello Dan, I'm BoBo. Glad to meet you too.

I like the fact your Gold response has 'glad' instead of 'nice', and I like the inclusion of 'too', and also with the bot understanding the two parts of the question.

What about if the bot said this, recognising that this is a common input:

Code: Select all

Diamond Response: Bobo here, and I'm glad to meet you too. Where are you from Dan?

The difference being (other than an arbitrary change of initial response), is that the bot shows it has understood the context 'Getting to know you'.

All I want to say here is, I think when the input is common and therefore a predicatable one to have (I'm sure most bots will have - here in Verbotese - Hi * I'm [youname]. How are you * as a whole input) then it is much harder to show intelligence.

My real-life answer would probably be "And you." .......... which makes me less intelligent than ALICE (although that may be true!).

If I had to pick winners based on that input alone, mine would be Bildgesmythe or Jeeny, because I think the bots show more 'getting' of the context (that it is just an elaborate 'hello').

[Mapestone]

Wow. Incredible critique MikeA. Love your diamond response. We're on a roll. I propose we put together an IQ test with a gold/diamond/platinum standard.

My "winners" were based on a bots' ability to identify and respond to all components of the user's input. This favors ALICE-like responses.

[MikeA]

................. I propose we put together an IQ test with a gold/diamond/platinum standard.

My "winners" were based on a bots' ability to identify and respond to all components of the user's input. This favors ALICE-like responses.

The IQ test is a great idea. This would make a good starting point to how the collaborative bot could be developed.

I suspect if those inputs and reponses were shown to a lot of people, most would agree with your choices. I'm just biased through my desire to see AI (particularly the pattern matching) do something a little different. That's not to say it's better or worse.

Taking the responses below, this could be a starting point for the input that matches this type. I've changed the responses to make them a bit more generic / normal. However, Bobo is not the name I'm suggesting for the bot!

User: Hi, I'm Dan. It's nice to meet you.
Bot Gold: Hi there Dan! It's nice meeting you too. (based on Alice's 'Hi there!. Hi, dan. It's nice meeting you too dan.') / I am glad to meet you too Dan. (based on Bildgesmythe's 'I am glad to meet you Dan.')
Bot Platinum: Hi there, it's good to meet you too Dan. I like to meet new people. (based on Lauren's 'Hi there . It's good to meet you, DAN. I like to meet new people.')
Bot Diamond: Bobo here, and I'm glad to meet you too. Where are you from Dan?

Just a starting point. Feel free to change.

[MikeA]

I've set up an online Verbot.

@Mapestone, I'll send details to you.

[Mapestone]

As soon as I finish my current project (Moby Thesaurus) I'll cleanup small_talk.txt and upload it, along with some ideas on chatbot IQ scoring.

[Mapestone]

Online Chatbot Survey
Objective: Identify chatbot performance benchmark(s).

Method

“Small Talk” was selected because it is a category of conversation in which chatbots should be fluent. 161 user input questions (with several non-question exceptions) were created which fit this category.

The internet was searched for web page based, publicly accessible chatbots. 30 were identified. 21 failed to provide reasonable responses to the first 5 questions and were disqualified. The full set of 161 questions was presented to the remaining nine chatbots.

Grading

Full Credit for any reasonable response: 2 points
Partial Credit for partly reasonable: 1 point
Dodge: 0 points
Wrong Answer: -1 point

Totaling 322 points for 100% “right” answers to all of the questions.

Ranking

Mitsuku 79%
http://www.mitsuku.com/

Cleverbot 66%
http://cleverbot.com/

Jaberwacky 62%
http://www.jabberwacky.com/

Bildgesmythe 57%
http://www.personalityforge.com/directc ... &MID=23957

Izar 56%
http://www.contechsolutions.net/btchat/

Talk-Bot 50%
http://www.frontiernet.net/~wcowart/index.shtml

Professsor 50%
http://demo.vhost.pandorabots.com/pando ... 771e340628

Rosette 46%
http://labs.telltalegames.com/rosette/

ALICE 46%
http://alice.pandorabots.com/


Challenging Questions
Challenging questions were identified by summing the chatbot scores for a given question: a total of -9 would indicate that all the chatbots gave a “wrong” answer. The actual lowest scoring questions, with sums of -4 or -3 are below:

What is one food you will never give up?
What is a food you can live without?
If you were stuck on a deserted Island, what would you take with you?
Name three things you have never done.
What is the worse movie you have ever seen?
What is my dog’s name?


Remarks

Mitsuku is my top pick. An example to back up my choice:

Question: If you were stuck on a deserted Island, what would you take with you?
Mitsuku: I would like to bring a raft so I could escape…
Cleverbot: No, you're stuck with me.
Jaberwacky: Food.

With a top score of 79%, all exhibit “character flaws”. However, I believe the chatbots perform almost as well as politicians in a press conference.

No credit was given for personality or creativity, just answers that suited the question. Cleverbot ranks second because it is more likely to give a simple “yes” or “no” response to a yes-or-no question: a reasonable strategy.

This method appears to work well: using a large set of questions and a general category reveals a real spread in talent. The question will need to be reworked prior to reuse: chatbots are updated to “fix flaws” uncovered by user input.

The details: questions, responses, and scores will be uploaded to the Tools category of the Forum Downloads.

[Mitsuku]

Mitsuku is my top pick.

Thanks for that. I am very pleased with Mitsuku and entered her in this year's Loebner Prize. Unfortunately Dr Loebner said he had "technical difficulties" when testing her saying that some files were missing. He later admitted that there was nothing wrong with the files he was sent.

This is a shame, as she unofficially got more questions correct than the highest scorer. The highest scorer in the preliminary rounds scored 45, Mitsuku would have scored 50.

[Mapestone]

...Thanks for that...

Ah, so you are the Mitsuku. I am glad to see you are paying attention to this forum. You've demonstrated that the gold medal is within reach. But as they say 90% complete for 90% of the project.