Nabaztag Emotional Intelligence (ei)

Long time ago (2006), in a kingdom far away (France), lived a rabbit. It wasn't a normal rabbit, it had very special powers. It could wiggle its ears, had RGB LED lights in his belly, and it could talk! Reading up the weather forecast, new emails, or just chit-chat, it was the most famous rabbit of all. So, it was given a special name: Nabaztag (Armenian for "hare").

But as with many fairy tales, the story took a twist and the rabbit died.

Nabaztag was the very first IoT device available on the market that humans could buy. But ahead of his time, the inventor went bankrupt and had to shut down their servers. The rabbit wasn't open source and heavily relied on the servers from its inventor.

Since then, the rabbit was sitting in a corner of my room, it didn't talk, it didn't wiggle its ears, it didn't bring any joy. Playing with Raspberry Pi's, I always had the idea to bring it alive one day, but it was on the long list of projects.

But today, using my Google AIY Voice HAT, I'm going to change that!

And the rabbit happily lived ever after!

Let me introduce you: Nabaztag Emotional Intelligence

What will the rabbit learn to do this time?

• Ask you questions about how you feel
• Sense what is around you
• Provide you feedback on what was sensed
• Generates a scoring card for you
• Solve all your issues by playing music

That is sorcery you say? Well, if this smart rabbit can do that, I'll believe it this time!

As Google did update all libraries and code since then, I have gone through all bits and bytes again, and updated the project on the Hackster.io website to let you build your own talking rabbit. If you want to build this project, read  this projectpage first. The project here, starts where that page ended.

Intercepting the Google Assistant

While looking closer at the Google Assistant logic, you can see what the assistant did hear. Once you say "Hey Google, can we talk about my emotions", you will get an EventType ON_RECOGNIZING_SPEECH_FINISHED returned, with the text "can we talk about my emotions".

Can you hack that? YES!

I'm listening for the text TALK, ABOUT, EMOTION. It doesn't matter if I say "please talk about my sad emotion" or "my neighbor is sick, can we talk about his emotions". Any other sentence without these words, just goes off to the Google Assistant. Google can deal with that, as there is no EI needed ;)

In the main() section of the python script, the Assistant is started:

    credentials = auth_helpers.get_assistant_credentials()
    try:
        with Board() as board, Assistant(credentials) as assistant:
            t_AssistantStartConversation = AssistantStartConversation(assistant)
            t_AssistantStartConversation.start()
            for event in assistant.start():
                process_event(assistant, board.led, event)
 
    except KeyboardInterrupt:
        pass
    except Exception as e:
        logging.warning(
            'An error occurred: %s',
            e)
    finally:
        LEDring('starting')

Line 3 starts the assistant. Only once that is done, I can start a thread that keeps listening for something that wants to start a conversation (like the button or any other sensor). That is what happens in line 4 and 5. Then in line 6 and 7, for each Event that happens, it triggers the process_event(). So that is where we have to build in our magic. Let's zoom in to that.

def process_event(assistant, led, event):
    logging.info(event)
    
    if event.type == EventType.ON_START_FINISHED:
        led.state = Led.BEACON_DARK  # Ready.
        logging.info('Say "OK, Google" then speak, or press Ctrl+C to quit...')
        Assistant_can_start_conversation = True
 
    elif (event.type == EventType.ON_CONVERSATION_TURN_STARTED
            or (event.type == EventType.ON_CONVERSATION_TURN_FINISHED and event.args and event.args['with_follow_on_turn'])):
        led.state = Led.ON  # Listening.
        Assistant_can_start_conversation = False
 
    elif event.type == EventType.ON_RECOGNIZING_SPEECH_FINISHED and event.args:
        text = event.args['text'].lower()
        if (("talk" in text and "about" in text and "emotion" in text)
                or Nabaztag_emotions_converstation == True):
            print('Nabaztag Emotions: you said >>', text, '<<')
            assistant.stop_conversation()
 
            emotions_conversation(text)
            time.sleep(0.1)
            
            if(Nabaztag_emotions_continue_conv == True or Nabaztag_emotions_text_query != ''):
                assistant.set_mic_mute(False)
                assistant.start_conversation()
 
                if(Nabaztag_emotions_text_query != ''):
                    assistant.send_text_query(Nabaztag_emotions_text_query)
                    print("Nabaztag emotions: sending >>", Nabaztag_emotions_text_query , "<<")
                    Nabaztag_emotions_text_query = ''

What are we doing here?

1. Line 14-21: Once I see these words being recognized by Google Assistant, I tell the Assistant to stop the conversation (assistant.stop_conversation()), and start my own logic. My own logic runs local code to find out what you talk about and ask you questions. These questions are returned by using the Google TTS (Text To Speech) library.
2. Line 24-26:  Then I turn back on the Google Assistant to wait for an answer (assistant.set_mic_mute(false); assistant.start_converstation()). The first bit is important; officially the microphone is said to be OFF, in reality it is of course ON but only listens for "Hey Google". Without turning it ON, Google doesn't record your talking, so don't forget this line!
3. Line 28-31:  And you can even inject text to Google Assistant (assistant.send_text_query("don't forget the milk")) for sending silent commands to Google Assistant.

In the first step, we jump to our own code where the Emotional Intelligence happens:

def emotions_conversation(text):
    Nabaztag_emotions_new_question = True
 
    if (Nabaztag_emotions_converstation == False):
        tts.say("I'm all ears.", volume=7)
        tts.say("Let me help you with some free advice!", volume=7)
        Nabaztag_emotions_converstation = True
        Nabaztag_emotions_continue_conv = True
        Nabaztag_emotions_number = ''
        Nabaztag_emotions_fruit = ''
        Nabaztag_emotions_color = ''
        random.shuffle(Nabaztag_emotions_array) # shuffle the order of tasks
        Nabaztag_emotions_array_pos = 0
        time.sleep(0.5)
 
    while Nabaztag_emotions_new_question == True:
        if (Nabaztag_emotions_array_pos < len(Nabaztag_emotions_array) and text != "no"):
            array_task = Nabaztag_emotions_array[Nabaztag_emotions_array_pos]
            print("Nabaztag emotions: task >>", array_task)