Questions about work in progress

ChatBot using speech recognition  p5jS > create a River script for the interaction between audience and computer , (D.Shiffman Tutorial)

The bot is triggered by proximity sensor monitored via Arduino. Arduino send via Serial to Computer , 1 if there is someone in front of the sculpture. If there is no one, bot is only displaying DataText.

Speaker sculpture is made of a sound sensor connected to an Arduino and a speaker plugged to computer. The audio speaker sculpture is acting as an actuator for the first servo which body is spinning and touching the other sculptures.

Still have to find the link between the proximity sensor with the bot and the speaker sculpture

DataText is produced by training Texts on masks and object in theatre and Text about components specifications. Training model is Markov Chain.

River script is based on waiting for keywords , and ability to display long sentences. This is a limitation to the conversation.

Need to set up a private router , to have the installation free from overcrowded network

Questions about training Neural Networks from the maker


The maker : For who’s pleasure do we train the models?

Who is the curator/ the audience?

What is the purpose?

“If we employ machine intelligence to augment our writing activities, it’s worth asking how such technology would affect how we think about writing as well as how we think in the general sense.” (Ross Godwin )

How to start

Things about Coding Resources

Tom Bocklish :

D.Shiffmann :
Using RiverScript

Machine Talking
Text to Speech p5 js :
Speech to Text p5 js

Machine Learning
Riverscript to process the input text to output text
Text Generator, All about Chatbot
Markov Chain with D.Shifman

App Recommanded by Janelle Shanne:
1. textgen-rnn – an open-source framework by Max Woolf, written in Python and powered by Tensorflow. It’s the easiest to install (though you still have to know your way around command line a bit) and comes pre-trained so you can get interesting results even from tiny datasets. It runs fine on an ordinary computer’s CPU, and lets you train the same network successively on different datasets, which is fun. It’s not as powerful as the other frameworks, but just fine for simple lists of names. (tested 15th may )
2. tensorflow char-rnn – an open-source framework by Chen Liang, written in Python. It has tons of flexibility, including the ability to adjust dropout, save frequency, and number of saved snapshots during training, and the ability to adjust temperature during sampling. There’s a tutorial here for running it on AWS, and I’m hoping to find a good tutorial for Google Cloud as well.
3. Andrej Karpathy’s char-rnn, an open-source neural network framework for torch (written in Lua). This one has great flexibility in training/sampling parameters, and it seems to run faster on my 2010 Macbook Pro’s CPU than the python/tensorflow models. I’ve been using this one for the simpler datasets.

Ross Goldwin General Method

  • prepend the seed with a pre-seed (another paragraph of text) to push the LSTM into a desired state.
  • Use high quality sample of output from the model you’re seeding with length approximately equal to the sequence length (see above) you set during training.
  • Seed the LSTM with a meaningful text, that the machine would complete
  • build a data set , corpus
  • Choosing the right settings for a given corpus
  • train the model
  • generate output
  • train again

James Loy : How to build your own Neural Network from scratch in Python- A beginner’s guide to understanding the inner workings of Deep Learning
Chatbot Personality by 5agado , github

Questions about the first actuator

  • recommendation from R. Fiebrink about sound and triggering the installation

Use sound recognition to find if there is someone talking to one of the artefact, will solve the problem of how many people are interacting effectively with the installation, play with the sensitivity level , to get the data sound we need

  • Build up one “ear” node, with microphone , check this  and that to understand wich type of microphone we need. > lavalier microphone
  • Build up a “mouth” to display the response of the chatbot installation, play with idea that a microphone is a reverse speaker?
    • version 1 simple version via p5js with the need to have access to cloud to do speech recognition ,
    • version 2 for speech recognition with BitVoicer API , speech recognition, check Arduino Tuto : speech recognition and synthesis,
    • would need to use a private router to secure access to the cloud for the text to speech > check authorisation for plugging the private router to internet, 

To generate responses from the ChatBot : build up different training set with different methods : Markov Chain , char RNN review and  basic char RNN training with github source,   , Tensor Flow‘s sequence to sequence library feed ,

write different scripts via RiverScript to compute the input text  to ouput text, use array in Riverscript? generate different characters , who you are talking to, use a list of trigger words in an array to output a certain sentence

or use the mic from MAC

  • after tutorial with H. Pritchard

produce a mix reality with the electronic node devices and human interpretation by the nodes Node A activate node B/ one sensor, one motor, one movement, / small (or big)size nodes

create the narrative: when (no one/ or not enough people) is (looking or talking) , the nodes are interacting  with each other- when there is enough people or one person( looking or talking), the nodes stop moving, and talk back

Using text to speech and maybe speech recognition



Question about the node language, surrealist legacy

The perspective is from the node artifacts not the human visitor. The piece is more about a bot for them to communicate with each other rather than building a chatbot to talk with us. It is machine and not human centered, The plan is  to give them a script to perform together as a network. I like the surrealist tone of the generative poetry bots and how the Dada movment was quite in advance in their practice of text collage and random writing. 

A list of tutorial to explore and start prototyping :

I have to learn Python , and how to work with Tensor Flow to manipulate text data sets

Things about the Project Theme

Sit down comfortably with your eyes closed. Try to imagine a scene that catches your topic/theme in a nutshell; you can try to imagine that your topic/ theme is being presented as a film or theatre play and construct a scene that presents crucial aspects of it. 
 Or you can try to think of a scene as part of a personal memory where you felt that crucial issues concerning your topic were at stake. Remember that even a purely theoretical topic/theme can be articulated in a scene. Try to mobilize all your senses when you imagine the scene. If possible, let your mind flow back to the work you did with the touching hands. 

proposed by Helen Pritchard based on Nina Lykke. Writing Academic Texts Differently: Intersectional Feminist Methodologies and the Playful Art of Writing (Routledge Advances in Feminist Studies and Intersectionality) (p. 155). Taylor and Francis. Kindle Edition.

-Behind the screen, we, the components, are talking, quietly so quietly that you could hardly hear us. Actually , we are constantly chatting , bubbling : “01101000 01100101 01101100 01101100 01101111 00100000 01110111 01101111 01110010 01101100 01100100 00100000 00100001” chanting “HOHA DADO BIDABI KOKA BIDEKA!” . We are so busy, sorting out your data: receive, store, delete, update, send. Little busy shadows, enchained together with no choice but to communicate and release the data we are taking from you. Behind the screen, there is a 24 hours show, a constant ballet of binaries , an infinite concerto of 0 and 1, sadly no one can hear us, except perhaps your EMF detector.

Suddenly a red light is activated, someone is looking at the screen, someone is typing on the keyboard or maybe it is a sound or a touch? Who or what is this?  I have a unique task, I know what to do …hey! this is my input! Let’s go and give it a byte!  Iterate, map, reiterate, variables, vectors, int,  float, strings loop and loop again and return the arrays to my algorithmic friend!

At the speed of an electron, we are sending feedback not only to you, the one with the keyboard, but also to our hosts: the cookies, in a protocol, a language , that no human could speak fluently. Luckily, we know how to speak human. And thanks to the continuous improvement of the human computer interfaces :   you, the users are by now,  perfectly trained to give us the correct input.

But with humans, there is always an end .  You are now releasing the keyboard and leaving the computer. Behind the screen, and until the next time we will meet, there is a new show going on , a new story we tell to each other through the wires or in the clouds, this is is a story based on the memory of our encounter with you. So come back , do not type on the keyboard, do not look at the screen,  try to listen to the story, because it is not only about you * but also about us. 

* Chun,W. 2016, Updating to remain the same,MIT Press.