Questions about work in progress

ChatBot using speech recognition  p5jS > create a River script for the interaction between audience and computer , (D.Shiffman Tutorial)

The bot is triggered by proximity sensor monitored via Arduino. Arduino send via Serial to Computer , 1 if there is someone in front of the sculpture. If there is no one, bot is only displaying DataText.

Speaker sculpture is made of a sound sensor connected to an Arduino and a speaker plugged to computer. The audio speaker sculpture is acting as an actuator for the first servo which body is spinning and touching the other sculptures.

Still have to find the link between the proximity sensor with the bot and the speaker sculpture

DataText is produced by training Texts on masks and object in theatre and Text about components specifications. Training model is Markov Chain.

River script is based on waiting for keywords , and ability to display long sentences. This is a limitation to the conversation.

Need to set up a private router , to have the installation free from overcrowded network

Questions about training Neural Networks from the maker

 

The maker : For who’s pleasure do we train the models?

Who is the curator/ the audience?

What is the purpose?

“If we employ machine intelligence to augment our writing activities, it’s worth asking how such technology would affect how we think about writing as well as how we think in the general sense.” (Ross Godwin )

How to start

https://towardsdatascience.com/how-to-build-a-neural-network-with-keras-e8faa33d0ae4

Questions about the first actuator

  • recommendation from R. Fiebrink about sound and triggering the installation

Use sound recognition to find if there is someone talking to one of the artefact, will solve the problem of how many people are interacting effectively with the installation, play with the sensitivity level , to get the data sound we need

  • Build up one “ear” node, with microphone , check this  and that to understand wich type of microphone we need. > lavalier microphone
  • Build up a “mouth” to display the response of the chatbot installation, play with idea that a microphone is a reverse speaker?
    • version 1 simple version via p5js with the need to have access to cloud to do speech recognition ,
    • version 2 for speech recognition with BitVoicer API , speech recognition, check Arduino Tuto : speech recognition and synthesis,
    • would need to use a private router to secure access to the cloud for the text to speech > check authorisation for plugging the private router to internet, 

To generate responses from the ChatBot : build up different training set with different methods : Markov Chain , char RNN review and  basic char RNN training with github source,   , Tensor Flow‘s sequence to sequence library feed ,

write different scripts via RiverScript to compute the input text  to ouput text, use array in Riverscript? generate different characters , who you are talking to, use a list of trigger words in an array to output a certain sentence

or use the mic from MAC

  • after tutorial with H. Pritchard

produce a mix reality with the electronic node devices and human interpretation by the nodes Node A activate node B/ one sensor, one motor, one movement, / small (or big)size nodes

create the narrative: when (no one/ or not enough people) is (looking or talking) , the nodes are interacting  with each other- when there is enough people or one person( looking or talking), the nodes stop moving, and talk back

Using text to speech and maybe speech recognition

 

 

Question about the node language, surrealist legacy

The perspective is from the node artifacts not the human visitor. The piece is more about a bot for them to communicate with each other rather than building a chatbot to talk with us. It is machine and not human centered, The plan is  to give them a script to perform together as a network. I like the surrealist tone of the generative poetry bots and how the Dada movment was quite in advance in their practice of text collage and random writing. 

A list of tutorial to explore and start prototyping :

I have to learn Python , and how to work with Tensor Flow to manipulate text data sets

Things about the Project Theme

Sit down comfortably with your eyes closed. Try to imagine a scene that catches your topic/theme in a nutshell; you can try to imagine that your topic/ theme is being presented as a film or theatre play and construct a scene that presents crucial aspects of it. 
 Or you can try to think of a scene as part of a personal memory where you felt that crucial issues concerning your topic were at stake. Remember that even a purely theoretical topic/theme can be articulated in a scene. Try to mobilize all your senses when you imagine the scene. If possible, let your mind flow back to the work you did with the touching hands. 

proposed by Helen Pritchard based on Nina Lykke. Writing Academic Texts Differently: Intersectional Feminist Methodologies and the Playful Art of Writing (Routledge Advances in Feminist Studies and Intersectionality) (p. 155). Taylor and Francis. Kindle Edition.

-Behind the screen, we, the components, are talking, quietly so quietly that you could hardly hear us. Actually , we are constantly chatting , bubbling : “01101000 01100101 01101100 01101100 01101111 00100000 01110111 01101111 01110010 01101100 01100100 00100000 00100001” chanting “HOHA DADO BIDABI KOKA BIDEKA!” . We are so busy, sorting out your data: receive, store, delete, update, send. Little busy shadows, enchained together with no choice but to communicate and release the data we are taking from you. Behind the screen, there is a 24 hours show, a constant ballet of binaries , an infinite concerto of 0 and 1, sadly no one can hear us, except perhaps your EMF detector.

Suddenly a red light is activated, someone is looking at the screen, someone is typing on the keyboard or maybe it is a sound or a touch? Who or what is this?  I have a unique task, I know what to do …hey! this is my input! Let’s go and give it a byte!  Iterate, map, reiterate, variables, vectors, int,  float, strings loop and loop again and return the arrays to my algorithmic friend!

At the speed of an electron, we are sending feedback not only to you, the one with the keyboard, but also to our hosts: the cookies, in a protocol, a language , that no human could speak fluently. Luckily, we know how to speak human. And thanks to the continuous improvement of the human computer interfaces :   you, the users are by now,  perfectly trained to give us the correct input.

But with humans, there is always an end .  You are now releasing the keyboard and leaving the computer. Behind the screen, and until the next time we will meet, there is a new show going on , a new story we tell to each other through the wires or in the clouds, this is is a story based on the memory of our encounter with you. So come back , do not type on the keyboard, do not look at the screen,  try to listen to the story, because it is not only about you * but also about us. 

* Chun,W. 2016, Updating to remain the same,MIT Press.

Questions about objects sensing the world

Imagine Communication between objects through their tactile properties

sensors to the world (other machines, human and environment)

plan to use a spin drum on the top of the servo A. work with catch dreamer format as it spin, and automata spinner Robert Race, carousel toy , spin

Plan to encapsulate Two servo motors, A and B

  • Environment produce shadow/ sound/ light …… and trigger A
  • A motor on
  • A turn and hit B ( every time or randomly)
  • B motor on
  • B produce sound
  •  
<When Sensing EMF from electronic devices you have on you
 The nodes move A / Nodes = textile + SERVO   / Osc control  / Nodes  sound B , if no EMF>

<Building a matrix ( resisitive or capacitive) sensors > , in order to map touch by machines,  exchanging  OSC messages

Tutorial, inspirations

Things about Motor and Sensors

02/10/2018

check conductivity of water , for input from viewer , Kobakant crying dress or Root Node

tuto EMF

02/05/2018

looking for a narrative, entanglement between a motor and a weaved W cell made of wire

01/30/2018

tutorial for  MPU6050-Arduino-Gyroscope GY-521

<put in a box , share info? to motor>

stepper, worked then freeze….. ????, servo more reliable, but/and making sound, stepper is soundless….. silent …..

01/29/2018

tutorial from Maker Show, Bret Stateham 

  •  stepper motor basics , , testing the order for the wire pins order
  • 28BYJ-48 stepper  common ground between IN1 and Micro Controller, power source independent motor and board , connect to arduino, find the bottom for the delay, not to quick , it will frieze
  • accelstepper arduino library : Supports acceleration and deceleration,Supports multiple simultaneous steppers, with independent concurrent stepping on each stepper, API functions never delay() or block, Supports 2, 3 and 4 wire steppers, plus 3 and 4 wire half steppers.
  • reduce speed DC motor with a PWM., < turning slowly and getting mad when sensor activated>

01/25/2018

Test Servo and sensors, check code, circuit and breadboard

  1. Servo+ Capactive sensors
  2. Servo+ Photo resistor

Hands on !

started from the arduino book and shadow theatre sensor installation

found some interesting movements from continuous rotation versus non continuous rotation  , and different motors , different “mood”,

to do : video and pictures , try different heads to check the movements, test how they relate to each other in term of movement/character