Research

Installation version IV work in progress

05/04/2018

ChatBot using speech recognition  p5jS > create a Script for the interaction between audience and computer , (D.Shiffman Tutorial)

Chatbot is triggered by ultra sound sensors monitored via Arduino. Arduino send via Serial to Computer , 1 if there is someone in front of the sculpture. If there is no one, Chatbot is only displaying DataText. The installation is moving mechanically after the first trigger by Audio Speaker Sculpture

DataText is produced by training Texts on masks and object in theatre and Text about components specifications. Training model is Markov Chain

Audio Speaker sculpture with microphone and speakers plugged to computer: role is to get the Chatbot starting, has to drive the audience in, to start the conversation,

Chatbot ‘s script should be based on waiting for keywords ,and ability to display long sentences

Need to set up a private router , to have the installation free from overcrowded network

Questions about training Neural Networks from the maker

The maker : For who’s pleasure do we train the models?

Who is the curator/ the audience?

What is the purpose?

“If we employ machine intelligence to augment our writing activities, it’s worth asking how such technology would affect how we think about writing as well as how we think in the general sense.” (Ross Godwin )

How to start

Coding Resources for the installation

Machine Talking

Machine Learning

  • Riverscript to process the input text to output text
  • Text Generator, All about Chatbot
  1. Markov Chain
  2. RNN
  3. Deep Learning
  4. TensorFlow’s sequence to sequence library

Neural Network Text

App Recommanded by Janelle Shanne:

1. textgen-rnn – an open-source framework by Max Woolf, written in Python and powered by Tensorflow. It’s the easiest to install (though you still have to know your way around command line a bit) and comes pre-trained so you can get interesting results even from tiny datasets. It runs fine on an ordinary computer’s CPU, and lets you train the same network successively on different datasets, which is fun. It’s not as powerful as the other frameworks, but just fine for simple lists of names. (tested 15th may )

 

2. tensorflow char-rnn – an open-source framework by Chen Liang, written in Python. It has tons of flexibility, including the ability to adjust dropout, save frequency, and number of saved snapshots during training, and the ability to adjust temperature during sampling. There’s a tutorial here for running it on AWS, and I’m hoping to find a good tutorial for Google Cloud as well.

3. Andrej Karpathy’s char-rnn, an open-source neural network framework for torch (written in Lua). This one has great flexibility in training/sampling parameters, and it seems to run faster on my 2010 Macbook Pro’s CPU than the python/tensorflow models. I’ve been using this one for the simpler datasets.

Ross Goldwin :

General Method

  1. prepend the seed with a pre-seed (another paragraph of text) to push the LSTM into a desired state.
  2. Use high quality sample of output from the model you’re seeding with length approximately equal to the sequence length (see above) you set during training.
  3. Seed the LSTM with a meaningful text, that the machine would complete
  4. build a data set , corpus
  5. Choosing the right settings for a given corpus
  6. train the model
  7. generate output
  8. train again

James Loy : How to build your own Neural Network from scratch in Python- A beginner’s guide to understanding the inner workings of Deep Learning

Chatbot Personality by 5agado , github

Installation version III

 

 

04/27/2018 after tutorial with R. Fiebrink

Use sound recognition to find if there is someone talking to one of the artefact, will solve the problem of how many people are interacting effectively with the installation, play with the sensitivity level , to get the data sound we need

  • Build up one “ear” node, with microphone , check this  and that to understand wich type of microphone we need. > lavalier microphone
  • Build up a “mouth” to display the response of the chatbot installation, play with idea that a microphone is a reverse speaker?
    • version 1 simple version via p5js with the need to have access to cloud to do speech recognition ,
    • version 2 for speech recognition with BitVoicer API , speech recognition, check Arduino Tuto : speech recognition and synthesis,
    • would need to use a private router to secure access to the cloud for the text to speech > check authorisation for plugging the private router to internet, 

To generate responses from the ChatBot : build up different training set with different methods : Markov Chain , char RNN review and  basic char RNN training with github source,   , Tensor Flow‘s sequence to sequence library feed ,

write different scripts via RiverScript to compute the input text  to ouput text, use array in Riverscript? generate different characters , who you are talking to, use a list of trigger words in an array to output a certain sentence

or use the mic from MAC

04/26/2018 after tutorial with H. Pritchard

produce a mix reality with the electronic node devices and human interpretation by the nodes Node A activate node B/ one sensor, one motor, one movement, / small (or big)size nodes

create the narrative: when (no one/ or not enough people) is (looking or talking) , the nodes are interacting  with each other- when there is enough people or one person( looking or talking), the nodes stop moving, and talk back

Using text to speech and maybe speech recognition

 

 

Artistic Research

  • Artistic Research methods by Jane Prophet : http://www.neuro-memento-mori.com/teaching/category/research-methods/

Action Based Research  by Kemmis :

  1. Planning in order to initiate change
  2. Implementing the change (acting) and observing the process of implementation and consequences
  3. Reflecting on processes of change and re-planning
  4. Acting and observing
  5. Reflecting

Artistic Research, by Friedman about challenges for PhD students and supervisors

Embodied Technologies by Sullivan “argue that the artistic approaches employed using embodied methodologies can be considered as a way to make meaning and that especially within participatory research, these approaches can strengthen validity. ”

  • Networking , conferences and seminars

http://newmaterialism.eu/, Networking European Scholarship on ‘How Matter Comes to Matter’. 

http://sarconference2018.org/ARWEI_SCHEDULE.pdf

  • Samples of artistic research ouput :

critter-compiler-prototype– by Helen Pritchard

 

Vocable Code

Material Aktiv Denken

  • Samples of Product based Research

Digital Lace, Sara Robertson and Sarah Taylor

Urban jacket Pablo Cesar

OpenFood Open Innovation Models, Sharon  Baurley

 

Installation version II

03/29/2018

produce a mix reality with the electronic node devices and human interpretation by the nodes

Node A activate node B

one sensor, one motor, one movement,

network create narrative reality based on simple neural network

 

Looking for the” language” of the installation

03/29/2018

Looking for a language for the installation, digging into Twitter bot ( Shakespeare sonnet from Twitter feed) , and generative text algorithm ( Markov Chain).

Started to learn Python , and Tensor Flow to manipulate text data sets

Goldsmiths : IS53051A: Machine Learning (2017-18)IS71074A: Data and Machine Learning for Artistic Practice (2017-18), IS71068A: Data Programming (2017-18)

 

Kadenze :     machine-learning-for-musicians-and-artists with Rebecca Friendrick   , deep-learning-with-tensorflow by P. Mital

Some of the very good blogs/ tutorials to get inspired or start prototyping

Narrative Reality by Ross Goodwin
Natural Language Processing Wikipedia

N grams and Markov Chain with Daniel Shiffmann