1、THE COOPER UNIONALBERT NERKEN SCHOOL OF ENGINEERINGA N E XPLORATION AND D EVELOPMENT OF C URRENTA RTIFICIAL N EURAL N ETWORK T HEORY AND A PPLICATIONSWITH E MPHASIS ON A RTIFICIAL L IFEbyDavid J. CavutoA thesis submitted in partial fulfillmentof the requirements for the degree ofMaster of Engineerin
2、gMay 6, 1997THE COOPER UNION FOR THE ADVANCEMENT OF SCIENCE AND ARTTHE COOPER UNION FOR THE ADVANCEMENT OF SCIENCE AND ARTALBERT NERKEN SCHOOL OF ENGINEERINGThis thesis was prepared under the direction of the Candidates ThesisAdvisor and has received approval. It was submitted to the Dean of theScho
3、ol of Engineering and the full Faculty, and was approved as partialfulfillment of the requirements for the degree of Master of Engineering._Dean, School of Engineering - Date_Prof. Simon Ben-Avi - DateCandidates Thesis AdvisoriAcknowledgmentsI would like to take this opportunity to thank, first and
4、foremost, my thesis advisor, Dr. SimonBen-Avi . His advice, both as a professor and as a friend, were and always will be invaluable.Moreover, I would like to thank the entire EE department faculty and staff for all the supportand encouragement (and toleration) they have shown me throughout the years
5、.I am deeply indebted to my friend and Big Brother Yashodhan Chandrahas Risbud (Yash!).Without the occasional smack in the head he needed to give me, I might not have made itthrough at all. Thanks for putting up with me.Kappa Phi Zeta Psi . My brothers supported me in the hard times and cheered me i
6、n thegood times. Can anyone ask for more?Special thanks to The Leib , Seamous , and of course, my Muffin .My utmost appreciation and thanks to my parents, George and Doris Cavuto . What can Isay? Thanks for everything. (Especially all that money!)And finally, a big old THANKS! to Peter Cooper for gi
7、ving me a place to work, learn, andgrow for the last six years. Anywhere else would have been just a school. The Cooper Unionhas been my home. DJCDisclaimer: this thesis is entirely a product of my imagination. Any resemblance to actual work is purely coincidental.ii1. AbstractThe purpose of this st
8、udy is to explore the possibilities offered by current Artificial Neural Net(ANN) structures and topologies and determine their strengths and weaknesses. The biologicalinspiration behind ANN structure is reviewed, and compared and contrasted with existingmodels. Traditional experiments are performed
9、 with these existing structures to verify theoryand investigate more possibilities . This study is conducted to the end of examining thepossibility of using ANNs to create “artificial life,” which is defined here as a structure oralgorithm which displays characteristics typically only attributed to
10、biological organisms,usually nonrepeating, nonrandom processes. Although some ANN topology is shown to behighly similar to that of biological systems, existing ANN algorithms are determined beinsufficient to generate the desired type of behavior. A new ANN structure, termed a“Temperon”, is designed,
11、 which encompasses more properties in common with biologicalneurons than did its predecessors. A virtual environment based on turtle graphics is used as atestbed for a neural net built with the new type of neuron. Experiments performed with theTemperon seem to confirm its ability to learn in an unas
12、sisted fashion.iiiTable of Contents1. ABSTRACT ii2. BACKGROUND 12.1 B IOLOGICAL N ATURE OF N EURAL C ELLS 12.1.1 P HYSICAL S TRUCTURE OF BIOLOGICAL NEURON 12.1.1.1 Body, Axon, Dendrites, Synapse 12.1.1.2 Neurotransmitter 32.1.1.3 Sodium/Potassium Pump 42.1.1.4 Ionized pulse 62.1.1.5 All-or-Nothing C
13、ausation 82.1.2 M ATHEMATICAL R EPRESENTATION OF N ERVE C ELL PROCESSES 102.1.2.1 Mathematical correlation to the physical interconnections 102.1.2.2 Linear combination of inputs 112.1.2.3 Thresholding resulting in binary or near-binary outputs 122.2 A RTIFICIAL N EURAL N ETS AND THEIR A PPLICATIONS
14、 142.2.1 G ENERAL T HEORY 142.2.1.1 Purpose 142.2.1.2 Structure 142.2.1.3 Weight Updating 152.2.2 P ERCEPTRONS - C LASSIFICATION 152.2.2.1 Single Layer 152.2.2.2 MLP - Feedforward 182.2.3 H OPFIELD N ET - P ATTERN R ECOGNITION 212.2.4 G ENERALIZATIONS 222.3 O UR FRIEND A PLYSIA 242.3.1 G ENERAL O BS
15、ERVATIONS 242.3.2 S UMMARY OF R ELEVANT E XPERIMENTS 252.3.2.1 Habituation 252.3.2.2 Sensitization 262.3.3 R ELEVANCE AND RELATION TO NEURAL NETS 26iv3. APPROACHES 273.1 G ENERAL M ETHODS AND T OOLS USED 273.1.1 MATLAB ANN TOOLBOX 273.1.2 J AVA 293.1.2.1 Neuron Package 293.1.2.2 TurtleMouse Environm
16、ent 303.2 P ERCEPTRON EXPLORATION 323.2.1 E XPLORATIONS 323.2.1.1 Test Set 1 - Network Size Limits 353.2.1.2 Test Set 2 - Disjoint Set A 353.2.1.3 Test Set 3 - Enclosed Region 363.2.1.4 Test Set 4 - Disjoint Set B 383.2.2 C ONCLUSIONS 413.2.2.1 Partitioning of n-space 413.2.2.2 Sensitivity of n th l
17、ayer to n-1 th layer underspecification 413.2.2.3 Tendency to find simplest solution results in sometimes non-useful heuristics 423.3 S PEAKER D IFFERENTIATION 433.3.1 G ENERAL I DEA 433.3.2 A PPROACHES 433.3.3 C ONCLUSIONS 453.4 T EMPERON 473.4.1 E VOLUTION OF M ODEL INSPIRED BY A PLYSIA 483.4.2 D
18、ESCRIPTION OF M ODEL 483.4.3 D ESCRIPTION OF T ESTBED 493.4.4 V ARIOUS TEST SETS O VERVIEW 513.4.4.1 Test Set 1: Learning rule adjustments 523.4.4.2 Test Set 2: Number of neurons 553.4.4.3 Test Set 3: Number/Types of senses 553.4.4.4 Test Set 4: Obstacle position 563.4.4.5 Test Set 5: SDIC 563.4.5 C
19、 ONCLUSIONS 563.4.5.1 Overall Behavior 563.4.5.2 Learning rule changes 573.4.5.3 Responses to test sets 593.4.5.4 General Conclusions 61v4. CONCLUSIONS 635. FUTURE CONSIDERATIONS 676. APPENDICES 706.1 A PPENDIX A: MATLAB CODE 716.1.1 P ERCEPTRON EXPLORATION 716.1.1.1 HINTONEM.M 716.1.1.2 PLOTEM.M 71
20、6.1.1.3 SET1.M 716.1.1.4 SET6.M 736.1.1.5 TESTNET.M 746.1.2 S PEAKER D IFFERENTIATION 766.1.2.1 HAMDIST.M 766.1.2.2 READDATA.M 766.1.3 T EMPERON 786.1.3.1 LCTEST.M 786.2 A PPENDIX B: JAVA CODE 796.2.1 NEURON P ACKAGE 796.2.1.1 Neuron.java 796.2.1.2 Perceptron.java 816.2.1.3 PercepFB.java 826.2.1.4 I
21、nput.java 836.2.1.5 Temperon.java 846.2.2 T EST PROGRAMS 896.2.2.1 MultiMouseApplet.java 896.2.2.2 TempApplet.java 1037. BIBLIOGRAPHY 117viTable of FiguresF IGURE 21. A TYPICAL NERVE CELL G UYTON , 4. 2F IGURE 22. A MAGNIFICATION OF THE STRUCTURES PRESENT IN THE SYNAPSE G UYTON , 126. 3F IGURE 23. D
22、 IFFUSION OF IONS DUE TO CONCENTRATION GRADIENTS AND VARYING PERMEABILITY OFMEMBRANE RESULTS IN A MEMBRANE POTENTIAL G UYTON , 64. 5F IGURE 24. S ODIUM - POTASSIUM PUMP MOVES IONS AGAINST THEIR GRADIENTS TO CREATE ADEPOLARIZATION OF THE NORMAL REST MEMBRANE POTENTIAL G UYTON , 64. 6F IGURE 25. C ONV
23、ERSION OF ATP TO ADP IN ACTION OF ION PUMP TO EXCHANGE THREE SODIUMANIONS FOR TWO POTASSIUM ONES G UYTON , 68. 6F IGURE 26. D EPICTION OF DEPOLARIZED ZONE PROPAGATING ALONG A NERVE FIBER . 7F IGURE 27. E XCITATORY AND INHIBITORY STIMULI AT THE SYNAPSE G UYTON , 131. 8F IGURE 28. S UMMING ACTION OF S
24、OMA PRESENTED WITH BOTH EXCITATORY (E) AND INHIBITORY (I)STIMULI G UYTON , 134. 9F IGURE 29. G RAPH SHOWING THE ALL - OR - NOTHING RESPONSE OF THE ACTION POTENTIAL G UYTON ,79. 10F IGURE 210. E ACH INPUT X IS ATTENUATED ( OR AMPLIFIED ) BY A WEIGHT CONSTANT W, WHICHRELATES TO THE PHYSICAL ATTENUATIO
25、N IMPOSED AT THE SYNAPSE . 11F IGURE 211. S UMMING AMPLIFIER EFFECT AT SOMA CAN BE MODELED AS A WEIGHTED SUM OF IMPUTS . 12F IGURE 212. C OMPLETE BLOCK DIAGRAM OF N EURAL MODEL , WITH BIAS AND NONLINARTHRESHOLDING FUNCTION . 13F IGURE 213. S INGLE NEURAL LAYER . E ACH CIRCLE REPRESENTS AN ENTIRE N E
26、URAL MODEL , EACHWITH N IMPUTS , AND ONE OUTPUT . 16F IGURE 214. G RAPHICAL DEPICTION OF THE XOR PROBLEM . T HE TWO SETS (X S AND O S ) ARE LINEARLYINSEPARABLE AND THEREFORE CANNOT BE PARTITIONED BY A SINGLE PERCEPTRON LAYER . 18F IGURE 215. T HREE - LAYER MLP N EURAL N ETWORK . 19F IGURE 216. F IVE
27、 NEURONS IN A FULLY - CONNECTED H OPFIELD NETWORK . A LL NEURON OUTPUTS FEEDTO ALL OTHER NEURONS . 22F IGURE 217. B OTTOM VIEW OF A PLYSIA C ALIFORNICA . 24F IGURE 31. C LASS HIERARCHY FOR THE N EURON PACKAGE . A LL CLASSES INHERIT FROM THE N EURONABSTRACT CLASS . 29F IGURE 32. T URTLEMOUSE VIRTUAL
28、ENVIRONMENT WINDOW . T URTLE ( TRIANGLE IN THE MIDDLE )MOVES AROUND WINDOW , LEAVING A TRAIL BEHIND . W INDOW EDGES LOOP AROUND . 30F IGURE 33. T HE XOR PROBLEM ( ABOVE ) AND ITS SOLUTION ( BELOW ) ILLUSTRATED GRAPHICALLY . 33F IGURE 34. T RAINING STATISTICS FOR THE XOR SOLUTION ( WITH RANDOM INITIA
29、LIZATION ). 34F IGURE 35. (R OUGHLY ) CIRCULAR SELECTION REGION SOLUTION . 37viiF IGURE 36. T RAINING STATISTICS FOR THE CIRCULAR SELECTION REGION SOLUTION . 38F IGURE 37. S ET WITH A “ HOLE ” SUCCESSFULLY CLASSIFIED BY THE 3- LAYER MLP. 39F IGURE 38. T RAINING DATA FOR THE PREVIOUS REGION . 40F IGU
30、RE 39. B LOCK DIAGRAM OF SPEAKER DIFFERENTIATION PREPROCESSING TO GENERATE INPUTVECTORS FOR THE ANN PROCESSOR . 44F IGURE 310. M AIN CONTROLLING WINDOW IN THE T EMPERON TESTBED APPLET . W INDOW SIZE ( I . E .NUMBER OF NEURONS IN MATRIX ) IS CONTROLLED BY HTML PARAMETER . 50F IGURE 311. T URTLEMOUSE
31、VIRTUAL ENVIRONMENT . 51F IGURE 312. W EIGHT SET ONE . 52F IGURE 313. B ASIC EXECUTION OF TURTLEMOUSE OVER TIME L - R , T - B , USING ABOVE INITIALIZATIONWEIGHTS . 57F IGURE 314. T IME SERIES OF SAME NETWORK AS PREVIOUS EXAMPLE ( WITH DENTICAL INITIAL VALUES ),WITH MAX / MIN WEIGHT CONDITION ADDED .
32、 59F IGURE 315. C ONTRAST TOP TWO PICTURES WITH BOTTOM TWO PICTURES . 60F IGURE 316. I LLUSTRATION OF SENSITIVE DEPENDANCE ON INITIAL CONDITIONS . 61Section 2 Background12. Background2.1 Biological Nature of Neural CellsSince a good deal of this thesis centers around an mathematical construct known
33、as anartificial neural net (ANN), it will be useful to expand upon the nature of these constructs, aswell as the physiological inspiration for their design. As such, we look into the structure ofactual nerve cells as found in many living creatures, including man. It is important to point outthat alt
34、hough many of the following explanations are generally accepted as fact in the field ofneuroscience, the ultimate function of each of the component parts of the nervous system is notprecisely defined, and many are not even known for certain. However, some such conclusionsare repeated here for the pu
35、rpose of providing insight into the development of artificial neuralstructures. More discussion regarding ANNs can be found in a later chapter.2.1.1 Physical Structure of biological neuronFor simplicity and familiarity, the following discussion will center around human neuralstructures. However, the
36、 facts and conclusions presented herein do generalize to most of theanimal world. In fact, we will later look at one specific example of these neural structures in asnail known as Aplysia . Also, please note that the following text is in no way a completedescription of neural cell structure and beha
37、vior. It merely attempts to familiarize the reader tothe specific structures and processes which inspire the work in later chapters.2.1.1.1 Body, Axon, Dendrites, SynapseNerve cells are thought to be the main processing element in our central nervous system.Humans generally have about 100 billion ne
38、rve cells in the entire nervous system.The nerve cell, or neuron , has four general regions, each defined by its physical position in thecell as well as its function. The cell body, or soma , provides the basic foundation on which theother parts of the cell can grow. It also provides the basic life-
39、supporting functionscharacteristic of any biological cell - nourishment, replenishment, reproduction, etc.The cell body has two types of interconnection structures which emerge from it: dendrites andthe axon . Each neuron generally has only one axon, but typically has many dendrites. Theaxon carries the nerve signal away from the cell body to other neurons. Dendrites carry signalsin to wards the cell body from the axons of other neurons. As such, the basic nerve cell can be
