Part Ⅰ Traces of History and A Neuroscience Briefer 3
1．Brain Style Computing：Origins and Issues 3
1.1 From the Greeks to the Renaissance 3
1.2 The Advent of Modern Neuroscience 6
1.3 On the Road to Artificial Intelligence 9
1.4 Classical AI and Neural Networks 12
1.5 Hybrid Intelligent Systems 14
Chapter Summary 15
Bibliographic Remarks 16
2．Lessons from Neuroscience 17
2.1 The Human Brain 17
2.2 Biological Neurons 23
Chapter Summary 37
Bibliographic Remarks 38
Part Ⅱ Feedforward Neural Networks and Supervised Learning 38
3．Artificial Neurons,Neural Networks and Architectures 41
3.1 Neuron Abstraction 41
3.2 Neuron Signal Functions 44
3.3 Mathematical Preliminaries 53
3.4 Neural Networks Defined 61
3.5 Architectures：Feedforward and Feedback 62
3.6 Salient Properties and Application Domains of Neural Networks 65
Chapter Summary 68
Bibliographic Remarks 69
Review Questions 69
4．Geometry of Binary Threshold Neurons and Their Networks 72
4.1 Pattern Recognition and Data Classification 72
4.2 Convex Sets.Convex Hulls and Linear Separability 76
4.3 Space of Boolean Functions 78
4.4 Binary Neurons are Pattern Dichotomizers 80
4.5 Non-linearly Separable Problems 83
4.6 Capacity of a Simple Threshold Logic Neuron 87
4.7 Revisiting thie XOR Problem 92
4.8 Multilayer Networks 95
4.9 How Many Hidden Nodes are Enough? 97
Chapter Summary 99
Bibliographic Remarks 100
Review Questions 100
5．Supervised Learning Ⅰ：Perceptrons and LMS 104
5.1 Learning and Memory 104
5.2 From Synapses to Behaviour：The Case of Aplysia 106
5.3 Learning Algorithms 110
5.4 Error Correction and Gradient Descent Rules 114
5.5 The Learning Objective for TLNs 115
5.6 Pattern Space and Weight Space 117
5.7 Perceptron Learning Algorithm 119
5.8 Perceptron Convergence Theorem 122
5.9 A Handworked Example and MATLAB Simulation 125
5.10 Perceptron Learning and Non-separable Sets 128
5.11 Handling Linearly Non-separable Sets 130
5.12 α-Least Mear Square Learning 132
5.13 MSE Error Surface and its Geometry 137
5.14 Steepest Descent Search with Exact Gradient Information 143
5.15 μ-LMS：Approximate Gradient Descent 147
5.16 Application of LMS to Noise Cancellation 152
Chapter Summary 156
Bibliographic Remarks 157
Review Questions 158
6．Supervised Learning Ⅱ：Backpropagation and Beyond 164
6.1 Multilayered Network Architectures 164
6.2 Backpropagation Learning Algorithm 167
6.3 Handworked Example 177
6.4 MATLAB Simulation Examples 181
6.5 Practical Considerations in Implementing the BP Algorithm 187
6.6 Structure Growing Algorithms 196
6.7 Fast Relatives of Backpropagation 198
6.8 Universal Function Approximation and Neural Networks 199
6.9 Applications of Feedforward Neural Networks 201
6.10 Reinforcement Learning:A Brief Review 205
Chapter Summary 212
Bibliographic Remarks 213
Review Questions 214
7．Neural Networks:A Statistical Pattern Recognition Perspective 218
7.1 Introduction 218
7.2 Bayes'Theorem 219
7.3 Two Instructive MATLAB Simulations 222
7.4 Implementing Classification Decisions with Bayes'Theorem 227
7.5 Probabilistic Interpretation of a Neuron Discriminant Function 230
7.6 MATLAB Simulation:Plotting Bayesian Decision Boundaries 232
7.7 Interpreting Neuron Signals as Probabilities 236
7.8 Multilayered Networks,Error Functions and Posterior Probabilities 239
7.9 Error Functions for Classification Problems 245
Chapter Summary 254
Bibliographic Remarks 255
Review Questions 255
8．Focussing on Generalization:Support Vector Machines and Radial Basis Function Networks 259
8.1 Learning From Examples and Generalization 259
8.2 Statistical Learning Theory Briefer 264
8.3 Support Vector Machines 273
8.4 Radial Basis Function Networks 304
8.5 Regularization Theory Route to RBFNs 314
8.6 Generalized Radial Basis Function Network 323
8.7 Learning in RBFN's 326
8.8 Image Classification Application 329
8.9 Other Models For Valid Generalization 334
Chapter Summary 339
Bibliographic Remarks 341
Review Questions 341
Part Ⅲ Recurrent Neurodynamical Systems 347
9．Dynamical Systems Review 347
9.1 States,State Vectors and Dynamics 347
9.2 State Equations 350
9.3 Attractors and Stability 352
9.4 Linear Dynamical Systems 354
9.5 Non-linear Dynamical Systems 358
9.6 Lyapunov Stability 363
9.7 Neurodynamical Systems 369
9.8 The Cohen-Grossberg Theorem 373
Chapter Summary 375
Bibliographic Remarks 376
Review Questions 376
10．Attractor Neural Networks 378
10.1 Introduction 378
10.2 Associative Learning 379
10.3 Attractor Neural Network Associative Memory 382
10.4 Linear Associative Memory 386
10.5 Hopfield Network 389
10.6 Content Addressable Memory 397
10.7 Two Handworked Examples 400
10.8 Example of Recall of Memories in Continuous Time 404
10.9 Spurious Attractors 405
10.10 Error Correction with Bipolar Encoding 407
10.11 Error Performance of Hopfield Networks 409
10.12 Applications of Hopfield Networks 412
10.13 Brain-State-in-a-Box Neural Network 419
10.14 Simulated Annealing 426
10.15 Boltzmann Machine 431
10.16 Bidirectional Associative Memory 440
10.17 Handworked Example 443
10.18 BAM Stability Analysis 447
10.19 Error Correction in BAMs 448
10.20 Memory Annihilation of Structured Maps in BAMs 450
10.21 Continuous BAMs 457
10.22 Adaptive BAMs 458
10.23 Application：Pattern Association 461
Chapter Summary 462
Bibliographic Remarks 464
Review Questions 464
11．Adaptive Resonance Theory 469
11.1 Noise-Saturation Dilemma 469
11.2 Solving the Noise-Saturation Dilemma 471
11.3 Recurrent On-center-Off-surround Networks 477
11.4 Building Blocks of Adaptive Resonance 482
11.5 Substrate of Resonance 487
11.6 Structural Details of the Resonance Model 489
11.7 Adaptive Resonance Theory Ⅰ（ART Ⅰ） 491
11.8 Handworked Example 502
11.9 MATLAB Code Description 504
11.10 A Breezy Review of ART Operating Principles 506
11.11 Neurophysiological Evidence for ART Mechanisms 507
11.12 Applications 511
Chapter Summary 516
Bibliographic Remarks 517
Review Questions 518
12．Towards the Self-organizing Feature Map 521
12.1 Self-organization 521
12.2 Maximal Eigenvector Filtering 522
12.3 Extracting Principal Components：Sanger's Rule 530
12.4 Generalized Learning Laws 532
12.5 Competitive Learning Revisited 537
12.6 Vector Quantization 540
12.7 Mexican Hat Networks 546
12.8 Self-organizing Feature Maps 552
12.9 Applications of the Self Organizing Map 563
Chapter Summary 569
Bibliographic Remarks 570
Review Questions 571
Part Ⅳ Contemporary Topics 577
13．Pulsed Neuron Models：The New Generation 577
13.1 Introduction 577
13.2 Spiking Neuron Model 578
13.3 Integrate-and-Fire Neurons 586
13.4 Conductance Based Models 594
13.5 Computing with Spiking Neurons 608
13.6 Reflections... 616
Chapter Summary 617
Bibliographic Remarks 618
14．Fuzzy Sets,Fuzzy Systems and Applications 620
14.1 Need for Numeric and Linguistic Processing 620
14.2 Fuzzy Uncertainty and the Linguistic Variable 621
14.3 Fuzzy Set 622
14.4 Membership Functions 624
14.5 Geometry of Fuzzy Sets 627
14.6 Simple Operations on Fuzzy Sets 628
14.7 Fuzzy Rules for Approximate Reasoning 632
14.8 Rule Composition and Deffuzification 634
14.9 Fuzzy Engineering 638
14.10 Applications 644
Chapter Summary 649
Bibliographic Remarks 650
Review Questions 650
15．Neural Networks and the Soft Computing Paradigm 652
15.1 Soft Computing=Neural+Fuzzy+Evolutionary 652
15.2 Neural Networks：A Summary 654
15.3 Fuzzy Sets and Systems：A Summary 656
15.4 Genetic Algorithms 658
15.5 Neural Networks and Fuzzy Logic 662
15.6 Neuro-Fuzzy-Genetic Integration 671
15.7 Integration Example：Subsethood-Product Based Fuzzy-Neural Inference System 675
15.8 A Concluding Note 683
Chapter Summary 684
Bibliographic Remarks 685
Appendix A：Neural Network Hardware 686
A.1 Motivation and Issues 686
A.2 Analog Building Blocks for Neuromorphic Networks 687
A.3 Digital Techniques 691
A.4 Bibliographic Remarks 692
Appendix B：Web Pointers 694
Bibliography 697
Index 729