目次

Preface x Introduction 1 1. What is a Neural Network? 1 2. The HumanBrain 6 3. Models of a Neuron 10 4. Neural Networks Viewed As Directed Graphs15 5. Feedback 18 6. Network Architectures 21 7. Knowledge Representation 248. Learning Processes 34 9. Learning Tasks 38 10. Concluding Remarks 45 Notesand References 46 Chapter 1 Rosenblatt's Perceptron 47 1.1 Introduction 471.2. Perceptron 48 1.3. The Perceptron Convergence Theorem 50 1.4. RelationBetween the Perceptron and Bayes Classifier for a Gaussian Environment 551.5. Computer Experiment: Pattern Classification 60 1.6. The Batch PerceptronAlgorithm 62 1.7. Summary and Discussion 65 Notes and References 66 Problems66 Chapter 2 Model Building through Regression 68 2.1 Introduction 68 2.2Linear Regression Model: Preliminary Considerations 69 2.3 Maximum aPosteriori Estimation of the Parameter Vector 71 2.4 Relationship BetweenRegularized Least-Squares Estimation and MAP Estimation 76 2.5 ComputerExperiment: Pattern Classification 77 2.6 The Minimum-Description-LengthPrinciple 79 2.7 Finite Sample-Size Considerations 82 2.8 TheInstrumental-Variables Method 86 2.9 Summary and Discussion 88 Notes andReferences 89 Problems 89 Chapter 3 The Least-Mean-Square Algorithm 91 3.1Introduction 91 3.2 Filtering Structure of the LMS Algorithm 92 3.3Unconstrained Optimization: a Review 94 3.4 The Wiener Filter 100 3.5 TheLeast-Mean-Square Algorithm 102 3.6 Markov Model Portraying the Deviation ofthe LMS Algorithm from the Wiener Filter 104 3.7 The Langevin Equation:Characterization of Brownian Motion 106 3.8 Kushner's Direct-Averaging Method107 3.9 Statistical LMS Learning Theory for Small Learning-Rate Parameter 1083.10 Computer Experiment I: Linear Prediction 110 3.11 Computer ExperimentII: Pattern Classification 112 3.12 Virtues and Limitations of the LMSAlgorithm 113 3.13 Learning-Rate Annealing Schedules 115 3.14 Summary andDiscussion 117 Notes and References 118 Problems 119 Chapter 4 MultilayerPerceptrons 122 4.1 Introduction 123 4.2 Some Preliminaries 124 4.3 BatchLearning and On-Line Learning 126 4.4 The Back-Propagation Algorithm 129 4.5XOR Problem 141 4.6 Heuristics for Making the Back-Propagation AlgorithmPerform Better 144 4.7 Computer Experiment: Pattern Classification 150 4.8Back Propagation and Differentiation 153 4.9 The Hessian and Its Role inOn-Line Learning 155 4.10 Optimal Annealing and Adaptive Control of theLearning Rate 157 4.11 Generalization 164 4.12 Approximations of Functions166 4.13 Cross-Validation 171 4.14 Complexity Regularization and NetworkPruning 175 4.15 Virtues and Limitations of Back-Propagation Learning 1804.16 Supervised Learning Viewed as an Optimization Problem 186 4.17Convolutional Networks 201 4.18 Nonlinear Filtering 203 4.19 Small-ScaleVersus Large-Scale Learning Problems 209 4.20 Summary and Discussion 217Notes and References 219 Problems 221 Chapter 5 Kernel Methods andRadial-Basis Function Networks 230 5.1 Introduction 230 5.2 Cover's Theoremon the Separability of Patterns 231 5.3 The Interpolation Problem 236 5.4Radial-Basis-Function Networks 239 5.5 K-Means Clustering 242 5.6 RecursiveLeast-Squares Estimation of the Weight Vector 245 5.7 Hybrid LearningProcedure for RBF Networks 249 5.8 Computer Experiment: PatternClassification 250 5.9 Interpretations of the Gaussian Hidden Units 252 5.10Kernel Regression and Its Relation to RBF Networks 255 5.11 Summary andDiscussion 259 Notes and References 261 Problems 263 Chapter 6 SupportVector Machines 268 6.1 Introduction 268 6.2 Optimal Hyperplane for LinearlySeparable Patterns 269 6.3 Optimal Hyperplane for Nonseparable Patterns 2766.4 The Support Vector Machine Viewed as a Kernel Machine 281 6.5 Design ofSupport Vector Machines 284 6.6 XOR Problem 286 6.7 Computer Experiment:Pattern Classification 289 6.8 Regression: Robustness Considerations 289 6.9Optimal Solution of the Linear Regression Problem 293 6.10 The RepresenterTheorem and Related Issues 296 6.11 Summary and Discussion 302 Notes andReferences 304 Problems 307 Chapter 7 Regularization Theory 313 7.1Introduction 313 7.2 Hadamard's Conditions for Well-Posedness 314 7.3Tikhonov's Regularization Theory 315 7.4 Regularization Networks 326 7.5Generalized Radial-Basis-Function Networks 327 7.6 The RegularizedLeast-Squares Estimator: Revisited 331 7.7 Additional Notes of Interest onRegularization 335 7.8 Estimation of the Regularization Parameter 336 7.9Semisupervised Learning 342 7.10 Manifold Regularization: PreliminaryConsiderations 343 7.11 Differentiable Manifolds 345 7.12 GeneralizedRegularization Theory 348 7.13 Spectral Graph Theory 350 7.14 GeneralizedRepresenter Theorem 352 7.15 Laplacian Regularized Least-Squares Algorithm354 7.16 Experiments on Pattern Classification Using Semisupervised Learning356 7.17 Summary and Discussion 359 Notes and References 361 Problems 363Chapter 8 Principal-Components Analysis 367 8.1 Introduction 367 8.2Principles of Self-Organization 368 8.3 Self-Organized Feature Analysis 3728.4 Principal-Components Analysis: Perturbation Theory 373 8.5 Hebbian-BasedMaximum Eigenfilter 383 8.6 Hebbian-Based Principal-Components Analysis 3928.7 Case Study: Image Coding 398 8.8 Kernel Principal-Components Analysis 4018.9 Basic Issues Involved in the Coding of Natural Images 406 8.10 KernelHebbian Algorithm 407 8.11 Summary and Discussion 412 Notes and References415 Problems 418 Chapter 9 Self-Organizing Maps 425 9.1 Introduction 425 9.2Two Basic Feature-Mapping Models 426 9.3 Self-Organizing Map 428 9.4Properties of the Feature Map 437 9.5 Computer Experiments I: DisentanglingLattice Dynamics Using SOM 445 9.6 Contextual Maps 447 9.7 HierarchicalVector Quantization 450 9.8 Kernel Self-Organizing Map 454 9.9 ComputerExperiment II: Disentangling Lattice Dynamics Using Kernel SOM 462 9.10Relationship Between Kernel SOM and Kullback--Leibler Divergence 464 9.11Summary and Discussion 466 Notes and References 468 Problems 470 Chapter 10Information-Theoretic Learning Models 475 10.1 Introduction 476 10.2 Entropy477 10.3 Maximum-Entropy Principle 481 10.4 Mutual Information 484 10.5Kullback--Leibler Divergence 486 10.6 Copulas 489 10.7 Mutual Information asan Objective Function to be Optimized 493 10.8 Maximum Mutual InformationPrinciple 494 10.9 Infomax and Redundancy Reduction 499 10.10 SpatiallyCoherent Features 501 10.11 Spatially Incoherent Features 504 10.12Independent-Components Analysis 508 10.13 Sparse Coding of Natural Images andComparison with ICA Coding 514 10.14 Natural-Gradient Learning forIndependent-Components Analysis 516 10.15 Maximum-Likelihood Estimation forIndependent-Components Analysis 526 10.16 Maximum-Entropy Learning for BlindSource Separation 529 10.17 Maximization of Negentropy forIndependent-Components Analysis 534 10.18 Coherent Independent-ComponentsAnalysis 541 10.19 Rate Distortion Theory and Information Bottleneck 54910.20 Optimal Manifold Representation of Data 553 10.21 Computer Experiment:Pattern Classification 560 10.22 Summary and Discussion 561 Notes andReferences 564 Problems 572 Chapter 11 Stochastic Methods Rooted inStatistical Mechanics 579 11.1 Introduction 580 11.2 Statistical Mechanics580 11.3 Markov Chains 582 11.4 Metropolis Algorithm 591 11.5 SimulatedAnnealing 594 11.6 Gibbs Sampling 596 11.7 Boltzmann Machine 598 11.8Logistic Belief Nets 604 11.9 Deep Belief Nets 606 11.10 DeterministicAnnealing 610 11.11 Analogy of Deterministic Annealing withExpectation-Maximization Algorithm 616 11.12 Summary and Discussion 617 Notesand References 619 Problems 621 Chapter 12 Dynamic Programming 627 12.1Introduction 627 12.2 Markov Decision Process 629 12.3 Bellman's OptimalityCriterion 631 12.4 Policy Iteration 635 12.5 Value Iteration 637 12.6Approximate Dynamic Programming: Direct Methods 642 12.7 Temporal-DifferenceLearning 643 12.8 Q-Learning 648 12.9 Approximate Dynamic Programming:Indirect Methods 652 12.10 Least-Squares Policy Evaluation 655 12.11Approximate Policy Iteration 660 12.12 Summary and Discussion 663 Notes andReferences 665 Problems 668 Chapter 13 Neurodynamics 672 13.1 Introduction672 13.2 Dynamic Systems 674 13.3 Stability of Equilibrium States 678 13.4Attractors 684 13.5 Neurodynamic Models 686 13.6 Manipulation of Attractorsas a Recurrent Network Paradigm 689 13.7 Hopfield Model 690 13.8 TheCohen--Grossberg Theorem 703 13.9 Brain-State-In-A-Box Model 705 13.10Strange Attractors and Chaos 711 13.11 Dynamic Reconstruction of a ChaoticProcess 716 13.12 Summary and Discussion 722 Notes and References 724Problems 727 Chapter 14 Bayseian Filtering for State Estimation of DynamicSystems 731 14.1 Introduction 731 14.2 State-Space Models 732 14.3 KalmanFilters 736 14.4 The Divergence-Phenomenon and Square-Root Filtering 744 14.5The Extended Kalman Filter 750 14.6 The Bayesian Filter 755 14.7 CubatureKalman Filter: Building on the Kalman Filter 759 14.8 Particle Filters 76514.9 Computer Experiment: Comparative Evaluation of Extended Kalman andParticle Filters 775 14.10 Kalman Filtering in Modeling of Brain Functions777 14.11 Summary and Discussion 780 Notes and References 782 Problems 784Chapter 15 Dynamically Driven Recurrent Networks 790 15.1 Introduction 79015.2 Recurrent Network Architectures 791 15.3 Universal Approximation Theorem797 15.4 Controllability and Observability 799 15.5 Computational Power ofRecurrent Networks 804 15.6 Learning Algorithms 806 15.7 Back PropagationThrough Time 808 15.8 Real-Time Recurrent Learning 812 15.9 VanishingGradients in Recurrent Networks 818 15.10 Supervised Training Framework forRecurrent Networks Using Nonlinear Sequential State Estimators 822 15.11Computer Experiment: Dynamic Reconstruction of Mackay--Glass Attractor 82915.12 Adaptivity Considerations 831 15.13 Case Study: Model Reference Appliedto Neurocontrol 833 15.14 Summary and Discussion 835 Notes and References 839Problems 842 Bibliography 845 Index 889