1 Probability Theory 1
1.1 Set Theory 1
1.2 Basics of Probability Theory 5
1.2.1 Axiomatic Foundations 5
1.2.2 The Calculus of Probabilities 9
1.2.3 Counting 13
1.2.4 Enumerating Outcomes 16
1.3 Conditional Probability and Independence 20
1.4 Random Variables 27
1.5 Distribution Functions 29
1.6 Density and Mass Functions 34
1.7 Exercises 37
1.8 Miscellanea 44
2 Transformations and Expectations 47
2.1 Distributions of Functions of a Random Variable 47
2.2 Expected Values 55
2.3 Moments and Moment Generating Functions 59
2.4 Differentiating Under an Integral Sign 68
2.5 Exercises 76
2.6 Miscellanea 82
3 Common Families of Distributions 85
3.1 Introduction 85
3.2 Discrete Distributions 85
3.3 Continuous Distributions 98
3.4 Exponential Families 111
3.5 Location and Scale Families 116
3.6 Inequalities and Identities 121
3.6.1 Probability Inequalities 122
3.6.2 Identities 123
3.7 Exercises 127
3.8 Miscellanea 135
4 Multiple Random Variables 139
4.1 Joint and Marginal Distributions 139
4.2 Conditional Distributions and Independence 147
4.3 Bivariate Transformations 156
4.4 Hierarchical Models and Mixture Distributions 162
4.5 Covariance and Correlation 169
4.6 Multivariate Distributions 177
4.7 Inequalities 186
4.7.1 Numerical Inequalities 186
4.7.2 Functional Inequalities 189
4.8 Exercises 192
4.9 Miscellanea 203
5 Properties of a Random Sample 207
5.1 Basic Concepts of Random Samples 207
5.2 Sums of Random Variables from a Random Sample 211
5.3 Sampling from the Normal Distribution 218
5.3.1 Properties of the Sample Mean and Variance 218
5.3.2 The Derived Distributions: Student's t and Snedecor's F 222
5.4 Order Statistics 226
5.5 Convergence Concepts 232
5.5.1 Convergence in Probability 232
5.5.2 Almost Sure Convergence 234
5.5.3 Convergence in Distribution 235
5.5.4 The Delta Method 240
5.6 Generating a Random Sample 245
5.6.1 Direct Methods 247
5.6.2 Indirect Methods 251
5.6.3 The Accept/Reject Algorithm 253
5.7 Exercises 255
5.8 Miscellanea 267
6 Principles of Data Reduction 271
6.1 Introduction 271
6.2 The Sufficiency Principle 272
6.2.1 Sufficient Statistics 272
6.2.2 Minimal Sufficient Statistics 279
6.2.3 Ancillary Statistics 282
6.2.4 Sufficient, Ancillary, and Complete Statistics 284
6.3 The Likelihood Principle 290
6.3.1 The Likelihood Function 290
6.3.2 The Formal Likelihood Principle 292
6.4 The Equivariance Principle 296
6.5 Exercises 300
6.6 Miscellanea 307
7 Point Estimation 311
7.1 Introduction 311
7.2 Methods of Finding Estimators 312
7.2.1 Method of Moments 312
7.2.2 Maximum Likelihood Estimators 315
7.2.3 Bayes Estimators 324
7.2.4 The EM Algorithm 326
7.3 Methods of Evaluating Estimators 330
7.3.1 Mean Squared Error 330
7.3.2 Best Unbiased Estimators 334
7.3.3 Sufficiency and Unbiasedness 342
7.3.4 Loss Function Optimality 348
7.4 Exercises 355
7.5 Miscellanea 367
8 Hypothesis Testing 373
8.1 Introduction 373
8.2 Methods of Finding Tests 374
8.2.1 Likelihood Ratio Tests 374
8.2.2 Bayesian Tests 379
8.2.3 Union-Intersection and Intersection-Union Tests 380
8.3 Methods of Evaluating Tests 382
8.3.1 Error Probabilities and the Power Function 382
8.3.2 Most Powerful Tests 387
8.3.3 Sizes of Union-Intersection and Intersection-Union Tests 394
8.3.4 p-Values 397
8.3.5 Loss Function Optimality 400
8.4 Exercises 402
8.5 Miscellanea 413
9 Interval Estimation 417
9.1 Introduction 417
9.2 Methods of Finding Interval Estimators 420
9.2.1 Inverting a Test Statistic 420
9.2.2 Pivotal Quantities 427
9.2.3 Pivoting the CDF 430
9.2.4 Bayesian Intervals 435
9.3 Methods of Evaluating Interval Estimators 440
9.3.1 Size and Coverage Probability 440
9.3.2 Test-Related Optimality 444
9.3.3 Bayesian Optimality 447
9.3.4 Loss Function Optimality 449
9.4 Exercises 451
9.5 Miscellanea 463
10 Asymptotic Evaluations 467
10.1 Point Estimation 467
10.1.1 Consistency 467
10.1.2 Efficiency 470
10.1.3 Calculations and Comparisons 473
10.1.4 Bootstrap Standard Errors 478
10.2 Robustness 481
10.2.1 The Mean and the Median 482
10.2.2 M-Estimators 484
10.3 Hypothesis Testing 488
10.3.1 Asymptotic Distribution of LRTs 488
10.3.2 Other Large-Sample Tests 492
10.4 Interval Estimation 496
10.4.1 Approximate Maximum Likelihood Intervals 496
10.4.2 Other Large-Sample Intervals 499
10.5 Exercises 504
10.6 Miscellanea 515
11 Analysis of Variance and Regression 521
11.1 Introduction 521
11.2 Oneway Analysis of Variance 522
11.2.1 Model and Distribution Assumptions 524
11.2.2 The Classic ANOVA Hypothesis 525
11.2.3 Inferences Regarding Linear Combinations of Means 527
11.2.4 The ANOVA F Test 530
11.2.5 Simultaneous Estimation of Contrasts 534
11.2.6 Partitioning Sums of Squares 536
11.3 Simple Linear Regression 539
11.3.1 Least Squares: A Mathematical Solution 542
11.3.2 Best Linear Unbiased Estimators: A Statistical Solution 544
11.3.3 Models and Distribution Assumptions 548
11.3.4 Estimation and Testing with Normal Errors 550
11.3.5 Estimation and Prediction at a Specified x = x0 557
11.3.6 Simultaneous Estimation and Confidence Bands 559
11.4 Exercises 563
11.5 Miscellanea 572
12 Regression Models 577
12.1 Introduction 577
12.2 Regression with Errors in Variables 577
12.2.1 Functional and Structural Relationships 579
12.2.2 A Least Squares Solution 581
12.2.3 Maximum Likelihood Estimation 583
12.2.4 Confidence Sets 588
12.3 Logistic Regression 591
12.3.1 The Model 591
12.3.2 Estimation 593
12.4 Robust Regression 597
12.5 Exercises 602
12.6 Miscellanea 608
Appendix: Computer Algebra 613
Table of Common Distributions 621
References 629
Author Index 645
Subject Index 649