
@@ 0,0 +1,457 @@


1

+{


2

+ "cells": [


3

+ {


4

+ "cell_type": "markdown",


5

+ "metadata": {},


6

+ "source": [


7

+ "# Goodness of Fit Metrics\n",


8

+ "\n",


9

+ "Link: [http://charlesreid1.com/wiki/Response_Surface_Methodology](http://charlesreid1.com/wiki/Response_Surface_Methodology)\n",


10

+ "\n",


11

+ "## Analysis of Variance (ANOVA Table)\n",


12

+ "\n",


13

+ "ANOVA is a statistical technique that builds a table attributing the variance in the system response to various degrees of freedom. Each piece of data about the real system contributes one degree of freedom, which can go toward estimating a new coefficient, or can go toward estimating the variance in the model.\n",


14

+ "\n",


15

+ "Bibliography note: Mason ch. 6, 8  derived in 6.1\n",


16

+ "\n",


17

+ "For ANOVA of linear regression models, need to define a few quantities.\n",


18

+ "\n",


19

+ "Total sum of squares: \n",


20

+ "\n",


21

+ "$$\n",


22

+ "TSS = \\sum \\left( y_i  \\overline{y} \\right)^2\n",


23

+ "$$\n",


24

+ "\n",


25

+ "Error sum of squares:\n",


26

+ "\n",


27

+ "$$\n",


28

+ "SSE = \\sum \\left( y_i  \\hat{y}_i \\right)^2\n",


29

+ "$$\n",


30

+ "\n",


31

+ "Model sum of squares (regression sum of squares):\n",


32

+ "\n",


33

+ "$$\n",


34

+ "SSR = \\sum \\left( \\hat{y}_i  \\overline{y} \\right)^2\n",


35

+ "$$\n",


36

+ "\n",


37

+ "### Univariate Linear Model\n",


38

+ "\n",


39

+ "(Sample ANOVA table constructed for a linear univariate function)\n",


40

+ "\n",


41

+ "### Multiple Linear Model\n",


42

+ "\n",


43

+ "Important not just to assess overall fit of prediction equation\n",


44

+ "\n",


45

+ "Also important to assess contribution of individual predictor variables to the fit\n",


46

+ "\n",


47

+ "Many commonlyreported measures of model fitness are part of ANOVA table\n",


48

+ "\n",


49

+ "Multivariate models: p degrees of freedom for sum of squares due to regression (because p coefficients {\\beta_1, \\beta_2, \\dots \\beta_p} must be estimated to obtain regression sum of squares\n",


50

+ "\n",


51

+ "(Sample ANOVA table constructed for multiple linear function)\n",


52

+ "\n",


53

+ "\n",


54

+ "\n",


55

+ "$p$ is number of predictor variables. To measure the adequacy of fitted model, determine the error standard deviation:\n",


56

+ "\n",


57

+ "$$\n",


58

+ "s_e = (MSE)^{1/2}\n",


59

+ "$$\n",


60

+ "\n",


61

+ "where $MSE = SSE/(np1)$.\n",


62

+ "\n",


63

+ "Small $s_e$ means predicted responses closely approximate observed responses.\n",


64

+ "\n",


65

+ "Large $s_e$ means large random error, or poor selection of model form."


66

+ ]


67

+ },


68

+ {


69

+ "cell_type": "markdown",


70

+ "metadata": {},


71

+ "source": [


72

+ "## FStatistic\n",


73

+ "\n",


74

+ "Main article: FStatistic\n",


75

+ "\n",


76

+ "I found this short YouTube video very helpful for illustrating what the Fstatistic means physically: [link](http://www.youtube.com/watch?v=TMwSS8DAVYk)\n",


77

+ "\n",


78

+ "The Fstatistic can be thought of as a frequentist metric for hypothesistesting. Once an Fstatistic and corresponding pvalue is calculated from the ANOVA table quantities, you can determine how confident you can be in a given hypothesis test (where the hypothesis is the model onto which you've chosen to regress your data).\n",


79

+ "\n",


80

+ "Mason: Different from tests for significance of factor effects in analysis of designed experiments\n",


81

+ "\n",


82

+ "Mason: example of acidcontent data...\n",


83

+ "\n",


84

+ "For multiple linear regression/model, can use Fstatistic to simultaneously test a hypothesis: whether all $\\beta_j = 0$ (the hypothesis), or whether at least one $\\beta_j \\neq 0$ (the null hypothesis, i.e., the outcome that our hypothesis is not correct).\n",


85

+ "\n",


86

+ "$\\beta_1 = \\beta_2 = \\dots = \\beta_p = 0$ versus $\\beta_j \\neq 0$\n",


87

+ "\n",


88

+ "(While this seems silly, it's much more useful if you're doing this for a subset of coefficients  such as testing the hypothesis of whether any of a subset of coefficients should be nonzero.)\n",


89

+ "\n",


90

+ "### Lack of Fit FStatistic\n",


91

+ "\n",


92

+ "For (deterministic) computer simulations (rather than experiments, which have random error), error is entirely due to lack of fit  there is no source of random error from real instruments or real experimental noise.\n",


93

+ "\n",


94

+ "In this case, an Fstatistic specifically for lackoffit is not possible to calculate (defined as $MSE_{LOF}/MSE_{P}$, and $MSE_P = 0 $for deterministic functions)\n",


95

+ "\n",


96

+ "### Partial FTest (Determination of Term Significance)\n",


97

+ "\n",


98

+ "Consider a full regresion model with $p$ predictor variables.\n",


99

+ "\n",


100

+ "Now consider a reduced regression model with $k < p$ predictor variables.\n",


101

+ "\n",


102

+ "Full model = $M_1$\n",


103

+ "\n",


104

+ "Reduced model = $M_2$\n",


105

+ "\n",


106

+ "Reduction in error sum of squares resulting from fit of additional terms in full model:\n",


107

+ "\n",


108

+ "$$\n",


109

+ "R(M_1 \\vert M_2) = SSE_2  SSE_1\n",


110

+ "$$\n",


111

+ "\n",


112

+ "$(p  k)$ more predictor variables\n",


113

+ "\n",


114

+ "Fstatistic for determining statistical significance of this subset is:\n",


115

+ "\n",


116

+ "$$\n",


117

+ "F = \\frac{ MSR(M_1 \\vert M_2) }{ MSE_1 }\n",


118

+ "$$\n",


119

+ "\n",


120

+ "where $MSR(M_1 \\vert M_2) = \\frac{ R(M_1 \\vert M_2) }{ (pk) }$\n",


121

+ "\n",


122

+ "If $k = p1$ then the Fstatistic is the square of the tstatistic from the full model corresponding to the term left out of the reduced model\n",


123

+ "\n",


124

+ "To determine if Fstatistic is highly significant, use p value (95% likelihood of being significant if $p < 0.05$, 99% likelihood if $p < 0.01$)\n",


125

+ "\n",


126

+ "Example: use to determine if interaction effect is important to surrogate model.\n",


127

+ "\n",


128

+ "Using this procedure and using the tstatistic to test the significance of a given term are equivalent!"


129

+ ]


130

+ },


131

+ {


132

+ "cell_type": "markdown",


133

+ "metadata": {},


134

+ "source": [


135

+ "## TStatistic\n",


136

+ "\n",


137

+ "**TODO:** \n",


138

+ "\n",


139

+ "Fix from here\n",


140

+ "\n",


141

+ "**Linear Univariate Model:**\n",


142

+ "\n",


143

+ "A tstatistic can be constructed to test $\\beta_1 = c$ vs. $\\beta_1 \\neq c$\n",


144

+ "\n",


145

+ "The following statistic has a Student tdistribution with $n2$ degrees of freedom:\n",


146

+ "\n",


147

+ "$$\n",


148

+ "\\begin{array}{rcl} t &=& \\frac{ b_1  \\beta_1 }{ s_e / s_xx^{1/2} } \\\\ s_{xx} &=& \\sum \\left( x_i  \\overline{x} \\right)^2 \\end{array}\n",


149

+ "$$\n",


150

+ "\n",


151

+ "and insert $\\beta_1 = c$.\n",


152

+ "\n",


153

+ "If you insert $c = 0$ and square the result you get the Fstatistic from the ANOVA table.\n",


154

+ "\n",


155

+ "This tvariate can be used to form confidence intervals on the slope parameter.\n",


156

+ "\n",


157

+ "Following Chapter 2.4: limits for $\\beta_1$ are\n",


158

+ "\n",


159

+ "$$\n",


160

+ "b_1  t_{\\alpha / 2} \\frac{s_e}{s_{xx}^{1/2}} \\leq \\beta_1 \\leq b_1 + t_{\\alpha / 2} \\frac{s_e}{s_{xx}^{1/2}}\n",


161

+ "$$\n",


162

+ "\n",


163

+ "where $t_{\\alpha / 2}$ is a $100(\\alpha / 2)%$ uppertail t critical value with n2 degrees of freedom\n",


164

+ "\n",


165

+ "Small model standard deviations will lead to small confidence intervals\n",


166

+ "\n",


167

+ "For the intercept parameter \\beta_0, use the following tvariate:\n",


168

+ "\n",


169

+ "$$\n",


170

+ "t = \\frac{ b_0  \\beta_0 }{ s_e \\left( n^{1} + \\overline{x}^2 / s_{xx} \\right)^2 }\n",


171

+ "$$\n",


172

+ "\n",


173

+ "**Multiple Linear Model:**\n",


174

+ "\n",


175

+ "Testing hypotheses on individual regression coefficients is of primary interest to someone performing regression analysis\n",


176

+ "\n",


177

+ "The tstatistic can be constructed to test $\\beta_j = c$ versus $\\beta_j \\neq c$.\n",


178

+ "\n",


179

+ "Test statistic used for this purpose is:\n",


180

+ "\n",


181

+ "$$\n",


182

+ "t = \\frac{ b_j  c }{ s_e c_{jj}^{1/2} }\n",


183

+ "$$\n",


184

+ "\n",


185

+ "where $s_e = (MSE)^{1/2}$ is the estimated error standard deviation.\n",


186

+ "\n",


187

+ "$$\n",


188

+ "c_{jj} = \\left[ (n  1) s_j^2 (1  R_j^2) \\right]^{1}\n",


189

+ "$$\n",


190

+ "\n",


191

+ "$s_j^2$ is the sample variance of the n values of the jth predictor variable\n",


192

+ "\n",


193

+ "$R_j^2$ is the coefficient of determination for regression of $x_j$ on the constant term and the other $p1$ predictor variables.\n",


194

+ "\n",


195

+ "Using tstatistics with $c = 0$ can be used to test statistical significance of of individual model parameters (usefulness of $x_j$ as predictor of response variable)\n",


196

+ "\n",


197

+ "NOTE: this test is only conditional, since $\\beta_j$ is partial regression coefficient, and $b_j$, $c_{jj}$ are functions of other predictor variable values\n",


198

+ "\n",


199

+ "Only determines significance of jth predictor variable conditional on the presence of the other predictor variables\n",


200

+ "\n",


201

+ "e.g. \"Each individual predictor variable contributes significantly to the given fits, given that the other two predictor variables are also included in the model\" "


202

+ ]


203

+ },


204

+ {


205

+ "cell_type": "markdown",


206

+ "metadata": {},


207

+ "source": [


208

+ "## Response Confidence Intervals\n",


209

+ "\n",


210

+ "Want confidence intervals for the response model $\\hat{y}$.\n",


211

+ "\n",


212

+ "**Linear Univariate Models:**\n",


213

+ "\n",


214

+ "Confidence interval constructed for the response model:\n",


215

+ "\n",


216

+ "$$\n",


217

+ "\\mu = \\beta_0 + \\beta_1 x\n",


218

+ "$$\n",


219

+ "\n",


220

+ "Mason: for fixed values of x...???\n",


221

+ "\n",


222

+ "The predicted response $\\hat{y}$ has a normal distribution (given certain assumptions).\n",


223

+ "\n",


224

+ "Thus mean and deviation given by:\n",


225

+ "\n",


226

+ "$$\n",


227

+ "\\mu = \\beta_0 + \\beta_1 x\n",


228

+ "$$\n",


229

+ "\n",


230

+ "$$\n",


231

+ "\\sigma \\left[ a_1 + \\left( x  a_2 \\right)^2 / s_{xx} \\right]^{1/2}\n",


232

+ "$$\n",


233

+ "\n",


234

+ "where:\n",


235

+ "\n",


236

+ "$$\n",


237

+ "a_1 = \\frac{1}{n}\n",


238

+ "$$\n",


239

+ "\n",


240

+ "$$\n",


241

+ "a_2 = \\overline{x}\n",


242

+ "$$\n",


243

+ "\n",


244

+ "$$\n",


245

+ "s_{xx} = \\sum \\left( x_i  \\overline{x} \\right)^2\n",


246

+ "$$\n",


247

+ "\n",


248

+ "And the following tvariate can be used to construct confidence intervals for $\\mu$:\n",


249

+ "\n",


250

+ "$$\n",


251

+ "t = \\frac{ \\hat{y}  \\mu }{ s_e \\left( a_1 + \\left( x  a_2 \\right) / s_{xx} \\right)^2 }\n",


252

+ "$$\n",


253

+ "\n",


254

+ "To form prediction interval for actual future response, not expected value of a response.\n",


255

+ "\n",


256

+ "Use this equation again, but replace $\\mu$ with $y_f$, and $a_1$ with $a_1 + 1$.\n",


257

+ "\n",


258

+ "$y_f$ is the future response\n",


259

+ "\n",


260

+ "$\\hat{y}$ is the predicted value of future response\n",


261

+ "\n",


262

+ "$a \\rightarrow a+1$ is because future response has standard deviation $\\sigma$ with an added variability\n",


263

+ "\n",


264

+ "Standard deviation of $\\hat{y}  y_f$ is:\n",


265

+ "\n",


266

+ "$$\n",


267

+ "\\sigma = \\left( 1 + a_1 + \\left( x  a_2 \\right)^2 / s_{xx} \\right)^{1/2}\n",


268

+ "$$\n",


269

+ "\n",


270

+ "**Multiple Linear Model:**\n",


271

+ "\n",


272

+ "Confidence interval for regression coefficients of multiple linear model:\n",


273

+ "\n",


274

+ "A $100(1\\alpha)%$ confidence interval for $\\beta_j$ is given by:\n",


275

+ "\n",


276

+ "$$\n",


277

+ "b_j  t_{\\alpha/2} s_e c_{jj}^{1/2} \\leq \\beta_j \\leq b_j + t_{\\alpha/2} s_e c_{jj}^{1/2}\n",


278

+ "$$\n",


279

+ "\n",


280

+ "where $t_{\\alpha/2}$ is twotailed $100 \\alpha %$ t critical value having $n  p  1$ degrees of freedom.\n",


281

+ "\n",


282

+ "Simultaneous confidence intervals for all coefficients in multiple linear regression model cannot be computed using individual coefficient intervals.\n",


283

+ "\n",


284

+ "They ignore systematic variation of predictor variables and consequent correlation among coefficient estimators."


285

+ ]


286

+ },


287

+ {


288

+ "cell_type": "markdown",


289

+ "metadata": {},


290

+ "source": [


291

+ "## Correlation Coefficient (R Squared)\n",


292

+ "\n",


293

+ "Measure of correlation between observed and predicted responses\n",


294

+ "\n",


295

+ "Univariate Linear Model\n",


296

+ "\n",


297

+ "$$\n",


298

+ "R^2 = \\left[ corr(y,\\hat{y}) \\right]^2 = 1  \\frac{SSE}{TSS}\n",


299

+ "$$\n",


300

+ "\n",


301

+ "**Multiple Linear Model:**\n",


302

+ "\n",


303

+ "Rsquared can be calculated as:\n",


304

+ "\n",


305

+ "$$\n",


306

+ "R^2 = 1  \\frac{SSE}{TSS}\n",


307

+ "$$\n",


308

+ "\n",


309

+ "It should be acknowledged that as the number of predictor variables approaches the number of observations, this can become arbitrarily close to 1\n",


310

+ "\n",


311

+ "if $n = p+1$ then $R^2 = 1$.\n",


312

+ "\n",


313

+ "Adjusted $R^2$, denoted $R_a^2$:\n",


314

+ "\n",


315

+ "$$\n",


316

+ "R_a^2 = 1  a \\frac{SSE}{TSS}\n",


317

+ "$$\n",


318

+ "\n",


319

+ "where\n",


320

+ "\n",


321

+ "$$\n",


322

+ "a = \\frac{ n1 }{ n  p  1 }\n",


323

+ "$$\n",


324

+ "\n",


325

+ "Differences between $R^2$ and $R_a^2$ are minor except for when $n$ and $p$ are close.\n",


326

+ "\n",


327

+ "Caution should be used in relying on single measure of fit (e.g. $R^2$)."


328

+ ]


329

+ },


330

+ {


331

+ "cell_type": "markdown",


332

+ "metadata": {},


333

+ "source": [


334

+ "### Contour Plots\n",


335

+ "\n",


336

+ "Contour plots can be used to determine sensitivities: if the response $y$ changes significantly in one parameter direction, it is sensitive to that parameter. If the contour shows a structure that is uniform in one parameter direction, the response is not sensitive to that parameter.\n",


337

+ "\n",


338

+ "For multiple responses, a contour plot for each response can be made, infeasible regions shaded gray, and the plots overlaid to yield the feasible region. "


339

+ ]


340

+ },


341

+ {


342

+ "cell_type": "markdown",


343

+ "metadata": {},


344

+ "source": [


345

+ "### Determination of Outlier Data Points\n",


346

+ "\n",


347

+ "Mason Ch. 18\n",


348

+ "\n",


349

+ "Various test for outliers (p. 629, 631, etc.)\n",


350

+ "\n",


351

+ "Tests for response variable outliers\n",


352

+ "\n",


353

+ "Tests for predictor variable outliers "


354

+ ]


355

+ },


356

+ {


357

+ "cell_type": "markdown",


358

+ "metadata": {},


359

+ "source": [


360

+ "## Other Things to Look At\n",


361

+ "\n",


362

+ "* Correlation between input variables  e.g., for time and temperature: $\\{t, T, tT, t^2, T^2\\}$"


363

+ ]


364

+ },


365

+ {


366

+ "cell_type": "markdown",


367

+ "metadata": {},


368

+ "source": [


369

+ "# Issues\n",


370

+ "\n",


371

+ "## Importance of Interaction Terms\n",


372

+ "\n",


373

+ "We mentioned above that screening and highly fractionated factorial designs tend to ignore interaction effects by aliasing them with main effects.\n",


374

+ "\n",


375

+ "This important note from Mason indicates that this can be avoided by only including the colinear interaction terms that are most likely to be significant:\n",


376

+ "\n",


377

+ "<blockquote>\n",


378

+ "Ordinarily, one should not routinely insert products of all the predictors in a regression model. To do so might create unnecessary complications in the analysis and interpretation of the fitted models due to collinear predictor variables. The purpose of including interaction terms in regression models is to improve the fit either because theoretical considerations require a modeling of the joint effects or because an analysis of the regression data indicates that joint effects are needed in addition to the linear terms of the individual variables.\n",


379

+ "</blockquote>"


380

+ ]


381

+ },


382

+ {


383

+ "cell_type": "markdown",


384

+ "metadata": {},


385

+ "source": [


386

+ "## Dealing with Multimodal Variables\n",


387

+ "\n",


388

+ "Sometimes, when constructing response surfaces, modal variables appear. Modal variables are variables that have multiple modes, or distinct sets of values. There are two variations of modal variables:\n",


389

+ "\n",


390

+ "* One uncertainty range\n",


391

+ "* N uncertainty range\n",


392

+ "\n",


393

+ "### One Uncertainty Range\n",


394

+ "\n",


395

+ "These types of modal variables have a single range of uncertainty assigned to them, but the values within that range of uncertainty are discrete. In order to sample the parameter within the range of uncertainty, the parameter must be sampled at distinct, discrete values.\n",


396

+ "\n",


397

+ "For example, if I am using the discrete ordinates model (DOM) for radiation calculations, the DOM requires a number of ordinate directions. This is a discrete value with distinct sets of values  e.g. 3, 6, 8, 24, etc.\n",


398

+ "\n",


399

+ "Each discrete value in this case composes a single range of uncertainty. Using the DOM example, that range of uncertainty would be $[3, 24]$.\n",


400

+ "\n",


401

+ "### N Uncertainty Ranges\n",


402

+ "\n",


403

+ "The other type of modal variables have several ranges of uncertainty assigned to them, with no restriction on values within that range of uncertainty being discrete or distinct. Essentially this can be thought of as a bimodal uncertainty distribution, where the two modes are distinct. Each mode can be sampled as usual, the only sticking point is that there is more than 1, and that they are distinct.\n",


404

+ "\n",


405

+ "The case of mixing in a chemically reacting system provides an example of how to think about this. Suppose we are investigating two mass flowrates of material into a reactor where there is mixing occurring  mass flowrates of 1 kg/s and 2 kg/s. \n",


406

+ "\n",


407

+ "Each mode also has a range of uncertainty. This can be an absolute range (as in, $\\pm 0.1$ kg/s) or a relative range (as in, $\\pm 5 \\%$).\n",


408

+ "\n",


409

+ "### How To Deal\n",


410

+ "\n",


411

+ "Multimodal variables can be dealt with in two ways:\n",


412

+ "\n",


413

+ "**Method 1: Separate Response Surfaces for Each Mode**\n",


414

+ "\n",


415

+ "The first way is to create a separate response surface for each distinct mode. This method works for both types of modal variables (1 uncertainty range represented by N distinct values, and N uncertainty ranges). This method is illustrated in the figures below. Each distinct mode (gray region) has its own computed response surface (blue dotted line), distinct from the response surface of the other modes.\n",


416

+ "\n",


417

+ "Of course, if the variable type is 1 uncertainty range represented by N distinct values, then there is no uncertainty range for each mode, and each gray region is, in fact, a delta function. As mentioned above, this means that the input variable is eliminated as a response surface parameter.\n",


418

+ "\n",


419

+ "If the variable type is N uncertainty ranges, then each uncertainty range is sampled as usual, and each response surface is constructed as usual. \n",


420

+ "\n",


421

+ "**Method 2: Single Response Surface (Ignore Modes)**\n",


422

+ "\n",


423

+ "A second way is to create a single response surface. This is typically only possible with N uncertainty ranges type of problems, because the parameter value is continuous, but it is only certain regions that are of interest. This approach is illustrated below.\n",


424

+ "\n",


425

+ "Essentially, this approach does away with any special treatment of modes. "


426

+ ]


427

+ },


428

+ {


429

+ "cell_type": "code",


430

+ "execution_count": null,


431

+ "metadata": {},


432

+ "outputs": [],


433

+ "source": []


434

+ }


435

+ ],


436

+ "metadata": {


437

+ "kernelspec": {


438

+ "display_name": "Python 3",


439

+ "language": "python",


440

+ "name": "python3"


441

+ },


442

+ "language_info": {


443

+ "codemirror_mode": {


444

+ "name": "ipython",


445

+ "version": 3


446

+ },


447

+ "file_extension": ".py",


448

+ "mimetype": "text/xpython",


449

+ "name": "python",


450

+ "nbconvert_exporter": "python",


451

+ "pygments_lexer": "ipython3",


452

+ "version": "3.6.3"


453

+ }


454

+ },


455

+ "nbformat": 4,


456

+ "nbformat_minor": 2


457

+}
