q Nonlinear regression models q Weight loss data q What to do? q Delta method
q Nonlinear ... p. 2/11. Today's class s Nonlinear regression. s Examples in R.
Statistics 203: Introduction to Regression and Analysis of Variance
Nonlinear regression Jonathan Taylor
- p. 1/11
Today’s class
● Today’s class
■
● Nonlinear regression models ● Weight loss data ● What to do?
■
Nonlinear regression. Examples in R.
● Delta method ● Nonlinear regression ● Nonlinear regression: details ● Iteration & Distribution ● Confidence intervals ● Weight loss data
- p. 2/11
Nonlinear regression models
● Today’s class
■
● Nonlinear regression models
We have usually assumed regression is of the form
● Weight loss data ● What to do? ● Delta method
Y i = β0 +
● Nonlinear regression ● Nonlinear regression: details
p−1 X
βj Xij + εi .
j=1
● Iteration & Distribution ● Confidence intervals ● Weight loss data
■
Or, the regression function f (x, β) = β0 +
p−1 X
β j xj
j=1
■
is linear in beta. Many real-life phenomena can be parameterized by non-linear regression functions. Example: ◆ Radioactive decay: half-life is a non-linear parameter f (t, θ) = C · 2−t/θ . - p. 3/11
Weight
0
50
100
Weight loss data
180 170 160 150 140 130 120 110
Days
150
200
250
- p. 4/11
What to do?
● Today’s class
■
● Nonlinear regression models ● Weight loss data ● What to do? ● Delta method
■
● Nonlinear regression ● Nonlinear regression: details ● Iteration & Distribution ● Confidence intervals
In some cases, it is possible to linearize to get the original model. Suppose Yi = C · 2−Xi /θ εi , then
● Weight loss data
log(Yi ) = C 0 − Xi /θ + ε∗i .
■
If ε∗i have approximately the same variance, than it looks like original model. What if Yi = C · 2−Xi /θ + εi where the εi ’s have the same variance?
- p. 5/11
Delta method
● Today’s class
■
● Nonlinear regression models
The delta method tells us that
● Weight loss data
Var(f (Yi )) ' f 0 (E(Yi ))2 Var(Yi ).
● What to do? ● Delta method ● Nonlinear regression ● Nonlinear regression: details
■
● Iteration & Distribution ● Confidence intervals
In this case Var(log(Yi )) '
● Weight loss data
■ ■
1
2 σ . /θ
C 2 2−2Xi Could use weighted least-squares if θ was known. One possibility: IRLS. bj , j ≥ 0, regress log(Yi ) onto Xi ◆ Starting at some initial θ solve for βb1j+1 , and set 1 j+1 b θ0 = − j+1 βb1
- p. 6/11
Nonlinear regression
● Today’s class
■
● Nonlinear regression models
Alternatively,
● Weight loss data ● What to do?
b = argmin b θ) (C,
● Delta method ● Nonlinear regression
(C,θ)
● Nonlinear regression: details ● Iteration & Distribution
n X
(Yi − C2−Xi /θ )2 .
i=1
● Confidence intervals ● Weight loss data
■
In general, given a possibly non-linear regression function f (x, θ), θ ∈ Rp : we can try to solve θb = argmin θ
■
n X
(Yi − f (xi , θ))2
i=1
where xi is the i-th row of the design matrix. Techniques: usually use a steepest descent approach instead of Newton-Raphson.
- p. 7/11
Nonlinear regression: details
● Today’s class ● Nonlinear regression models
■
p X ∂f 0 0 0 b f (xi , θ) ' f (xi , θ ) + (θ − θ ) = η i i i (θ). ∂θ j (xi ,θ)=(xi ,θb0 ) j=1
● Weight loss data ● What to do? ● Delta method ● Nonlinear regression ● Nonlinear regression: details ● Iteration & Distribution ● Confidence intervals ● Weight loss data
Given an initial value θb0
■
In matrix form η 0 (θ) = ω 0 + Z 0 θ where
∂f 0 Zij = ∂θj
(x,θ)=(xi ,θb0 ) p X
ωi0 = f (xi , θ0 ) −
∂ηi = ∂θj
θ=θb0
0 θi0 Zij
j=1
■
The vector η 0 (θ) is our approximation to the regression surface, we want to choose θ to get as close to Y as possible. Project onto “tangent space.” - p. 8/11
Iteration & Distribution
● Today’s class ● Nonlinear regression models
■
● Weight loss data ● What to do?
Want to choose θb1 as follows θb1 = argmin
● Delta method ● Nonlinear regression
θ
● Nonlinear regression: details ● Iteration & Distribution
n X
(Yi − ωi0 − Z 0 θ)2
i=1
● Confidence intervals ● Weight loss data
■
■
Solution Given θbj , set
θb1 = Z 0 t Z 0
−1
θbj+1 = Z j t Z j
■
Repeat until convergence.
■
b Approximate distribution of θ: where
Z 0 t (Y − ω 0 ).
−1
Z j t (Y − ω j ).
b −1 ), bt Z) θb ∼ N (θ, σ b 2 (Z
n X b 2 /(n − p). (Yi − f (xi , θ)) σ b2 =
- p. 9/11
Confidence intervals
● Today’s class
■
● Nonlinear regression models ● Weight loss data ● What to do? ● Delta method ● Nonlinear regression ● Nonlinear regression: details
■
● Iteration & Distribution ● Confidence intervals ● Weight loss data
■
If θ is restricted, say θ ≥ 0 the asymptotic confidence intervals may be inaccurate (may overlap into negative numbers). library(MASS) provides another method to obtain confidence intervals based on “inverting” an F -test. Basic idea, confidence interval for θ1 : for each fixed value θ1,0 we could compute the “extra sum of squares” between the unrestricted model and the model with θ1 fixed at θ1,0 . b SSEθ1 =θ1,0 (θb2:p ) − SSE(θ) ∼ F1,n−p F (θ1,0 ) = 2 σ b
■
■
at least approximately under H0 : θ1 = θ1,0 . Or, q T (θ1,0 ) = sign(θb − θb1,0 ) F (θ1,0 ) ∼ tn−p . Confidence interval:
{θ1 : −tn−p,1−α/2 < T (θ1 ) < tn−p,1−α/2 }.
- p. 10/11
150 140 130 120 110
Weight
160
170
180
Weight loss data
0
50
100
150 Days
200
250 - p. 11/11
