+ J(K) /
setting (d
_1 L
2
+
K
4
This expression can be simplified to (6.t). Also, note that (d 2 /dK 2 )
It remains to argue that if
R = AoB/ Ao,
then
R.E
K as n+
00.
Using the fact that K
is the minimizer of ep and AoB minimizes EASE B , it follows that ep(R)/ep(K)~1 as n+oo. Because ep is a continuous function, R~K otherwise a contradiction could be obtained. 0 Appendix
1 Proof of Lemma 6.1.
This appendix outlines a proof of Lemma 6.1 in Section 6. Most of the technical arguments have been adapted from Cox (1988) (subsequently abbreviated AMORE) and the reader is referred to this work for more details. In order to prove this result, it is useful to place this smoothing problem in the general setting treated in AMORE. After establishing two preliminary results (Lemmas A.l and A.2), the proof of Lemma 6.1 is given. The following Hilbert spaces and operators will be used in this analysis: 1 n  l; c.(. n i=1 "1 1
J
x"x"du 1 2
[0, 1]
Let W, U (: L(g;),
such that
J
and
x"x" 1 2
du
[0, 1]
From the arguments in Section 2 of AMORE, there is a basis {¢v}~g; such that (¢v, U ¢J.l}g; =
8v J.l
(¢v, W ¢J.l}g; =
'V 8VJ.l
This basis is just the continuous analog of the DemmlerReinsch basis given in Section 2. For cubic smoothing splines with natural boundary conditions IV= (a1l"v)4
(1+ 0(1))
as n+oo.
One important property of the DemlerReinsch construction is that the eigenvalues have the same asymptotic distribution as {'v}, From Speckman (1981) 19
if M= Ln 2/5 J then (A.1)
Illn = III (1+0(1)),
uniformly for lIE[l,M] as n+
00.
This limiting behaviour will be used to approximate the trace of A('\). The basis
{¢J~}
can also be used to define a scale of Hilbert spaces. Let 00 ~
1I=1
Clearly (x 1 ,x 1 )P > (xl' x2)$ for p~l and we can construct Hilbert spaces $p by completing $ with respect to the inner product ( , )p, Note that by the choice of G (see Assumption 0) $o=L 2 [O, 1]. Also $1 =W~[O, 1]. It is also possible to define spaces for p>l. The most important of these is
h" (0) =h"f (¢J) =0,
Now let
h" (1) =h"f(1) =O}.
T n :$_lR n
satisfy T n (h)=(h(t 1 ), h(t 2 ), ... , h(tn))t the minimization problem of (2.1) can be rewritten as {;;;i¥; (Y Tnh, Y Tnh)q;n +'\(Wh, h)$ and has the solution
(A.I)
(A.2)
The basic approximation used in Cox's representation of a spline estimate is to substitute U for Un in (A.2). Expressions involving only Wand U can then be analyzed using the simultaneous diagonalization of these operators by the basis {¢J1I}'
The results of the first lemma are phrased in terms of the smoothing operators: 1 A n ..\ f=PW+U n r U n f A..\ f=(,\W + Ur 1 U f and bias operators
20
Lemma A.l
i)
iii)
II(B ,A , B,A, )hll =O(A n", n",o '" "'0 P
2p
2p
v AO
)
iv)
all convergence being uniform for A e: [An, 00].
Proof Part i) follows from Theorem 2.3 (AMORE) and the characterization of 9;2 from Theorem 3.2 (AMORE). Part ii) is a consequence of Theorem 4.3 (AMORE) with the identifications:
{3=2, s=I/2, r=4 and k n ...... (I/n). From the first case of Theorem 4.3 (s+l/r) . AMORE, we must have k~ An 0. Given our choice of An, this convergence
condition will hold. The third part of this lemma is more specific to the material of this paper and requires a detailed argument. II(B , A , B,A, )hll ~ II(B ,B,)A , hll +IIB,(A , A, )hll n", n",o '" "'0 P n", '" n",o p '" n",o "'0 p and from part ii)
~ 0(..\2 p) IIA , h II + o(..\~ p) n",o
2
Thus iii) will follow provided that IIA , h II = 0(1). Now, n",o 2
and from equation (4.1) of AMORE we have
liB, hll n",
where
Pi =0, P2=1/4,
2
~ liB, hll
'"
2 1/2 + kn . ~ C (..\, 2+sp.) liB ,hll 2 1=1 1 n", Pi
s=I/2 and
00 (1+ lv )5 C(A,5)= ~ 2 v=1 (1+A IV)
,(5+l)
'"
21
Also by Theorems 2.3 and 4.2 of AMORE
liB, h II = o( A2 p ) n", p tJ.sing these rates combined with the assumptions on k n and An, it follows that
With this bound iii) now follows. The last part of this lemma can be proved using the results from iii) and an inequality developed by Cox (1984) Section 4. Specifically, by a simple modification of the inequality at the top of page 38, Cox (1984) we have II(B ,A , B,A, ) h 11 2 II(B ,A , B,A, ) h 11 2 ~ knll(B ,A , B,A, ) h 11 2 n", n",o '" "'0 a n", n",o '" "'0 q,Jn . n", n",o '" "'0 s with s= 1/2 and k n l/n. The bound on this expression will be 0(A2) for Af[An,OO]. 0 Lemma 4.2 Under Assumptions 02,
i)
1/4 tr A(A)k=\k1l" L k (1+0(1)) uniformly for Af[A n , An) as n+oo .
uniformly for Af[An, An] as n+oo .
Proof 2 5 Choose M  n / . From (A.I) and (A.2), for any w<1 there is an N such that for all
"Yvn «(}1l"v)4
Thus,
(A.a)
<
w
for
vE [I,M]
Now du
(A.l)
)k
(1+>.w(a1ru 4 )
1/4 JMW>'
a1r
(A.2)
dw
o and similarly M
(A.3)
1
I:
. v=1 (1+>,w(1rav)4)
dw
k
The absolute value of the difference of the integrals in (A.l) and (A.3) is
(A.4)
(W>.fl/4 a1r
uniformly for >'E[>'n, AnI as n
+ 00.
Note that by the choice of >'n the upper limits of the integrals will tend to
00
and by the
choice of An, the lower limits will converge to zero. Considering the second term in (A.O) and using the hypothesis that k;:::2,
(nM)/(>'(1raM)4t
= O( >'nk n k8 / 5+1) = 0(n k4 / 5+1)=
0(1)
Thus we have
Now if one assumes that
W> 1 one
can use the same arguments to show
1 0(>.4) Since the choice of
W
+
O( 1)
< tr A(>.)k
is arbitrary, part i) now follows from these upper and lower bounds.
Part ii) is established by the same methods. For example we, can bound
23
from below and above by 1/4 41 MWA o £1'11" (w'x°) £1'11"
J
and
'!/J(w, A/ Ao)dw
o
The difference between these bounds is
1/4 [
(A.5)
(WA~~
(M + l)w'xo 1/4
w'xo 1/4 £1'11"
J
'!/J(w,'x/ 'xo)dw
+
J
o
MWA 1/4 o and thus (A.5) is bounded by
From these estimates, ii) now follows.D
Proof of Lemma 6.1 Part i) is a standard result for splines and follows from Lemmas A.1 and A.2. Part ii) is obtained by minimizing the asymptotic expression for EASE('x) from part i). For part iii) 2
EASE(A)B
= k~(IA(A»)A(A) y II +
2 Sn
= kKIA(A») (A(A)A(Ao) + A(Ao» =
2
tr[~(A)
] 2
Y1 +
S~ tr[~(A)2]
k~(IA('x») A(Ao) f ~2 + O"n2t{(I_A(A»)2 A(AO)2] +
where
24
0"2 J.l2(A)
R 3 =ltll(IA(A)) (A(A)A(Ao))
Y
2 11
fi[yT A(,\o) (IA(,\) )2(A(A)A(,\O)) y ]
 R4 = and
2 2 2 (') _ Sn tr[A(A) ] R 5n  (7 J.l2 A
R 1 and R 2 can be analyzed using the same tools developed to study crossv~lidation for spline estimates. In particular the techniques in Speckman (1983) for normal errors and Cox (1984) for nonnormal errors can easily be adapted to the terms obtained when R 1 and R 2 are expanded. We thus have
R = Op(A 1/4/ n ) 1 and
From Lemma A.1 in Nychka (1988) it is shown that ltll(A(,\O)A(A) ) y11 2 =op( EASE(A o )) Therefore,
R3 <
lt~(A(A)A(AO)) Y 1 = 2
op(EASE(A O)).
Using the fact that the convergence rates for the average squared bias and average variance are balanced by AO' R 3 =op(Ab). From the CauchySchwartz inequality IR41~ IA(Ao) (IA(A)) yl IA(,\)) (A(A)A(Ao)) yll
fi
(A.6)
IR41~
in
K
II A(AO) (IA(,\)) yllfR3
Now from considering the rates on ltll(IA(A)) A(,\o) f
2
, R 1 and R 2 we see that the first norm in (A.6) will be op(,\ V AO)' Thus we have R 4 =op(,\2 V Ab). From the results of Nychka 11
(1987) S~ will converge in probability to (72. Now applying part i) from Lemma A.2 we see that R 5 is 0p(J.l2(A)).0
25
Appendix 2
MACSYMA source listing and output.
This appendix includes a listing of the MACSYMA program used to compute the matrices HI' H 2 and H 3 defined in Section 4. The representation of these expressions by MACSYMA is also given In these symbolic calculations the reader should make the following identifications:
A b(l)
(IA(A))
a(l)
A(A)
e
~
m, imi
i
trA(A)/n, ltrA(A)/n trA(A)2/ n
m2 con
HI
lin
H2
quad
H3
26
HACSY}fA
batch prograPl to calculate Hl' HZ' H3
phi: « b(l)+(la(l)*e )**2 )/n+ (1m(1»**2* (  e**2/n  ( b(1)**2 2*b(1)*a(1)*e + e**2*a(1)**2) In); phi: phi/iml(1)**2; phi: combine( expand(phi))$ dphi: diff(phi,l)$ dphi: ev(dphi, diff(a(1),1)(la(1»)*a(1)/1,diff(m(1),1)(mlm2)/1, m(l)ml,diff(b(l),l) a(1)*b(1)/1,diff(iml(1),1)(mlm2)/1)$ dphi: expand(dphi)$ dphi : ev(dphi,a(l) A,b(l)b,iml(l)iml)$ dphi: ratsubst( im3,iml**3,dphi); dphi: ratsubst( im2,iml**2,dphi)$ dphi: expand(ratsubst( Im1,im1,~phi))$ con: combine(coeff(dphi,b**2)); temp: expand( ratsubst(c,e*b,dphi) )$ lin: ratsimp(combine(coeff(temp,c»),a); quad: ratsimp(combine(coeff(dphi,e**2)),a,a**2,a**3);
MACSYMA
listing of the terms can, lin and quad.
(c86) con; 3
2 m2  2 a m1 (d86)
2
'+ 6 a ml
 4 a ml  2 m1
~im3 1 n
(c87) lin; 3
a (4 m2  2 ml
(d87)

2
+ 6 m1
 10 m1 + 2)  4 m2 + 4 m1
im3 1 n
(c88) quad; (d88) (a
2 3 2 (2 m2  2 ml + 6 ml  4 ml  2) + 2 m2 + a ( 4 m2 + 2 ml + 2) + a
332 (2 ml  6 ml Z6b
+
4 ml)  2 ml)/(im3 1 n
REFERENCES
Bates, D., Lindstrom, M., Wahba, G. and Yandell, B. (1986), "GCVPACKRoutines for Generalized Crossvalidation," Technical Report No. 775, Department of Statistics, University of Wisconsin/Madison. Bowman, A. and HardIe, W. (1988), "Bootstrapping in Nonparametric Regression: Local Adaptive Smoothing and Confidence Bands," Journal of the American Statistical Association, 83, 102110. Cox, D.D. (1984), Gaussian approximation of smoothing splines. Technical Report 743, University of Wisconsin/Madison, Department of Statistics. Chiu, S.T. (1989), "The Average Posterior Variance of a Smoothing Spline and a Consistent Estimate of the Average Squared Error," to appear in Annals of Statistics. Hall, P., Marron, J.S. and Park B.U. (1989), "Smoothed CrossValidation," unpublished manuscript. HardIe, W., Hall, P. and Marron, S. (1988), "How far are automatically chosen regression smoothing parameters from their optimum? ," Journal of the American Statistical Association 83, pp. 8695. Li, K. (1986), "Asymptotic Optimality of C L and generalized crossvalidation in ridge regression with an application to spline smoothing," Annals of Statistics 14, pp. 11011112. Messer, K. (1986), "A Comparison of a Spline Estimate to its 'Equivalent' Kernel Estimate," to appear in Annals of Statistics. Nychka, D.W. (1987), "A Frequency Interpretation of Bayesian Confidence Intervals for a Smoothing Spline," North Carolina Institute of Statistics Mimeo Series 1688. Nychka, D.W. (1989a), "Bayesian Confidence Intervals for Smooth Splines," to appear in Journal of the American Statistical Association. Nychka, D.W. (1989b), "The Average Posterior Variance of a Smoothing Spline and a Consistent Estimate of the Average Squared Error," to appear in Annals of Statistics. Rice, J. and Rosenblatt, M. (1983), Smoothing splines: regression, derivatives and deconvolution. Ann. Statist. 11, 141156. Scott, D. (1988)~ "Comment on: How Far Are Automatically Chosen Regression Smoothing Parameters From Their Optimum?" Hardie, W., Hall, P. and Marron, J.S., Journal of the American Statistical Association, 83, 9698. Silverman, B.W .. (1984), "Spline Smoothing: the equivalent variable kernel method," Annals of Statistics 12, pp. 898916. 27
Silverman, B.W .. (1985), "Some Aspects of the Spline Smoothing Approach to Nonparametric Regression Curve Fitting," Journal of the Royal Statistical Society B 47, pp. 152. Speckman, P. (1981), "The Asymptotic Integrated Mean Square Error for Smoothing Noisy Data by Splines," University of Oregon preprint.
Speckman, P. (1983), Efficient nonparametric regression with crossvalidated smoothing splines. Unpublished manuscript. vVahba, G. (1983). Bayesian "confidence intervals" for the crossvalidated smoothing spline. Journal of the Royal Statistical Society B 45 133150.
28
Table 1 Summary of the results of a simulation study comparing two types of confidence intervals for the smoothing parameter
Interval Type
Sample Size
Asymptotic Theory
Simulation
Distribution known
Test
(j=
Function
Type I
n=128
Type II
Type III
Coverage
Distribution of
Coverage
Distribution of
Fixed
Probability
interval widths
Probability
interval widths
Width
2.5%
50%
97.5%
2.5%
50%
97.5%
.2
.927
1.72
2.16
3.11
.905
1.05
1.81
5.52
.4
.925
1.81
2.27
3.29
.890
1.10
2.06
9.41
1.86
.6
.942
1.88
2.42
3.69
.873
1.11
2.06
8.53
2.67
.2
.950
0.90
1.13
1.67
.910
0.67
1.07
3.61
2.14
.4
.952
1.10
1.41
1.99
.952
0.77
1.29
4.65
2.95
.6
.967
1.29
1.64
2.42
.930
0.86
1.41
5.36
2.48 1.24
2.22
.2
.947
1.05
1.30
1.80
.932
0.76
1.19
4.73
.4
.962
1.22
1.57
2.26
.935
0.81
1.47
4.98
.86
.6
.977
1.48
2.11
3.06
.920
0.92
1.55
6.58
1.60
Interval Type
Sample Size
Asymptotic Theory
Simulation
Distribution known
Test
(j=
Function
Type I
n=256
Type II
Type III
Coverage
Distribution of
Coverage
Distribution of
Fixed
Probability
interval widths
Probability
interval widths
Width
2.5%
50%
97.5%
2.5%
50%
97.5%
.2
.962
1.55
2.02
2.80
.942
1.07
1.77
8.97
1.14
.4
.915
1.70
2.17
2.99
.885
1.12
2.05
7.40
1.62
.6
.915
1.72
2.19
3.34
.885
1.17
2.10
9.18
1.37 1.38
.2
.970
0.74
0.99
1.27
.980
0.67
1.02
2.35
.4
.960
0.94
1.19
1.59
.940
0.74
1.21
4.19
1.10
.6
.960
1.09
1.37
1.75
.957
0.81
1.32
3.95
1.68
.2
.962
0.88
1.15
1.50
.942
0.73
1.10
3.17
1.29
.4
.965
1.08
1.32
1.87
.942
0.83
1.36
5.12
1.89
.6
.970
1.27
1.56
2.33
.940
0.85
1.47
6.36
1.51
29
FIGURE LEGENDS
Figure 1.
Simulated data illustrating the dependence on the curve estimate to the smoothing parameter. Data have been simulated according to the additive model (1.1) with normal
er~ors,
17=.2, n=128 and f is function type II from Figure 7. The estimates
are cubic smoothing splines with In (A)=( 18.5, 15.7, 11.1). The effective number of parameters for these curves are tr A(,\)=(39.5, 18.9, 6.6).
Figure 2.
Surface illustrating the spline estimate as a function of t and the smoothing parameter.
Figure 3.
The average squared error for a spline estimate as a function of the log smoothing parameter. Plotted points corr.espond to the rough, optimal and oversmooth estimates in Figure 1.
Figure 4.
Spline estimates evaluated at the endpoints of a 95% confidence interval for the optimal smoothing parameter. For the data in this figure the smoothing parameter was estimated using generalized crossvalidation and a confidence interval for the smoothing parameter that minimizes ASF was constructed using a simulationbased method.
Figure 5.
Experimental data comparing the effect of an appetite suppressant drug on rats. The response is the median food intake measured on a daily basis (*=control group, o=treatment group). The dashed lines are the spline estimates where the smoothing parameter has been found using generalized crossvalidation. The solid lines are under and oversmoothed estimates derived from 90% confidence intervals for the optimal smoothing parameter. The wide range of the estimates indicates the variability of crossvalidation in small data sets.
Figure 6.
Approximate 90% simultaneous confidence bands for median food intake.
30
Figure 7.
Three types of test functions used in the simulation study. Each curve is a mixture of Beta densities: Type I
j ,81O,5+j ,87,7+j ,85,10
where ,8m,n is the standard Beta density on [0,1]. Figure 8.
A summary of the simulation results studying two methods of constructing confidence intervals for the smoothing parameter. This panel of 4 plots reports the performance of a simulationbased method and one based on asymptotic normality of the log smoothing parameter. Two sample sizes are considered. The 9 cases in each plot correspond to 3 different levels of 17 (.2, .4, .6) and 3 types of test functions (see Figure 7). The endpoints of the line segments are the 2.5% and 97.5% percentiles of the distribution of interval widths for a given case. The function type locates the median of the distribution. The level of each segment indicates the observed coverage probability. The confidence interval widths when the distribution of the smoothing parameter is known are also included. These serve as a reference for the two methods being studied and are located near the xaxis.
31
o o
o
0
0
... 0 0 0
"'T")
..... l.O
C
,
CD I>
o
o
'"
o
... o
o
o
0
0
0 0
0 0 U1
...
0
0
...
0 U1
0
0
0
...
U1
I IV 0
... I
0>
... I
'"
t"' 0
"l til
II
'"T"J
0 0
I.
et
::r
....
l:l "l
...... I
lO C
.,
CD
'"
OJ
W
'1 OJ
II CD
rt CD '1
... I
IV
... I
o
I 0>
....I
....
o
o
w
o
o
o o ;:,0
o o
o
~,
.~.
,\:,0 "
0
o o
o o
o
o 0
o 0
0
~
~'O
0
'0 0 0
0
;:.:
0
0 0
a:, O'
hi
o
0 0 0 0 0
(\0
.'
0 0
'~
0
0,
0 0 0
l.O
.,C
0
'"
0
CD
0
o
...... 0 0
0
\ /
o o
o
o
o o
o
.
o
CD
o
.... o
0
":
0 0
...
...
...
...o
o
\J1
\J1
0_
.
\
\ \ \ \ \
\
\
IV
0
'0
; \
0
.I i d
0
\ \ \ \
o·
0
\0
0
..
. .
\
...
\
.
~
0
\
0
.
\ \
0
\ 0
:::0
\
Q)
rt"
\
O'
\ o \
0 C III
'< en
'" 0
.
rt"
\ \
0 Q)
\ 0
rt"
\
Q)
\
\ \ CD
o
,0 \ · , .• 0
'.
\ \
0
'.
\ \
\ \ \
0
\
... o o
......
ro
\ 0
0
\
\ \ \
~\V~~ b
Rat Diet Data
35
30
M
e d
i a 25 n F 0 0
d I
n 20 t
a k
e
15
o
10
II o
I
I
I
I
I
20
40
60
80
100
Days
120
o
o
....
....
o
U1
o
U1
o
o
U1
U1
o o
\.
''''''
.
~"
0_. .
. _.~.'~"
" ..
")
o
..
/
I .'
)<,
\. o
"
......
\.
(Q
\
C
I
'" '"
'"
'"
'"
// /
J
ro
/
. ......
/ o
/ '"
/
/
/ I \
.
....
" "
........
"
......... :..:
..
'
.' .'
o
co
../
;'
.... o
.
Bootstrap n= 128 98
Asymptotic n= 128 98
3
96
96 2 2 3
.••   _.
_ ••.•• 
_•••.•• 
..
.
:l:
C 94
li: __
__
•
_
__
__
__
__ .
C 94
0
o
v
v
e 92 r
e 92
a
a
g 90 e
g 90
88
88
"2 3
r
2 :l:
e :l:
:l:
86
2

3
i
3 1
2
86
4
2
0
1 1
interval
6
8
3
o
10
i
3 1 2
96
I"
4
interval
width
6
8
10
8
10
width
Asymptotic n= 256
Bootstrap n= 256 98
1 1
I
2
98
r:r
l~
f
1
96
2
....    _ .......................  w. _._ •• _ ........... ___ ....... _ .. ___ .... "' .. ____ ...... _____ .. ______ .................... __ ..........
C 94
C 94
0
0
v
v
e 92
e 92
~
r
r
a
a g 90
g 90
e
e 88
88
2
86 0
133 2
1 1 1 2
86 4
interval
6
width
8
10
h. 0
3.33 222 1.11 2
4
interval
6
width