PAC-BAYESIAN SUPERVISED CLASSIFICATION (The thermodynamics of statistical learning)

Olivier Catoni

August 10, 2007

Abstract: This monograph deals with adaptive supervised classification, using tools borrowed from statistical mechanics and information theory, stemming from the PAC-Bayesian approach pioneered by David McAllester and applied to a conception of statistical learning theory forged by Vladimir Vapnik. Using convex analysis on the set of posterior probability measures, we show how to get local measures of the complexity of the classification model involving the relative entropy of posterior distributions with respect to Gibbs posterior measures. We then discuss relative bounds, comparing the generalization error of two classification rules, showing how the margin assumption of Mammen and Tsybakov can be replaced with some empirical measure of the covariance structure of the classification model. We show how to associate to any posterior distribution an effective temperature relating it to the Gibbs prior distribution with the same level of expected error rate, and how to estimate this effective temperature from data, resulting in an estimator whose expected error rate converges according to the best possible power of the sample size adaptively under any margin and parametric complexity assumptions. We describe and study an alternative selection scheme based on relative bounds between estimators, and present a two step localization technique which can handle the selection of a parametric model from a family of those. We show how to extend systematically all the results obtained in the inductive setting to transductive learning, and use this to improve Vapnik’s generalization bounds, extending them to the case when the sample is made of independent non-identically distributed pairs of patterns and labels. Finally we review briefly the construction of Support Vector Machines and show how to derive generalization bounds for them, measuring the complexity either through the number of support vectors or through the value of the transductive or inductive margin.

2000 Mathematics Subject Classification: 62H30, 68T05, 62B10.

Keywords: adaptive statistics, compression schemes, effective temperature, Gibbs posterior distributions, local complexity bounds, Mammen and Tsybakov margin assumption, margin bounds, PAC-Bayesian theorems, pattern recognition, randomized estimators, statistical learning theory, Support Vector Machines, transductive inference, Vapnik–Cervonenkis dimension.

CNRS – Laboratoire de Probabilités et Modèles Aléatoires, Université Paris 6 (site Chevaleret), 4 place Jussieu – Case 188, 75 252 Paris Cedex 05.

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . 5

1. Inductive PAC-Bayesian learning . . . . . . . . . . 13

1.1. Basic inequality . . . . . . . . . . . . . . . . . 14 1.2. Non local bounds . . . . . . . . . . . . . . . . 17

1.2.1. Unbiased empirical bounds . . . . . . . . . . . 18 1.2.2. Optimizing explicitly the exponential parameter λ . . 21 1.2.3. Non random bounds . . . . . . . . . . . . . 23 1.2.4. Deviation bounds . . . . . . . . . . . . . . 25

1.3. Local bounds . . . . . . . . . . . . . . . . . . 28

1.3.1. Choice of the prior . . . . . . . . . . . . . . 28 1.3.2. Unbiased local empirical bounds . . . . . . . . . 29 1.3.3. Non random local bounds . . . . . . . . . . . 32 1.3.4. Local deviation bounds . . . . . . . . . . . . 33 1.3.5. Partially local bounds . . . . . . . . . . . . . 38 1.3.6. Two step localization . . . . . . . . . . . . . 43

1.4. Relative bounds . . . . . . . . . . . . . . . . . 49

1.4.1. Basic inequalities . . . . . . . . . . . . . . 50 1.4.2. Non random bounds . . . . . . . . . . . . . 54 1.4.3. Unbiased empirical bounds . . . . . . . . . . . 57 1.4.4. Relative empirical deviation bounds . . . . . . . 61

2. Comparing posterior distributions to Gibbs priors . . 69

2.1. Bounds relative to a Gibbs distribution . . . . . . . 69

2.1.1. Comparing a posterior distribution with a Gibbs prior 70 2.1.2. The effective temperature of a posterior distribution . 74 2.1.3. Analysis of an empirical bound for the effective

temperature . . . . . . . . . . . . . . . . 75 2.1.4. Adaptation to parametric and margin assumptions . 81 2.1.5. Estimating the divergence of a posterior with re-

spect to a Gibbs prior . . . . . . . . . . . . . 87

2.2. Playing with two posterior and two local prior distributions. . . . . . . . . . . . . . . . . . . 88

2.2.1. Comparing two posterior distributions . . . . . . 88 2.2.2. Elaborate uses of relative bounds between posteriors . 90 2.2.3. Analysis of relative bounds . . . . . . . . . . . 96

2.3. Two step localization. . . . . . . . . . . . . . . 111

2.3.1. Two step localization of bounds relative to a Gibbs

prior . . . . . . . . . . . . . . . . . . . 111 2.3.2. Analysis of two step bounds relative to a Gibbs prior 119 2.3.3. Two step localization between posterior distributions 125

3. Transductive PAC-Bayesian learning . . . . . . . . 137

3.1. Basic inequalities. . . . . . . . . . . . . . . . . 137

3.1.1. The transductive setting . . . . . . . . . . . . 137 3.1.2. Absolute bound . . . . . . . . . . . . . . . 139 3.1.3. Relative bounds . . . . . . . . . . . . . . . 140

3.2. Vapnik bounds for transductive classification …. 142

3.2.1. With a shadow sample of arbitrary size . . . . . . 142 3.2.2. When the shadow sample has the same size as

the training sample . . . . . . . . . . . . . 145 3.2.3. When moreover the distribution of the augmented

sample is exchangeable . . . . . . . . . . . . 146

3.3. Vapnik bounds for inductive classification . . . . . . 148

3.3.1. Arbitrary shadow sample size . . . . . . . . . 148 3.3.2. A better minimization with respect to the expo-

nential parameter . . . . . . . . . . . . . . 150 3.3.3. Equal shadow and training sample sizes . . . . . 153 3.3.4. Improvement on the equal sample size bound in

the i.i.d. case . . . . . . . . . . . . . . . . 153

3.4. Gaussian approximation in Vapnik bounds. . . . . . . 155

3.4.1. Gaussian upper bounds of variance terms . . . . . 155 3.4.2. Arbitrary shadow sample size . . . . . . . . . 156 3.4.3. Equal sample sizes in the i.i.d. case . . . . . . . 156

4. Support Vector Machines . . . . . . . . . . . . . . 159

4.1. How to build them . . . . . . . . . . . . . . . . 159

4.1.1. The canonical hyperplane . . . . . . . . . . . 159 4.1.2. Computation of the canonical hyperplane . . . . . 160 4.1.3. Support vectors . . . . . . . . . . . . . . . 162 4.1.4. The non-separable case . . . . . . . . . . . . 163 4.1.5. Support Vector Machines . . . . . . . . . . . 167 4.1.6. Building kernels . . . . . . . . . . . . . . . 169

4.2. Bounds for Support Vector Machines . . . . . . . . 171

4.2.1. Compression scheme bounds . . . . . . . . . . 171 4.2.2. The Vapnik–Cervonenkis dimension of a family

of subsets . . . . . . . . . . . . . . . . . 173 4.2.3. Vapnik–Cervonenkis dimension of linear rules with

margin . . . . . . . . . . . . . . . . . . 176 4.2.4. Application to Support Vector Machines . . . . . 178 4.2.5. Inductive margin bounds . . . . . . . . . . . 179

Appendix . . . . . . . . . . . . . . . . . . . . . . . 185

5.1. Description of the model . . . . . . . . . . . . . 185 5.2. Computation of inductive bounds . . . . . . . . . . 186 5.3. Transductive bounds . . . . . . . . . . . . . . . 188

Bibliography . . . . . . . . . . . . . . . . . . . . . 191 4

Introduction

Among the possible approaches to pattern recognition, statistical learning theory has received a lot of attention in the last few years. Although a realistic pattern recognition scheme involves data pre-processing and postprocessing that need a theory of their own, a central role is often played by some kind of supervised learning algorithm. This central building block is the subject we are going to analyse in these notes.

Accordingly, we assume that we have prepared in some way or another a sample of N labelled patterns (X_{i}, Y_{i})_{i=1}^{N}, where X_{i} ranges in some pattern space 𝒳 and Y_{i} ranges in some finite label set 𝒴. We also assume that we have devised our experiment in such a way that the couples of random variables (X_{i}, Y_{i}) are independent (but not necessarily equidistributed). Here, randomness should be understood to come from the way the statistician has planned his experiment. He may for instance have drawn the X_{i}s at random from some larger population of patterns the algorithm is meant to be applied to in a second stage. The labels Y_{i} may have been set with the help of some external expertise (which may itself be faulty or contain some amount of randomness, so we do not assume that Y_{i} is a function of X_{i}, and allow the couple of random variables (X_{i}, Y_{i}) to follow any kind of joint distribution). In practice, patterns will be extracted from some high dimensional and highly structured data, such as digital images, speech signals, DNA sequences, etc. We will not discuss this pre-processing stage here, although it poses crucial problems dealing with segmentation and the choice of a representation. The aim of supervised classification is to choose some classification rule f : 𝒳 → 𝒴 which predicts Y from X making as few mistakes as possible on average.

The choice of f will be driven by a suitable use of the information provided by the sample (X_{i}, Y_{i})_{i=1}^{N} on the joint distribution of X and Y. Moreover, considering all the possible measurable functions f from 𝒳 to 𝒴 would not be feasible in practice and maybe more importantly not well founded from a statistical point of view, at least as soon as the pattern space 𝒳 is large and little is known in advance about the joint distribution of patterns X and labels Y. Therefore, we will consider parametrized subsets of classification rules {f_{θ} : 𝒳 → 𝒴; θ ∈ Θ_{m}}, m ∈ M, which may be grouped to form a big parameter set Θ = ⋃_{m∈M} Θ_{m}.

The subject of this monograph is to introduce to statistical learning theory, and more precisely to the theory of supervised classification, a number of technical tools akin to statistical mechanics and information theory, dealing with the concepts of entropy and temperature. A central task will in particular be to control the mutual information between an estimated parameter and the observed sample. The focus will not be directly on the description of the data to be classified, but on the description of the classification rules. As we want to deal with high dimensional data, we will be bound to consider high dimensional sets of candidate classification rules, and will analyse them with tools very similar to those used in statistical mechanics to describe particle systems with many degrees of freedom. More specifically, the sets of classification rules will be described by Gibbs measures defined on parameter sets and depending on the observed sample value. A Gibbs measure is the special kind of probability measure used in statistical mechanics to describe the state of a particle system driven by a given energy function at some given temperature. Here, Gibbs measures will emerge as minimizers of the average loss value under entropy (or mutual information) constraints. Entropy itself, more precisely the Kullback divergence function between probability measures, will emerge in conjunction with the use of exponential deviation inequalities: indeed, the log-Laplace transform may be seen as the Legendre transform of the Kullback divergence function, as will be stated in Lemma 1.1.3 (page 16).

To fix notation, let (X_{i}, Y_{i})_{i=1}^{N} be the canonical process on Ω = (𝒳 × 𝒴)^{N} (which means the coordinate process). Let the pattern space be provided with a sigma-algebra ℬ turning it into a measurable space (𝒳, ℬ). On the finite label space 𝒴, we will consider the trivial algebra ℬ′ made of all its subsets. Let ℳ_{+}^{1}[(𝒦 × 𝒴)^{N}, (ℬ ⊗ ℬ′)^{⊗N}] be our notation for the set of probability measures (i.e. of positive measures of total mass equal to 1) on the measurable space [(𝒳 × 𝒴)^{N}, (ℬ × ℬ′)^{⊗N}]. Once some probability distribution ℙ ∈ ℳ_{+}^{1}[(𝒳 × 𝒴)^{N}, (ℬ ⊗ ℬ′)^{⊗N}] is chosen, it turns (X_{i}, Y_{i})_{i=1}^{N} into the canonical realization of a stochastic process modelling the observed sample (also called the training set). We will assume that ℙ = ⨂_{i=1}^{N} P_{i}, where for each i = 1, …, N, P_{i} ∈ ℳ_{+}^{1}(𝒳 × 𝒴, ℬ ⊗ ℬ′), to reflect the assumption that we observe independent pairs of patterns and labels. We will also assume that we are provided with some indexed set of possible classification rules

ℛ_{Θ} = {f_{θ} : 𝒳 → 𝒴; θ ∈ Θ},

where (Θ, 𝒯) is some measurable index set. Assuming some indexation of the classification rules is just a matter of presentation. Although it leads to heavier notation, it allows us to integrate over the space of classification rules as well as over Ω, using the usual formalism of multiple integrals. For this matter, we will assume that (θ, x) ↦ f_{θ}(x) : (Θ × 𝒳, ℬ ⊗ 𝒯) → (𝒴, ℬ′) is a measurable function.

In many cases, as already mentioned, Θ = ⋃_{m∈M} Θ_{m} will be a finite (or more generally countable) union of subspaces, dividing the classification model ℛ_{Θ} = ⋃_{m∈M} ℛ_{Θ_{m}} into a union of sub-models. The importance of introducing such a structure has been put forward by V. Vapnik, as a way to avoid making strong hypotheses on the distribution ℙ of the sample. If neither the distribution of the sample nor the set of classification rules were constrained, it is well known that no kind of statistical inference would be possible. Considering a family of sub-models is a way to provide for adaptive classification where the choice of the model depends on the observed sample. Restricting the set of classification rules is more realistic than restricting the distribution of patterns, since the classification rules are a processing tool left to the choice of the statistician, whereas the distribution of the patterns is not fully under his control, except for some planning of the learning experiment which may enforce some weak properties like independence, but not the precise shapes of the marginal distributions P_{i} which are as a rule unknown distributions on some high dimensional space.

In these notes, we will concentrate on general issues concerned with a natural measure of risk, namely the expected error rate of each classification rule f_{θ}, expressed as

R(θ) = \frac{1}{N} ∑_{i=1}^{N} ℙ[f_{θ}(X_{i}) ≠ Y_{i}].  (0.1)

As this quantity is unobserved, we will be led to work with the corresponding empirical error rate

r(θ, ω) = \frac{1}{N} ∑_{i=1}^{N} 𝟙[f_{θ}(X_{i}) ≠ Y_{i}].  (0.2)

This does not mean that practical learning algorithms will always try to minimize this criterion. They often on the contrary try to minimize some other criterion which is linked with the structure of the problem and has some nice additional properties (like smoothness and convexity, for example). Nevertheless, and independently of the precise form of the estimator θ̂: Ω → Θ under study, the analysis of R(θ̂) is a natural question, and often corresponds to what is required in practice.

Answering this question is not straightforward because, although R(θ) is the expectation of r(θ), a sum of independent Bernoulli random variables, R(θ̂) is not the expectation of r(θ̂), because of the dependence of θ̂ on the sample, and neither is r(θ̂) a sum of independent random variables. To circumvent this unfortunate situation, some uniform control over the deviations of r from R is needed.

We will follow the PAC-Bayesian approach to this problem, originated in the machine learning community and pioneered by McAllester (1998, 1999). It can be seen as some variant of the more classical approach of M-estimators relying on empirical process theory — as described for instance in Van de Geer (2000).

It is built on some general principles:

• One idea is to embed the set of estimators of the type θ̂ : Ω → Θ into the larger set of regular conditional probability measures ρ : (Ω, (ℬ ⊗ ℬ′)^{⊗N}) → ℳ_{+}^{1}(Θ, 𝒯). We will call these conditional probability measures posterior distributions, to follow standard terminology.

• A second idea is to measure the fluctuations of ρ with respect to the sample, using some prior distribution π ∈ ℳ_{+}^{1}(Θ, 𝒯), and the Kullback divergence function 𝒦(ρ, π). The expectation ℙ{𝒦(ρ, π)} measures the randomness of ρ. The optimal choice of π would be ℙ(ρ), resulting in a measure of the randomness of ρ equal to the mutual information between the sample and the estimated parameter drawn from ρ. Anyhow, since ℙ(ρ) is usually not better known than ℙ, we will have to be content with some less concentrated prior distribution π, resulting in some looser measure of randomness, as shown by the identity ℙ[𝒦(ρ, π)] = ℙ{𝒦[ρ, ℙ(ρ)]} + 𝒦[ℙ(ρ), π].

• A third idea is to analyse the fluctuations of the random process θ ↦ r(θ) from its mean process θ ↦ R(θ) through the log-Laplace transform

−\frac{1}{λ} log \{∫∫ exp[−λr(θ, ω)]π(dθ)ℙ(dω)\} ,

as would be done in statistical mechanics, where this is called the free energy. This transform is well suited to relate min_{θ∈Θ} r(θ) to inf_{θ∈Θ} R(θ), since for large enough values of the parameter λ, corresponding to low enough values of the temperature, the system has small fluctuations around its ground state.

• A fourth idea deals with localization. It consists of considering a prior distribution \overline{π} depending on the unknown expected error rate function R. Thus some central result of the theory will consist in an empirical upper bound for 𝒦[ρ, π_{exp(−βR)}], where π_{exp(−βR)}, defined by its density

\frac{d}{dπ} [π_{exp(−βR)}] = \frac{exp(−βR)}{π[exp(−βR)]},

is a Gibbs distribution built from a known prior distribution π ∈ ℳ_{+}^{1}(Θ, 𝒯), some inverse temperature parameter β ∈ ℝ_{+} and the expected error rate R. This bound will in particular be used when ρ is a posterior Gibbs distribution, of the form π_{exp(−βr)}. The general idea will be to show that in the case when ρ is not too random, in the sense that it is possible to find a prior (that is non-random) distribution \overline{π} such that 𝒦(ρ, \overline{π}) is small, then ρ(r) can be reliably taken for a good approximation of ρ(R).

This monograph is divided into four chapters. The first deals with the inductive setting presented in these lines. The second is devoted to relative bounds. It shows that it is possible to obtain a tighter estimate of the mutual information between the sample and the estimated parameter by comparing prior and posterior Gibbs distributions. It shows how to use this idea to obtain adaptive model selection schemes under very weak hypotheses.

The third chapter introduces the transductive setting of V. Vapnik (Vapnik, 1998), which consists in comparing the performance of classification rules on the learning sample with their performance on a test sample instead of their average performance. The fourth one is a fast introduction to Support Vector Machines. It is the occasion to show the implications of the general results discussed in the three first chapters when some particular choice is made about the structure of the classification rules.

In the first chapter, two types of bounds are shown. Empirical bounds are useful to build, compare and select estimators. Non random bounds are useful to assess the speed of convergence of estimators, relating this speed to the behaviour of the Gibbs prior expected error rate β ↦ π_{exp(−βR)}(R) and to covariance factors related to the margin assumption of Mammen and Tsybakov when a finer analysis is performed. We will proceed from the most straightforward bounds towards more elaborate ones, built to achieve a better asymptotic behaviour. In this course towards more sophisticated inequalities, we will introduce local bounds and relative bounds.

The study of relative bounds is expanded in the third chapter, where tighter comparisons between prior and posterior Gibbs distributions are proved. Theorems 2.1.3 (page 72) and 2.2.4 (page 93) present two ways of selecting some nearly optimal classification rule. They are both proved to be adaptive in all the parameters under Mammen and Tsybakov margin assumptions and parametric complexity assumptions. This is done in Corollary 2.1.17 (page 86) of Theorem 2.1.15 (page 85) and in Theorem 2.2.11 (page 110). In the first approach, the performance of a randomized estimator modelled by a posterior distribution is compared with the performance of a prior Gibbs distribution. In the second approach posterior distributions are directly compared between themselves (and leads to slightly stronger results, to the price of using a more complex algorithm). When there are more than one parametric model, it is appropriate to use also some doubly localized scheme: two step localization is presented for both approaches, in Theorems 2.3.2 (page 116) and 2.3.9 (page 131) and provides bounds with a decreased influence of the number of empirically inefficient models included in the selection scheme.

We would not like to induce the reader into thinking that the most sophisticated results presented in these first two chapters are necessarily the most useful ones, they are as a rule only more efficient asymptotically, whereas, being more involved, they use looser constants leading to less precision for small sample sizes. In practice whether a sample is to be considered small is a question of the ratio between the number of examples and the complexity (roughly speaking the number of parameters) of the model used for classification. Since our aim here is to describe methods appropriate for complex data (images, speech, DNA, …), we suspect that practitioners wanting to make use of our proposals will often be confronted with small sample sizes; thus we would advise them to try the simplest bounds first and only afterwards see whether the asymptotically better ones can bring some improvement.

We would also like to point out that the results of the first two chapters are not of a purely theoretical nature: posterior parameter distributions can indeed be computed effectively, using Monte Carlo techniques, and there is well-established know-how about these computations in Bayesian statistics. Moreover, non-randomized estimators of the classical form θ̂: Ω → Θ can be efficiently approximated by posterior distributions ρ : Ω → ℳ_{+}^{1}(Θ) supported by a fairly narrow neighbourhood of θ̂, more precisely a neighbourhood of the size of the typical fluctuations of θ̂, so that this randomized approximation of θ̂ will most of the time provide the same classification as θ̂ itself, except for a small amount of dubious examples for which the classification provided by θ̂ would anyway be unreliable. This is explained on page 20.

As already mentioned, the third chapter is about the transductive setting, that is about comparing the performance of estimators on a training set and on a test set. We show first that this comparison can be based on a set of exponential deviation inequalities which parallels the one used in the inductive case. This gives the opportunity to transport all the results obtained in the inductive case in a systematic way. In the transductive setting, the use of prior distributions can be extended to the use of partially exchangeable posterior distributions depending on the union of training and test patterns, bringing increased possibilities to adapt to the data and giving rise to such crucial notions of complexity as the Vapnik–Cervonenkis dimension.

Having done so, we more specifically focus on the small sample case, where local and relative bounds are not expected to be of great help. Introducing a fictitious (that is unobserved) shadow sample, we study Vapnik-type generalization bounds, showing how to tighten and extend them with some original ideas, like making no Gaussian approximation to the log-Laplace transform of Bernoulli random variables, using a shadow sample of arbitrary size. shrinking from the use of any symmetrization trick, and using a suitable subset of the group of permutations to cover the case of independent non-identically distributed data. The culminating result of the third chapter is Theorem 3.3.3 (page 153), subsequent bounds showing the separate influence of the above ideas and providing an easier comparison with Vapnik’s original results. Vapnik-type generalization bounds have a broad applicability, not only through the concept of Vapnik–Cervonenkis dimension, but also through the use of compression schemes (Little et al., 1986), which are briefly described on page 144.

The beginning of the fourth chapter introduces Support Vector Machines, both in the separable and in the non-separable case (using the box constraint). We then describe different types of bounds. We start with compression scheme bounds, to proceed with margin bounds. We begin with transductive margin bounds, recalling on this occasion in Theorem 4.2.2 (page 174) the growth bound for a family of classification rules with given Vapnik–Cervonenkis dimension. In Theorem 4.2.4 (page 176) we give the usual estimate of the Vapnik–Cervonenkis dimension of a family of separating hyperplanes with a given transductive margin (we mean by this that the margin is computed on the union of the training and test sets). We present an original probabilistic proof inspired by a similar one from Cristianini et al. (2000), whereas other proofs available usually rely on the informal claim that the simplex is the worst case. We end this short review of Support Vector Machines with a discussion of inductive margin bounds. Here the margin is computed on the training set only, and a more involved combinatorial lemma, due to Alon et al. (1997) and recalled in Lemma 4.2.6 (page 180) is used. We use this lemma and the results of the third chapter to establish a bound depending on the margin of the training set alone.

In appendix, we finally discuss the textbook example of classification by thresholding: in this setting, each classification rule is built by thresholding a series of measurements and taking a decision based on these thresholded values. This relatively simple example (which can be considered as an introduction to the more technical case of classification trees) can be used to give more flesh to the results of the first three chapters.

It is a pleasure to end this introduction with my greatest thanks to Anthony Davison, for his careful reading of the manuscript and his numerous suggestions.

12  Introduction

Chapter 1

Inductive PAC-Bayesian learning

The setting of inductive inference (as opposed to transductive inference to be discussed later) is the one described in the introduction.

When we will have to take the expectation of a random variable Z : Ω → ℝ as well as of a function of the parameter h : Θ → ℝ with respect to some probability measure, we will as a rule use short functional notation instead of resorting to the integral sign: thus we will write ℙ(Z) for ∫_{Ω} Z(ω)ℙ(dω) and π(h) for ∫_{Θ} h(θ)π(dθ).

A more traditional statistical approach would focus on estimators θ̂ : Ω → Θ of the parameter θ and be interested on the relationship between the empirical error rate r(θ̂), defined by equation (0.1, page 7), which is the number of errors made on the sample, and the expected error rate R(θ̂), defined by equation (0.2, page 7), which is the expected probability of error on new instances of patterns. The PAC-Bayesian approach instead chooses a broader perspective and allows the estimator θ̂ to be drawn at random using some auxiliary source of randomness to smooth the dependence of θ̂ on the sample. One way of representing the supplementary randomness allowed in the choice of θ̂, is to consider what it is usual to call posterior distributions on the parameter space, that is probability measures ρ : Ω → ℳ_{+}^{1}(Θ, 𝒯), depending on the sample, or from a technical perspective, regular conditional (or transition) probability measures. Let us recall that we use the model described in the introduction: the training sample is modelled by the canonical process (X_{i}, Y_{i})_{i=1}^{N} on Ω = (𝒳 × 𝒴)^{N}, and a product probability measure ℙ = ⨂_{i=1}^{N} P_{i} on Ω is considered to reflect the assumption that the training sample is made of independent pairs of patterns and labels. The transition probability measure ρ, along with ℙ ∈ ℳ_{+}^{1}(Ω), defines a probability distribution on Ω × Θ and describes the conditional distribution of the estimated parameter θ̂ knowing the sample (X_{i}, Y_{i})_{i=1}^{N}.

The main subject of this broadened theory becomes to investigate the relationship between ρ(r), the average error rate of θ̂ on the training sample, and ρ(R), the expected error rate of θ̂ on new samples. The first step towards using some kind of thermodynamics to tackle this question, is to consider the Laplace transform of ρ(R) − ρ(r), a well known provider of non-asymptotic deviation bounds. This transform takes the form

ℙ{exp[λ[ρ(R) − ρ(r)]]},

where some inverse temperature parameter λ ∈ ℝ_{+}, as a physicist would call it, is introduced. This Laplace transform would be easy to bound if ρ did not depend on ω ∈ Ω (namely on the sample), because ρ(R) would then be non-random, and

ρ(r) = \frac{1}{N} ∑_{i=1}^{N} ρ[Y_{i} ≠ f_{θ}(X_{i})],

would be a sum of independent random variables. It turns out, and this will be the subject of the next section, that this annoying dependence of ρ on ω can be quantified, using the inequality

ρ(R) − ρ(r) ≤ λ^{−1} log{π[exp[λ(R − r)]]} + λ^{−1}𝒦(ρ, π),

which holds for any probability measure π ∈ ℳ_{+}^{1}(Θ) on the parameter space; for our purpose it will be appropriate to consider a prior distribution π that is non-random, as opposed to ρ, which depends on the sample. Here, 𝒦(ρ, π) is the Kullback divergence of ρ from π, whose definition will be recalled when we will come to technicalities; it can be seen as an upper bound for the mutual information between the (X_{i}, Y_{i})_{i=1}^{N} and the estimated parameter θ̂. This inequality will allow us to relate the penalized difference ρ(R)−ρ(r)−λ^{−1}𝒦(ρ, π) with the Laplace transform of sums of independent random variables.

1.1. Basic inequality

Let us now come to the details of the investigation sketched above. The first thing we will do is to study the Laplace transform of R(θ) − r(θ), as a starting point for the more general study of ρ(R) − ρ(r): it corresponds to the simple case where θ̂ is not random at all, and therefore where ρ is a Dirac mass at some deterministic parameter value θ.

In the setting described in the introduction, let us consider the Bernoulli random variables σ_{i}(θ) = 𝟙[Y_{i} ≠ f_{θ}(X_{i})], which indicates whether the classification rule f_{θ} made an error on the ith component of the training sample. Using independence and the concavity of the logarithm function, it is readily seen that for any real constant λ

log\{ℙ\{exp[−λr(θ)]\}\} = ∑_{i=1}^{N} log\{ℙ[exp(−\frac{λ}{N} σ_{i})]\}

≤ N log\{\frac{1}{N} ∑_{i=1}^{N} ℙ[exp(−\frac{λ}{N} σ_{i})]\}.

The right-hand side of this inequality is the log-Laplace transform of a Bernoulli distribution with parameter \frac{1}{N} ∑_{i=1}^{N} ℙ(σ_{i}) = R(θ). As any Bernoulli distribution is fully defined by its parameter, this log-Laplace transform is necessarily a function of R(θ). It can be expressed with the help of the family of functions

Φ_{a}(p) = −a^{−1} log{1 − [1 − exp(−a)]p}, a ∈ ℝ, p ∈ (0, 1).  (1.1)

It is immediately seen that Φ_{a} is an increasing one-to-one mapping of the unit interval onto itself, and that it is convex when a > 0, concave when a < 0 and can be defined by continuity to be the identity when a = 0. Moreover the inverse of Φ_{a} is given by the formula

Φ_{a}^{−1}(q) = \frac{1 − exp(−aq)}{1 − exp(−a)}, a ∈ ℝ, q ∈ (0, 1).

This formula may be used to extend Φ_{a}^{−1} to q ∈ ℝ, and we will use this extension without further notice when required.

Using this notation, the previous inequality becomes

log\{ℙ\{exp[−λr(θ)]\}\} ≤ −λΦ_{\frac{λ}{N}} [R(θ)], proving

Lemma 1.1.1. For any real constant λ and any parameter θ ∈ Θ,

ℙ\{exp\{λ[Φ_{\frac{λ}{N}} [R(θ)] − r(θ)]\}\} ≤ 1.

In previous versions of this study, we had used some Bernstein bound, instead of this lemma. Anyhow, as it will turn out, keeping the log-Laplace transform of a Bernoulli instead of approximating it provides simpler and tighter results.

Lemma 1.1.1 implies that for any constants λ ∈ ℝ_{+} and ϵ ∈)0, 1),

ℙ[Φ_{\frac{λ}{N}} [R(θ)] + \frac{log(ϵ)}{λ} ≤ r(θ)] ≥ 1 − ϵ.

Choosing \overline{λ} ∈ arg max_{ℝ_{+}}Φ_{\frac{λ}{N}} [R(θ)] + \frac{log(ϵ)}{λ}, we deduce

Lemma 1.1.2. For any ϵ ∈)0, 1), any θ ∈ Θ,

ℙ\{R(θ) ≤ inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}}[r(θ) − \frac{log(ϵ)}{λ}]\} ≥ 1 − ϵ.

We will illustrate throughout these notes the bounds we prove with a small numerical example: in the case where N = 1000, ϵ = 0.01 and r(θ) = 0.2, we get with a confidence level of 0.99 that R(θ) ≤ .2402, this being obtained for λ = 234.

Now, to proceed towards the analysis of posterior distributions, let us put U_{λ}(θ, ω) = λ[Φ_{\frac{λ}{N}} [R(θ)] − r(θ, ω)] for short, and let us consider some prior probability distribution π ∈ ℳ_{+}^{1}(Θ, 𝒯). A proper choice of π will be an important question, underlying much of the material presented in this monograph, so for the time being, let us only say that we will let this choice be as open as possible by writing inequalities which hold for any choice of π . Let us insist on the fact that when we say that π is a prior distribution, we mean that it does not depend on the training sample (X_{i}, Y_{i})_{i=1}^{N}. The quantity of interest to obtain the bound we are looking for is log{ℙ[π[exp(U_{λ})]]}. Using Fubini’s theorem for non-negative functions, we see that

log{ℙ[π[exp(U_{λ})]]} = log{π[ℙ[exp(U_{λ})]]} ≤ 0.

To relate this quantity to the expectation ρ(U_{λ}) with respect to any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), we will use the properties of the Kullback divergence 𝒦(ρ, π) of ρ with respect to π, which is defined as

𝒦(ρ, π) = \begin{cases}∫ log(\frac{dρ}{dπ})dρ, & when ρ is absolutely continuous with respect to π, \\ +∞, & otherwise. \end{cases}

The following lemma shows in which sense the Kullback divergence function can be thought of as the dual of the log-Laplace transform.

Lemma 1.1.3. For any bounded measurable function h : Θ → ℝ, and any probability distribution ρ ∈ ℳ_{+}^{1}(Θ) such that 𝒦(ρ, π) < ∞,

log{π[exp(h)]} = ρ(h) − 𝒦(ρ, π) + 𝒦(ρ, π_{exp(h)}),

where by definition \frac{dπ_{exp(h)}}{dπ} = \frac{exp[h(θ)]}{π[exp(h)]}. Consequently

log{π[exp(h)]]} = sup_{ρ∈ℳ_{+}^{1}(Θ)} ρ(h) − 𝒦(ρ, π).

The proof is just a matter of writing down the definition of the quantities involved and using the fact that the Kullback divergence function is non-negative, and can be found in Catoni (2004, page 160). In the duality between measurable functions and probability measures, we thus see that the log-Laplace transform with respect to π is the Legendre transform of the Kullback divergence function with respect to π. Using this, we get

ℙ{exp{ sup_{ρ∈ℳ_{+}^{1}(Θ)} ρ[U_{λ}(θ)] − 𝒦(ρ, π)}} ≤ 1,

which, combined with the convexity of λΦ_{\frac{λ}{N}}, proves the basic inequality we were looking for.

Theorem 1.1.4. For any real constant λ,

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ[Φ_{\frac{λ}{N}} [ρ(R)] − ρ(r)] − 𝒦(ρ, π)]\}

≤ ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ[ρ(Φ_{\frac{λ}{N}} ∘R) − ρ(r)] − 𝒦(ρ, π)]\} ≤ 1.

We insist on the fact that in this theorem, we take a supremum in ρ ∈ ℳ_{+}^{1}(Θ) inside the expectation with respect to ℙ, the sample distribution. This means that the proved inequality holds for any ρ depending on the training sample, that is for any posterior distribution: indeed, measurability questions set aside,

ℙ{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ[ρ[U_{λ}(θ)] − 𝒦(ρ, π)]]}

= sup_{ρ:Ω→ℳ_{+}^{1}(Θ)} ℙ{exp[λ[ρ[U_{λ}(θ)] − 𝒦(ρ, π)]]]},

and more formally,

sup_{ρ:Ω→ℳ_{+}^{1}(Θ)} ℙ{exp[λ[ρ[U_{λ}(θ)] − 𝒦(ρ, π)]]]}

≤ ℙ{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ[ρ[U_{λ}(θ)] − 𝒦(ρ, π)]]},

where the supremum in ρ taken in the left-hand side is restricted to regular conditional probability distributions.

The following sections will show how to use this theorem.

1.2. Non local bounds

At least three sorts of bounds can be deduced from Theorem 1.1.4.

The most interesting ones with which to build estimators and tune parameters, as well as the first that have been considered in the development of the PAC-Bayesian approach, are deviation bounds. They provide an empirical upper bound for ρ(R) — that is a bound which can be computed from observed data — with some probability 1 − ϵ, where ϵ is a presumably small and tunable parameter setting the desired confidence level.

Anyhow, most of the results about the convergence speed of estimators to be found in the statistical literature are concerned with the expectation ℙ[ρ(R)], therefore it is also enlightening to bound this quantity. In order to know at which rate it may be approaching inf_{Θ} R, a non-random upper bound is required, which will relate the average of the expected risk ℙ[ρ(R)] with the properties of the contrast function θ ↦ R(θ).

Since the values of constants do matter a lot when a bound is to be used to select between various estimators using classification models of various complexities, a third kind of bound, related to the first, may be considered for the sake of its hopefully better constants: we will call them unbiased empirical bounds, to stress the fact that they provide some empirical quantity whose expectation under ℙ can be proved to be an upper bound for ℙ[ρ(R)], the average expected risk. The price to pay for these better constants is of course the lack of formal guarantee given by the bound: two random variables whose expectations are ordered in a certain way may very well be ordered in the reverse way with a large probability, so that basing the estimation of parameters or the selection of an estimator on some unbiased empirical bound is a hazardous business. Anyhow, since it is common practice to use the inequalities provided by mathematical statistical theory while replacing the proven constants with smaller values showing a better practical efficiency, considering unbiased empirical bounds as well as deviation bounds provides an indication about how much the constants may be decreased while not violating the theory too much.

1.2.1. Unbiased empirical bounds. Let ρ : Ω → ℳ_{+}^{1}(Θ) be some fixed (and arbitrary) posterior distribution, describing some randomized estimator θ̂: Ω → Θ. As we already mentioned, in these notes a posterior distribution will always be a regular conditional probability measure. By this we mean that

• for any A ∈ 𝒯, the map ω ↦ ρ(ω, A) : (Ω, (ℬ ⊗ ℬ′)^{⊗N}) → ℝ_{+} is assumed to be measurable;

• for any ω ∈ Ω, the map A ↦ ρ(ω, A) : 𝒯 → ℝ_{+} is assumed to be a probability measure.

We will also assume without further notice that the σ-algebras we deal with are always countably generated. The technical implications of these assumptions are standard and discussed for instance in Catoni (2004, pages 50-54), where, among other things, a detailed proof of the decomposition of the Kullback Liebler divergence is given.

Let us restrict to the case when the constant λ is positive. We get from Theorem 1.1.4 that

exp[λ\{Φ_{\frac{λ}{N}} [ℙ[ρ(R)]] − ℙ[ρ(r)]\} − ℙ[𝒦(ρ, π)]] ≤ 1,  (1.2)

where we have used the convexity of the exp function and of Φ_{\frac{λ}{N}}. Since we have restricted our attention to positive values of the constant λ, equation (1.2) can also be written

ℙ[ρ(R)] ≤ Φ−1_{\frac{λ}{N}}\{ℙ[ρ(r) + λ^{−1}𝒦(ρ, π)]\},

leading to

Theorem 1.2.1. For any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), for any positive parameter λ,

ℙ[ρ(R)] ≤ \frac{1 − exp[−N^{−1}ℙ[λρ(r) + 𝒦(ρ, π)]]}{1 − exp(−\frac{λ}{N})}

≤ ℙ\{\frac{λ}{N[1 − exp(−\frac{λ}{N})]} [ρ(r) + \frac{𝒦(ρ, π)}{λ}]\}.

The last inequality provides the unbiased empirical upper bound for ρ(R) we were looking for, meaning that the expectation of \frac{λ}{N[1−exp(− \frac{λ}{N})]} [ρ(r) + \frac{𝒦(ρ,π)}{λ} ] is larger than the expectation of ρ(R). Let us

notice that 1 ≤ _{\frac{λ}{N[1−exp(− \frac{λ}{N})]}} ≤ [1 − \frac{λ}{2N} ]^{−1} and therefore that this coeffi-

cient is close to 1 when λ is significantly smaller than N.

If we are ready to believe in this bound (although this belief is not mathematically well founded, as we already mentioned), we can use it to optimize λ and to choose ρ. While the optimal choice of ρ when λ is fixed is, according to Lemma 1.1.3 (page 16), to take it equal to π_{exp(−λr)}, a Gibbs posterior distribution, as it is sometimes called, we may for computational reasons be more interested in choosing ρ in some other class of posterior distributions.

For instance, our real interest may be to select some non-randomized estimator from a family θ̂_{m} : Ω → Θ_{m}, m ∈ M, of possible ones, where Θ_{m} are measurable subsets of Θ and where M is an arbitrary (non necessarily countable) index set. We may for instance think of the case when θ̂_{m} ∈ arg min_{Θ_{m}} r. We may slightly randomize the estimators to start with, considering for any θ ∈ Θ_{m} and any m ∈ M,

Δ_{m}(θ) = {θ′ ∈ Θ_{m} : [f_{θ′}(X_{i})]_{i=1}^{N} = [f_{θ}(X_{i})]_{i=1}^{N}},

and defining ρ_{m} by the formula

\frac{dρ_{m}}{dπ} (θ) = \frac{𝟙[θ ∈ Δ_{m}(θ̂_{m})]}{π[Δ_{m}(θ̂_{m})]}.

Our posterior minimizes 𝒦(ρ, π) among those distributions whose support is restricted to the values of θ in Θ_{m} for which the classification rule f_{θ} is identical to the estimated one fθ̂_{m} on the observed sample. Presumably, in many practical situations, f_{θ}(x) will be ρ_{m} almost surely identical to fθ̂_{m}(x) when θ is drawn from ρ_{m}, for the vast majority of the values of x ∈ 𝒳 and all the sub-models Θ_{m} not plagued with too much overfitting (since this is by construction the case when x ∈ {X_{i} : i = 1, …, N}). Therefore replacing θ̂_{m} with ρ_{m} can be expected to be a minor change in many situations. This change by the way can be estimated in the (admittedly not so common) case when the distribution of the patterns (X_{i})_{i=1}^{N} is known. Indeed, introducing the pseudo distance

D(θ, θ′) = \frac{1}{N} ∑_{i=1}^{N} ℙ[f_{θ}(X_{i}) ≠ f_{θ′}(X_{i})],  θ, θ′ ∈ Θ,  (1.3)

one immediately sees that R(θ′) ≤ R(θ) + D(θ, θ′), for any θ, θ′ ∈ Θ, and therefore that

R(θ̂_{m}) ≤ ρ_{m}(R) + ρ_{m}[D(·, θ̂_{m})].

Let us notice also that in the case where Θ_{m} ⊂ ℝ^{d_{m}}, and R happens to be convex on Δ_{m}(θ̂_{m}), then ρ_{m}(R) ≥ R[∫ θρ_{m}(dθ)], and we can replace θ̂_{m} with θ̃_{m} = ∫ θρ_{m}(dθ), and obtain bounds for R(θ̃_{m}). This is not a very heavy assumption about R, in the case where we consider θ̂_{m} ∈ arg min_{Θ_{m}} r. Indeed, θ̂_{m}, and therefore Δ_{m}(θ̂_{m}), will presumably be close to arg min_{Θ_{m}} R, and requiring a function to be convex in the neighbourhood of its minima is not a very strong assumption.

Since r(θ̂_{m}) = ρ_{m}(r), and 𝒦(ρ_{m}, π) = − log{π[Δ_{m}(θ̂_{m})]}, our unbiased empirical upper bound in this context reads as

\frac{λ}{N[1 − exp(−\frac{λ}{N})]} \{r(θ̂_{m}) − \frac{log\{π[Δ_{m}(θ̂_{m})]\}}{λ}\} .

Let us notice that we obtain a complexity factor − log{π[Δ_{m}(θ̂_{m})]} which may be compared with the Vapnik–Cervonenkis dimension. Indeed, in the case of binary classification, when using a classification model with Vapnik– Cervonenkis dimension not greater than h_{m}, that is when any subset of 𝒳 which can be split in any arbitrary way by some classification rule f_{θ} of the model Θ_{m} has at most h_{m} points, then

{Δ_{m}(θ) : θ ∈ Θ_{m}}

is a partition of Θ_{m} with at most (\frac{eN}{h_{m}} )^{h_{m}} components: these facts, if not already familiar to the reader, will be proved in Theorems 4.2.2 and 4.2.3 (page 174). Therefore

inf_{θ∈Θ_{m}} − log\{π[Δ_{m}(θ)]\} ≤ h_{m} log (\frac{eN}{h_{m}} ) − log[π(Θ_{m})].

Thus, if the model and prior distribution are well suited to the classification task, in the sense that there is more “room” (where room is measured with π) between the two clusters defined by θ̂_{m} than between other partitions of the sample of patterns (X_{i})_{i=1}^{N}, then we will have

− log\{π[Δ_{m}(θ̂)]\} ≤ h_{m} log (\frac{eN}{h_{m}} ) − log[π(Θ_{m})].

An optimal value m̂ may be selected so that

m̂ ∈ arg min_{m∈M} \{ inf_{λ∈ℝ_{+}} \frac{λ}{N[1 − exp(−\frac{λ}{N})]} (r(θ̂_{m}) − \frac{log\{π[Δ_{m}(θ̂_{m})]\}}{λ})\} .

Since ρm̂ is still another posterior distribution, we can be sure that

ℙ{R(θ̂m̂) − ρm̂[D(·, θ̂m̂)]} ≤ ℙ[ρm̂(R)]

≤ inf_{λ∈ℝ_{+}} ℙ \{\frac{λ}{N[1 − exp(−\frac{λ}{N})]} (r(θ̂m̂) − \frac{log\{π[Δm̂(θ̂m̂)]\}}{λ})\} .

Taking the infimum in λ inside the expectation with respect to ℙ would be possible at the price of some supplementary technicalities and a slight increase of the bound that we prefer to postpone to the discussion of deviation bounds, since they are the only ones to provide a rigorous mathematical foundation to the adaptive selection of estimators.

1.2.2. Optimizing explicitly the exponential parameter λ. In this section we address some technical issues we think helpful to the understanding of Theorem 1.2.1 (page 19): namely to investigate how the upper bound it provides could be optimized, or at least approximately optimized, in λ. It turns out that this can be done quite explicitly.

So we will consider in this discussion the posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ) to be fixed, and our aim will be to eliminate the constant λ from the bound by choosing its value in some nearly optimal way as a function of ℙ[ρ(r)], the average of the empirical risk, and of ℙ[𝒦(ρ, π)], which controls overfitting.

Let the bound be written as

φ(λ) = [1 − exp(−\frac{λ}{N})]^{−1} \{1 − exp[−\frac{λ}{N} ℙ[ρ(r)] − N^{−1}ℙ[𝒦(ρ, π)]]\} .

We see that

N \frac{∂}{∂λ} log[φ(λ)] = \frac{ℙ[ρ(r)]}{exp[\frac{λ}{N} ℙ[ρ(r)] + N^{−1}ℙ[𝒦(ρ, π)]] − 1} − \frac{1}{exp(\frac{λ}{N}) − 1}.

Thus, the optimal value for λ is such that

[exp(\frac{λ}{N}) − 1]ℙ[ρ(r)] = exp[\frac{λ}{N} ℙ[ρ(r)] + N^{−1}ℙ[𝒦(ρ, π)]] − 1.

Assuming that 1 ≫ \frac{λ}{N} ℙ[ρ(r)] ≫ \frac{ℙ[𝒦(ρ,π)]}{N} , and keeping only higher order terms, we are led to choose

λ = \sqrt{\frac{2Nℙ[𝒦(ρ, π)]}{ℙ[ρ(r)]\{1 − ℙ[ρ(r)]\}}},

obtaining

Theorem 1.2.2. For any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ[ρ(R)] ≤ \frac{1 − exp \{−\sqrt{\frac{2ℙ[𝒦(ρ,π)]ℙ[ρ(r)]}{N\{1−ℙ[ρ(r)]\}}} − \frac{ℙ[𝒦(ρ,π)]}{N} \}}{1 − exp \{−\sqrt{\frac{2ℙ[𝒦(ρ,π)]}{Nℙ[ρ(r)]\{1−ℙ[ρ(r)]\}}} \}}.

This result of course is not very useful in itself, since neither of the two quantities ℙ[ρ(r)] and ℙ[𝒦(ρ, π)] are easy to evaluate. Anyhow it gives a hint that replacing them boldly with ρ(r) and 𝒦(ρ, π) could produce something close to a legitimate empirical upper bound for ρ(R). We will see in the subsection about deviation bounds that this is indeed essentially true.

Let us remark that in the third chapter of this monograph, we will see another way of bounding

inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}} (q + \frac{d}{λ}) , leading to

Theorem 1.2.3. For any prior distribution π ∈ ℳ_{+}^{1}(Θ), for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ[ρ(R)] ≤ (1 + \frac{2ℙ[𝒦(ρ, π)]}{N})^{−1} \{ℙ[ρ(r)] + \frac{ℙ[𝒦(ρ, π)]}{N}

+ \sqrt{\frac{2ℙ[𝒦(ρ, π)]ℙ[ρ(r)]\{1 − ℙ[ρ(r)]\}}{N} + \frac{ℙ[𝒦(ρ, π)]^{2}}{N^{2}}} \},

as soon as ℙ[ρ(r)] + \sqrt{\frac{ℙ[𝒦(ρ, π)]}{2N}} ≤ \frac{1}{2},

and ℙ[ρ(R)] ≤ ℙ[ρ(r)] + \sqrt{\frac{ℙ[𝒦(ρ, π)]}{2N}} otherwise.

This theorem enlightens the influence of three terms on the average expected risk:

• the average empirical risk, ℙ[ρ(r)], which as a rule will decrease as the size of the classification model increases, acts as a bias term, grasping the ability of the model to account for the observed sample itself;

• a variance term \frac{1}{N} ℙ[ρ(r)]\{1 − ℙ[ρ(r)]\} is due to the random fluctuations of ρ(r);

• a complexity term ℙ[𝒦(ρ, π)], which as a rule will increase with the size of the classification model, eventually acts as a multiplier of the variance term.

We observed numerically that the bound provided by Theorem 1.2.2 is better than the more classical Vapnik-like bound of Theorem 1.2.3. For instance, when N = 1000, ℙ[ρ(r)] = 0.2 and ℙ[𝒦(ρ, π)] = 10, Theorem 1.2.2 gives a bound lower than 0.2604, whereas the more classical Vapnik-like approximation of Theorem 1.2.3 gives a bound larger than 0.2622. Numerical simulations tend to suggest the two bounds are always ordered in the same way, although this could be a little tedious to prove mathematically.

1.2.3. Non random bounds. It is time now to come to less tentative results and see how far is the average expected error rate ℙ[ρ(R)] from its best possible value inf_{Θ} R.

Let us notice first that

λρ(r) + 𝒦(ρ, π) = 𝒦(ρ, π_{exp(−λr)}) − log{π[exp(−λr)]}.

Let us remark moreover that r ↦ log[π[exp(−λr)]] is a convex functional, a property which from a technical point of view can be dealt with in the following way:

ℙ{log[π[exp(−λr)]]} = ℙ{ sup_{ρ∈ℳ_{+}^{1}(Θ)} −λρ(r) − 𝒦(ρ, π)}

≥ sup_{ρ∈ℳ_{+}^{1}(Θ)} ℙ{−λρ(r) − 𝒦(ρ, π)} = sup_{ρ∈ℳ_{+}^{1}(Θ)} −λρ(R) − 𝒦(ρ, π)

= log{π[exp(−λR)]} = −∫_{0}^{λ}π_{exp(−βR)}(R)dβ. (1.4)

These remarks applied to Theorem 1.2.1 lead to

Theorem 1.2.4. For any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), for any positive parameter λ,

ℙ[ρ(R)] ≤ \frac{1 − exp \{−\frac{1}{N} ∫_{0}^{λ} π_{exp(−βR)}(R)dβ − \frac{1}{N} ℙ[𝒦(ρ, π_{exp(−λr)})]\}}{1 − exp(−\frac{λ}{N})}

≤ \frac{1}{N[1 − exp(−\frac{λ}{N})]}\{∫_{0}^{λ}π_{exp(−βR)}(R)dβ + ℙ[𝒦(ρ, π_{exp(−λr)})]\}.

This theorem is particularly well suited to the case of the Gibbs posterior distribution ρ = π_{exp(−λr)}, where the entropy factor cancels and where ℙ[π_{exp(−λr)}(R)] is shown to get close to inf_{Θ} R when N goes to +∞, as soon as λ/N goes to 0 while λ goes to +∞.

We can elaborate on Theorem 1.2.4 and define a notion of dimension of (Θ, R), with margin η ≥ 0 putting

d_{η}(Θ, R) = sup_{β∈ℝ_{+}} β[π_{exp(−βR)}(R) − ess inf_{π} R − η]

≤ − log{π[R ≤ ess inf_{π} R + η]}. (1.5)

This last inequality can be established by the chain of inequalities:

βπ_{exp(−βR)}(R) ≤ ∫_{0}^{β}π_{exp(−γR)}(R)dγ = − log{π[exp(−βR)]}

≤ β(ess inf_{π} R + η) − log[π(R ≤ ess inf_{π} R + η)],

where we have used successively the fact that λ ↦ π_{exp(−λR)}(R) is decreasing (because it is the derivative of the concave function λ ↦ − log{π[exp(−λR)]}) and the fact that the exponential function takes positive values.

In typical “parametric” situations d_{0}(Θ, R) will be finite, and in all circumstances d_{η}(Θ, R) will be finite for any η > 0 (this is a direct consequence of the definition of the essential infimum). Using this notion of dimension, we see that

∫_{0}^{λ} π_{exp(−βR)}(R)dβ ≤ λ(ess inf_{π} R + η)

+ ∫_{0}^{λ} [\frac{d_{η}}{β} ∧ (1 − ess inf_{π} R − η)] dβ

= λ(ess inf_{π} R + η) + d_{η}(Θ, R) log [\frac{eλ}{d_{η}(Θ, R)}(1 − ess inf_{π} R − η)] .

This leads to

Corollary 1.2.5 With the above notation, for any margin η ∈ ℝ_{+}, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ[ρ(R)] ≤ inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}} [ess inf_{π} R + η + \frac{d_{η}}{λ} log (\frac{eλ}{d_{η}}) + \frac{ℙ\{𝒦[ρ, π_{exp(−λr)}]\}}{λ}] .

If one wants a posterior distribution with a small support, the theorem can also be applied to the case when ρ is obtained by truncating π_{exp(−λr)} to some level set to reduce its support: let Θ_{p} = {θ ∈ Θ : r(θ) ≤ p}, and let us define for any q ∈)0, 1) the level p_{q} = inf{p : π_{exp(−λr)}(Θ_{p}) ≥ q}, let us then define ρ_{q} by its density

\frac{dρ_{q}}{dπ_{exp(−λr)}} (θ) = \frac{𝟙(θ ∈ Θ_{p_{q}})}{π_{exp(−λr)}(Θ_{p_{q}})},

then ρ_{0} = π_{exp(−λr)} and for any q ∈ (0, 1(,

ℙ[ρ_{q}(R)] ≤ \frac{1 − exp \{−\frac{1}{N} ∫_{0}^{λ} π_{exp(−βR)}(R)dβ − \frac{log(q)}{N}\}}{1 − exp(−\frac{λ}{N})}

≤ \frac{1}{N[1 − exp(−\frac{λ}{N})]}\{∫_{0}^{λ}π_{exp(−βR)}(R)dβ − log(q)\}.

1.2.4. Deviation bounds. They provide results holding under the distribution ℙ of the sample with probability at least 1 − ϵ, for any given confidence level, set by the choice of ϵ ∈)0, 1(. Using them is the only way to be quite (i.e. with probability 1 − ϵ) sure to do the right thing, although this right thing may be over-pessimistic, since deviation upper bounds are larger than corresponding non-biased bounds.

Starting again from Theorem 1.1.4 (page 17), and using Markov’s inequality ℙ[exp(h) ≥ 1] ≤ ℙ[exp(h)], we obtain

Theorem 1.2.6. For any positive parameter λ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ρ(R) ≤ Φ−1_{\frac{λ}{N}} \{ρ(r) + \frac{𝒦(ρ, π) − log(ϵ)}{λ}\}

= \frac{1 − exp \{−\frac{λρ(r)}{N} − \frac{𝒦(ρ, π) − log(ϵ)}{N}\}}{1 − exp(−\frac{λ}{N} )}

≤ \frac{λ}{N [1 − exp (−\frac{λ}{N} )]} [ρ(r) + \frac{𝒦(ρ, π) − log(ϵ)}{λ}] .

We see that for a fixed value of the parameter λ, the upper bound is optimized when the posterior is chosen to be the Gibbs distribution ρ = π_{exp(−λr)}.

In this theorem, we have bounded ρ(R), the average expected risk of an estimator θ̂ drawn from the posterior ρ. This is what we will do most of the time in this study. This is the error rate we will get if we classify a large number of test patterns, drawing a new θ̂ for each one. However, we can also be interested in the error rate we get if we draw only one θ̂ from ρ and use this single draw of θ̂ to classify a large number of test patterns. This error rate is R(θ̂). To state a result about its deviations, we can start back from Lemma 1.1.1 (page 15) and integrate it with respect to the prior distribution π to get for any real constant λ

ℙ\{π[exp\{λ[Φ_{\frac{λ}{N}} (R) − r]\}]\} ≤ 1.

For any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), this can be rewritten as

ℙ\{ρ[exp\{λ[Φ_{\frac{λ}{N}} (R) − r] − log(\frac{dρ}{dπ} ) + log(ϵ)]\}]\} ≤ ϵ,

proving

Theorem 1.2.7 For any positive real parameter λ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), with ℙρ probability at least 1 − ϵ,

R(θ̂) ≤ Φ−1_{\frac{λ}{N}}\{r(θ̂) + λ^{−1} log(ϵ^{−1} \frac{dρ}{dπ} )\}

≤ \frac{λ}{N[1 − exp(−\frac{λ}{N})]}[r(θ̂) + λ^{−1} log(ϵ^{−1} \frac{dρ}{dπ} )].

Let us remark that the bound provided here is the exact counterpart of the bound of Theorem 1.2.6, since log(\frac{dρ}{dπ} ) appears as a disintegrated version of the divergence 𝒦(ρ, π). The parallel between the two theorems is particularly striking in the special case when ρ = π_{exp(−λr)}. Indeed Theorem 1.2.6 proves that with ℙ probability at least 1 − ϵ,

π_{exp(−λr)}(R) ≤ Φ−1_{\frac{λ}{N}}\{−\frac{log\{π[exp(−λr)]\} + log(ϵ)}{λ}\},

whereas Theorem 1.2.7 proves that with ℙπ_{exp(−λr)} probability at least 1−ϵ

R(θ̂) ≤ Φ−1_{\frac{λ}{N}}\{−\frac{log\{π[exp(−λr)]\} + log(ϵ)}{λ}\},

showing that we get the same deviation bound for π_{exp(−λr)}(R) under ℙ and for θ̂ under ℙπ_{exp(−λr)}.

We would like to show now how to optimize with respect to λ the bound given by Theorem 1.2.6 (the same discussion would apply to Theorem 1.2.7). Let us notice first that values of λ less than 1 are not interesting (because they provide a bound larger than one, at least as soon as ϵ ≤ exp(−1)). Let us consider some real parameter α > 1, and the set Λ = {α^{k}; k ∈ ℕ}, on which we put the probability measure ν(α^{k}) = [(k + 1)(k + 2)]^{−1}. Applying Theorem 1.2.6 to λ = α^{k} at confidence level 1 − \frac{ϵ}{(k+1)(k+2)}, and using a union bound, we see that with probability at least 1 − ϵ, for any posterior distribution ρ,

ρ(R) ≤ inf_{λ′∈Λ} Φ−1_{\frac{λ′}{N}} \{ρ(r) + \frac{𝒦(ρ, π) − log(ϵ) + 2 log[\frac{log(α^{2}λ′)}{log(α)}]}{λ′} \} .

Now we can remark that for any λ ∈ (1, +∞(, there is λ′ ∈ Λ such that α^{−1}λ ≤ λ′ ≤ λ. Moreover, for any q ∈ (0, 1), β ↦ Φ_{β}^{−1}(q) is increasing on ℝ_{+}. Thus with probability at least 1 − ϵ, for any posterior distribution ρ,

ρ(R) ≤ inf_{λ∈(1,∞(} Φ−1_{\frac{λ}{N}} \{ρ(r) + \frac{α}{λ} [𝒦(ρ, π) − log(ϵ) + 2 log(\frac{log(α^{2}λ)}{log(α)})]\}

= inf_{λ∈(1,∞(} \frac{1 − exp \{−\frac{λ}{N} ρ(r) − \frac{α}{N} [𝒦(ρ, π) − log(ϵ) + 2 log(\frac{log(α^{2}λ)}{log(α)})]\}}{1 − exp(−\frac{λ}{N})}.

Taking the approximately optimal value

λ = \sqrt{\frac{2Nα [𝒦(ρ, π) − log(ϵ)]}{ρ(r)[1 − ρ(r)]}},

we obtain

Theorem 1.2.8. With probability 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), putting d(ρ, ϵ) = 𝒦(ρ, π) − log(ϵ),

ρ(R) ≤ inf_{k∈ℕ} \frac{1 − exp \{−\frac{α^{k}}{N} ρ(r) − \frac{1}{N} [d(ρ, ϵ) + log[(k + 1)(k + 2)]]\}}{1 − exp (−\frac{α^{k}}{N} )}

≤ \frac{1 − exp \{−\sqrt{\frac{2αρ(r)d(ρ, ϵ)}{N[1 − ρ(r)]}} − \frac{α}{N} [d(ρ, ϵ) + 2 log(^{\frac{log(α^{2}\sqrt{\frac{2Nαd(ρ,ϵ)}{ρ(r)[1−ρ(r)]}} )}{log(α)}})]\}}{1 − exp [−\sqrt{\frac{2αd(ρ, ϵ)}{Nρ(r)[1 − ρ(r)]}}]}.

Moreover with probability at least 1 − ϵ, for any posterior distribution ρ such that ρ(r) = 0,

ρ(R) ≤ 1 − exp [−\frac{𝒦(ρ, π) − log(ϵ)}{N}] .

We can also elaborate on the results in an other direction by introducing the empirical dimension

d_{e} = sup_{β∈ℝ_{+}} β[π_{exp(−βr)}(r) − ess inf_{π} r] ≤ − log[π(r = ess inf_{π} r)]. (1.6)

There is no need to introduce a margin in this definition, since r takes at most N values, and therefore π(r = ess inf_{π} r) is strictly positive. This leads to

Corollary 1.2.9. For any positive real constant λ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ρ(R) ≤ Φ−1_{\frac{λ}{N}} [ess inf_{π} r + \frac{d_{e}}{λ} log (\frac{eλ}{d_{e}}) + \frac{𝒦[ρ, π_{exp(−λr)}] − log(ϵ)}{λ}] .

We could then make the bound uniform in λ and optimize this parameter in a way similar to what was done to obtain Theorem 1.2.8.

1.3. Local bounds

In this section, better bounds will be achieved through a better choice of the prior distribution. This better prior distribution turns out to depend on the unknown sample distribution ℙ, and some work is required to circumvent this and obtain empirical bounds.

1.3.1. Choice of the prior. As mentioned in the introduction, if one is willing to minimize the bound in expectation provided by Theorem 1.2.1 (page 19), one is led to consider the optimal choice π = ℙ(ρ). However, this is only an ideal choice, since ℙ is in all conceivable situations unknown. Nevertheless it shows that it is possible through Theorem 1.2.1 to measure the complexity of the classification model with ℙ{𝒦[ρ, ℙ(ρ)]}, which is nothing but the mutual information between the random sample (X_{i}, Y_{i})_{i=1}^{N} and the estimated parameter θ̂, under the joint distribution ℙρ.

In practice, since we cannot choose π = ℙ(ρ), we have to be content with a flat prior π, resulting in a bound measuring complexity according to ℙ[𝒦(ρ, π)] = ℙ{𝒦[ρ, ℙ(ρ)]} + 𝒦[ℙ(ρ), π] larger by the entropy factor 𝒦[ℙ(ρ), π] than the optimal one (we are still commenting on Theorem 1.2.1).

If we want to base the choice of π on Theorem 1.2.4 (page 23), and if we choose ρ = π_{exp(−λr)} to optimize this bound, we will be inclined to choose some π such that

\frac{1}{λ}∫_{0}^{λ}π_{exp(−βR)}(R)dβ = −\frac{1}{λ} log\{π[exp(−λR)]\}

is as far as possible close to inf_{θ∈Θ} R(θ) in all circumstances. To give a more specific example, in the case when the distribution of the design (X_{i})_{i=1}^{N} is known, one can introduce on the parameter space Θ the metric D already defined by equation (1.3, page 20) (or some available upper bound for this distance). In view of the fact that R(θ) − R(θ′) ≤ D(θ, θ′), for any θ, θ′ ∈ Θ, it can be meaningful, at least theoretically, to choose π as

π = ∑_{k=1}^{∞} \frac{1}{k(k + 1)}π_{k},

where π_{k} is the uniform measure on some minimal (or close to minimal) 2^{−k}-net 𝒩(Θ, D, 2^{−k}) of the metric space (Θ, D). With this choice

− \frac{1}{λ} log\{π[exp(−λR)]\} ≤ inf_{θ∈Θ} R(θ)

+ inf_{k} \{2^{−k} + \frac{log(|𝒩(Θ, D, 2^{−k})|) + log[k(k + 1)]}{λ}\} .

Another possibility, when we have to deal with real valued parameters, meaning that Θ ⊂ ℝ^{d}, is to code each real component θ_{i} ∈ ℝ of θ = (θ_{i})_{i=1}^{d} to some precision and to use a prior μ which is atomic on dyadic numbers. More precisely let us parametrize the set of dyadic real numbers as

𝒟 = {r[s, m, p, (b_{j})_{j=1}^{p}] = s2^{m}(1 + ∑_{j=1}^{p} b_{j}2^{−j})

: s ∈ {−1, +1}, m ∈ ℤ, p ∈ ℕ, b_{j} ∈ {0, 1}},

where, as can be seen, s codes the sign, m the order of magnitude, p the precision and (b_{j})_{j=1}^{p} the binary representation of the dyadic number r[s, m, p, (b_{j})_{j=1}^{p}]. We can for instance consider on 𝒟 the probability distribution

μ{r[s, m, p, (b_{j})_{j=1}^{p}]} = [3(|m| + 1)(|m| + 2)(p + 1)(p + 2)2^{p}]^{−1}, (1.7)

and define π ∈ ℳ_{+}^{1}(ℝ^{d}) as π = μ^{⊗d}. This kind of “coding” prior distribution can be used also to define a prior on the integers (by renormalizing the restriction of μ to integers to get a probability distribution). Using μ is somehow equivalent to picking up a representative of each dyadic interval, and makes it possible to restrict to the case when the posterior ρ is a Dirac mass without losing too much (when Θ = (0, 1), this approach is somewhat equivalent to considering as prior distribution the Lebesgue measure and using as posterior distributions the uniform probability measures on dyadic intervals, with the advantage of obtaining non-randomized estimators). When one uses in this way an atomic prior and Dirac masses as posterior distributions, the bounds proven so far can be obtained through a simpler union bound argument. This is so true that some of the detractors of the PAC-Bayesian approach (which, as a newcomer, has sometimes received a suspicious greeting among statisticians) have argued that it cannot bring anything that elementary union bound arguments could not essentially provide. We do not share of course this derogatory opinion, and while we think that allowing for non atomic priors and posteriors is worthwhile, we also would like to stress that the upcoming local and relative bounds could hardly be obtained with the only help of union bounds.

Although the choice of a flat prior seems at first glance to be the only alternative when nothing is known about the sample distribution ℙ, the previous discussion shows that this type of choice is lacking proper localisation, and namely that we loose a factor 𝒦{ℙ[π_{exp(−λr)}], π}, the divergence between the bound-optimal prior ℙ[π_{exp(−λr)}], which is concentrated near the minima of R in favourable situations, and the flat prior π. Fortunately, there are technical ways to get around this difficulty and to obtain more local empirical bounds.

1.3.2. Unbiased local empirical bounds. The idea is to start with some flat prior π ∈ ℳ_{+}^{1}(Θ), and the posterior distribution ρ = π_{exp(−λr)} minimizing the bound of Theorem 1.2.1 (page 19), when π is used as a prior. To improve the bound, we would like to use ℙ[π_{exp(−λr)}] instead of π, and we are going to make the guess that we could approximate it with π_{exp(−βR)} (we have replaced the parameter λ with some distinct parameter β to give some more freedom to our investigation, and also because, intuitively, ℙ[π_{exp(−λr)}] may be expected to be less concentrated than each of the π_{exp(−λr)} it is mixing, which suggests that the best approximation of ℙ[π_{exp(−λr)}] by some π_{exp(−βR)} may be obtained for some parameter β < λ). We are then led to look for some empirical upper bound of 𝒦[ρ, π_{exp(−βR)}]. This is happily provided by the following computation

ℙ{𝒦[ρ, π_{exp(−βR)}]} = ℙ[𝒦(ρ, π)] + βℙ[ρ(R)] + log{π[exp(−βR)]}

= ℙ{𝒦[ρ, π_{exp(−βr)}]} + βℙ[ρ(R − r)]

+ log{π[exp(−βR)]} − ℙ{log π[exp(−βr)]}.

Using the convexity of r ↦ log{π[exp(−βr)]} as in equation (1.4) on page 23, we conclude that

0 ≤ ℙ{𝒦[ρ, π_{exp(−βR)}]} ≤ βℙ[ρ(R − r)] + ℙ{𝒦[ρ, π_{exp(−βr)}]}.

This inequality has an interest of its own, since it provides a lower bound for ℙ[ρ(R)]. Moreover we can plug it into Theorem 1.2.1 (page 19) applied to the prior distribution π_{exp(−βR)} and obtain for any posterior distribution ρ and any positive parameter λ that

Φ_{\frac{λ}{N}} \{ℙ[ρ(R)]\} ≤ ℙ\{ρ(r) + \frac{β}{λ}ρ(R − r) + \frac{1}{λ}ℙ\{𝒦[ρ, π_{exp(−βr)}]\}\}.

In view of this, it it convenient to introduce the function

Φ̃_{a,b}(p) = (1 − b)^{−1}[Φ_{a}(p) − bp]

= −(1 − b)^{−1}{a^{−1} log{1 − p[1 − exp(−a)]} + bp},
p ∈ (0, 1), a ∈)0, ∞(, b ∈ (0, 1(.

This is a convex function of p, moreover

Φ̃′_{a,b}(0) = {a^{−1}[1 − exp(−a)] − b}(1 − b)^{−1},

showing that it is an increasing one to one convex map of the unit interval unto itself as soon as b ≤ a^{−1}[1 − exp(−a)]. Its convexity, combined with the value of its derivative at the origin, shows that

Φ̃_{a,b}(p) ≥ \frac{a^{−1}[1 − exp(−a)] − b}{1 − b}p.

Using this notation and remarks, we can state Theorem 1.3.1. For any positive real constants β and λ such that 0 ≤ β < N[1 − exp(−\frac{λ}{N})], for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ\{ρ(r) − \frac{𝒦[ρ, π_{exp(−βr)}]}{β}\} ≤ ℙ[ρ(R)]

≤ Φ̃_{\frac{λ}{N} , \frac{β}{λ}}^{−1} \{ℙ[ρ(r) + \frac{𝒦[ρ, π_{exp(−βr)}]}{λ − β}]\}

≤ \frac{λ − β}{N[1 − exp(−\frac{λ}{N})] − β} ℙ[ρ(r) + \frac{𝒦[ρ, π_{exp(−βr)}]}{λ − β}].

Thus (taking λ = 2β), for any β such that 0 ≤ β < \frac{N}{2} ,

ℙ[ρ(R)] ≤ \frac{1}{1 − \frac{2β}{N}} ℙ\{ρ(r) + \frac{𝒦[ρ, π_{exp(−βr)}]}{β}\}.

Note that the last inequality is obtained using the fact that 1 − exp(−x) ≥ x − \frac{x^{2}}{2} , x ∈ ℝ_{+}.

Corollary 1.3.2. For any β ∈ (0, N(,

ℙ[π_{exp(−βr)}(r)] ≤ ℙ[π_{exp(−βr)}(R)]

≤ inf_{λ∈(−N log(1− \frac{β}{N}),∞(} \frac{λ − β}{N[1 − exp(−\frac{λ}{N})] − β} ℙ[π_{exp(−βr)}(r)]

≤ \frac{1}{1 − \frac{2β}{N}} ℙ[π_{exp(−βr)}(r)],

the last inequality holding only when β < \frac{N}{2} .

It is interesting to compare the upper bound provided by this corollary with Theorem 1.2.1 (page 19) when the posterior is a Gibbs measure ρ = π_{exp(−βr)}. We see that we have got rid of the entropy term 𝒦[π_{exp(−βr)}, π], but at the price of an increase of the multiplicative factor, which for small values of \frac{β}{N} grows from (1 − \frac{β}{2N})^{−1} (when we take λ = β in Theorem 1.2.1), to (1 − \frac{2β}{N})^{−1}. Therefore non-localized bounds have an interest of their own, and are superseded by localized bounds only in favourable circumstances (presumably when the sample is large enough when compared with the complexity of the classification model).

Corollary 1.3.2 shows that when \frac{2β}{N} is small, π_{exp(−βr)}(r) is a tight approximation of π_{exp(−βr)}(R) in the mean (since we have an upper bound and a lower bound which are close together).

Another corollary is obtained by optimizing the bound given by Theorem 1.3.1 in ρ, which is done by taking ρ = π_{exp(−λr)}.

Corollary 1.3.3. For any positive real constants β and λ such that 0 ≤ β < N[1 − exp(−\frac{λ}{N})],

ℙ[π_{exp(−λr)}(R)] ≤ Φ̃_{\frac{λ}{N} , \frac{β}{λ}}^{−1} \{ℙ[\frac{1}{λ − β} ∫_{β}^{λ} π_{exp(−γr)}(r)dγ]\}

≤ \frac{1}{N[1 − exp(−\frac{λ}{N})] − β} ℙ[∫_{β}^{λ}π_{exp(−γr)}(r)dγ].

Although this inequality gives by construction a better upper bound for inf_{λ∈ℝ_{+}} ℙ[π_{exp(−λr)}(R)] than Corollary 1.3.2, it is not easy to tell which one of the two inequalities is the best to bound ℙ[π_{exp(−λr)}(R)] for a fixed (and possibly suboptimal) value of λ, because in this case, one factor is improved while the other is worsened.

Using the empirical dimension d_{e} defined by equation (1.6) on page 27, we see that

\frac{1}{λ − β} ∫_{β}^{λ} π_{exp(−γr)}(r)dγ ≤ ess inf_{π} r + d_{e} log (\frac{λ}{β}) .

Therefore, in the case when we keep the ratio \frac{λ}{β} bounded, we get a better dependence on the empirical dimension d_{e} than in Corollary 1.2.9 (page 27).

1.3.3. Non random local bounds. Let us come now to the localization of the non-random upper bound given by Theorem 1.2.4 (page 23). According to Theorem 1.2.1 (page 19) applied to the localized prior π_{exp(−βR)},

λΦ_{\frac{λ}{N}} \{ℙ[ρ(R)]\} ≤ ℙ\{λρ(r) + 𝒦(ρ, π) + βρ(R)\} + log\{π[exp(−βR)]\}

= ℙ{𝒦[ρ, π_{exp(−λr)}] − log{π[exp(−λr)]} + βρ(R)} + log{π[exp(−βR)]}

≤ ℙ{𝒦[ρ, π_{exp(−λr)}]+βρ(R)}−log{π[exp(−λR)]}+log{π[exp(−βR)]},

where we have used as previously inequality (1.4) (page 23). This proves

Theorem 1.3.4. For any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), for any real parameters β and λ such that 0 ≤ β < N[1 − exp(−\frac{λ}{N})],

ℙ[ρ(R)] ≤ Φ̃_{\frac{λ}{N} , \frac{β}{λ}}^{−1} \{\frac{1}{λ − β} ∫_{β}^{λ} π_{exp(−γR)}(R)dγ + ℙ[\frac{𝒦[ρ, π_{exp(−λr)}]}{λ − β}]\}

≤ \frac{1}{N[1 − exp(−\frac{λ}{N})] − β} \{∫_{β}^{λ} π_{exp(−γR)}(R)dγ + ℙ\{𝒦[ρ, π_{exp(−λr)}]\}\}.

Let us notice in particular that this theorem contains Theorem 1.2.4 (page 23) which corresponds to the case β = 0. As a corollary, we see also, taking ρ = π_{exp(−λr)} and λ = 2β, and noticing that γ ↦ π_{exp(−γR)}(R) is decreasing, that

ℙ[π_{exp(−λr)}(R)] ≤ inf_{β,β<N[1−exp(− \frac{λ}{N})]} \frac{β}{N[1 − exp(−\frac{λ}{N})] − β} π_{exp(−βR)}(R)

≤ \frac{1}{1 − \frac{λ}{N}} π_{exp(− \frac{λ}{2} R)}(R).

We can use this inequality in conjunction with the notion of dimension with margin η introduced by equation (1.5) on page 24, to see that the Gibbs posterior achieves for a proper choice of λ and any margin parameter η ≥ 0 (which can be chosen to be equal to zero in parametric situations)

inf_{λ} ℙ[π_{exp(−λr)}(R)] ≤ ess inf_{π} R + η + \frac{4d_{η}}{N}

+ 2\sqrt{\frac{2d_{η}(ess inf_{π} R + η)}{N} + \frac{4d_{η}^{2}}{N^{2}}} . (1.8)

Deviation bounds to come next will show that the optimal λ can be estimated from empirical data.

Let us propose a little numerical example as an illustration: assuming that d_{0} = 10, N = 1000 and ess inf_{π} R = 0.2, we obtain from equation (1.8) that inf_{λ} ℙ[π_{exp(−λr)}(R)] ≤ 0.373.

1.3.4. Local deviation bounds. When it comes to deviation bounds, for technical reasons we will choose a slightly more involved change of prior distribution and apply Theorem 1.2.6 (page 25) to the prior π_{exp[−βΦ_{− \frac{β}{N}} ∘R]}.

The advantage of tweaking R with the nonlinear function Φ_{− \frac{β}{N}} will appear in the search for an empirical upper bound of the local entropy term. Theorem 1.1.4 (page 17), used with the above-mentioned local prior, shows that

ℙ\{ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ\{ρ(Φ_{\frac{λ}{N}} ∘R) − ρ(r)\} − 𝒦[ρ, π_{exp(−βΦ_{− \frac{β}{N}}∘R)}]\} ≤ 1. (1.9)

Moreover

𝒦[ρ, π_{exp[−βΦ_{− \frac{β}{N}} ∘R]}] = 𝒦[ρ, π_{exp(−βr)}] + βρ[Φ_{− \frac{β}{N}} ∘R − r]

+ log\{π[exp(−βΦ_{− \frac{β}{N}} ∘R)]\} − log\{π[exp(−βr)]\}, (1.10)

which is an invitation to find an upper bound for log\{π[exp[−βΦ_{− \frac{λ}{N}}∘R]]\}−

log{π[exp(−βr)]}. For conciseness, let us call our localized prior distribution \overline{π}, thus defined by its density

\frac{d\overline{π}}{dπ} (θ) = \frac{exp\{−βΦ_{− \frac{β}{N}} [R(θ)]\}}{π\{exp[−βΦ_{− \frac{β}{N}} ∘R]\}}.

Applying once again Theorem 1.1.4 (page 17), but this time to −β, we see that

ℙ\{exp[log\{π[exp(−βΦ_{− \frac{β}{N}} ∘R)]\} − log\{π[exp(−βr)]\}]\}

= ℙ\{exp[log\{π[exp(−βΦ_{− \frac{β}{N}} ∘R))]\} + inf_{ρ∈ℳ_{+}^{1}(Θ)} βρ(r) + 𝒦(ρ, π)]\}

≤ ℙ\{exp[log\{π[exp(−βΦ_{− \frac{β}{N}} ∘R))]\} + β\overline{π}(r) + 𝒦(\overline{π}, π)]\}

= ℙ\{exp[β[\overline{π}(r) − \overline{π}(Φ_{− \frac{β}{N}} ∘R)] + 𝒦(\overline{π}, \overline{π})]\} ≤ 1. (1.11)

Combining equations (1.10) and (1.11) and using the concavity of Φ_{− \frac{β}{N}}, we see that with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

0 ≤ 𝒦(ρ, \overline{π}) ≤ 𝒦[ρ, π_{exp(−βr)}] + β[Φ_{− \frac{β}{N}} [ρ(R)] − ρ(r)] − log(ϵ).

We have proved a lower deviation bound:

Theorem 1.3.5 For any positive real constant β, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

\frac{exp\{\frac{β}{N} [ρ(r) − \frac{𝒦[ρ, π_{exp(−βr)}] − log(ϵ)}{β}]\} − 1}{exp(\frac{β}{N} ) − 1} ≤ ρ(R).

We can also obtain a lower deviation bound for θ̂. Indeed equation (1.11) can also be written as

ℙ\{π_{exp(−βr)}[exp\{β[r − Φ_{− \frac{β}{N}} ∘ R]\}]\} ≤ 1.

This means that for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ\{ρ[exp\{β[r − Φ_{− \frac{β}{N}} ∘ R] − log(\frac{dρ}{dπ_{exp(−βr)}} )\}]\} ≤ 1.

We have proved

Theorem 1.3.6 For any positive real constant β, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), with ℙρ probability at least 1 − ϵ,

R(θ̂) ≥ Φ_{− \frac{β}{N}}^{−1} [r(θ̂) − \frac{log(\frac{dρ}{dπ_{exp(−βr)}} ) − log(ϵ)}{β}]

= \frac{exp\{\frac{β}{N} [r(θ̂) − \frac{log(\frac{dρ}{dπ_{exp(−βr)}} ) − log(ϵ)}{β}]\} − 1}{exp(\frac{β}{N} ) − 1}.

Let us now resume our investigation of the upper deviations of ρ(R). Using the Cauchy-Schwarz inequality to combine equations (1.9, page 33) and (1.11, page 34), we obtain

ℙ\{exp[\frac{1}{2} sup_{ρ∈ℳ_{+}^{1}(Θ)} λρ(Φ_{\frac{λ}{N}}∘R)−βρ(Φ_{− \frac{β}{N}}∘R)−(λ−β)ρ(r)−𝒦[ρ, π_{exp(−βr)}]]\}

= ℙ\{exp[\frac{1}{2} sup_{ρ∈ℳ_{+}^{1}(Θ)} (λ\{ρ(Φ_{\frac{λ}{N}} ∘R) − ρ(r)\} − 𝒦(ρ, \overline{π}))]

× exp[\frac{1}{2}(log\{π[exp(−βΦ_{− \frac{β}{N}} ∘R)]\} − log\{π[exp(−βr)]\})]\}

≤ ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} (λ\{ρ(Φ_{\frac{λ}{N}} ∘R) − ρ(r)\} − 𝒦(ρ, \overline{π}))]\}^{1/2}

× ℙ\{exp[(log\{π[exp(−βΦ_{− \frac{β}{N}} ∘R)]\} − log\{π[exp(−βr)]\})]\}^{1/2} ≤ 1. (1.12)

Thus with ℙ probability at least 1 − ϵ, for any posterior distribution ρ,

λΦ_{\frac{λ}{N}} [ρ(R)] − βΦ_{− \frac{β}{N}} [ρ(R)]

≤ λρ(Φ_{\frac{λ}{N}} ∘ R) − βρ(Φ_{− \frac{β}{N}} ∘ R)

≤ (λ − β)ρ(r) + 𝒦(ρ, π_{exp(−βr)}) − 2 log(ϵ).

(It would have been more straightforward to use a union bound on deviation inequalities instead of the Cauchy-Schwarz inequality on exponential moments, anyhow, this would have led to replace −2 log(ϵ) with the worse factor 2 log(\frac{2}{ϵ}).) Let us now recall that

λΦ_{\frac{λ}{N}} (p) − βΦ_{− \frac{β}{N}} (p) = −N log\{1 − [1 − exp(−\frac{λ}{N} )]p\}

− N log\{1 + [exp(\frac{β}{N} ) − 1]p\},

and let us put

B = (λ − β)ρ(r) + 𝒦[ρ, π_{exp(−βr)}] − 2 log(ϵ)

= 𝒦[ρ, π_{exp(−λr)}] + ∫_{β}^{λ}π_{exp(−ξr)}(r)dξ − 2 log(ϵ).

Let us consider moreover the change of variables α = 1 − exp(−\frac{λ}{N}) and γ = exp(\frac{β}{N}) − 1. We obtain [1 − αρ(R)][1 + γρ(R)] ≥ exp(−\frac{B}{N}), leading to

Theorem 1.3.7. For any positive constants α, γ, such that 0 ≤ γ < α < 1, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), the bound

M(ρ) = −\frac{log[(1 − α)(1 + γ)]}{α − γ}ρ(r) + \frac{𝒦(ρ, π_{exp[−N log(1+γ)r]}) − 2 log(ϵ)}{N(α − γ)}

= \frac{𝒦[ρ, π_{exp[N log(1−α)r]}] + ∫_{N log(1+γ)}^{−N log(1−α)} π_{exp(−ξr)}(r)dξ − 2 log(ϵ)}{N(α − γ)},

is such that

ρ(R) ≤ \frac{α − γ}{2αγ} (\sqrt{1 + \frac{4αγ}{(α − γ)^{2}} \{1 − exp[−(α − γ)M(ρ)]\}} − 1) ≤ M(ρ),

Let us now give an upper bound for R(θ̂). Equation (1.12 page 35) can also be written as

ℙ\{[π_{exp(−βr)}\{exp[λΦ_{\frac{λ}{N}} ∘ R − βΦ_{− \frac{β}{N}} ∘ R − (λ − β)r]\}]^{\frac{1}{2}} \} ≤ 1.

This means that for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ\{[ρ\{exp[λΦ_{\frac{λ}{N}} ∘ R − βΦ_{− \frac{β}{N}} ∘ R − (λ − β)r − log(\frac{dρ}{dπ_{exp(−βr)}} )]\}]^{\frac{1}{2}} \} ≤ 1.

Using the concavity of the square root function, this inequality can be weakened to

ℙ\{ρ[exp\{\frac{1}{2}[λΦ_{\frac{λ}{N}} ∘ R − βΦ_{− \frac{β}{N}} ∘ R − (λ − β)r − log(\frac{dρ}{dπ_{exp(−βr)}} )]\}]\} ≤ 1.

We have proved

Theorem 1.3.8. For any positive real constants λ and β and for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), with ℙρ probability at least 1 − ϵ,

λΦ_{\frac{λ}{N}} [R(θ̂)] − βΦ_{− \frac{β}{N}} [R(θ̂)] ≤ (λ − β) r(θ̂) + log[\frac{dρ}{dπ_{exp(−βr)}} (θ̂)] − 2 log(ϵ).

Putting α = 1 − exp(−\frac{λ}{N} ), γ = exp(\frac{β}{N} ) − 1 and

M(θ) = −\frac{log[(1 − α)(1 + γ)]}{α − γ}r(θ) + \frac{log[\frac{dρ}{dπ_{exp[−N log(1+γ)r]}} (θ)] − 2 log(ϵ)}{N(α − γ)}

= \frac{log[\frac{dρ}{dπ_{exp[N log(1−α)r]}} (θ)] + ∫_{N log(1+γ)}^{−N log(1−α)} π_{exp(−ξr)}(r) dξ − 2 log(ϵ)}{N(α − γ)},

we can also, in the case when γ < α, write this inequality as

R(θ̂) ≤ \frac{α − γ}{2αγ} (\sqrt{1 + \frac{4αγ}{(α − γ)^{2}} \{1 − exp[−(α − γ)M(θ̂)]\}} − 1) ≤ M(θ̂).

It may be enlightening to introduce the empirical dimension d_{e} defined by equation (1.6) on page 27. It provides the upper bound

∫_{β}^{λ} π_{exp(−ξr)}(r)dξ ≤ (λ − β) ess inf_{π} r + d_{e} log (\frac{λ}{β}) ,

which shows that in Theorem 1.3.7 (page 35),

M(ρ) ≤ \frac{log[(1 + γ)(1 − α)]}{γ − α} ess inf_{π} r

+ \frac{d_{e} log [\frac{− log(1−α)}{log(1+γ)}] + 𝒦[ρ, π_{exp[N log(1−α)r]}] − 2 log(ϵ)}{N(α − γ)}.

Similarly, in Theorem 1.3.8 above,

M(θ) ≤ \frac{log[(1 + γ)(1 − α)]}{γ − α} ess inf_{π} r

+ \frac{d_{e} log[\frac{− log(1−α)}{log(1+γ)}] + log[\frac{dρ}{dπ_{exp[N log(1−α)r]}} (θ)] − 2 log(ϵ)}{N(α − γ)}

Let us give a little numerical illustration: assuming that d_{e} = 10, N = 1000, and ess inf_{π} r = 0.2, taking ϵ = 0.01, α = 0.5 and γ = 0.1, we obtain from Theorem 1.3.7 π_{exp[N log(1−α)r]}(R) ≃ π_{exp(−693r)}(R) ≤ 0.332 ≤ 0.372, where we have given respectively the non-linear and the linear bound. This shows the practical interest of keeping the non-linearity. Optimizing the values of the parameters α and γ would not have yielded a significantly lower bound.

The following corollary is obtained by taking λ = 2β and keeping only the linear bound; we give it for the sake of its simplicity:

Corollary 1.3.9. For any positive real constant β such that exp(\frac{β}{N}) + exp(−\frac{2β}{N}) < 2, which is the case when β < 0.48N, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ρ(R) ≤ \frac{βρ(r) + 𝒦[ρ, π_{exp(−βr)}] − 2 log(ϵ)}{N[2 − exp(\frac{β}{N} ) − exp(−\frac{2β}{N} )]}

= \frac{∫_{β}^{2β} π_{exp(−ξr)}(r)dξ + 𝒦[ρ, π_{exp(−2βr)}] − 2 log(ϵ)}{N[2 − exp(\frac{β}{N}) − exp(−\frac{2β}{N})]}.

Let us mention that this corollary applied to the above numerical example gives π_{exp(−200r)}(R) ≤ 0.475 (when we take β = 100, consistently with the choice γ = 0.1).

1.3.5. Partially local bounds. Local bounds are suitable when the lowest values of the empirical error rate r are reached only on a small part of the parameter set Θ. When Θ is the disjoint union of sub-models of different complexities, the minimum of r will as a rule not be “localized” in a way that calls for the use of local bounds. Just think for instance of the case when Θ = ⨆_{m=1}^{M} Θ_{m}, where the sets Θ_{1} ⊂ Θ_{2} ⊂ ⋯ ⊂ Θ_{M} are nested. In this case we will have inf_{Θ_{1}} r ≥ inf_{Θ_{2}} r ≥ ⋯ ≥ inf_{Θ_{M}} r, although Θ_{M} may be too large to be the right model to use. In this situation, we do not want to localize the bound completely. Let us make a more specific fanciful but typical pseudo computation. Just imagine we have a countable collection (Θ_{m})_{m∈M} of sub-models. Let us assume we are interested in choosing between the estimators θ̂_{m} ∈ arg min_{Θ_{m}} r, maybe randomizing them (e.g. replacing them with π_{exp(−λr)}^{m}). Let us imagine moreover that we are in a typically parametric situation, where, for some priors π^{m} ∈ ℳ_{+}^{1}(Θ_{m}), m ∈ M, there is a “dimension” d_{m} such that λ[π_{exp(−λr)}^{m}(r) − r(θ̂_{m})] ≃ d_{m}. Let μ ∈ ℳ_{+}^{1}(M) be some distribution on the index set M. It is easy to see that (μπ)_{exp(−λr)} will typically not be properly local, in the sense that typically

(μπ)_{exp(−λr)}(r) = \frac{μ\{π_{exp(−λr)}(r)π[exp(−λr)]\}}{μ\{π[exp(−λr)]\}}

≃ \frac{∑_{m∈M}[(inf_{Θ_{m}} r) + \frac{d_{m}}{λ} ] exp[−λ(inf_{Θ_{m}} r) − d_{m} log(\frac{eλ}{d_{m}} )]μ(m)}{∑_{m∈M} exp[−λ(inf_{Θ_{m}} r) − d_{m} log(\frac{eλ}{d_{m}} )]μ(m)}

≃ \{ inf_{m∈M}(inf_{Θ_{m}} r) + \frac{d_{m}}{λ} log(\frac{eλ}{d_{m}} ) − \frac{1}{λ} log[μ(m)]\}

+ log\{ ∑_{m∈M} exp[−d_{m} log(\frac{λ}{d_{m}})]μ(m)\}.

where we have used the approximations

− log{π[exp(−λr)]} = ∫_{0}^{λ} π_{exp(−βr)}(r)dβ

≃ ∫_{0}^{λ}(inf_{Θ_{m}} r) + [\frac{d_{m}}{β} ∧ 1]dβ ≃ λ(inf_{Θ_{m}} r) + d_{m}[log(\frac{λ}{d_{m}} ) + 1],

and \frac{∑_{m} h(m) exp[−h(m)]ν(m)}{∑_{m} exp[−h(m)]ν(m)} ≃ inf_{m} h(m) − log[ν(m)], ν ∈ ℳ_{+}^{1}(M), taking

ν(m) = \frac{μ(m) exp[−d_{m} log(\frac{λ}{d_{m}} )]}{∑_{m′} μ(m′) exp[−d_{m′} log(\frac{λ}{d_{m′}} )]}.

These approximations have no pretension to be rigorous or very accurate, but they nevertheless give the best order of magnitude we can expect in typical situations, and show that this order of magnitude is not what we are looking for: mixing different models with the help of μ spoils the localization, introducing a multiplier log(\frac{λ}{d_{m}} ) to the dimension d_{m} which is precisely what we would have got if we had not localized the bound at all. What we would really like to do in such situations is to use a partially localized posterior distribution, such as πm̂ ; exp(−λr), where m̂ is an estimator of the best sub-model to be used. While the most straightforward way to do this is to use a union bound on results obtained for each sub-model Θ_{m}, here we are going to show how to allow arbitrary posterior distributions on the index set (corresponding to a randomization of the choice of m̂).

Let us consider the framework we just mentioned: let the measurable parameter set (Θ, 𝒯) be a union of measurable sub-models, Θ = ⋃_{m∈M} Θ_{m}. Let the index set (M, ℳ) be some measurable space (most of the time it will be a countable set). Let μ ∈ ℳ_{+}^{1}(M) be a prior probability distribution on (M, ℳ). Let π : M → ℳ_{+}^{1}(Θ) be a regular conditional probability measure such that π(m, Θ_{m}) = 1, for any m ∈ M. Let μπ ∈ ℳ_{+}^{1}(M × Θ) be the product probability measure defined for any bounded measurable function h : M × Θ → ℝ by

μπ(h) = ∫_{m∈M} (∫_{θ∈Θ} h(m, θ)π(m, dθ)) μ(dm).

For any bounded measurable function h : Ω × M × Θ → ℝ, let π_{exp(h)} : Ω × M → ℳ_{+}^{1}(Θ) be the regular conditional posterior probability measure defined by

\frac{dπ_{exp(h)}}{dπ} (m, θ) = \frac{exp[h(m, θ)]}{π[m, exp(h)]},

where consistently with previous notation π(m, h) = ∫_{Θ} h(m, θ)π(m, dθ) (we will also often use the less explicit notation π(h)). For short, let

U(θ, ω) = λΦ_{\frac{λ}{N}} [R(θ)] − βΦ_{− \frac{β}{N}} [R(θ)] − (λ − β)r(θ, ω).

Integrating with respect to μ equation (1.12, page 35), written in each submodel Θ_{m} using the prior distribution π(m, ·), we see that

ℙ\{exp[ sup_{ν∈ℳ_{+}^{1}(M)} sup_{ρ:M→ℳ_{+}^{1}(Θ)} \frac{1}{2}[(νρ)(U)−ν\{𝒦([ρ, π_{exp(−βr)}]\}]−𝒦(ν, μ)]\}

≤ ℙ\{exp[ sup_{ν∈ℳ_{+}^{1}(M)} \frac{1}{2}ν( sup_{ρ:M→ℳ_{+}^{1}(Θ)} ρ(U) − 𝒦(ρ, π_{exp(−βr)})) − 𝒦(ν, μ)]\}

= ℙ\{μ[exp\{\frac{1}{2} sup_{ρ:M→ℳ_{+}^{1}(Θ)} [ρ(U) − 𝒦[ρ, π_{exp(−βr)}]]\}]\}

= μ\{ℙ[exp\{\frac{1}{2} sup_{ρ:M→ℳ_{+}^{1}(Θ)} [ρ(U) − 𝒦[ρ, π_{exp(−βr)}]]\}]\} ≤ 1.

This proves that

ℙ\{exp[\frac{1}{2} sup_{ν∈ℳ_{+}^{1}(M)} sup_{ρ:M→ℳ_{+}^{1}(Θ)} νρ[λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R)]

− (λ − β)νρ(r) − 2𝒦(ν, μ) − ν{𝒦[ρ, π_{exp(−βr)}]}]} ≤ 1. (1.13)

Introducing the optimal value of r on each sub-model r^{⋆}(m) = ess inf_{π(m,·)} r and the empirical dimensions

d_{e}(m) = sup_{ξ∈ℝ_{+}} ξ[π_{exp(−ξr)}(m, r) − r^{⋆}(m)],

we can thus state

Theorem 1.3.10. For any positive real constants β < λ, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M), for any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

νρ[λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R)] ≤ λΦ_{\frac{λ}{N}} [νρ(R)] − βΦ_{− \frac{β}{N}} [νρ(R)] ≤ B_{1}(ν, ρ),

where B_{1}(ν, ρ) = (λ − β)νρ(r) + 2𝒦(ν, μ) + ν{𝒦[ρ, π_{exp(−βr)}]} − 2 log(ϵ)

= ν[∫_{β}^{λ} π_{exp(−αr)}(r)dα] + 2𝒦(ν, μ) + ν{𝒦[ρ, π_{exp(−λr)}]} − 2 log(ϵ)

= −2 log\{μ[exp(−\frac{1}{2} ∫_{β}^{λ} π_{exp(−αr)}(r)dα)]\}

+ 2𝒦[ν, μ_{( \frac{π[exp(−λr)]}{π[exp(−βr)]} )}_{1/2}] + ν\{𝒦[ρ, π_{exp(−λr)}]\} − 2 log(ϵ),

and therefore B_{1}(ν, ρ) ≤ ν[(λ − β)r^{⋆} + log(\frac{λ}{β})d_{e}] + 2𝒦(ν, μ)

+ ν{𝒦[ρ, π_{exp(−λr)}]} − 2 log(ϵ),

as well as B_{1}(ν, ρ) ≤ −2 log\{μ[exp(−\frac{(λ−β)}{2}r^{⋆} − \frac{1}{2} log(\frac{λ}{β})d_{e})]\}

+ 2𝒦[ν, μ_{(\frac{π[exp(−λr)]}{π[exp(−βr)]})1/2}] + ν\{𝒦[ρ, π_{exp(−λr)}] − 2 log(ϵ).

Thus, for any real constants α and γ such that 0 ≤ γ < α < 1, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M) and any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ), the bound

B_{2}(ν, ρ) = −^{\frac{log[(1−α)(1+γ)]}{α−γ}}νρ(r) + ^{\frac{2𝒦(ν,μ)+ν\{𝒦[ρ,π_{(1+γ)^{−Nr}}]\}−2 log(ϵ)}{N(α−γ)}}

= \frac{1}{N(α − γ)}\{2𝒦[ν, μ_{( \frac{π[(1−α)^{Nr}]}{π[(1+γ)^{−Nr}]} )}_{1/2}] + ν\{𝒦[ρ, π_{(1−α)^{Nr}}]\}\}

− \frac{2}{N(α − γ)} log\{μ[exp[−\frac{1}{2} ∫_{N log(1+γ)}^{−N log(1−α)} π_{exp(−ξr)}(·, r)dξ]]\}

− \frac{2 log(ϵ)}{N(α − γ)}

satisfies

νρ(R) ≤ \frac{α − γ}{2αγ} (\sqrt{1 + \frac{4αγ}{(α − γ)^{2}} \{1 − exp[−(α − γ)B_{2}(ν, ρ)]\}} − 1)

≤ B_{2}(ν, ρ).

If one is willing to bound the deviations with respect to ℙνρ, it is enough to remark that the equation preceding equation (1.13, page 40) can also be written as

ℙ\{μ[\{π_{exp(−βr)}[exp\{λΦ_{\frac{λ}{N}} ∘ R − βΦ_{− \frac{β}{N}} ∘ R − (λ − β)r\}]\}^{1/2}]\} ≤ 1.

Thus for any posterior distributions ν : Ω → ℳ_{+}^{1}(M) and ρ : Ω × M → ℳ_{+}^{1}(Θ),

ℙ\{ν[\{ρ[exp\{λΦ_{\frac{λ}{N}} ∘ R − βΦ_{− \frac{β}{N}} ∘ R

− (λ − β)r − 2 log(\frac{dν}{dμ}) − log(\frac{dρ}{dπ_{exp(−βr)}} )\}]\}^{1/2}]\} ≤ 1.

Using the concavity of the square root function to pull the integration with respect to ρ out of the square root, we get

ℙνρ\{exp[\frac{1}{2}\{λΦ_{\frac{λ}{N}} ∘ R − βΦ_{− \frac{β}{N}} ∘ R

− (λ − β)r − 2 log(\frac{dν}{dπ} ) − log(\frac{dρ}{dπ_{exp(−βr)}} )\}]\} ≤ 1.

This leads to

Theorem 1.3.11. For any positive real constants β < λ, for any posterior distributions ν : Ω → ℳ_{+}^{1}(M) and ρ : Ω×M → ℳ_{+}^{1}(Θ), with ℙνρ probability at least 1 − ϵ,

λΦ_{\frac{λ}{N}} [R(m̂, θ̂)] − βΦ_{− \frac{β}{N}} [R(m̂, θ̂)] ≤ (λ − β)r(m̂, θ̂)

+ 2 log[\frac{dν}{dμ}(m̂)] + log[\frac{dρ}{dπ_{exp(−βr)}} (m̂, θ̂)] − 2 log(ϵ)
= ∫_{β}^{λ} π_{exp(−αr)}(r)dα

+ 2 log[\frac{dν}{dμ}(m̂)] + log[\frac{dρ}{dπ_{exp(−λr)}} (m̂, θ̂)] − 2 log(ϵ)

= 2 log\{μ[exp(−\frac{1}{2} ∫_{β}^{λ} π_{exp(−αr)}(r)dα)]\}

+ 2 log[_{\frac{dν}{dμ_{(\frac{π[exp(−λr)]}{π[exp(−βr)]})_{1/2}}}} (m̂)] + log[\frac{dρ}{dπ_{exp(−λr)}} (m̂, θ̂)] − 2 log(ϵ).

Another way to state the same inequality is to say that for any real constants α and γ such that 0 ≤ γ < α < 1, with ℙνρ probability at least 1 − ϵ,

R(m̂, θ̂)

≤ \frac{α − γ}{2αγ} (\sqrt{1 + \frac{4αγ}{(α − γ)^{2}} \{1 − exp[−(α − γ)B(m̂, θ̂)]\}} − 1)

≤ B(m̂, θ̂),

where

B(m̂, θ̂) = −\frac{log[(1 − α)(1 + γ)]}{α − γ}r(m̂, θ̂)

+ \frac{2 log[\frac{dν}{dμ}(m̂)] + log[\frac{dρ}{dπ_{(1+γ)^{−Nr}}} (m̂, θ̂)] − 2 log(ϵ)}{N(α − γ)}

= \frac{2}{N(α − γ)} log[\frac{dν}{dμ_{(\frac{π[(1−α)^{Nr}]}{π[(1+γ)^{−Nr}]})1/2}} (m̂)]

+ \frac{log[\frac{dρ}{dπ_{(1−α)^{Nr}}} (m̂, θ̂)] − 2 log(ϵ)}{N(α − γ)}

+ \frac{2}{N(α − γ)} log\{μ[exp(−\frac{1}{2} ∫_{β}^{λ} π_{exp(−αr)}(r)dα)]\}.

Let us remark that in the case when ν = μ_{( \frac{π[(1−α)^{Nr}]}{π[(1+γ)^{−Nr}]} )}_{1/2} and ρ =

π_{(1−α)^{Nr}}, we get as desired a bound that is adaptively local in all the Θ_{m} (at least when M is countable and μ is atomic):

B(ν, ρ) ≤ −\frac{2}{N(α−γ)} log\{μ\{exp[\frac{N}{2} log[(1 + γ)(1 − α)]r^{⋆}

− log (\frac{− log(1−α)}{log(1+γ)}) \frac{d_{e}}{2} ]\}\} − \frac{2 log(ϵ)}{N(α − γ)}

≤ inf_{m∈M}\{−^{\frac{log[(1−α)(1+γ)]}{α−γ}}r^{⋆}(m)

+ log (\frac{− log(1−α)}{log(1+γ)}) \frac{d_{e}(m)}{N(α−γ)} − 2^{\frac{log[ϵμ(m)]}{N(α−γ)}}\}.

The penalization by the empirical dimension d_{e}(m) in each sub-model is as desired linear in d_{e}(m). Non random partially local bounds could be obtained in a way that is easy to imagine. We leave this investigation to the reader.

1.3.6. Two step localization. We have seen that the bound optimal choice of the posterior distribution ν on the index set in Theorem 1.3.10 (page 40) is such that

\frac{dν}{dμ}(m) ∼ (\frac{π[exp(−λr(m, ·))]}{π[exp(−βr(m, ·))]})^{\frac{1}{2}} = exp[−\frac{1}{2} ∫_{β}^{λ} π_{exp(−αr)}(m, r)dα].

This suggests replacing the prior distribution μ with \overline{μ} defined by its density

\frac{d\overline{μ}}{dμ}(m) = \frac{exp[−h(m)]}{μ[exp(−h)]},

where h(m) = −ξ ∫_{β}^{γ} π_{exp(−αΦ_{− \frac{η}{N}} ∘R)}[Φ_{− \frac{η}{N}} ∘R(m, ·)]dα. (1.14)

The use of Φ_{− \frac{η}{N}} ∘R instead of R is motivated by technical reasons which will appear in subsequent computations. Indeed, we will need to bound

ν[∫_{β}^{λ} π_{exp(−αΦ_{− \frac{η}{N}} ∘R)}(Φ_{− \frac{η}{N}} ∘R)dα]

in order to handle 𝒦(ν, \overline{μ}). In the spirit of equation (1.9, page 33), starting back from Theorem 1.1.4 (page 17), applied in each sub-model Θ_{m} to the prior distribution π_{exp(−γΦ_{− \frac{η}{N}} ∘R)} and integrated with respect to \overline{μ}, we see that for any positive real constants λ, γ and η, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M) on the index set and any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

νρ(λΦ_{\frac{λ}{N}} ∘R − γΦ_{− \frac{η}{N}} ∘R) ≤ λνρ(r)

+ ν𝒦(ρ, π) + 𝒦(ν, \overline{μ}) + ν\{log[π[exp(−γΦ_{− \frac{η}{N}} ∘R)]]\} − log(ϵ). (1.15)

Since x ↦ f(x) =^{def} λΦ_{\frac{λ}{N}} − γΦ_{− \frac{η}{N}} (x) is a convex function, it is such that

f(x) ≥ xf′(0) = xN\{[1 − exp(−\frac{λ}{N})] + \frac{γ}{η} [exp(\frac{η}{N}) − 1]\}.

Thus if we put

γ = \frac{η[1 − exp(−\frac{λ}{N})]}{exp(\frac{η}{N}) − 1},  (1.16)

we obtain that f(x) ≥ 0, x ∈ ℝ, and therefore that the left-hand side of equation (1.15) is non-negative. We can moreover introduce the prior conditional distribution \overline{π} defined by

\frac{d\overline{π}}{dπ} (m, θ) = \frac{exp[−βΦ_{− \frac{η}{N}} ∘ R(θ)]}{π\{m, exp[−βΦ_{− \frac{η}{N}} ∘ R]\}}.

With ℙ probability at least 1 − ϵ, for any posterior distributions ν : Ω → ℳ_{+}^{1}(M) and ρ : Ω × M → ℳ_{+}^{1}(Θ),

βνρ(r) + ν[𝒦(ρ, π)] = ν{𝒦[ρ, π_{exp(−βr)}]} − ν[log{π[exp(−βr)]}]

≤ ν\{𝒦[ρ, π_{exp(−βr)}]\} + βν\overline{π}(r) + ν[𝒦(\overline{π}, π)]
≤ ν\{𝒦[ρ, π_{exp(−βr)}]\} + βν\overline{π}(Φ_{− \frac{η}{N}} ∘R)
+ \frac{β}{η} [𝒦(ν, \overline{μ}) − log(ϵ)] + ν[𝒦(\overline{π}, π)]
= ν\{𝒦[ρ, π_{exp(−βr)}]\} − ν\{log[π[exp(−βΦ_{− \frac{η}{N}} ∘R)]]\}

+ \frac{β}{η} [𝒦(ν, \overline{μ}) − log(ϵ)].

Thus, coming back to equation (1.15), we see that under condition (1.16), with ℙ probability at least 1 − ϵ,

0 ≤ (λ − β)νρ(r) + ν{𝒦[ρ, π_{exp(−βr)}]}

− ν[∫_{β}^{γ} π_{exp(−αΦ_{− \frac{η}{N}} ∘R)}(Φ_{− \frac{η}{N}} ∘R)dα] + (1 + \frac{β}{η})[𝒦(ν, \overline{μ}) + log(\frac{2}{ϵ})].

Noticing moreover that

(λ − β)νρ(r) + ν{𝒦[ρ, π_{exp(−βr)}]}

= ν{𝒦[ρ, π_{exp(−λr)}]} + ν[∫_{β}^{λ} π_{exp(−αr)}(r)dα],

and choosing ρ = π_{exp(−λr)}, we have proved

Theorem 1.3.12. For any positive real constants β, γ and η, such that γ < η[exp(\frac{η}{N}) − 1]^{−1}, defining λ by condition (1.16), so that λ = −N log\{1 − \frac{γ}{η} [exp(\frac{η}{N}) − 1]\}, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M), any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

ν[∫_{β}^{γ} π_{exp(−αΦ_{− \frac{η}{N}} ∘R)}(Φ_{− \frac{η}{N}} ∘R)dα]

≤ ν[∫_{β}^{λ} π_{exp(−αr)}(r)dα] + (1 + \frac{β}{η} )[𝒦(ν, \overline{μ}) + log(\frac{2}{ϵ})].

Let us remark that this theorem does not require that β < γ, and thus provides both an upper and a lower bound for the quantity of interest:

Corollary 1.3.13. For any positive real constants β, γ and η such that max\{β, γ\} < η[exp(\frac{η}{N}) − 1]^{−1}, with ℙ probability at least 1 − ϵ, for any posterior distributions ν : Ω → ℳ_{+}^{1}(M) and ρ : Ω × M → ℳ_{+}^{1}(Θ),

ν[∫_{−N log\{1− \frac{β}{N} [exp( \frac{η}{N})−1]\}}^{γ} π_{exp(−αr)}(r)dα] − (1 + \frac{γ}{η} )[𝒦(ν, \overline{μ}) + log(\frac{3}{ϵ})]

≤ ν[∫_{β}^{γ} π_{exp(−αΦ_{− \frac{η}{N}} ∘R)}(Φ_{− \frac{η}{N}} ∘R)dα]
≤ ν[∫_{β}^{−N log\{1− \frac{γ}{η} [exp( \frac{η}{N})−1]\}} π_{exp(−αr)}(r)dα]
+ (1 + \frac{β}{η} )[𝒦(ν, \overline{μ}) + log(\frac{3}{ϵ})].

We can then remember that

𝒦(ν, \overline{μ}) = ξ(ν − \overline{μ})[∫_{β}^{γ} π_{exp(−αΦ_{− \frac{η}{N}} ∘R)}(Φ_{− \frac{η}{N}} ∘R)dα] + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ),

to conclude that, putting

G_{η}(α) = −N log\{1 − \frac{α}{η}[exp(\frac{η}{N}) − 1]\} ≥ α,  α ∈ ℝ_{+},  (1.17)

and

\frac{dν̂}{dμ}(m) =^{def} \frac{exp[−h(m)]}{μ[exp(−h)]} where h(m) = ξ ∫_{G_{η}(β)}^{γ} π_{exp(−αr)}(m, r)dα, (1.18)

the divergence of ν with respect to the local prior \overline{μ} is bounded by

[1 − ξ(1 + \frac{β}{η} )]𝒦(ν, \overline{μ})

≤ ξν[∫_{β}^{G_{η}(γ)} π_{exp(−αr)}(r)dα] − ξ\overline{μ}[∫_{G_{η}(β)}^{γ} π_{exp(−αr)}(r)dα]
+ 𝒦(ν, μ) − 𝒦(\overline{μ}, μ) + ξ(2 + \frac{β+γ}{η} ) log(\frac{3}{ϵ})
≤ ξν[∫_{β}^{G_{η}(γ)} π_{exp(−αr)}(r)dα] + 𝒦(ν, μ)
+ log{μ[exp(−ξ ∫_{G_{η}(β)}^{γ} π_{exp(−αr)}(r)dα)]}
+ ξ(2 + \frac{β+γ}{η} ) log(\frac{3}{ϵ})
= 𝒦(ν, ν̂) + ξν[(∫_{β}^{G_{η}(β)} + ∫_{γ}^{G_{η}(γ)})π_{exp(−αr)}(r)dα]
+ ξ(2 + \frac{β+γ}{η} ) log(\frac{3}{ϵ}).

Theorem 1.3.14. For any positive constants β, γ and η such that max\{β, γ\} < η[exp(\frac{η}{N}) − 1]^{−1}, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M) and any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

𝒦(ν, \overline{μ}) ≤ [1 − ξ(1 + \frac{β}{η} )]^{−1}\{𝒦(ν, ν̂)

+ ξν[(∫_{β}^{G_{η}(β)} + ∫_{γ}^{G_{η}(γ)})π_{exp(−αr)}(r)dα]
+ ξ(2 + \frac{β+γ}{η} ) log(\frac{3}{ϵ})\}

≤ [1 − ξ(1 + \frac{β}{η} )]^{−1}\{𝒦(ν, ν̂)

+ ξν[[G_{η}(γ) − γ + G_{η}(β) − β]r^{⋆} + log(\frac{G_{η}(β)G_{η}(γ)}{βγ})d_{e}]

+ ξ(2 + \frac{β+γ}{η} ) log(\frac{3}{ϵ})\},

where the local prior \overline{μ} is defined by equation (1.14, page 43) and the local posterior ν̂ and the function G_{η} are defined by equation (1.18, page 45).

We can then use this theorem to give a local version of Theorem 1.3.10 (page 40). To get something pleasing to read, we can apply Theorem 1.3.14 with constants β′, γ′ and η chosen so that _{\frac{2ξ}{1−ξ(1+ \frac{β′}{η})}} = 1, G_{η}(β′) = β and γ′ = λ,

where β and λ are the constants appearing in Theorem 1.3.10. This gives

Theorem 1.3.15. For any positive real constants β < λ and η such that λ < η[exp(\frac{η}{N}) − 1]^{−1}, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M), for any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

νρ[λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R)] ≤ λΦ_{\frac{λ}{N}} [νρ(R)] − βΦ_{− \frac{β}{N}} [νρ(R)] ≤ B_{3}(ν, ρ),

where B_{3}(ν, ρ) = ν[∫_{G_{η}^{−1}(β)}^{G_{η}(λ)} π_{exp(−αr)}(r)dα]

+ (3 + \frac{G_{η}^{−1}(β)}{η})𝒦[ν, μ_{exp[−(3+ \frac{G{−1 \atop η}(β)}{η})^{−1} ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}]

+ ν\{𝒦(ρ, π_{exp(−λr)}]\} + (4 + \frac{G_{η}^{−1}(β)+λ}{η}) log(\frac{4}{ϵ})

≤ ν[[G_{η}(λ) − G_{η}^{−1}(β)]r^{⋆} + log(\frac{G_{η}(λ)}{G_{η}^{−1}(β)})d_{e}]
+ (3 + \frac{G_{η}^{−1}(β)}{η})𝒦[ν, μ_{exp[−(3+ \frac{G{−1 \atop η}(β)}{η})^{−1} ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}]

+ ν\{𝒦(ρ, π_{exp(−λr)}]\} + (4 + \frac{G_{η}^{−1}(β)+λ}{η}) log(\frac{4}{ϵ}), and where the function G_{η} is defined by equation (1.17, page 45).

A first remark: if we had the stamina to use Cauchy Schwarz inequalities (or more generally Hölder inequalities) on exponential moments instead of using weighted union bounds on deviation inequalities, we could have replaced log(\frac{4}{ϵ}) with − log(ϵ) in the above inequalities.

We see that we have achieved the desired kind of localization of Theorem 1.3.10 (page 40), since the new empirical entropy term

𝒦[ν, μ_{exp[−ξ ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}]

cancels for a value of the posterior distribution on the index set ν which is of the same form as the one minimizing the bound B_{1}(ν, ρ) of Theorem 1.3.10 (with a decreased constant, as could be expected). In a typical parametric setting, we will have

∫_{β}^{λ} π_{exp(−αr)}(r)dα ≃ (λ − β)r^{⋆}(m) + log (\frac{λ}{β}) d_{e}(m),

and therefore, if we choose for ν the Dirac mass at

m̂ ∈ arg min_{m∈M} r^{⋆}(m) + \frac{log( \frac{λ}{β})}{λ−β}d_{e}(m),

and ρ(m, ·) = π_{exp(−λr)}(m, ·), we will get, in the case when the index set M is countable,

B_{3}(ν, ρ) ≲ max \{[G_{η}(λ) − G_{η}^{−1}(β)], (λ − β)^{\frac{log[\frac{Gη(λ)}{G{−1 \atop η}(β)}]}{log( \frac{λ}{β})}}\}

× [r^{⋆}(m̂) + \frac{log( \frac{λ}{β})}{λ−β}d_{e}(m̂)]

+ (3 + \frac{G_{η}^{−1}(β)}{η}) log\{ ∑_{m∈M} \frac{μ(m)}{μ(m̂)} exp[−(3 + \frac{G_{η}^{−1}(β)}{η})^{−1}

× \{(λ − β)[r^{⋆}(m) − r^{⋆}(m̂)] + log(\frac{λ}{β})[d_{e}(m) − d_{e}(m̂)]\}]\}

+ (4 + \frac{G_{η}^{−1}(β)+λ}{η}) log(\frac{4}{ϵ}).

This shows that the impact on the bound of the addition of supplementary models depends on their penalized minimum empirical risk r^{⋆}(m) +

\frac{log( \frac{λ}{β})}{λ−β} d_{e}(m). More precisely the adaptive and local complexity factor

log\{ ∑_{m∈M} \frac{μ(m)}{μ(m̂)} exp[−(3 + \frac{G_{η}^{−1}(β)}{η})^{−1}

× \{(λ − β)[r^{⋆}(m) − r^{⋆}(m̂)] + log(\frac{λ}{β})[d_{e}(m) − d_{e}(m̂)]\}]\} replaces in this bound the non local factor

𝒦(ν, μ) = − log[μ(m̂)] = log[ ∑_{m∈M} \frac{μ(m)}{μ(m̂)}]

which appears when applying Theorem 1.3.10 (page 40) to the Dirac mass ν = δm̂. Thus in the local bound, the influence of models decreases exponentially fast when their penalized empirical risk increases.

One can deduce a result about the deviations with respect to the posterior νρ from Theorem 1.3.15 (page 46) without much supplementary work: it is enough for that purpose to remark that with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M),

ν[log\{π_{exp(−λr)}[exp\{λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R)\}]\}]

− ν (∫_{G_{η}^{−1}(β)}^{G_{η}(λ)} π_{exp(−αr)}(r)dα)
− (3 + \frac{G_{η}^{−1}(β)}{η})𝒦[ν, μ_{exp[−(3+ \frac{G{−1 \atop η}(β)}{η})^{−1} ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}]

− (4 + \frac{G_{η}^{−1}(β)+λ}{η}) log(\frac{4}{ϵ}) ≤ 0,

this inequality being obtained by taking a supremum in ρ in Theorem 1.3.15 (page 46). One can then take a supremum in ν, to get, still with ℙ probability at least 1 − ϵ,

log\{μ_{exp[−(3+ \frac{G{−1 \atop η}(β)}{η})^{−1} ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}[

\{π_{exp(−λr)}[exp\{λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R)\}]\}^{(3+ \frac{G{−1 \atop η}(β)}{η})^{−1}}
× exp(−(3 + \frac{G_{η}^{−1}(β)}{η})^{−1} ∫_{G_{η}^{−1}(β)}^{G_{η}(λ)} π_{exp(−αr)}(r)dα)]\}

≤ \frac{4 + \frac{G_{η}^{−1}(β)+λ}{η}}{3 + \frac{G_{η}^{−1}(β)}{η}} log(\frac{4}{ϵ}).

Using the fact that x ↦ x^{α} is concave when α = (3 + \frac{G_{η}^{−1}(β)}{η})^{−1} < 1, we get for any posterior conditional distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

μ_{exp[−(3+ \frac{G{−1 \atop η}(β)}{η})^{−1} ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}ρ\{ exp[(3 + \frac{G_{η}^{−1}(β)}{η})^{−1}(λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R) − ∫_{G_{η}^{−1}(β)}^{G_{η}(λ)} π_{exp(−αr)}(r)dα

+ log [\frac{dρ}{dπ_{exp(−λr)}} (m̂, θ̂)])]\}

≤ exp(\frac{4 + \frac{G_{η}^{−1}(β)+λ}{η}}{3 + \frac{G_{η}^{−1}(β)}{η}} log(\frac{4}{ϵ})).

We can thus state

Theorem 1.3.16. For any ϵ ∈)0, 1(, with ℙ probability at least 1−ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M) and conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ), for any ξ ∈)0, 1(, with νρ probability at least 1 − ξ,

λΦ_{\frac{λ}{N}} (R) − βΦ_{− \frac{β}{N}} (R) ≤ ∫_{G_{η}^{−1}(β)}^{G_{η}(λ)} π_{exp(−αr)}(r)dα

+ (3 + \frac{G_{η}^{−1}(β)}{η}) log [\frac{dν}{dμ_{exp[−(3+ \frac{G{−1 \atop η}(β)}{η})^{−1} ∫_{β}^{λ} π_{exp(−αr)}(r)dα]}} (m̂)]

+ log [\frac{dρ}{dπ_{exp(−λr)}} (m̂, θ̂)] + (4 + \frac{G_{η}^{−1}(β)+λ}{η}) log(\frac{4}{ϵ}) − (3 + \frac{G_{η}^{−1}(β)}{η}) log(ξ).

Note that the given bound consequently holds with ℙνρ probability at least (1 − ϵ)(1 − ξ) ≥ 1 − ϵ − ξ.

1.4. Relative bounds

The behaviour of the minimum of the empirical process θ ↦ r(θ) is known to depend on the covariances between pairs [r(θ), r(θ′)], θ, θ′ ∈ Θ. In this respect, our previous study, based on the analysis of the variance of r(θ) (or technically on some exponential moment playing quite the same role), loses some accuracy in some circumstances (namely when inf_{Θ} R is not close enough to zero).

In this section, instead of bounding the expected risk ρ(R) of any posterior distribution, we are going to upper bound the difference ρ(R) − inf_{Θ} R, and more generally ρ(R) − R(θ̃), where θ̃ ∈ Θ is some fixed parameter value.

In the next section we will analyse ρ(R) − π_{exp(−βR)}(R), allowing us to compare the expected error rate of a posterior distribution ρ with the error rate of a Gibbs prior distribution. We will also analyse ρ_{1}(R)−ρ_{2}(R), where ρ_{1} and ρ_{2} are two arbitrary posterior distributions, using comparison with a Gibbs prior distribution as a tool, and in particular as a tool to establish the required Kullback divergence bounds.

Relative bounds do not provide the same kind of results as direct bounds on the error rate: it is not possible to estimate ρ(R) with an order of precision higher than (ρ(R)/N)^{1/2}, so that relative bounds cannot of course achieve that, but they provide a way to reach a faster rate for ρ(R) − inf_{Θ} R, that is for the relative performance of the estimator within a restricted model.

The study of PAC-Bayesian relative bounds was initiated in the second and third parts of J.-Y. Audibert’s dissertation (Audibert, 2004b).

In this section and the next, we will suggest a series of possible uses of relative bounds. As usual, we will start with the simplest inequalities and proceed towards more sophisticated techniques with better theoretical properties, but at the same time less precise constants, so that which one is the more fitted will depend on the size of the training sample.

The first thing we will do is to compute for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ) a relative performance bound bearing on ρ(R) − inf_{Θ} R. We will also compare the classification model indexed by Θ with a sub-model indexed by one of its measurable subsets Θ_{1} ⊂ Θ. For this purpose we will form the difference ρ(R)−R(θ̃), where θ̃ ∈ Θ_{1} is some possibly unobservable value of the parameter in the sub-model defined by Θ_{1}, typically chosen in arg min_{Θ_{1}} R. If this is so and ρ(R) − R(θ̃) = ρ(R) − inf_{Θ_{1}} R, a negative upper bound indicates that it is definitely worth using a randomized estimator ρ supported by the larger parameter set Θ instead of using only the classification model defined by the smaller set Θ_{1}.

1.4.1. Basic inequalities. Relative bounds in this section are based on the control of r(θ) − r(θ̃), where θ, θ̃ ∈ Θ. These differences are related to the random variables

ψ_{i}(θ, θ̃) = σ_{i}(θ) − σ_{i}(θ̃) = 𝟙[f_{θ}(X_{i}) ≠ Y_{i}] − 𝟙[fθ̃(X_{i}) ≠ Y_{i}].

Some supplementary technical difficulties, as compared to the previous sections, come from the fact that ψ_{i}(θ, θ̃) takes three values, whereas σ_{i}(θ) takes only two. Let

r′(θ, θ̃) = r(θ) − r(θ̃) = \frac{1}{N} ∑_{i=1}^{N} ψ_{i}(θ, θ̃), θ, θ̃ ∈ Θ,  (1.19)

and R′(θ, θ̃) = R(θ) − R(θ̃) = ℙ[r′(θ, θ̃)]. We have as usual from independence that

log\{ℙ[exp[−λr′(θ, θ̃)]]\} = ∑_{i=1}^{N} log\{ℙ[exp[−\frac{λ}{N} ψ_{i}(θ, θ̃)]]\}

≤ N log\{\frac{1}{N} ∑_{i=1}^{N} ℙ\{exp[−\frac{λ}{N} ψ_{i}(θ, θ̃)]\}\}.

Let C_{i} be the distribution of ψ_{i}(θ, θ̃) under ℙ and let C̄ = \frac{1}{N} ∑_{i=1}^{N} C_{i} ∈ ℳ_{+}^{1}({−1, 0, 1}). With this notation

log\{ℙ[exp[−λr′(θ, θ̃)]]\} ≤ N log\{∫_{ψ∈\{−1,0,1\}} exp(−\frac{λ}{N} ψ)C̄(dψ)\}.

(1.20)

The right-hand side of this inequality is a function of C̄. On the other hand, C̄ being a probability measure on a three point set, is defined by two parameters, that we may take equal to ∫ ψC̄(dψ) and ∫ ψ^{2}C̄(dψ). To this purpose, let us introduce

M′(θ, θ̃) = ∫ ψ^{2}C̄(dψ) = C̄(+1) + C̄(−1) = \frac{1}{N} ∑_{i=1}^{N} ℙ[ψ_{i}^{2}(θ, θ̃)], θ, θ̃ ∈ Θ.

It is a pseudo distance (meaning that it is symmetric and satisfies the triangle inequality), since it can also be written as

M′(θ, θ̃) = \frac{1}{N} ∑_{i=1}^{N} ℙ\{|𝟙[f_{θ}(X_{i}) ≠ Y_{i}] − 𝟙[fθ̃(X_{i}) ≠ Y_{i}]|\}, θ, θ̃ ∈ Θ.

It is readily seen that

N log \{∫ exp (−\frac{λ}{N} ψ) C̄(dψ)\} = −λΨ_{\frac{λ}{N}} [R′(θ, θ̃), M′(θ, θ̃)],

where

Ψ_{a}(p, m) = −a^{−1} log[(1 − m) + \frac{m + p}{2} exp(−a) + \frac{m − p}{2} exp(a)]
= −a^{−1} log\{1 − sinh(a)[p − m tanh(\frac{a}{2})]\}.  (1.21)

Thus plugging this equality into inequality (1.20, page 51) we get

Theorem 1.4.1. For any real parameter λ,

log\{ℙ[exp[−λr′(θ, θ̃)]]\} ≤ −λΨ_{\frac{λ}{N}} [R′(θ, θ̃), M′(θ, θ̃)], θ, θ̃ ∈ Θ,

where r′ is defined by equation (1.19, page 50) and Ψ and M′ are defined just above.

To make a link with previous work of Mammen and Tsybakov — see e.g. Mammen et al. (1999) and Tsybakov (2004) — we may consider the pseudodistance D on Θ defined by equation (1.3, page 20). This distance only depends on the distribution of the patterns. It is often used to formulate margin assumptions, in the sense of Mammen and Tsybakov. Here we are going to work rather with M′: as it is dominated by D in the sense that M′(θ, θ̃) ≤ D(θ, θ̃), θ, θ̃ ∈ Θ, with equality in the important case of binary classification, hypotheses formulated on D induce hypotheses on M′, and working with M′ may only sharpen the results when compared to working with D.

Using the same reasoning as in the previous section, we deduce

Theorem 1.4.2. For any real parameter λ, any θ̃ ∈ Θ, any prior distribution π ∈ ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ[ρ\{Ψ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)]\} − ρ[r′(·, θ̃)]] − 𝒦(ρ, π)]\} ≤ 1.

We are now going to derive some other type of relative exponential inequality. In Theorem 1.4.2 we obtained an inequality comparing one observed quantity ρ[r′(·, θ̃)] with two unobserved ones, ρ[R′(·, θ̃)] and ρ[M′(·, θ̃)],

— indeed, because of the convexity of the function λΨ_{\frac{λ}{N}},

λρ\{Ψ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)]\} ≥ λΨ_{\frac{λ}{N}} \{ρ[R′(·, θ̃)], ρ[M′(·, θ̃)]\}.

This may be inconvenient when looking for an empirical bound for ρ[R′(·, θ̃)], and we are going now to seek an inequality comparing ρ[R′(·, θ̃)] with empirical quantities only.

This is possible by considering the log-Laplace transform of some modified random variable χ_{i}(θ, θ̃). We may consider more precisely the change of variable defined by the equation

exp (−\frac{λ}{N} χ_{i}) = 1 − \frac{λ}{N} ψ_{i},

which is possible when \frac{λ}{N} ∈ )−1, 1( and leads to define

χ_{i} = −\frac{N}{λ} log (1 − \frac{λ}{N} ψ_{i}) .

We may then work on the log-Laplace transform

log\{ℙ[exp\{−\frac{λ}{N} ∑_{i=1}^{N} χ_{i}(θ, θ̃)\}]\} = log\{ℙ[∏_{i=1}^{N}(1 − \frac{λ}{N} ψ_{i}(θ, θ̃))]\}

= log\{ℙ[exp\{∑_{i=1}^{N} log[1 − \frac{λ}{N} ψ_{i}(θ, θ̃)]\}]\}.

We may now follow the same route as previously, writing

log\{ℙ[exp\{∑_{i=1}^{N} log[1 − \frac{λ}{N} ψ_{i}(θ, θ̃)]\}]\}

= ∑_{i=1}^{N} log[1 − \frac{λ}{N} ℙ[ψ_{i}(θ, θ̃)]] ≤ N log[1 − \frac{λ}{N} R′(θ, θ̃)].

Let us also introduce the random pseudo distance

m′(θ, θ̃) = \frac{1}{N} ∑_{i=1}^{N} ψ_{i}(θ, θ̃)^{2}

= \frac{1}{N} ∑_{i=1}^{N}|𝟙[f_{θ}(X_{i}) ≠ Y_{i}] − 𝟙[fθ̃(X_{i}) ≠ Y_{i}]|, θ, θ̃ ∈ Θ. (1.22)

This is the empirical counterpart of M′, implying that ℙ(m′) = M′. Let us notice that

\frac{1}{N} ∑_{i=1}^{N} log[1 − \frac{λ}{N} ψ_{i}(θ, θ̃)] = \frac{log(1 − \frac{λ}{N}) − log(1 + \frac{λ}{N})}{2}r′(θ, θ̃)

+ \frac{log(1 − \frac{λ}{N}) + log(1 + \frac{λ}{N})}{2}m′(θ, θ̃)

= \frac{1}{2} log (\frac{1 − \frac{λ}{N}}{1 + \frac{λ}{N}} ) r′(θ, θ̃) + \frac{1}{2} log(1 − \frac{λ^{2}}{N^{2}} )m′(θ, θ̃).

Let us put γ = \frac{N}{2} log(\frac{1+ \frac{λ}{N}}{1− \frac{λ}{N}} ), so that

λ = N tanh(\frac{γ}{N} ) and \frac{N}{2} log(1 − \frac{λ^{2}}{N^{2}} ) = −N log[cosh(\frac{γ}{N})].

With this notation, we can conveniently write the previous inequality as

ℙ\{exp[−N log[1 − tanh(\frac{γ}{N} )R′(θ, θ̃)]

− γr′(θ, θ̃) − N log[cosh(\frac{γ}{N})]m′(θ, θ̃)]\} ≤ 1.

Integrating with respect to a prior probability measure π ∈ ℳ_{+}^{1}(Θ), we obtain

Theorem 1.4.3. For any real parameter γ, for any θ̃ ∈ Θ, for any prior probability distribution π ∈ ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} \{−Nρ\{log[1 − tanh(\frac{γ}{N} )R′(·, θ̃)]\}

− γρ[r′(·, θ̃)] − N log[cosh(\frac{γ}{N})]ρ[m′(·, θ̃)] − 𝒦(ρ, π)\}]\} ≤ 1.

1.4.2. Non random bounds. Let us first deduce a non-random bound from Theorem 1.4.2 (page 52). This theorem can be conveniently taken advantage of by throwing the non-linearity into a localized prior, considering the prior probability measure μ defined by its density

\frac{dμ}{dπ}(θ) = \frac{exp\{−λΨ_{\frac{λ}{N}} [R′(θ, θ̃), M′(θ, θ̃)] + βR′(θ, θ̃)\}}{π\{exp\{−λΨ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)] + βR′(·, θ̃)\}\}}.

Indeed, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

𝒦(ρ, μ) = 𝒦(ρ, π) + λρ\{Ψ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)]\} − βρ[R′(·, θ̃)]

+ log\{π[exp\{−λΨ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)] + βR′(·, θ̃)]\}]\}.

Plugging this into Theorem 1.4.2 (page 52) and using the convexity of the exponential function, we see that for any posterior probability distribution ρ : Ω → ℳ_{+}^{1}(Θ),

βℙ{ρ[R′(·, θ̃)]} ≤ λℙ{ρ[r′(·, θ̃)]} + ℙ[𝒦(ρ, π)]

+ log\{π[exp\{−λΨ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)] + βR′(·, θ̃)]\}]\}.

We can then recall that

λρ[r′(·, θ̃)] + 𝒦(ρ, π) = 𝒦[ρ, π_{exp(−λr)}] − log{π[exp[−λr′(·, θ̃)]]},

and notice moreover that

−ℙ{log{π[exp[−λr′(·, θ̃)]]}} ≤ − log{π[exp[−λR′(·, θ̃)]]},

since R′ = ℙ(r′) and h ↦ log{π[exp(h)]} is a convex functional. Putting these two remarks together, we obtain

Theorem 1.4.4. For any real positive parameter λ, for any prior distribution π ∈ ℳ_{+}^{1}(Θ), for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ\{ρ[R′(·, θ̃)]\} ≤ \frac{1}{β} ℙ[𝒦(ρ, π_{exp(−λr)})]

+ \frac{1}{β} log\{π[exp\{−λΨ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)] + βR′(·, θ̃)]\}]\}

− \frac{1}{β} log\{π[exp[−λR′(·, θ̃)]]\}

≤ \frac{1}{β} ℙ[𝒦(ρ, π_{exp(−λr)})]

+ \frac{1}{β} log\{π[exp\{−[N sinh(\frac{λ}{N}) − β]R′(·, θ̃)

+ 2N sinh(\frac{λ}{2N})^{2}M′(·, θ̃)\}]\}

− \frac{1}{β} log\{π[exp[−λR′(·, θ̃)]]\}.

It may be interesting to derive some more suggestive (but slightly weaker) bound in the important case when Θ_{1} = Θ and R(θ̃) = inf_{Θ} R. In this case, it is convenient to introduce the expected margin function

φ(x) = sup_{θ∈Θ} M′(θ, θ̃) − xR′(θ, θ̃), x ∈ ℝ_{+}.  (1.23)

We see that φ is convex and non-negative on ℝ_{+}. Using the bound M′(θ, θ̃) ≤ xR′(θ, θ̃) + φ(x), we obtain

ℙ\{ρ[R′(·, θ̃)]\} ≤ \frac{1}{β} ℙ[𝒦(ρ, π_{exp(−λr)})]

+ \frac{1}{β} log\{π[exp\{−\{N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − β\}R′(·, θ̃)\}]\}

+ \frac{N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})}{β}φ(x) − \frac{1}{β} log\{π[exp[−λR′(·, θ̃)]]\}.

Let us make the change of variable γ = N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − β to obtain

Corollary 1.4.5. For any real positive parameters x, γ and λ such that x ≤ tanh(\frac{λ}{2N})^{−1} and 0 ≤ γ < N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})],

ℙ[ρ(R)] − inf_{Θ} R ≤ \{N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − γ\}^{−1}

× {∫_{γ}^{λ}[π_{exp(−αR)}(R) − inf_{Θ} R]dα

+ N sinh(\frac{λ}{N} ) tanh(\frac{λ}{2N} )φ(x) + ℙ[𝒦(ρ, π_{exp(−λr)})]\}.

Let us remark that these results, although well suited to study Mammen and Tsybakov’s margin assumptions, hold in the general case: introducing the convex expected margin function φ is a substitute for making hypotheses about the relations between R and D.

Using the fact that R′(θ, θ̃) ≥ 0, θ ∈ Θ and that φ(x) ≥ 0, x ∈ ℝ_{+}, we can weaken and simplify the preceding corollary even more to get

Corollary 1.4.6. For any real parameters β, λ and x such that x ≥ 0 and 0 ≤ β < λ − x\frac{λ^{2}}{2N}, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ[ρ(R)] ≤ inf_{Θ} R

+ [λ − x\frac{λ^{2}}{2N} − β]^{−1}\{∫_{β}^{λ}[π_{exp(−αR)}(R) − inf_{Θ} R]dα

+ ℙ\{𝒦[ρ, π_{exp(−λr)}]\} + φ(x)\frac{λ^{2}}{2N} \}.

Let us apply this bound under the margin assumption first considered by Mammen and Tsybakov (Mammen et al., 1999; Tsybakov, 2004), which says that for some real positive constant c and some real exponent κ ≥ 1,

R′(θ, θ̃) ≥ cD(θ, θ̃)^{κ},  θ ∈ Θ.  (1.24)

In the case when κ = 1, then φ(c^{−1}) = 0, proving that

ℙ\{π_{exp(−λr)}[R′(·, θ̃)]\} ≤ \frac{∫_{β}^{λ} π_{exp(−γR)}[R′(·, θ̃)]dγ}{N sinh(\frac{λ}{N})[1 − c^{−1} tanh(\frac{λ}{2N})] − β}

≤ \frac{∫_{β}^{λ} π_{exp(−γR)}[R′(·, θ̃)]dγ}{λ − \frac{λ^{2}}{2cN} − β} .

Taking for example λ = \frac{cN}{2} , β = \frac{λ}{2} = \frac{cN}{4} , we obtain

ℙ[π_{exp(−2^{−1}cNr)}(R)] ≤ inf R + \frac{8}{cN} ∫ \frac{cN}{2} ; \frac{cN}{4} π_{exp(−γR)}[R′(·, θ̃)]dγ
≤ inf R + 2π_{exp(− \frac{cN}{4} R)}[R′(·, θ̃)].

If moreover the behaviour of the prior distribution π is parametric, meaning that π_{exp(−βR)}[R′(·, θ̃)] ≤ \frac{d}{β}, for some positive real constant d linked with the dimension of the classification model, then

ℙ[π_{exp(− \frac{cN}{2} r)}(R)] ≤ inf R + \frac{8 log(2)d}{cN} ≤ inf R + \frac{5.55 d}{cN}.

In the case when κ > 1,

φ(x) ≤ (κ − 1)κ^{− \frac{κ}{κ−1}} (cx)^{− \frac{1}{κ−1}} = (1 − κ^{−1})(κcx)^{− \frac{1}{κ−1}} ,

thus ℙ{π_{exp(−λr)}[R′(·, θ̃)]}

≤ \frac{∫_{β}^{λ} π_{exp(−γR)}[R′(·, θ̃)]dγ + (1 − κ^{−1})(κcx)^{− \frac{1}{κ−1}} \frac{λ^{2}}{2N}}{λ − \frac{xλ^{2}}{2N} − β} .

Taking for instance β = \frac{λ}{2}, x = \frac{N}{2λ}, and putting b = (1 − κ^{−1})(cκ)^{− \frac{1}{κ−1}} , we obtain

ℙ[π_{exp(−λr)}(R)] − inf R ≤ \frac{4}{λ} ∫_{λ/2}^{λ} π_{exp(−γR)}[R′(·, θ̃)]dγ + b (\frac{2λ}{N})^{\frac{κ}{κ−1}} .

In the parametric case when π_{exp(−γR)}[R′(·, θ̃)] ≤ \frac{d}{γ}, we get

ℙ[π_{exp(−λr)}(R)] − inf R ≤ \frac{4 log(2)d}{λ} + b (\frac{2λ}{N})^{\frac{κ}{κ−1}} .

Taking

\overline{λ} = 2^{−1}[8 log(2)d]^{\frac{κ−1}{2κ−1}} (κc)^{\frac{1}{2κ−1}} N^{\frac{κ}{2κ−1}} ,

we obtain

ℙ[π_{exp(−\overline{λ}r)}(R)] − inf R ≤ (2 − κ^{−1})(κc)^{− \frac{1}{2κ−1}} (\frac{8 log(2)d}{N})^{\frac{κ}{2κ−1}} .

We see that this formula coincides with the result for κ = 1. We can thus reduce the two cases to a single one and state

Corollary 1.4.7. Let us assume that for some θ̃ ∈ Θ, some positive real constant c, some real exponent κ ≥ 1 and for any θ ∈ Θ, R(θ) ≥ R(θ̃) + cD(θ, θ̃)^{κ}. Let us also assume that for some positive real constant d and any positive real parameter γ, π_{exp(−γR)}(R) − inf R ≤ \frac{d}{γ}. Then

ℙ[π_{exp\{−2^{−1}[8 log(2)d]^{\frac{κ−1}{2κ−1}} (κc)^{\frac{1}{2κ−1}} N^{\frac{κ}{2κ−1}} r\}}(R)]

≤ inf R + (2 − κ^{−1})(κc)^{− \frac{1}{2κ−1}} (\frac{8 log(2)d}{N})^{\frac{κ}{2κ−1}} .

Let us remark that the exponent of N in this corollary is known to be the minimax exponent under these assumptions: it is unimprovable, whatever estimator is used in place of the Gibbs posterior shown here (at least in the worst case compatible with the hypotheses). The interest of the corollary is to show not only the minimax exponent in N, but also an explicit nonasymptotic bound with reasonable and simple constants. It is also clear that we could have got slightly better constants if we had kept the full strength of Theorem 1.4.4 (page 54) instead of using the weaker Corollary 1.4.6 (page 55).

We will prove in the following empirical bounds showing how the constant λ can be estimated from the data instead of being chosen according to some margin and complexity assumptions.

1.4.3. Unbiased empirical bounds. We are going to define an empirical counterpart for the expected margin function φ. It will appear in empirical bounds having otherwise the same structure as the non-random bound we just proved. Anyhow, we will not launch into trying to compare the behaviour of our proposed empirical margin function with the expected margin function, since the margin function involves taking a supremum which is not straightforward to handle. When we will touch the issue of building provably adaptive estimators, we will instead formulate another type of bounds based on integrated quantities, rather than try to analyse the properties of the empirical margin function.

Let us start as in the previous subsection with the inequality

βℙ{ρ[R′(·, θ̃)]} ≤ ℙ{λρ[r′(·, θ̃)] + 𝒦(ρ, π)}

+ log\{π[exp\{−λΨ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)] + βR′(·, θ̃) \}]\}.

We have already defined by equation (1.22, page 53) the empirical pseudodistance

m′(θ, θ̃) = \frac{1}{N} ∑_{i=1}^{N} ψ_{i}(θ, θ̃)^{2}.

Recalling that ℙ[m′(θ, θ̃)] = M′(θ, θ̃), and using the convexity of h ↦ log{π[exp(h)]}, leads to the following inequalities:

log\{π[exp\{−λΨ_{\frac{λ}{N}} [R′(·, θ̃), M′(·, θ̃)] + βR′(·, θ̃)\}]\}

≤ log\{π[exp\{−N sinh(\frac{λ}{N})R′(·, θ̃)
+ N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})M′(·, θ̃) + βR′(·, θ̃)]\}]\}
≤ ℙ\{log\{π[exp\{−[N sinh(\frac{λ}{N}) − β]r′(·, θ̃)
+ N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})m′(·, θ̃)\}]\}\}.

We may moreover remark that

λρ[r′(·, θ̃)] + 𝒦(ρ, π) = [β − N sinh(\frac{λ}{N}) + λ]ρ[r′(·, θ̃)] + 𝒦[ρ, π_{exp\{−[N sinh( \frac{λ}{N})−β]r\}}]

− log\{π[exp\{−[N sinh(\frac{λ}{N}) − β]r′(·, θ̃)\}]\}.

This establishes

Theorem 1.4.8. For any positive real parameters β and λ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ\{ρ[R′(·, θ̃)]\} ≤ ℙ\{[1 − \frac{N sinh(\frac{λ}{N}) − λ}{β}]ρ[r′(·, θ̃)]

+ \frac{𝒦[ρ, π_{exp\{−[N sinh( \frac{λ}{N})−β]r\}}]}{β}

+ β^{−1} log\{π_{exp\{−[N sinh( \frac{λ}{N})−β]r\}}[exp[N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})m′(·, θ̃)]]\}\}.

Taking β = \frac{N}{2} sinh(\frac{λ}{N}), using the fact that sinh(a) ≥ a, a ≥ 0 and expressing tanh(\frac{a}{2}) = a^{−1}[\sqrt{1 + sinh(a)^{2}} − 1] and a = log[\sqrt{1 + sinh(a)^{2}} + sinh(a)], we deduce

Corollary 1.4.9. For any positive real constant β and any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ\{ρ[R′(·, θ̃)]\} ≤ ℙ\{\underbrace{[\frac{N}{β} log(\sqrt{1 + \frac{4β^{2}}{N^{2}}} + \frac{2β}{N} ) − 1]}_{≤1} ρ[r′(·, θ̃)]

+ \frac{1}{β} \{𝒦[ρ, π_{exp(−βr)}]

+ log[π_{exp(−βr)}\{exp[N(\sqrt{1 + \frac{4β^{2}}{N^{2}}} − 1)m′(·, θ̃)]\}]\}\}.

This theorem and its corollary are really analogous to Theorem 1.4.4 (page 54), and it could easily be proved that under Mammen and Tsybakov margin assumptions we obtain an upper bound of the same order as Corollary 1.4.7 (page 57). Anyhow, in order to obtain an empirical bound, we are now going to take a supremum over all possible values of θ̃, that is over Θ_{1}. Although we believe that taking this supremum will not spoil the bound in cases when over-fitting remains under control, we will not try to investigate precisely if and when this is actually true, and provide our empirical bound as such. Let us say only that on qualitative grounds, the values of the margin function quantify the steepness of the contrast function R or its empirical counterpart r, and that the definition of the empirical margin function is obtained by substituting ℙ, the true sample distribution, with \overline{ℙ} = (\frac{1}{N} ∑_{i=1}^{N} δ_{(X_{i},Y_{i})})^{⊗N}, the empirical sample distribution, in the definition of the expected margin function. Therefore, on qualitative grounds, it seems hopeless to presume that R is steep when r is not, or in other words that a classification model that would be inefficient at estimating a bootstrapped sample according to our non-random bound would be by some miracle efficient at estimating the true sample distribution according to the same bound. To this extent, we feel that our empirical bounds bring a satisfactory counterpart of our non-random bounds. Anyhow, we will also produce estimators which can be proved to be adaptive using PAC-Bayesian tools in the next section, at the price of a more sophisticated construction involving comparisons between a posterior distribution and a Gibbs prior distribution or between two posterior distributions.

Let us now restrict discussion to the important case when θ̃ ∈ arg min_{Θ_{1}} R. To obtain an observable bound, let θ̂ ∈ arg min_{θ∈Θ} r(θ) and let us introduce the empirical margin functions

\overline{φ}(x) = sup_{θ∈Θ} m′(θ, θ̂) − x[r(θ) − r(θ̂)], x ∈ ℝ_{+},

φ̃(x) = sup_{θ∈Θ_{1}} m′(θ, θ̂) − x[r(θ) − r(θ̂)], x ∈ ℝ_{+}.

Using the fact that m′(θ, θ̃) ≤ m′(θ, θ̂) + m′(θ̂, θ̃), we get

Corollary 1.4.10. For any positive real parameters β and λ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ[ρ(R)] − inf_{Θ_{1}} R ≤ ℙ\{[1 − \frac{N sinh( \frac{λ}{N})−λ}{β}][ρ(r) − r(θ̂)]

+ \frac{𝒦[ρ, π_{exp\{−[N sinh( \frac{λ}{N})−β]r\}}]}{β}

+ β^{−1} log\{π_{exp\{−[N sinh( \frac{λ}{N})−β]r\}}[exp[N sinh(\frac{λ}{N} ) tanh(\frac{λ}{2N} )m′(·, θ̂)]]\}

+β^{−1}N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})φ̃[\frac{β}{N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})} (1 − \frac{N sinh(\frac{λ}{N}) − λ}{β})]\}.

Taking β = \frac{N}{2} sinh(\frac{λ}{N}), we also obtain

ℙ[ρ(R)] − inf_{Θ_{1}} R ≤ ℙ\{\underbrace{[\frac{N}{β} log(\sqrt{1 + \frac{4β^{2}}{N^{2}}} + \frac{2β}{N} ) − 1]}_{≤1}[ρ(r) − r(θ̂)]

+ \frac{1}{β} \{𝒦[ρ, π_{exp(−βr)}]

+ log[π_{exp(−βr)}\{exp[N(\sqrt{1 + \frac{4β^{2}}{N^{2}}} − 1)m′(·, θ̂)]\}]\}

+ \frac{N}{β} (\sqrt{1 + \frac{4β^{2}}{N^{2}}} − 1)φ̃[\frac{log(\sqrt{1 + \frac{4β^{2}}{N^{2}}} + \frac{2β}{N} ) − \frac{β}{N}}{(\sqrt{1 + \frac{4β^{2}}{N^{2}}} − 1)} ]\}.

Note that we could also use the upper bound m′(θ, θ̂) ≤ x[r(θ)−r(θ̂)]+\overline{φ}(x) and put α = N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − β, to obtain

Corollary 1.4.11. For any non-negative real parameters x, α and λ, such that α < N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})], for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

ℙ[ρ(R)] − inf_{Θ_{1}} R

≤ ℙ\{[1 − \frac{N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − λ}{N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − α}][ρ(r) − r(θ̂)]

+ \frac{𝒦[ρ, π_{exp(−αr)}]}{N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − α}

+ \frac{N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})}{N sinh(\frac{λ}{N})[1 − x tanh(\frac{λ}{2N})] − α}

× [\overline{φ}(x) + φ̃(\frac{λ − α}{N sinh(\frac{λ}{N}) tanh(\frac{λ}{2N})})]\}.

Let us notice that in the case when Θ_{1} = Θ, the upper bound provided by this corollary has the same general form as the upper bound provided by Corollary 1.4.5 (page 55), with the sample distribution ℙ replaced with the empirical distribution of the sample \overline{ℙ} = (\frac{1}{N} ∑_{i=1}^{N} δ_{(X_{i},Y_{i})})^{⊗N}. Therefore, our empirical bound can be of a larger order of magnitude than our nonrandom bound only in the case when our non-random bound applied to the bootstrapped sample distribution \overline{ℙ} would be of a larger order of magnitude than when applied to the true sample distribution ℙ. In other words, we can say that our empirical bound is close to our non-random bound in every situation where the bootstrapped sample distribution \overline{ℙ} is not harder to bound than the true sample distribution ℙ. Although this does not prove that our empirical bound is always of the same order as our non-random bound, this is a good qualitative hint that this will be the case in most practical situations of interest, since in situations of “under-fitting”, if they exist, it is likely that the choice of the classification model is inappropriate to the data and should be modified.

Another reassuring remark is that the empirical margin functions \overline{φ} and φ̃ behave well in the case when inf_{Θ} r = 0. Indeed in this case m′(θ, θ̂) = r′(θ, θ̂) = r(θ), θ ∈ Θ, and thus \overline{φ}(1) = φ̃(1) = 0, and

φ̃(x) ≤ −(x − 1) inf_{Θ_{1}} r, x ≥ 1.

This shows that in this case we recover the same accuracy as with nonrelative local empirical bounds. Thus the bound of Corollary 1.4.11 does not collapse in presence of massive over-fitting in the larger model, causing r(θ̂) = 0, which is another hint that this may be an accurate bound in many situations.

1.4.4. Relative empirical deviation bounds. It is natural to make use of Theorem 1.4.3 (page 53) to obtain empirical deviation bounds, since this theorem provides an empirical variance term.

Theorem 1.4.3 is written in a way which exploits the fact that ψ_{i} takes only the three values −1, 0 and +1. However, it will be more convenient for the following computations to use it in its more general form, which only makes use of the fact that ψ_{i} ∈ (−1, 1). With notation to be explained hereafter, it can indeed also be written as

ℙ{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} {−Nρ{log[1 − λP (ψ)]}

+ Nρ\{\overline{P} [log(1 − λψ)]\} − 𝒦(ρ, π)\}]\} ≤ 1. (1.25)

We have used the following notation in this inequality. We have put

\overline{P} = \frac{1}{N} ∑_{i=1}^{N} δ_{(X_{i},Y_{i})},

so that \overline{P} is our notation for the empirical distribution of the process (X_{i}, Y_{i})_{i=1}^{N}. Moreover we have also used

P = ℙ(\overline{P}) = \frac{1}{N} ∑_{i=1}^{N} P_{i},

where it should be remembered that the joint distribution of the process (X_{i}, Y_{i})_{i=1}^{N} is ℙ = ⨂_{i=1}^{N} P_{i}. We have considered ψ(θ, θ̃) as a function defined on 𝒳 × 𝒴 as ψ(θ, θ̃)(x, y) = 𝟙[y ≠ f_{θ}(x)] − 𝟙[y ≠ fθ̃(x)], (x, y) ∈ 𝒳 × 𝒴 so that it should be understood that

P (ψ) = \frac{1}{N} ∑_{i=1}^{N} ℙ[ψ_{i}(θ, θ̃)]

= \frac{1}{N} ∑_{i=1}^{N} ℙ\{𝟙[Y_{i} ≠ f_{θ}(X_{i})] − 𝟙[Y_{i} ≠ fθ̃(X_{i})]\} = R′(θ, θ̃).

In the same way

\overline{P} [log(1 − λψ)] = \frac{1}{N} ∑_{i=1}^{N} log[1 − λψ_{i}(θ, θ̃)].

Moreover integration with respect to ρ bears on the index θ, so that

ρ\{log[1 − λP (ψ)]\} = ∫_{θ∈Θ} log\{1 − \frac{λ}{N} ∑_{i=1}^{N} ℙ[ψ_{i}(θ, θ̃)]\}ρ(dθ),

ρ\{\overline{P} [log(1 − λψ)]\} = ∫_{θ∈Θ}\{\frac{1}{N} ∑_{i=1}^{N} log[1 − λψ_{i}(θ, θ̃)]\}ρ(dθ).

We have chosen concise notation, as we did throughout these notes, in order to make the computations easier to follow.

To get an alternate version of empirical relative deviation bounds, we need to find some convenient way to localize the choice of the prior distribution π in equation (1.25, page 61). Here we propose replacing π with μ = π_{exp{−N log[1+βP (ψ)]}}, which can also be written π_{exp{−N log[1+βR′(·,θ̃)]}}. Indeed we see that

𝒦(ρ, μ) = Nρ{log[1 + βP (ψ)]} + 𝒦(ρ, π)

+ log{π[exp{−N log[1 + βP (ψ)]}]}.

Moreover, we deduce from our deviation inequality applied to −ψ, that (as long as β > −1),

ℙ\{exp[Nμ\{\overline{P} [log(1 + βψ)]\} − Nμ\{log[1 + βP (ψ)]\}]\} ≤ 1.

Thus

ℙ{exp[log{π[exp{−N log[1 + βP (ψ)]}]}

− log\{π[exp\{−N\overline{P} [log(1 + βψ)]\}]\}]\}
≤ ℙ{exp[−Nμ{log[1 + βP (ψ)]} − 𝒦(μ, π)

+ Nμ\{\overline{P} [log(1 + βψ)]\} + 𝒦(μ, π)]\} ≤ 1.

This can be used to handle 𝒦(ρ, μ), making use of the Cauchy–Schwarz inequality as follows

ℙ\{exp[\frac{1}{2}[−N log\{(1 − λρ[P (ψ)])(1 + βρ[P (ψ)])\}

+Nρ\{\overline{P} [log(1 − λψ)]\}

− 𝒦(ρ, π) − log\{π[exp\{−N\overline{P} [log(1 + βψ)]\}]\}]]\}

≤ ℙ{exp[−N log{(1 − λρ[P (ψ)])}

+ Nρ\{\overline{P} [log(1 − λψ)]\} − 𝒦(ρ, μ)]\}^{1/2}

× ℙ{exp[log{π[exp{−N log[1 + βP (ψ)]}]}

− log\{π[exp\{−N\overline{P} [log(1 + βψ)]\}]\}]\}^{1/2} ≤ 1.

This implies that with ℙ probability at least 1 − ϵ,

− N log{(1 − λρ[P (ψ)])(1 + βρ[P (ψ)])}

≤ −Nρ\{\overline{P} [log(1 − λψ)]\}

+ 𝒦(ρ, π) + log\{π[exp\{−N\overline{P} [log(1 + βψ)]\}]\} − 2 log(ϵ).

It is now convenient to remember that

\overline{P} [log(1 − λψ)] = \frac{1}{2} log (\frac{1 − λ}{1 + λ}) r′(θ, θ̃) + \frac{1}{2} log(1 − λ^{2})m′(θ, θ̃).

We thus can write the previous inequality as

− N log{(1 − λρ[R′(·, θ̃)])(1 + βρ[R′(·, θ̃)])}

≤ \frac{N}{2} log (\frac{1 + λ}{1 − λ}) ρ[r′(·, θ̃)] − \frac{N}{2} log(1 − λ^{2})ρ[m′(·, θ̃)] + 𝒦(ρ, π)

+ log\{π[exp\{ − \frac{N}{2} log(\frac{1 + β}{1 − β} )r′(·, θ̃)

− \frac{N}{2} log(1 − β^{2})m′(·, θ̃)\}]\} − 2 log(ϵ).

Let us assume now that θ̃ ∈ arg min_{Θ_{1}} R. Let us introduce θ̂ ∈ arg min_{Θ} r. Decomposing r′(θ, θ̃) = r′(θ, θ̂) + r′(θ̂, θ̃) and considering that

m′(θ, θ̃) ≤ m′(θ, θ̂) + m′(θ̂, θ̃),

we see that with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

− N log{(1 − λρ[R′(·, θ̃)])(1 + βρ[R′(·, θ̃))}

≤ \frac{N}{2} log(\frac{1 + λ}{1 − λ})ρ[r′(·, θ̂)] − \frac{N}{2} log(1 − λ^{2})ρ[m′(·, θ̂)] + 𝒦(ρ, π)

+ log\{π[exp\{−\frac{N}{2} log(\frac{1+β}{1−β} )[r′(·, θ̂)] − \frac{N}{2} log(1 − β^{2})m′(·, θ̂)\}]\}

+ \frac{N}{2} log[\frac{(1+λ)(1−β)}{(1−λ)(1+β)}][r(θ̂) − r(θ̃)]

− \frac{N}{2} log[(1 − λ^{2})(1 − β^{2})]m′(θ̂, θ̃) − 2 log(ϵ).

Let us now define for simplicity the posterior ν : Ω → ℳ_{+}^{1}(Θ) by the identity

\frac{dν}{dπ} (θ) = \frac{exp\{−\frac{N}{2} log(\frac{1+λ}{1−λ})r′(θ, θ̂) + \frac{N}{2} log(1 − λ^{2})m′(θ, θ̂)\}}{π[exp\{−\frac{N}{2} log(\frac{1+λ}{1−λ})r′(·, θ̂) + \frac{N}{2} log(1 − λ^{2})m′(·, θ̂)\}]}.

Let us also introduce the random bound

B = \frac{1}{N} log\{ν[exp[\frac{N}{2} log[\frac{(1+λ)(1−β)}{(1−λ)(1+β)}]r′(·, θ̂)

− \frac{N}{2} log[(1 − λ^{2})(1 − β^{2})]m′(·, θ̂)]]\}

+ sup_{θ∈Θ_{1}} \frac{1}{2} log [\frac{(1−λ)(1+β)}{(1+λ)(1−β)}]r′(θ, θ̂)

− \frac{1}{2} log[(1 − λ^{2})(1 − β^{2})]m′(θ, θ̂) − \frac{2}{N} log(ϵ).

Theorem 1.4.12. Using the above notation, for any real constants 0 ≤ β < λ < 1, for any prior distribution π ∈ ℳ_{+}^{1}(Θ), for any subset Θ_{1} ⊂ Θ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

− log\{(1 − λ[ρ(R) − inf_{Θ_{1}} R])(1 + β[ρ(R) − inf_{Θ_{1}} R])\} ≤ \frac{𝒦(ρ, ν)}{N} + B.

Therefore,

ρ(R) − inf_{Θ_{1}} R

≤ \frac{λ − β}{2λβ} (\sqrt{1 + 4\frac{λβ}{(λ − β)^{2}} [1 − exp (−B − \frac{𝒦(ρ, ν)}{N})]} − 1)

≤ \frac{1}{λ − β} (B + \frac{𝒦(ρ, ν)}{N}) .

Let us define the posterior ν̂ by the identity

\frac{dν̂}{dπ}(θ) = \frac{exp[−\frac{N}{2} log (\frac{1+β}{1−β} ) r′(θ, θ̂) − \frac{N}{2} log(1 − β^{2})m′(θ, θ̂)]}{π\{exp[−\frac{N}{2} log (\frac{1+β}{1−β} ) r′(·, θ̂) − \frac{N}{2} log(1 − β^{2})m′(·, θ̂)]\}}.

It is useful to remark that

\frac{1}{N} log\{ν[exp[\frac{N}{2} log(\frac{(1 + λ)(1 − β)}{(1 − λ)(1 + β)})r′(·, θ̂)

− \frac{N}{2} log[(1 − λ^{2})(1 − β^{2})]m′(·, θ̂)]]\}

≤ ν̂\{\frac{1}{2} log(\frac{(1 + λ)(1 − β)}{(1 − λ)(1 + β)})r′(·, θ̂)

− \frac{1}{2} log[(1 − λ^{2})(1 − β^{2})]m′(·, θ̂)\}.

This inequality is a special case of

log{π[exp(g)]} − log{π[exp(h)]}

= ∫_{α=0}^{1} π_{exp[h+α(g−h)]}(g − h)dα ≤ π_{exp(g)}(g − h),

which is a consequence of the convexity of α ↦ log{π[exp[h + α(g − h)]]}.

Let us introduce as previously \overline{φ}(x) = sup_{θ∈Θ} m′(θ, θ̂)−x r′(θ, θ̂), x ∈ ℝ_{+}. Let us moreover consider φ̃(x) = sup_{θ∈Θ_{1}} m′(θ, θ̂)−x r′(θ, θ̂), x ∈ ℝ_{+}. These functions can be used to produce a result which is slightly weaker, but maybe easier to read and understand. Indeed, we see that, for any x ∈ ℝ_{+}, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ,

− N log{(1 − λρ[R′(·, θ̃)])(1 + βρ[R′(·, θ̃)])}

≤ \frac{N}{2} log [\frac{(1 + λ)}{(1 − λ)(1 − λ^{2})^{x}} ] ρ[r′(·, θ̂)]

− \frac{N}{2} log[(1 − λ^{2})(1 − β^{2})]\overline{φ}(x) + 𝒦(ρ, π)

+ log\{π[exp\{−\frac{N}{2} log[\frac{(1+β)}{(1−β)(1−β^{2})^{x}} ]r′(·, θ̂)\}]\}

− \frac{N}{2} log[(1 − λ^{2})(1 − β^{2})]φ̃ (\frac{log [\frac{(1+λ)(1−β)}{(1−λ)(1+β)}]}{− log [(1 − λ^{2})(1 − β^{2})]})

− 2 log(ϵ)

= ∫\begin{matrix}\frac{N}{2} & log & [ \frac{(1+λ)}{(1−λ)(1−λ^{2})^{x}} ] \\ \frac{N}{2} & log & [ \frac{(1+β)}{(1−β)(1−β^{2})^{x}} ] \end{matrix} π_{exp(−αr)}[r′(·, θ̂)]dα

+ 𝒦(ρ, π_{exp\{− \frac{N}{2} log[ \frac{(1+λ)}{(1−λ)(1−λ^{2})^{x}}]r\}}) − 2 log(ϵ)

− \frac{N}{2} log[(1 − λ^{2})(1 − β^{2})] [\overline{φ}(x) + φ̃ (\frac{log [\frac{(1+λ)(1−β)}{(1−λ)(1+β)}]}{− log[(1 − λ^{2})(1 − β^{2})]})] .

Theorem 1.4.13. With the previous notation, for any real constants 0 ≤ β < λ < 1, for any positive real constant x, for any prior probability distribution π ∈ ℳ_{+}^{1}(Θ), for any subset Θ_{1} ⊂ Θ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), putting

B(ρ) = \frac{1}{N(λ − β)} ∫\begin{matrix}\frac{N}{2} & log & [ \frac{(1+λ)}{(1−λ)(1−λ^{2})^{x}} ] \\ \frac{N}{2} & log & [ \frac{(1+β)}{(1−β)(1−β^{2})^{x}} ] \end{matrix} π_{exp(−αr)}[r′(·, θ̂)]dα

+ \frac{𝒦(ρ, π_{exp\{− \frac{N}{2} log[ \frac{(1+λ)}{(1−λ)(1−λ^{2})^{x}}]r\}}) − 2 log(ϵ)}{N(λ − β)}

− \frac{1}{2(λ − β)} log[(1 − λ^{2})(1 − β^{2})] [\overline{φ}(x) + φ̃ (\frac{log [\frac{(1+λ)(1−β)}{(1−λ)(1+β)}]}{− log[(1 − λ^{2})(1 − β^{2})]})]

≤ \frac{1}{N(λ − β)}d_{e} log (\begin{matrix}log & [ & \frac{(1+λ)}{(1−λ)(1−λ^{2})^{x}} & ] \\ log & ( & \frac{(1+β)}{(1−β)(1−β^{2})^{x}} & ) \end{matrix})

+ \frac{𝒦(ρ, π_{exp\{− \frac{N}{2} log[ \frac{(1+λ)}{(1−λ)(1−λ^{2})^{x}}]r\}}) − 2 log(ϵ)}{N(λ − β)}

− \frac{1}{2(λ − β)} log[(1 − λ^{2})(1 − β^{2})] [\overline{φ}(x) + φ̃ (\frac{log [\frac{(1+λ)(1−β)}{(1−λ)(1+β)}]}{− log[(1 − λ^{2})(1 − β^{2})]})] ,

the following bounds hold true:

ρ(R) − inf_{Θ_{1}} R

≤ \frac{λ − β}{2λβ} (\sqrt{1 + \frac{4λβ}{(λ − β)^{2}} \{1 − exp[−(λ − β)B(ρ)]\}} − 1)

≤ B(ρ).

Let us remark that this alternative way of handling relative deviation bounds made it possible to carry on with non-linear bounds up to the final result. For instance, if λ = 0.5, β = 0.2 and B(ρ) = 0.1, the non-linear bound gives ρ(R) − inf_{Θ_{1}} R ≤ 0.096.

68  Chapter 1. Inductive PAC-Bayesian learning

Comparing posterior distributions to Gibbs priors

2.1. Bounds relative to a Gibbs distribution

We now come to an approach to relative bounds whose performance can be analysed with PAC-Bayesian tools.

The empirical bounds at the end of the previous chapter involve taking suprema in θ ∈ Θ, and replacing the expected margin function φ with some empirical counterparts \overline{φ} or φ̃, which may prove unsafe when using very complex classification models.

We are now going to focus on the control of the divergence 𝒦[ρ, π_{exp(−βR)}]. It is already obvious, we hope, that controlling this divergence is the crux of the matter, and that it is a way to upper bound the mutual information between the training sample and the parameter, which can be expressed as 𝒦[ρ, ℙ(ρ)] = 𝒦[ρ, π_{exp(−βR)}]−𝒦[ℙ(ρ), π_{exp(−βR)}], as explained on page 28.

Through the identity

𝒦[ρ, π_{exp(−βR)}] = β[ρ(R) − π_{exp(−βR)}(R)]

+ 𝒦(ρ, π) − 𝒦[π_{exp(−βR)}, π], (2.1)

we see that the control of this divergence is related to the control of the difference ρ(R) − π_{exp(−βR)}(R). This is the route we will follow first.

Thus comparing any posterior distribution with a Gibbs prior distribution will provide a first way to build an estimator which can be proved to reach adaptively the best possible asymptotic error rate under Mammen and Tsybakov margin assumptions and parametric complexity assumptions (at least as long as orders of magnitude are concerned, we will not discuss the question of asymptotically optimal constants).

Then we will provide an empirical bound for the Kullback divergence 𝒦[ρ, π_{exp(−βR)}] itself. This will serve to address the question of model selection, which will be achieved by comparing the performance of two posterior distributions possibly supported by two different models. This will also provide a second way to build estimators which can be proved to be adaptive under Mammen and Tsybakov margin assumptions and parametric complexity assumptions (somewhat weaker than with the first method).

Finally, we will present two-step localization strategies, in which the performance of the posterior distribution to be analysed is compared with a two-step Gibbs prior.

2.1.1. Comparing a posterior distribution with a Gibbs prior. Similarly to Theorem 1.4.3 (page 53) we can prove that for any prior distribution π̃ ∈ ℳ_{+}^{1}(Θ),

ℙ\{π̃ ⊗ π̃\{exp[−N log(1 − N tanh(\frac{γ}{N} )R′)

− γr′ − N log[cosh(\frac{γ}{N})]m′]\}\} ≤ 1. (2.2)

Replacing π̃ with π_{exp(−βR)} and considering the posterior distribution ρ ⊗ π_{exp(−βR)}, provides a starting point in the comparison of ρ with π_{exp(−βR)}; we can indeed state with ℙ probability at least 1 − ϵ that

− N log\{1 − tanh(\frac{γ}{N} )[ρ(R) − π_{exp(−βR)}(R)]\}

≤ γ[ρ(r) − π_{exp(−βR)}(r)] + N log[cosh(\frac{γ}{N})][ρ ⊗ π_{exp(−βR)}](m′)

+ 𝒦[ρ, π_{exp(−βR)}] − log(ϵ). (2.3)

Using equation (2.1, page 69) to handle the entropy term, we get

− N log\{1 − tanh(\frac{γ}{N})[ρ(R) − π_{exp(−βR)}(R)]\} − β[ρ(R) − π_{exp(−βR)}(R)]

≤ γ[ρ(r) − π_{exp(−βR)}(r)] + N log[cosh(\frac{γ}{N} )]ρ ⊗ π_{exp(−βR)}(m′)

+ 𝒦(ρ, π) − 𝒦[π_{exp(−βR)}, π] − log(ϵ). (2.4)

We can then decompose in the right-hand side γ[ρ(r)−π_{exp(−βR)}(r)] into (γ − λ)[ρ(r) − π_{exp(−βR)}(r)] + λ[ρ(r) − π_{exp(−βR)}(r)] for some parameter λ to be set later on and use the fact that

λ[ρ(r) − π_{exp(−βR)}(r)] + N log[cosh(\frac{γ}{N})]ρ ⊗ π_{exp(−βR)}(m′)

+ 𝒦(ρ, π) − 𝒦[π_{exp(−βR)}, π]

≤ λρ(r) + 𝒦(ρ, π) + log\{π[exp\{−λr + N log[cosh(\frac{γ}{N})]ρ(m′)\}]\}

= 𝒦[ρ, π_{exp(−λr)}] + log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]ρ(m′)\}]\},

to get rid of the appearance of the unobserved Gibbs prior π_{exp(−βR)} in most places of the right-hand side of our inequality, leading to Theorem 2.1.1. For any real constants β and γ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), for any real constant λ,

[N tanh(\frac{γ}{N}) − β][ρ(R) − π_{exp(−βR)}(R)]

≤ −N log\{1 − tanh(\frac{γ}{N})[ρ(R) − π_{exp(−βR)}(R)]\}
− β[ρ(R) − π_{exp(−βR)}(R)]
≤ (γ − λ)[ρ(r) − π_{exp(−βR)}(r)] + 𝒦[ρ, π_{exp(−λr)}]

+ log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]ρ(m′)\}]\} − log(ϵ)

= 𝒦[ρ, π_{exp(−γr)}]
+ log\{π_{exp(−γr)}[exp\{(γ − λ)r + N log[cosh(\frac{γ}{N})]ρ(m′)\}]\}

− (γ − λ)π_{exp(−βR)}(r) − log(ϵ).

We would like to have a fully empirical upper bound even in the case when λ ≠ γ. This can be done by using the theorem twice. We will need a lemma.

Lemma 2.1.2 For any probability distribution π ∈ ℳ_{+}^{1}(Θ), for any bounded measurable functions g, h : Θ → ℝ,

π_{exp(−g)}(g) − π_{exp(−h)}(g) ≤ π_{exp(−g)}(h) − π_{exp(−h)}(h).

Proof. Let us notice that

0 ≤ 𝒦(π_{exp(−g)}, π_{exp(−h)}) = π_{exp(−g)}(h)+log{π[exp(−h)]}+𝒦(π_{exp(−g)}, π)

= π_{exp(−g)}(h) − π_{exp(−h)}(h) − 𝒦(π_{exp(−h)}, π) + 𝒦(π_{exp(−g)}, π)

= π_{exp(−g)}(h)−π_{exp(−h)}(h)−𝒦(π_{exp(−h)}, π)−π_{exp(−g)}(g)−log{π[exp(−g)]}.

Moreover

− log{π[exp(−g)]} ≤ π_{exp(−h)}(g) + 𝒦(π_{exp(−h)}, π),

which ends the proof. □

For any positive real constants β and λ, we can then apply Theorem 2.1.1 to ρ = π_{exp(−λr)}, and use the inequality

\frac{λ}{β}[π_{exp(−λr)}(r) − π_{exp(−βR)}(r)] ≤ π_{exp(−λr)}(R) − π_{exp(−βR)}(R) (2.5)

provided by the previous lemma. We thus obtain with ℙ probability at least 1 − ϵ

− N log\{1 − tanh(\frac{γ}{N})\frac{λ}{β}[π_{exp(−λr)}(r) − π_{exp(−βR)}(r)]\}

− γ[π_{exp(−λr)}(r) − π_{exp(−βR)}(r)]

≤ log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λr)}(m′)\}]\} − log(ϵ).

Let us introduce the convex function

F_{γ,α}(x) = −N log[1 − tanh(\frac{γ}{N})x] − αx ≥ [N tanh(\frac{γ}{N}) − α]x.

With ℙ probability at least 1 − ϵ,

− π_{exp(−βR)}(r) ≤ inf_{λ∈ℝ_{+}^{∗}} {−π_{exp(−λr)}(r)

+ \frac{β}{λ}F_{γ, \frac{βγ}{λ}}^{−1} [log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λr)}(m′)\}]\}

− log(ϵ)]}.

Since Theorem 2.1.1 holds uniformly for any posterior distribution ρ, we can apply it again to some arbitrary posterior distribution ρ. We can moreover make the result uniform in β and γ by considering some atomic measure ν ∈ ℳ_{+}^{1}(ℝ) on the real line and using a union bound. This leads to

Theorem 2.1.3. For any atomic probability distribution on the positive real line ν ∈ ℳ_{+}^{1}(ℝ_{+}), with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), for any positive real constants β and γ,

[N tanh(\frac{γ}{N}) − β][ρ(R) − π_{exp(−βR)}(R)]

≤ F_{γ,β}[ρ(R) − π_{exp(−βR)}(R)] ≤ B(ρ, β, γ), where

B(ρ, β, γ) = inf_{\begin{matrix}λ_{1}∈ℝ_{+},λ_{1}≤γ \\ λ_{2}∈ℝ,λ_{2}> \frac{βγ}{N} tanh( \frac{γ}{N})^{−1} \end{matrix}} \{𝒦[ρ, π_{exp(−λ_{1}r)}]

+ (γ − λ_{1})[ρ(r) − π_{exp(−λ_{2}r)}(r)]

+ log\{π_{exp(−λ_{1}r)}[exp\{N log[cosh(\frac{γ}{N})]ρ(m′)\}]\} − log[ϵν(β)ν(γ)]

+ (γ − λ_{1})\frac{β}{λ_{2}} F_{γ, \frac{βγ}{λ_{2}}}^{−1} [log\{

π_{exp(−λ_{2}r)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λ_{2}r)}(m′)\}]\}
− log[ϵν(β)ν(γ)]]}

≤ inf_{\begin{matrix}λ_{1}∈ℝ_{+},λ_{1}≤γ \\ λ_{2}∈ℝ,λ_{2}> \frac{βγ}{N} tanh( \frac{γ}{N})^{−1} \end{matrix}} \{𝒦[ρ, π_{exp(−λ_{1}r)}]

+ (γ − λ_{1})[ρ(r) − π_{exp(−λ_{2}r)}(r)]
+ log\{π_{exp(−λ_{1}r)}[exp\{N log[cosh(\frac{γ}{N})]ρ(m′)\}]\}

+ \frac{β}{λ_{2}} \frac{(1 − \frac{λ_{1}}{γ})}{[\frac{N}{γ} tanh(\frac{γ}{N}) − \frac{β}{λ_{2}} ]} log\{π_{exp(−λ_{2}r)}[

exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λ_{2}r)}(m′)\}]\}

− \{1 + \frac{β}{λ_{2}} \frac{(1− \frac{λ_{1}}{γ})}{[ \frac{N}{γ} tanh( \frac{γ}{N})− \frac{β}{λ_{2}} ]}\} log[ϵν(β)ν(γ)]\},

where we have written for short ν(β) and ν(γ) instead of ν({β}) and ν({γ}).

Let us notice that B(ρ, β, γ) = +∞ when ν(β) = 0 or ν(γ) = 0, the uniformity in β and γ of the theorem therefore necessarily bears on a countable number of values of these parameters. We can typically choose distributions for ν such as the one used in Theorem 1.2.8 (page 27): namely we can put for some positive real ratio α > 1

ν(α^{k}) = \frac{1}{(k + 1)(k + 2)}, k ∈ ℕ,

or alternatively, since we are interested in values of the parameters less than N, we can prefer

ν(α^{k}) = \frac{log(α)}{log(αN)}, 0 ≤ k < \frac{log(N)}{log(α)}.

We can also use such a coding distribution on dyadic numbers as the one defined by equation (1.7, page 29).

Following the same route as for Theorem 1.3.15 (page 46), we can also prove the following result about the deviations under any posterior distribution ρ:

Theorem 2.1.4 For any ϵ ∈)0, 1(, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), with ρ probability at least 1 − ξ,

F_{γ,β}[R(θ̂) − π_{exp(−βR)}(R)] ≤ inf_{\begin{matrix}λ_{1}∈ℝ_{+},λ_{1}≤γ, \\ λ_{2}∈ℝ,λ_{2}> \frac{βγ}{N} tanh( \frac{γ}{N})^{−1} \end{matrix}} \{log[\frac{dρ}{dπ_{exp(−λ_{1}r)}} (θ̂)]

+ (γ − λ_{1})[r(θ̂) − π_{exp(−λ_{2}r)}(r)]

+ log\{π_{exp(−λ_{1}r)}[exp\{N log[cosh(\frac{γ}{N})]m′(·, θ̂)\}]\} − log[ϵξν(β)ν(γ)]

+ (γ − λ_{1})\frac{β}{λ_{2}} F_{γ, \frac{βγ}{λ_{2}}}^{−1} [log\{

π_{exp(−λ_{2}r)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λ_{2}r)}(m′)\}]\}

− log[ϵν(β)ν(γ)]]}.

The only tricky point is to justify that we can still take an infimum in λ_{1} without using a union bound. To justify this, we have to notice that the following variant of Theorem 2.1.1 (page 71) holds: with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), for any real constant λ,

ρ{F_{γ,β}[R − π_{exp(−βR)}(R)]} ≤ 𝒦[ρ, π_{exp(−γr)}]

+ ρ[ inf_{λ∈ℝ} log\{π_{exp(−γr)}[exp\{(γ − λ)r + N log[cosh(\frac{γ}{N} )]m′(·, θ̂)\}]\}

− (γ − λ)π_{exp(−βR)}(r)] − log(ϵ).

We leave the details as an exercise.

2.1.2. The effective temperature of a posterior distribution. Using the parametric approximation π_{exp(−αr)}(r) − inf_{Θ} r ≃ \frac{d_{e}}{α}, we get as an order of magnitude

B(π_{exp(−λ_{1}r)}, β, γ) ≲ −(γ − λ_{1})d_{e}[λ_{2}^{−1} − λ_{1}^{−1}]

+ 2d_{e} log \frac{λ_{1}}{λ_{1} − N log[cosh(\frac{γ}{N})]x}

+ 2\frac{β}{λ_{2}} \frac{(1 − \frac{λ_{1}}{γ})}{[\frac{N}{γ} tanh(\frac{γ}{N}) − \frac{β}{λ_{2}} ]}d_{e} log (\frac{λ_{2}}{λ_{2} − N log[cosh(\frac{γ}{N})]x})

+ 2N log[cosh(\frac{γ}{N})][1 + \frac{β}{λ_{2}} \frac{(1 − \frac{λ_{1}}{γ})}{[\frac{N}{γ} tanh(\frac{γ}{N}) − \frac{β}{λ_{2}} ]}]φ̃(x)

− \{1 + \frac{β}{λ_{2}} \frac{(1 − \frac{λ_{1}}{γ})}{[\frac{N}{γ} tanh(\frac{γ}{N}) − \frac{β}{λ_{2}} ]}\} log[ν(β)ν(γ)ϵ].

Therefore, if the empirical dimension d_{e} stays bounded when N increases, we are going to obtain a negative upper bound for any values of the constants λ_{1} > λ_{2} > β, as soon as γ and \frac{N}{γ} are chosen to be large enough. This ability to obtain negative values for the bound B(π_{exp(−λ_{1}r)}, γ, β), and more generally B(ρ, γ, β), leads the way to introducing the new concept of the effective temperature of an estimator.

Definition 2.1.1 For any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ) we define the effective temperature T (ρ) ∈ ℝ ∪ {−∞, +∞} of ρ by the equation

ρ(R) = π_{exp(− \frac{R}{T (ρ)} )}(R).

Note that β ↦ π_{exp(−βR)}(R) : ℝ ∪ {−∞, +∞} → (0, 1) is continuous and strictly decreasing from ess sup_{π} R to ess inf_{π} R (as soon as these two bounds do not coincide). This shows that the effective temperature T (ρ) is a welldefined random variable.

Theorem 2.1.3 provides a bound for T (ρ), indeed:

Proposition 2.1.5. Let

β̂(ρ) = sup\{β ∈ ℝ; inf_{γ,N tanh( \frac{γ}{N})>β} B(ρ, β, γ) ≤ 0\},

where B(ρ, β, γ) is as in Theorem 2.1.3 (page 72). Then with ℙ probability at least 1−ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ), T (ρ) ≤ β̂(ρ)^{−1}, or equivalently ρ(R) ≤ π_{exp[−β̂(ρ)R]}(R).

This notion of effective temperature of a (randomized) estimator ρ is interesting for two reasons:

• the difference ρ(R)−π_{exp(−βR)}(R) can be estimated with better accuracy than ρ(R) itself, due to the use of relative deviation inequalities, leading to convergence rates up to 1/N in favourable situations, even when inf_{Θ} R is not close to zero;

• and of course π_{exp(−βR)}(R) is a decreasing function of β, thus being able to estimate ρ(R)−π_{exp(−βR)}(R) with some given accuracy, means being able to discriminate between values of ρ(R) with the same accuracy, although doing so through the parametrization β ↦ π_{exp(−βR)}(R), which can neither be observed nor estimated with the same precision!

2.1.3. Analysis of an empirical bound for the effective temperature. We are now going to launch into a mathematically rigorous analysis of the bound B(π_{exp(−λ_{1}r),β,γ}) provided by Theorem 2.1.3 (page 72), to show that inf_{ρ∈ℳ_{+}^{1}(Θ)} π_{exp[−β̂(ρ)R]}(R) converges indeed to inf_{Θ} R at some optimal rate in favourable situations.

It is more convenient for this purpose to use deviation inequalities involving M′ rather than m′. It is straightforward to extend Theorem 1.4.2 (page 52) to

Theorem 2.1.6. For any real constants β and γ, for any prior distributions π, μ ∈ ℳ_{+}^{1}(Θ), with ℙ probability at least 1−η, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

γρ ⊗ π_{exp(−βR)}[Ψ_{\frac{γ}{N}} (R′, M′)] ≤ γρ ⊗ π_{exp(−βR)}(r′) + 𝒦(ρ, μ) − log(η).

In order to transform the left-hand side into a linear expression and in the same time localize this theorem, let us choose μ defined by its density

\frac{dμ}{dπ}(θ_{1}) = C^{−1} exp[−βR(θ_{1})

− γ ∫_{Θ}\{Ψ_{\frac{γ}{N}} [R′(θ_{1}, θ_{2}), M′(θ_{1}, θ_{2})]

− \frac{N}{γ} sinh(\frac{γ}{N})R′(θ_{1}, θ_{2})\}π_{exp(−βR)}(dθ_{2})],

76  Chapter 2. Comparing posterior distributions to Gibbs priors

where C is such that μ(Θ) = 1. We get

𝒦(ρ, μ) = βρ(R) + γρ ⊗ π_{exp(−βR)}[Ψ_{\frac{γ}{N}} (R′, M′) − \frac{N}{γ} sinh(\frac{γ}{N})R′] + 𝒦(ρ, π)

+ log{∫_{Θ} exp[−βR(θ_{1})

− γ ∫_{Θ}\{Ψ_{\frac{γ}{N}} [R′(θ_{1}, θ_{2}), M′(θ_{1}, θ_{2})]

− \frac{N}{γ} sinh(\frac{γ}{N})R′(θ_{1}, θ_{2})\}π_{exp(−βR)}(dθ_{2})]π(dθ_{1})\}

= β[ρ(R) − π_{exp(−βR)}(R)]

+ γρ ⊗ π_{exp(−βR)}[Ψ_{\frac{γ}{N}} (R′, M′) − \frac{N}{γ} sinh(\frac{γ}{N})R′]
+ 𝒦(ρ, π) − 𝒦(π_{exp(−βR)}, π)
+ log\{∫_{Θ} exp[−γ ∫_{Θ}\{Ψ_{\frac{γ}{N}} [R′(θ_{1}, θ_{2}), M′(θ_{1}, θ_{2})]

− \frac{N}{γ} sinh(\frac{γ}{N})R′(θ_{1}, θ_{2})\}π_{exp(−βR)}(dθ_{2})]π_{exp(−βR)}(dθ_{1})\}.

Thus with ℙ probability at least 1 − η,

[N sinh(\frac{γ}{N}) − β][ρ(R) − π_{exp(−βR)}(R)]

≤ γ[ρ(r)−π_{exp(−βR)}(r)]+𝒦(ρ, π)−𝒦(π_{exp(−βR)}, π)−log(η)+C(β, γ) where C(β, γ) = log\{∫_{Θ} exp[−γ ∫_{Θ}\{Ψ_{\frac{γ}{N}} [R′(θ_{1}, θ_{2}), M′(θ_{1}, θ_{2})]

− \frac{N}{γ} sinh(\frac{γ}{N})R′(θ_{1}, θ_{2})\}π_{exp(−βR)}(dθ_{2})]π_{exp(−βR)}(dθ_{1})\}. (2.6)

Remarking that

𝒦[ρ, π_{exp(−βR)}] = β[ρ(R) − π_{exp(−βR)}(R)] + 𝒦(ρ, π) − 𝒦(π_{exp(−βR)}, π), we deduce from the previous inequality Theorem 2.1.7. For any real constants β and γ, with ℙ probability at least 1 − η, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

N sinh(\frac{γ}{N})[ρ(R) − π_{exp(−βR)}(R)] ≤ γ[ρ(r) − π_{exp(−βR)}(r)]

+ 𝒦[ρ, π_{exp(−βR)}] − log(η) + C(β, γ).

We can also go into a slightly different direction, starting back again from equation (2.6, page 76) and remarking that for any real constant λ,

λ[ρ(r) − π_{exp(−βR)}(r)] + 𝒦(ρ, π) − 𝒦(π_{exp(−βR)}, π)

≤ λρ(r) + 𝒦(ρ, π) + log{π[exp(−λr)]} = 𝒦[ρ, π_{exp(−λr)}].

This leads to Theorem 2.1.8. For any real constants β and γ, with ℙ probability at least 1 − η, for any real constant λ,

[N sinh(\frac{γ}{N}) − β][ρ(R) − π_{exp(−βR)}(R)]

≤ (γ − λ)[ρ(r) − π_{exp(−βR)}(r)] + 𝒦[ρ, π_{exp(−λr)}] − log(η) + C(β, γ),

where the definition of C(β, γ) is given by equation (2.6, page 76).

We can now use this inequality in the case when ρ = π_{exp(−λr)} and combine it with Inequality (2.5, page 71) to obtain

Theorem 2.1.9 For any real constants β and γ, with ℙ probability at least 1 − η, for any real constant λ,

[\frac{Nλ}{β} sinh(\frac{γ}{N}) − γ][π_{exp(−λr)}(r) − π_{exp(−βR)}(r)] ≤ C(β, γ) − log(η).

We deduce from this theorem

Proposition 2.1.10 For any real positive constants β_{1}, β_{2} and γ, with ℙ probability at least 1 − η, for any real constants λ_{1} and λ_{2}, such that λ_{2} < β_{2} \frac{γ}{N} sinh(\frac{γ}{N})^{−1} and λ_{1} > β_{1} \frac{γ}{N} sinh(\frac{γ}{N})^{−1},

π_{exp(−λ_{1}r)}(r) − π_{exp(−λ_{2}r)}(r) ≤ π_{exp(−β_{1}R)}(r) − π_{exp(−β_{2}R)}(r)

+ \frac{C(β_{1}, γ) + log(2/η)}{\frac{Nλ_{1}}{β_{1}} sinh(\frac{γ}{N}) − γ} + \frac{C(β_{2}, γ) + log(2/η)}{γ − \frac{Nλ_{2}}{β_{2}} sinh(\frac{γ}{N})}.

Moreover, π_{exp(−β_{1}R)} and π_{exp(−β_{2}R)} being prior distributions, with ℙ probability at least 1 − η,

γ[π_{exp(−β_{1}R)}(r) − π_{exp(−β_{2}R)}(r)]

≤ γπ_{exp(−β_{1}R)} ⊗ π_{exp(−β_{2}R)}[Ψ_{− \frac{γ}{N}} (R′, M′)] − log(η).

Hence

Proposition 2.1.11 For any positive real constants β_{1}, β_{2} and γ, with ℙ probability at least 1 − η, for any positive real constants λ_{1} and λ_{2} such that λ_{2} < β_{2} \frac{γ}{N} sinh(\frac{γ}{N})^{−1} and λ_{1} > β_{1} \frac{γ}{N} sinh(\frac{γ}{N})^{−1},

π_{exp(−λ_{1}r)}(r) − π_{exp(−λ_{2}r)}(r)

≤ π_{exp(−β_{1}R)} ⊗ π_{exp(−β_{2}R)}[Ψ_{− \frac{γ}{N}} (R′, M′)]

+ \frac{log(\frac{3}{η})}{γ} + \frac{C(β_{1}, γ) + log(\frac{3}{η})}{\frac{Nλ_{1}}{β_{1}} sinh(\frac{γ}{N}) − γ} + \frac{C(β_{2}, γ) + log(\frac{3}{η})}{γ − \frac{Nλ_{2}}{β_{2}} sinh(\frac{γ}{N})}.

In order to achieve the analysis of the bound B(π_{exp(−λ_{1}r)}, β, γ) given by Theorem 2.1.3 (page 72), it now remains to bound quantities of the general form

log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λr)}(m′)\}]\}

= sup_{ρ∈ℳ_{+}^{1}(Θ)} N log[cosh(\frac{γ}{N})]ρ ⊗ π_{exp(−λ)}(m′) − 𝒦[ρ, π_{exp(−λr)}].

Let us consider the prior distribution μ ∈ ℳ_{+}^{1}(Θ × Θ) on couples of parameters defined by the density

\frac{dμ}{d(π ⊗ π)}(θ_{1}, θ_{2}) = C^{−1} exp\{−βR(θ_{1}) − βR(θ_{2}) + αΦ_{− \frac{α}{N}} [M′(θ_{1}, θ_{2})]\},

where the normalizing constant C is such that μ(Θ × Θ) = 1. Since for fixed values of the parameters θ and θ′ ∈ Θ, m′(θ, θ′), like r(θ), is a sum of independent Bernoulli random variables, we can easily adapt the proof of Theorem 1.1.4 on page 17, to establish that with ℙ probability at least 1 − η, for any posterior distribution ρ and any real constant λ,

αρ ⊗ π_{exp(−λr)}(m′) ≤ αρ ⊗ π_{exp(−λr)}[Φ_{− \frac{α}{N}} (M′)]

+ 𝒦(ρ ⊗ π_{exp(−λr)}, μ) − log(η)

= 𝒦[ρ, π_{exp(−βR)}] + 𝒦[π_{exp(−λr)}, π_{exp(−βR)}]

+ log\{π_{exp(−βR)} ⊗ π_{exp(−βR)}[exp(αΦ_{− \frac{α}{N}} ∘M′)]\} − log(η).

Thus for any real constant β and any positive real constants α and γ, with ℙ probability at least 1 − η, for any real constant λ,

log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λr)}(m′)\}]\}

≤ sup_{ρ∈ℳ_{+}^{1}(Θ)} (\frac{N}{α} log[cosh(\frac{γ}{N})]\{𝒦[ρ, π_{exp(−βR)}] + 𝒦[π_{exp(−λr)}, π_{exp(−βR)}]

+ log\{π_{exp(−βR)} ⊗ π_{exp(−βR)}[exp(αΦ_{− \frac{α}{N}} ∘M′)]\}

− log(η)} − 𝒦[ρ, π_{exp(−λr)}]). (2.7)

To finish, we need some appropriate upper bound for the entropy 𝒦[ρ, π_{exp(−βR)}]. This question can be handled in the following way: using Theorem 2.1.7 (page 76), we see that for any positive real constants γ and β, with ℙ probability at least 1 − η, for any posterior distribution ρ,

𝒦[ρ, π_{exp(−βR)}] = β[ρ(R) − π_{exp(−βR)}(R)] + 𝒦(ρ, π) − 𝒦(π_{exp(−βR)}, π)

≤ \frac{β}{N sinh(\frac{γ}{N})}[γ[ρ(r) − π_{exp(−βR)}(r)]

+ 𝒦[ρ, π_{exp(−βR)}] − log(η) + C(β, γ)]

+ 𝒦(ρ, π) − 𝒦(π_{exp(−βR)}, π)
≤ 𝒦[ρ, π_{exp(−_{\frac{βγ}{N sinh( \frac{γ}{N} )}} r)}]

+ \frac{β}{N sinh(\frac{γ}{N})}\{𝒦[ρ, π_{exp(−βR)}] + C(β, γ) − log(η)\}.

In other words,

Theorem 2.1.12. For any positive real constants β and γ such that β < N sinh(\frac{γ}{N}), with ℙ probability at least 1 − η, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

𝒦[ρ, π_{exp(−βR)}] ≤ \frac{𝒦[ρ, π_{exp[−β \frac{γ}{N} sinh( \frac{γ}{N})^{−1}r]}]}{1 − \frac{β}{N sinh(\frac{γ}{N})}} + \frac{C(β, γ) − log(η)}{\frac{N sinh(\frac{γ}{N})}{β} − 1},

where the quantity C(β, γ) is defined by equation (2.6, page 76). Equivalently, it will be in some cases more convenient to use this result in the form: for any positive real constants λ and γ, with ℙ probability at least 1 − η, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

𝒦[ρ, π_{exp[−λ \frac{N}{γ} sinh( \frac{γ}{N})R]}] ≤ \frac{𝒦[ρ, π_{exp(−λr)}]}{1 − \frac{λ}{γ}} + \frac{C(λ\frac{N}{γ} sinh(\frac{γ}{N}), γ) − log(η)}{\frac{λ}{β} − 1}.

Choosing in equation (2.7, page 78) α = \frac{N log[cosh(\frac{γ}{N})]}{1 − \frac{β}{N sinh( \frac{γ}{N})}} and

β = λ\frac{N}{γ} sinh(\frac{γ}{N}), so that α = \frac{N log[cosh(\frac{γ}{N})]}{1 − \frac{λ}{γ}}, we obtain with ℙ proba-

bility at least 1 − η,

log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λr)}(m′)\}]\}

≤ \frac{2λ}{γ}[C(β, γ) + log(\frac{2}{η})]
+ (1 − \frac{λ}{γ})[log\{π_{exp(−βR)} ⊗ π_{exp(−βR)}[exp(αΦ_{− \frac{α}{N}} ∘M′)]\}
+ log(\frac{2}{η})].

This proves

Proposition 2.1.13. For any positive real constants λ < γ, with ℙ probability at least 1 − η, log\{π_{exp(−λr)}[exp\{N log[cosh(\frac{γ}{N})]π_{exp(−λr)}(m′)\}]\}

≤ \frac{2λ}{γ}[C(\frac{Nλ}{γ} sinh(\frac{γ}{N}), γ) + log(\frac{2}{η})]

+ (1 − \frac{λ}{γ}) log\{π_{exp[− \frac{Nλ}{γ} sinh( \frac{γ}{N})R]}^{⊗2}[

exp(\frac{N log[cosh(\frac{γ}{N})]}{1 − \frac{λ}{γ}}Φ_{− \frac{log[cosh( \frac{γ}{N} )]}{1− \frac{λ}{γ}}} ∘M′)]\}

+ (1 − \frac{λ}{γ}) log(\frac{2}{η}).

We are now ready to analyse the bound B(π_{exp(−λ_{1}r)}, β, γ) of Theorem 2.1.3 (page 72).

Theorem 2.1.14. For any positive real constants λ_{1}, λ_{2}, β_{1}, β_{2}, β and γ, such that

λ_{1} < γ,  β_{1} < \frac{Nλ_{1}}{γ} sinh(\frac{γ}{N}),
λ_{2} < γ,  β_{2} > \frac{Nλ_{2}}{γ} sinh(\frac{γ}{N}),
β < \frac{Nλ_{2}}{γ} tanh(\frac{γ}{N}),

with ℙ probability 1−η, the bound B(π_{exp(−λ_{1}r)}, β, γ) of Theorem 2.1.3 (page 72) satisfies

B(π_{exp(−λ_{1}r)}, β, γ)

≤ (γ − λ_{1})\{π_{exp(−β_{1}R)} ⊗ π_{exp(−β_{2}R)}[Ψ_{− \frac{γ}{N}} (R′, M′)] + \frac{log(\frac{7}{η})}{γ}

+ \frac{C(β_{1}, γ) + log(\frac{7}{η})}{\frac{Nλ_{1}}{β_{1}} sinh(\frac{γ}{N}) − γ} + \frac{C(β_{2}, γ) + log(\frac{7}{η})}{γ − \frac{Nλ_{2}}{β_{2}} sinh(\frac{γ}{N})}\}

+ \frac{2λ_{1}}{γ} [C(\frac{Nλ_{1}}{γ} sinh(\frac{γ}{N}), γ) + log(\frac{7}{η})]

+ (1 − \frac{λ_{1}}{γ} ) log\{π_{exp[− \frac{Nλ_{1}}{γ} sinh( \frac{γ}{N})R]}^{⊗2}[

exp(\frac{N log[cosh( \frac{γ}{N})]}{1− \frac{λ_{1}}{γ}}Φ_{− \frac{log[cosh( \frac{γ}{N} )]}{1− \frac{λ_{1}}{γ}}} ∘M′)]\}

+ (1 − \frac{λ_{1}}{γ} ) log(\frac{7}{η}) − log[ν(\{β\})ν(\{γ\})ϵ]

+ (γ − λ_{1})\frac{β}{λ_{2}} F_{γ, \frac{βγ}{λ_{2}}}^{−1} \{\frac{2λ_{2}}{γ} [C(\frac{Nλ_{2}}{γ} sinh(\frac{γ}{N}), γ) + log(\frac{7}{η})]

+ (1 − \frac{λ_{2}}{γ} ) log\{π_{exp[− \frac{Nλ_{2}}{γ} sinh( \frac{γ}{N})R]}^{⊗2}[

exp(\frac{N log[cosh(\frac{γ}{N})]}{1 − \frac{λ_{2}}{γ}}Φ_{− \frac{log[cosh( \frac{γ}{N} )]}{1− \frac{λ_{2}}{γ}}} ∘M′)]\}

+ (1 − \frac{λ_{2}}{γ} ) log(\frac{7}{η}) − log[ν(\{β\})ν(\{γ\})ϵ]\},

where the function C(β, γ) is defined by equation (2.6, page 76).

2.1.4. Adaptation to parametric and margin assumptions. To help understand the previous theorem, it may be useful to give linear upperbounds to the factors appearing in the right-hand side of the previous inequality. Introducing θ̃ such that R(θ̃) = inf_{Θ} R (assuming that such a parameter exists) and remembering that

Ψ_{−a}(p, m) ≤ a^{−1} sinh(a)p + 2a^{−1} sinh(\frac{a}{2})^{2}m,  a ∈ ℝ_{+},
Φ_{−a}(p) ≤ a^{−1}[exp(a) − 1]p,  a ∈ ℝ_{+},
Ψ_{a}(p, m) ≥ a^{−1} sinh(a)p − 2a^{−1} sinh(\frac{a}{2})^{2}m,  a ∈ ℝ_{+},
M′(θ_{1}, θ_{2}) ≤ M′(θ_{1}, θ̃) + M′(θ_{2}, θ̃),  θ_{1}, θ_{2} ∈ Θ,
M′(θ_{1}, θ̃) ≤ xR′(θ_{1}, θ̃) + φ(x),  x ∈ ℝ_{+}, θ_{1} ∈ Θ,

the last inequality being rather a consequence of the definition of φ than a property of M′, we easily see that

π_{exp(−β_{1}R)} ⊗ π_{exp(−β_{2}R)}[Ψ_{− \frac{γ}{N}} (R′, M′)]

≤ \frac{N}{γ} sinh(\frac{γ}{N})[π_{exp(−β_{1}R)}(R) − π_{exp(−β_{2}R)}(R)]

+ \frac{2N}{γ} sinh(\frac{γ}{2N})^{2}π_{exp(−β_{1}R)} ⊗ π_{exp(−β_{2}R)}(M′)

≤ \frac{N}{γ} sinh(\frac{γ}{N})[π_{exp(−β_{1}R)}(R) − π_{exp(−β_{2}R)}(R)]

+ \frac{2xN}{γ} sinh(\frac{γ}{2N})^{2}\{π_{exp(−β_{1}R)}[R′(·, θ̃)] + π_{exp(−β_{2}R)}[R′(·, θ̃)]\}

+ \frac{4N}{γ} sinh(\frac{γ}{2N})^{2}φ(x),

that

C(β, γ) ≤ log\{π_{exp(−βR)}\{exp[2N sinh(\frac{γ}{2N} )^{2}π_{exp(−βR)}(M′)]\}\}

≤ log\{π_{exp(−βR)}\{exp[2N sinh(\frac{γ}{2N} )^{2}M′(·, θ̃)]\}\}

+ 2N sinh(\frac{γ}{2N})^{2}π_{exp(−βR)}[M′(·, θ̃)]

≤ log\{π_{exp(−βR)}\{exp[2xN sinh(\frac{γ}{2N})^{2}R′(·, θ̃)]\}\}

+ 2xN sinh(\frac{γ}{2N})^{2}π_{exp(−βR)}[R′(·, θ̃)] + 4N sinh(\frac{γ}{2N})^{2}φ(x)

= ∫_{β−2xN sinh( \frac{γ}{2N})^{2}}^{β} π_{exp(−αR)}[R′(·, θ̃)]dα
+ 2xN sinh(\frac{γ}{2N})^{2}π_{exp(−βR)}[R′(·, θ̃)] + 4N sinh(\frac{γ}{2N})^{2}φ(x)
≤ 4xN sinh(\frac{γ}{2N})^{2}π_{exp[−(β−2xN sinh( \frac{γ}{2N})^{2})R]}[R′(·, θ̃)]
+ 4N sinh(\frac{γ}{2N})^{2}φ(x),

and that

log{π_{exp(−βR)}^{⊗2}[exp(NαΦ_{−α}∘M′)]}

≤ 2 log{π_{exp(−βR)}[exp(N[exp(α) − 1]M′(·, θ̃))]}
≤ 2xN[exp(α) − 1]π_{exp[−(β−xN[exp(α)−1])R]}[R′(·, θ̃)]
+ 2xN[exp(α) − 1]φ(x).

Let us push further the investigation under the parametric assumption that for some positive real constant d

lim_{β→+∞} βπ_{exp(−βR)}[R′(·, θ̃)] = d,  (2.8)

This assumption will for instance hold true with d = \frac{n}{2} when R : Θ → (0, 1) is a smooth function defined on a compact subset Θ of ℝ^{n} that reaches its minimum value on a finite number of non-degenerate (i.e. with a positive definite Hessian) interior points of Θ, and π is absolutely continuous with respect to the Lebesgue measure on Θ and has a smooth density.

In case of assumption (2.8), if we restrict ourselves to sufficiently large values of the constants β, β_{1}, β_{2}, λ_{1}, λ_{2} and γ (the smaller of which is as a rule β, as we will see), we can use the fact that for some (small) positive constant δ, and some (large) positive constant A,

\frac{d}{α}(1 − δ) ≤ π_{exp(−αR)}[R′(·, θ̃)] ≤ \frac{d}{α}(1 + δ),  α ≥ A.  (2.9)

Under this assumption,

π_{exp(−β_{1}R)} ⊗ π_{exp(−β_{2}R)}[Ψ_{− \frac{γ}{N}} (R′, M′)]

≤ \frac{N}{γ} sinh(\frac{γ}{N})[\frac{d}{β_{1}} (1 + δ) − \frac{d}{β_{2}} (1 − δ)]

+ \frac{2xN}{γ} sinh(\frac{γ}{2N})^{2}(1 + δ)[\frac{d}{β_{1}} + \frac{d}{β_{2}} ] + \frac{4N}{γ} sinh(\frac{γ}{2N})^{2}φ(x).

C(β, γ) ≤ d(1 + δ) log(\frac{β}{β−2xN sinh( \frac{γ}{2N})^{2}} )

+ 2xN sinh(\frac{γ}{2N})^{2} \frac{(1+δ)d}{β} + 4N sinh(\frac{γ}{2N})^{2}φ(x).

log{π_{exp(−βR)}^{⊗2}[exp(NαΦ_{−α}∘M′)]}

≤ 2xN[exp(α) − 1]\frac{d(1 + δ)}{β − xN[exp(α) − 1]} + 2N[exp(α) − 1]φ(x).

2.1. Bounds relative to a Gibbs distribution  83

Thus with ℙ probability at least 1 − η, B(π_{exp(−λ_{1}r)}, β, γ) ≤ −(γ − λ_{1})\frac{N}{γ} sinh(\frac{γ}{N})\frac{d}{β_{2}} (1 − δ) + (γ − λ_{1})\{\frac{N}{γ} sinh(\frac{γ}{N})\frac{(1+δ)d}{β_{1}}

+ \frac{2xN}{γ} sinh(\frac{γ}{2N})^{2}(1 + δ)[\frac{d}{β_{1}} + \frac{d}{β_{2}} ] + \frac{4N}{γ} sinh(\frac{γ}{2N})^{2}φ(x) + \frac{log(\frac{7}{η})}{γ}

+ \frac{4xN sinh(\frac{γ}{2N})^{2} \frac{(1+δ)d}{β_{1}−2xN sinh( \frac{γ}{2N})^{2}} + 4N sinh(\frac{γ}{2N})^{2}φ(x) + log(\frac{7}{η})}{\frac{Nλ_{1}}{β_{1}} sinh(\frac{γ}{N}) − γ}

+ \frac{4xN sinh(\frac{γ}{2N})^{2} \frac{(1+δ)d}{β_{2}−2xN sinh( \frac{γ}{2N})^{2}} + 4N sinh(\frac{γ}{2N})^{2}φ(x) + log(\frac{7}{η})}{γ − \frac{Nλ_{2}}{β_{2}} sinh(\frac{γ}{N})}\}

+ \frac{2λ_{1}}{γ} \{4xN sinh(\frac{γ}{2N})^{2} _{\frac{(1+δ)d}{\frac{Nλ_{1}}{γ} sinh( \frac{γ}{N} )−2xN sinh( \frac{γ}{2N})^{2}}}

+ 4N sinh(\frac{γ}{2N})^{2}φ(x) + log(\frac{7}{η})\}

+ (1 − \frac{λ_{1}}{γ} )\{2d(1 + δ)(_{\frac{λ_{1} sinh(\frac{γ}{N} )}{xγ[exp(\frac{log[cosh( \frac{γ}{N} )]}{1− \frac{λ_{1}}{γ}})−1]}} − 1)^{−1}

+ 2N[exp(\frac{log[cosh( \frac{γ}{N})]}{1− \frac{λ_{1}}{γ}}) − 1]φ(x)\}

+ (1 − \frac{λ_{1}}{γ} ) log(\frac{7}{η}) − log[ν(\{β\})ν(\{γ\})ϵ]

+ \frac{1 − \frac{λ_{1}}{γ}}{\frac{Nλ_{2}}{βγ} tanh(\frac{γ}{N}) − 1}\{\frac{2λ_{2}}{γ} \{4xN sinh(\frac{γ}{2N})^{2} _{\frac{(1+δ)d}{\frac{Nλ_{2}}{γ} sinh( \frac{γ}{N} )−2xN sinh( \frac{γ}{2N})^{2}}}

+ 4N sinh(\frac{γ}{2N})^{2}φ(x) + log(\frac{7}{η})\}

+ (1 − \frac{λ_{2}}{γ} )[2d(1 + δ)(_{\frac{λ_{2} sinh(\frac{γ}{N} )}{xγ[exp(\frac{log[cosh( \frac{γ}{N} )]}{1− \frac{λ_{2}}{γ}})−1]}} − 1)^{−1}

+ 2N[exp(\frac{log[cosh( \frac{γ}{N})]}{1− \frac{λ_{2}}{γ}}) − 1]φ(x)]

+ (1 − \frac{λ_{2}}{γ} ) log(\frac{7}{η}) − log[ν(β)ν(γ)ϵ]\}.

Now let us choose for simplicity β_{2} = 2λ_{2} = 4β, β_{1} = λ_{1}/2 = γ/4, and let us introduce the notation

C_{1} = \frac{N}{γ} sinh(\frac{γ}{N}),

84  Chapter 2. Comparing posterior distributions to Gibbs priors
C_{2} = \frac{N}{γ} tanh(\frac{γ}{N}),
C_{3} = \frac{N^{2}}{γ^{2}} [exp(\frac{γ^{2}}{N^{2}} ) − 1]
and C_{4} = \frac{2N^{2}(1 − \frac{2β}{γ})}{γ^{2}}[exp(\frac{γ^{2}}{2N^{2}(1 − \frac{2β}{γ})}) − 1],

to obtain

B(π_{exp(−λ_{1}r)}, β, γ) ≤ −\frac{C_{1}γ}{8β} (1 − δ)d

+ \frac{C_{1}γ}{2} \{\frac{4(1+δ)d}{γ} + x\frac{γ}{2N} (1 + δ)[\frac{4d}{γ} + \frac{d}{4β} ] + \frac{γ}{N} φ(x)\} + \frac{1}{2} log(\frac{7}{η})

+ \frac{1}{2C_{1} − 1}[(1 + δ)d(\frac{N}{2xC_{1}γ} − 1)^{−1} + C_{1} \frac{γ^{2}}{2N} φ(x) + \frac{1}{2} log(\frac{7}{η})]

+ \frac{1}{2 − C_{1}} [2(1 + δ)d(\frac{8Nβ}{xC_{1}γ^{2}} − 1)^{−1} + C_{1} \frac{γ^{2}}{N} φ(x) + log(\frac{7}{η})]

+ \frac{2xγ(1 + δ)d}{N − xγ} + C_{1} \frac{γ^{2}}{N} φ(x) + log(\frac{7}{η})

+ d(1 + δ)\frac{xγ}{N} (\frac{C_{1}}{2C_{3}} − \frac{xγ}{N} )^{−1} + \frac{γ^{2}}{N} C_{3}φ(x) + \frac{log(\frac{7}{η})}{2} − log[ν(β)ν(γ)ϵ]

+ (4C_{2} − 2)^{−1}\{\frac{4β}{γ} \{x\frac{γ^{2}}{N} C_{1}(1 + δ)d(2βC_{1} − xC_{1} \frac{γ^{2}}{2N} )^{−1}

+ \frac{γ^{2}}{N} φ(x) + log(\frac{7}{η})\}

+ (1 − \frac{2β}{γ} )\{2d(1 + δ)\frac{xγ}{N} [\frac{4βC_{1}}{γC_{4}} (1 − \frac{2β}{γ} ) − \frac{xγ}{N} ]^{−1}

+ \frac{γ^{2}}{N(1 − \frac{2β}{γ})}C_{4}φ(x)\}

+ (1 − \frac{2β}{γ} ) log(\frac{7}{η}) − log[ν(β)ν(γ)ϵ]\}.

This simplifies to

B(π_{exp(−λ_{1}r)}, β, γ) ≤ −\frac{C_{1}}{8} (1 − δ)d\frac{γ}{β}

+ 2C_{1}(1 + δ)d + log(\frac{7}{η})[2 + \frac{3C_{1}}{(4C_{1}−2)(2−C_{1})} + \frac{1 + \frac{2β}{γ}}{4C_{2} − 2}]

− (1 + \frac{1}{4C_{2}−2}) log[ν(β)ν(γ)ϵ]

+ \frac{(1 + δ)dxγ}{N} \{C_{1} + \frac{1}{2C_{1}−1}(\frac{1}{2C_{1}} − \frac{γx}{N})^{−1}

+ 2(1 − \frac{γx}{N})^{−1} + (\frac{C_{1}}{2C_{3}} − \frac{γx}{N})^{−1} + \frac{4C_{1}β}{γ(4C_{2}−2)}\}

+ \frac{(1 + δ)dxγ^{2}}{Nβ} \{\frac{C_{1}}{16} + \frac{2}{2−C_{1}} (\frac{8}{C_{1}} − \frac{xγ^{2}}{Nβ} )^{−1}

+ (1 − \frac{2β}{γ} )\frac{1}{2C_{2}−1}[\frac{4C_{1}}{C_{4}} (1 − \frac{2β}{γ} ) − \frac{γ^{2}x}{βN}]^{−1}\}

+ \frac{γ^{2}}{N} φ(x)\{\frac{3C_{1}}{2} + \frac{C_{1}}{4C_{1}−2} + \frac{C_{1}}{2−C_{1}} + C_{3} + \frac{4β}{γ(4C_{2}−2)} + \frac{C_{4}}{4C_{2}−2}\}.

This shows that there exist universal positive real constants A_{1}, A_{2}, B_{1}, B_{2}, B_{3}, and B_{4} such that as soon as \frac{γ max\{x,1\}}{N} ≤ A_{1} \frac{β}{γ} ≤ A_{2},

B(π_{exp(−λ_{1}r)}, β, γ) ≤ −B_{1}(1 − δ)d\frac{γ}{β} + B_{2}(1 + δ)d

− B_{3} log[ν(β)ν(γ)ϵ η] + B_{4} \frac{γ^{2}}{N} φ(x).

Thus π_{exp(−λ_{1}r)}(R) ≤ π_{exp(−βR)}(R) ≤ inf_{Θ} R + \frac{(1+δ)d}{β} as soon as

\frac{β}{γ} ≤ \frac{B_{1}}{B_{2} \frac{(1+δ)}{(1−δ)} + ^{\frac{B_{4} \frac{γ^{2}}{N} φ(x)−B_{3} log[ν(β)ν(γ)ϵη]}{(1−δ)d}}} .

Choosing some real ratio α > 1, we can now make the above result uniform for any

β, γ ∈ Λ_{α} =^{def} \{α^{k}; k ∈ ℕ, 0 ≤ k < \frac{log(N)}{log(α)}\},  (2.10)

by substituting ν(β) and ν(γ) with \frac{log(α)}{log(αN)} and − log(η) with − log(η) +

2 log [\frac{log(αN)}{log(α)}].

Taking η = ϵ for simplicity, we can summarize our result in

Theorem 2.1.15. There exist positive real universal constants A, B_{1}, B_{2}, B_{3} and B_{4} such that for any positive real constants α > 1, d and δ, for any prior distribution π ∈ ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any β, γ ∈ Λ_{α} (where Λ_{α} is defined by equation (2.10) above) such that

sup_{β′∈ℝ,β′≥β} |\frac{β′}{d} [π_{exp(−β′R)}(R) − inf_{Θ} R] − 1| ≤ δ

and such that also for some positive real parameter x

\frac{γ max\{x, 1\}}{N} ≤ \frac{Aβ}{γ} and \frac{β}{γ} ≤ \frac{B_{1}}{B_{2} \frac{(1+δ)}{(1−δ)} + ^{\frac{B_{4} \frac{γ^{2}}{N} φ(x)−2B_{3} log(ϵ)+4B_{3} log[\frac{log(N)}{log(α)}]}{(1−δ)d}}} ,

the bound B(π_{exp(− \frac{γ}{2} r)}, β, γ) given by Theorem 2.1.3 on page 72 in the case where we have chosen ν to be the uniform probability measure on Λ_{α}, satisfies B(π_{exp(− \frac{γ}{2} r)}, β, γ) ≤ 0, proving that β̂(π_{exp(− \frac{γ}{2} r)}) ≥ β and therefore that

π_{exp(−γ \frac{r}{2} )}(R) ≤ π_{exp(−βR)}(R) ≤ inf_{Θ} R + \frac{(1 + δ)d}{β}.

What is important in this result is that we do not only bound π_{exp(− \frac{γ}{2} r)}(R), but also B(π_{exp(− \frac{γ}{2} r)}, β, γ), and that we do it uniformly on a grid of values of β and γ, showing that we can indeed set the constants β and γ adaptively using the empirical bound B(π_{exp(− \frac{γ}{2} r)}, β, γ).

Let us see what we get under the margin assumption (1.24, page 56). When κ = 1, we have φ(c^{−1}) ≤ 0, leading to

Corollary 2.1.16. Assuming that the margin assumption (1.24, page 56) is satisfied for κ = 1, that R : Θ → (0, 1) is independent of N (which is the case for instance when ℙ = P^{⊗N}), and is such that

lim_{β′→+∞} β′[π_{exp(−β′R)}(R) − inf_{Θ} R] = d,

there are universal positive real constants B_{5} and B_{6} and N_{1} ∈ ℕ such that for any N ≥ N_{1}, with ℙ probability at least 1 − ϵ

π_{exp(−γ̂ \frac{r}{2} )}(R) ≤ inf_{Θ} R + \frac{B_{5}d}{cN} [1 + \frac{B_{6}}{d} log(\frac{log(N)}{ϵ})]^{2} ,

where γ̂ ∈ arg max_{γ∈Λ_{2}} max\{β ∈ Λ_{2}; B(π_{exp(−γ \frac{r}{2} )}, β, γ) ≤ 0\}, where Λ_{2} is defined by equation (2.10, page 85), and B is the bound of Theorem 2.1.3 (page 72).

When κ > 1, φ(x) ≤ (1 − κ^{−1})(κcx)^{− \frac{1}{κ−1}} , and we can choose γ and x such that \frac{γ^{2}}{N} φ(x) ≃ d to prove

Corollary 2.1.17. Assuming that the margin assumption (1.24, page 56) is satisfied for some exponent κ > 1, that R : Θ → (0, 1) is independent of N (which is for instance the case when ℙ = P^{⊗N}), and is such that

lim_{β′→+∞} β′[π_{exp(−β′R)}(R) − inf_{Θ} R] = d,

there are universal positive constants B_{7} and B_{8} and N_{1} ∈ ℕ such that for any N ≥ N_{1}, with ℙ probability at least 1 − ϵ,

π_{exp(−γ̂ \frac{r}{2} )}(R) ≤ inf_{Θ} R + B_{7}c^{− \frac{1}{2κ−1}} [1 + \frac{B_{8}}{d} log(\frac{log(N)}{ϵ})]^{\frac{2κ}{2κ−1}} (\frac{d}{N} )^{\frac{κ}{2κ−1}} ,

where γ̂ ∈ arg max_{γ∈Λ_{2}} max\{β ∈ Λ_{2}; B(π_{exp(−γ \frac{r}{2} )}, β, γ) ≤ 0\}, Λ_{2} being defined by equation (2.10, page 85) and B by Theorem 2.1.3 (page 72).

We find the same rate of convergence as in Corollary 1.4.7 (page 57), but this time, we were able to provide an empirical posterior distribution π_{exp(−γ̂ \frac{r}{2} )} which achieves this rate adaptively in all the parameters (meaning in particular that we do not need to know d, c or κ). Moreover, as already mentioned, the power of N in this rate of convergence is known to be optimal in the worst case (see Mammen et al. (1999); Tsybakov (2004); Tsybakov et al. (2005), and more specifically in Audibert (2004b) — downloadable from its author’s web page — Theorem 3.3, page 132).

2.1.5. Estimating the divergence of a posterior with respect to a Gibbs prior. Another interesting question is to estimate 𝒦[ρ, π_{exp(−βR)}] using relative deviation inequalities. We follow here an idea to be found first in (Audibert, 2004b, page 93). Indeed, combining equation (2.3, page 70) with equation (2.1, page 69), we see that for any positive real parameters β and λ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

𝒦[ρ, π_{exp(−βR)}] ≤ \frac{β}{N tanh(\frac{γ}{N})}\{γ[ρ(r) − π_{exp(−βR)}(r)]

+ N log[cosh(\frac{γ}{N})]ρ ⊗ π_{exp(−βR)}(m′)

+ 𝒦[ρ, π_{exp(−βR)}] − log(ϵ)} + 𝒦(ρ, π) − 𝒦[π_{exp(−βR)}, π]

≤ 𝒦[ρ, π_{exp[−_{\frac{βγ}{N tanh( \frac{γ}{N} )}} r]}] + \frac{β}{N tanh(\frac{γ}{N})}\{𝒦[ρ, π_{exp(−βR)}] − log(ϵ)\}

+ log[π_{exp[−_{\frac{βγ}{N tanh( \frac{γ}{N} )}} r]}\{exp[\frac{β}{tanh(\frac{γ}{N})} log[cosh(\frac{γ}{N})]ρ(m′)]\}].

We thus obtain

Theorem 2.1.18. For any positive real constants β and γ such that β < N tanh(\frac{γ}{N}), with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

𝒦[ρ, π_{exp(−βR)}] ≤ (1 − \frac{β}{N} tanh (\frac{γ}{N} )^{−1})^{−1}

× \{𝒦[ρ, π_{exp[− \frac{βγ}{N} tanh( \frac{γ}{N})^{−1}r]}] − \frac{β}{N tanh(\frac{γ}{N})} log(ϵ)

+ log\{π_{exp[− \frac{βγ}{N} tanh( \frac{γ}{N})^{−1}r]}[exp\{β tanh(\frac{γ}{N})^{−1} log[cosh(\frac{γ}{N})]ρ(m′)\}]\}\}.

This theorem provides another way of measuring over-fitting, since it gives an upper bound for 𝒦[π_{exp[− \frac{βγ}{N} tanh( \frac{γ}{N})^{−1}r]}, π_{exp(−βR)}]. It may be used in combination with Theorem 1.2.6 (page 25) as an alternative to Theorem 1.3.7 (page 35). It will also be used in the next section.

An alternative parametrization of the same result providing a simpler right-hand side is also useful:

Corollary 2.1.19. For any positive real constants β and γ such that β < γ, with ℙ probability at least 1 − ϵ, for any posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ),

𝒦[ρ, π_{exp[−N \frac{β}{γ} tanh( \frac{γ}{N})R]}] ≤ (1 − \frac{β}{γ} )^{−1}\{𝒦[ρ, π_{exp(−βr)}] − \frac{β}{γ} log(ϵ)

+ log\{π_{exp(−βr)}[exp\{N \frac{β}{γ} log[cosh(\frac{γ}{N})]ρ(m′)\}]\}\}.

2.2. Playing with two posterior and two local prior distributions

2.2.1. Comparing two posterior distributions. Estimating the effective temperature of an estimator provides an efficient way to tune parameters in a model with parametric behaviour. On the other hand, it will not be fitted to choose between different models, especially when they are nested, because as we already saw in the case when Θ is a union of nested models, the prior distribution π_{exp(−βR)} does not provide an efficient localization of the parameter in this case, in the sense that π_{exp(−βR)}(R) does not go down to inf_{Θ} R at the desired rate when β goes to +∞, requiring a resort to partial localization.

Once some estimator (in the form of a posterior distribution) has been chosen in each sub-model, these estimators can be compared between themselves with the help of the relative bounds that we will establish in this section. It is also possible to choose several estimators in each sub-model, to tune parameters in the same time (like the inverse temperature parameter if we decide to use Gibbs posterior distributions in each sub-model).

From equation (2.2 page 70) (slightly modified by replacing π ⊗ π with π^{1} ⊗ π^{2}), we easily obtain

Theorem 2.2.1. For any positive real constant λ, for any prior distributions π^{1}, π^{2} ∈ ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any posterior distributions ρ_{1} and ρ_{2} : Ω → ℳ_{+}^{1}(Θ),

− N log\{1 − tanh(\frac{λ}{N} )[ρ_{2}(R) − ρ_{1}(R)]\} ≤ λ[ρ_{2}(r) − ρ_{1}(r)]

+ N log[cosh(\frac{λ}{N} )]ρ_{1} ⊗ ρ_{2}(m′)

+ 𝒦(ρ_{1}, π^{1}) + 𝒦(ρ_{2}, π^{2}) − log(ϵ).

This is where the entropy bound of the previous section enters into the game, providing a localized version of Theorem 2.2.1 (page 88). We will use the notation

Ξ_{a}(q) = tanh(a)^{−1}[1 − exp(−aq)] ≤ \frac{a}{tanh(a)}q,  a, q ∈ ℝ.  (2.11)

Theorem 2.2.2. For any ϵ ∈)0, 1(, any sequence of prior distributions (π^{i})_{i∈ℕ} ∈ ℳ_{+}^{1}(Θ)^{ℕ} , any probability distribution μ on ℕ, any atomic probability distribution ν on ℝ_{+}, with ℙ probability at least 1−ϵ, for any posterior distributions ρ_{1}, ρ_{2} : Ω → ℳ_{+}^{1}(Θ),

ρ_{2}(R) − ρ_{1}(R) ≤ B(ρ_{1}, ρ_{2}), where

B(ρ_{1}, ρ_{2}) = inf_{λ,β_{1}<γ_{1},β_{2}<γ_{2}∈ℝ_{+},i,j∈ℕ} Ξ_{\frac{λ}{N}} \{[ρ_{2}(r) − ρ_{1}(r)]

+ \frac{N}{λ} log[cosh(\frac{λ}{N})]ρ_{1} ⊗ ρ_{2}(m′)

+ \frac{1}{λ(1 − \frac{β_{1}}{γ_{1}} )}\{𝒦[ρ_{1}, π_{exp(−β_{1}r)}^{i}]

+ log\{π_{exp(−β_{1}r)}^{i}[exp\{β_{1} \frac{N}{γ_{1}} log[cosh(\frac{γ_{1}}{N})]ρ_{1}(m′)\}]\}

− \frac{β_{1}}{γ_{1}} log[ν(γ_{1})]\}

+ \frac{1}{λ(1 − \frac{β_{2}}{γ_{2}} )}\{𝒦[ρ_{2}, π_{exp(−β_{2}r)}^{j}]

+ log\{π_{exp(−β_{2}r)}^{j}[exp\{β_{2} \frac{N}{γ_{2}} log[cosh(\frac{γ_{2}}{N})]ρ_{2}(m′)\}]\}

− \frac{β_{2}}{γ_{2}} log[ν(γ_{2})]\}

− [(\frac{γ_{1}}{β_{1}} − 1)^{−1} + (\frac{γ_{2}}{β_{2}} − 1)^{−1} + 1]\frac{log[3^{−1}ν(β_{1})ν(β_{2})ν(λ)μ(i)μ(j)ϵ]}{λ}\}.

The sequence of prior distributions (π^{i})_{i∈ℕ} should be understood to be typically supported by subsets of Θ corresponding to parametric sub-models, that is sub-models for which it is reasonable to expect that

lim_{β→+∞} β[π_{exp(−βR)}^{i}(R) − ess inf_{π^{i}} R]

exists and is positive and finite. As there is no reason why the bound B(ρ_{1}, ρ_{2}) provided by the previous theorem should be sub-additive (in the sense that B(ρ_{1}, ρ_{3}) ≤ B(ρ_{1}, ρ_{2})+B(ρ_{2}, ρ_{3})), it is adequate to consider some workable subset 𝒫 of posterior distributions (for instance the distributions of the form π_{exp(−βr)}^{i}, i ∈ ℕ, β ∈ ℝ_{+}), and to define the sub-additive chained bound

B̃(ρ, ρ′) = inf{∑_{k=0}^{n−1} B(ρ_{k}, ρ_{k+1}); n ∈ ℕ^{∗}, (ρ_{k})_{k=0}^{n} ∈ 𝒫^{n+1},

ρ_{0} = ρ, ρ_{n} = ρ′}, ρ, ρ′ ∈ 𝒫. (2.12)

Proposition 2.2.3. With ℙ probability at least 1 − ϵ, for any posterior distributions ρ_{1}, ρ_{2} ∈ 𝒫, ρ_{2}(R) − ρ_{1}(R) ≤ B̃(ρ_{1}, ρ_{2}). Moreover for any posterior distribution ρ_{1} ∈ 𝒫, any posterior distribution ρ_{2} ∈ 𝒫 such that B̃(ρ_{1}, ρ_{2}) = inf_{ρ_{3}∈𝒫} B̃(ρ_{1}, ρ_{3}) is unimprovable with the help of B̃ in 𝒫 in the sense that inf_{ρ_{3}∈𝒫} B̃(ρ_{2}, ρ_{3}) ≥ 0.

Proof. The first assertion is a direct consequence of the previous theorem, so only the second assertion requires a proof: for any ρ_{3} ∈ 𝒫, we deduce from the optimality of ρ_{2} and the sub-additivity of B̃ that

B̃(ρ_{1}, ρ_{2}) ≤ B̃(ρ_{1}, ρ_{3}) ≤ B̃(ρ_{1}, ρ_{2}) + B̃(ρ_{2}, ρ_{3}).

□ This proposition provides a way to improve a posterior distribution ρ_{1} ∈ 𝒫

by choosing ρ_{2} ∈ arg min_{ρ∈𝒫} B̃(ρ_{1}, ρ) whenever B̃(ρ_{1}, ρ_{2}) < 0. This improvement is proved by Proposition 2.2.3 to be one-step: the obtained improved posterior ρ_{2} cannot be improved again using the same technique.

Let us give some examples of possible starting distributions ρ_{1} for this improvement scheme: ρ_{1} may be chosen as the best posterior Gibbs distribution according to Proposition 2.1.5 (page 75). More precisely, we may build from the prior distributions π^{i}, i ∈ ℕ, a global prior π = ∑_{i∈ℕ} μ(i)π^{i}. We can then define the estimator of the inverse effective temperature as in Proposition 2.1.5 (page 75) and choose ρ_{1} ∈ arg min_{ρ∈𝒫} β̂(ρ), where 𝒫 is as suggested above the set of posterior distributions

𝒫 = {π_{exp(−βr)}^{i}; i ∈ ℕ, β ∈ ℝ_{+}}.

This starting point ρ_{1} should already be pretty good, at least in an asymptotic perspective, the only gain in the rate of convergence to be expected bearing on spurious log(N) factors.

2.2.2. Elaborate uses of relative bounds between posteriors. More elaborate uses of relative bounds are described in the third section of the second chapter of Audibert (2004b), where an algorithm is proposed and analysed, which allows one to use relative bounds between two posterior distributions as a stand-alone estimation tool.

Let us give here some alternative way to address this issue. We will assume for simplicity and without great loss of generality that the working set of posterior distributions 𝒫 is finite (so that among other things any ordering of it has a first element).

It is natural to define the estimated complexity of any given posterior distribution ρ ∈ 𝒫 in our working set as the bound for inf_{i∈ℕ} 𝒦(ρ, π^{i}) used in Theorem 2.2.1 (page 88). This leads to set (given some confidence level 1 − ϵ)

𝒞(ρ) = inf_{β<γ∈ℝ_{+},i∈ℕ} (1 − \frac{β}{γ} )^{−1}\{𝒦[ρ, π_{exp(−βr)}^{i}]

+ log\{π_{exp(−βr)}^{i}[exp\{β \frac{N}{γ} log[cosh(\frac{γ}{N})]ρ(m′)\}]\}

− \frac{β}{γ} log[3^{−1}ν(γ)ν(β)μ(i)ϵ]\}.

Let us moreover call γ(ρ), β(ρ) and i(ρ) the values achieving this infimum, or nearly achieving it, which requires a slight change of the definition of 𝒞(ρ) to take this modification into account. For the sake of simplicity, we can assume without substantial loss of generality that the supports of ν and μ are large but finite, and thus that the minimum is reached.

To understand how this notion of complexity comes into play, it may be interesting to keep in mind that for any posterior distributions ρ and ρ′ we can write the bound in Theorem 2.2.2 (page 89) as

B(ρ, ρ′) = inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [ρ′(r) − ρ(r) + S_{λ}(ρ, ρ′)],  (2.13)

where

S_{λ}(ρ, ρ′) = S_{λ}(ρ′, ρ) ≤ \frac{N}{λ} log[cosh(\frac{λ}{N})]ρ⊗ρ′(m′)+\frac{𝒞(ρ) + 𝒞(ρ′)}{λ}−\frac{log(3^{−1}ϵ)}{λ}

− \frac{log\{ν[β(ρ)]μ[i(ρ)]\}}{λ(1 − \frac{β(ρ′)}{γ(ρ′)})} − \frac{log\{ν[β(ρ′)]μ[i(ρ′)]\}}{λ(1 − \frac{β(ρ)}{γ(ρ)})}

− [(\frac{γ(ρ)}{β(ρ)} − 1)^{−1} + (\frac{γ(ρ′)}{β(ρ′)} − 1)^{−1} + 1]\frac{log[ν(λ)]}{λ}.

(Let us recall that the function Ξ is defined by equation (2.11, page 89).) Thus for any ρ, ρ′ such that B(ρ′, ρ) > 0, we can deduce from the monotonicity of Ξ_{\frac{λ}{N}} that

ρ′(r) − ρ(r) ≤ inf_{λ∈ℝ_{+}} S_{λ}(ρ, ρ′),

proving that the left-hand side is small, and consequently that B(ρ, ρ′) and its chained counterpart defined by equation (2.12, page 90) are small:

B̃(ρ, ρ′) ≤ B(ρ, ρ′) ≤ inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ, ρ′)].

It is also worth noticing that B(ρ, ρ′) and B̃(ρ, ρ′) are upper bounded in terms of variance and complexity only.

The presence of the ratios \frac{γ(ρ)}{β(ρ)} should not be obnoxious, since their values should be automatically tamed by the fact that β(ρ) and γ(ρ) should make the estimate of the complexity of ρ optimal.

As an alternative, it is possible to restrict to set of parameter values β and γ such that, for some fixed constant ζ > 1, the ratio \frac{γ}{β} is bounded away from 1 by the inequality \frac{γ}{β} ≥ ζ. This leads to an alternative definition of 𝒞(ρ):

𝒞(ρ) = inf_{γ≥ζβ∈ℝ_{+},i∈ℕ} (1 − \frac{β}{γ} )^{−1}\{𝒦[ρ, π_{exp(−βr)}^{i}]

+ log\{π_{exp(−βr)}^{i}[exp\{β \frac{N}{γ} log[cosh(\frac{γ}{N})]ρ(m′)\}]\}

− \frac{β}{γ} log[3^{−1}ν(γ)ν(β)μ(i)ϵ]\} − \frac{log[ν(β)μ(i)]}{(1 − ζ^{−1})} − \frac{log(3^{−1}ϵ)}{2}.

We can even push simplification a step further, postponing the optimization of the ratio \frac{γ}{β}, and setting it to the fixed value ζ. This leads us to adopt the definition

𝒞(ρ) = inf_{β∈ℝ_{+},i∈ℕ} (1 − ζ^{−1})^{−1}{𝒦[ρ, π_{exp(−βr)}^{i}]

+ log\{π_{exp(−βr)}^{i}[exp\{\frac{N}{ζ} log[cosh(\frac{ζβ}{N})]ρ(m′)\}]\}\}

− \frac{ζ + 1}{ζ − 1}\{log[ν(β)μ(i)] + 2^{−1} log(3^{−1}ϵ)\}. (2.14)

With either of these modified definitions of the complexity 𝒞(ρ), we get the upper bound

S_{λ}(ρ, ρ′) ≤ S̃_{λ}(ρ, ρ′) =^{def} \frac{N}{λ} log[cosh(\frac{λ}{N})]ρ ⊗ ρ′(m′)

+ \frac{1}{λ}\{𝒞(ρ) + 𝒞(ρ′) − \frac{ζ + 1}{ζ − 1} log[ν(λ)]\}. (2.15)

With these definitions, we have for any posterior distributions ρ and ρ′

B(ρ, ρ′) ≤ inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} \{ρ′(r) − ρ(r) + S̃_{λ}(ρ, ρ′)\}.

Consequently in the case when B(ρ′, ρ) > 0, we get

B̃(ρ, ρ′) ≤ B(ρ, ρ′) ≤ inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S̃_{λ}(ρ, ρ′)].

To select some nearly optimal posterior distribution in 𝒫, it is appropriate to order the posterior distributions of 𝒫 according to increasing values of their complexity 𝒞(ρ) and consider some indexation 𝒫 = {ρ_{1}, …, ρ_{M}}, where 𝒞(ρ_{k}) ≤ 𝒞(ρ_{k+1}), 1 ≤ k < M.

Let us now consider for each ρ_{k} ∈ 𝒫 the first posterior distribution in 𝒫 which cannot be proved to be worse than ρ_{k} according to the bound B̃:

t(k) = min{j ∈ {1, … M} : B̃(ρ_{j}, ρ_{k}) > 0}.  (2.16)

In this definition, which uses the chained bound defined by equation (2.12, page 90), it is appropriate to assume by convention that B̃(ρ, ρ) = 0, for any posterior distribution ρ. Let us now define our estimated best ρ ∈ 𝒫 as ρk̂, where

k̂ = min(arg max t).  (2.17)

Thus we take the posterior with smallest complexity which can be proved to be better than the largest starting interval of 𝒫 in terms of estimated relative classification error.

The following theorem is a simple consequence of the chosen optimisation scheme. It is valid for any arbitrary choice of the complexity function ρ ↦ 𝒞(ρ).

Theorem 2.2.4. Let us put t̂= t(k̂), where t is defined by equation (2.16) and k̂ is defined by equation (2.17). With ℙ probability at least 1 − ϵ,

ρk̂(R) ≤ ρ_{j}(R) + \begin{cases}0, 1 ≤ j < t̂, \\ B̃(ρ_{j}, ρ_{t(j)}), t̂≤ j < k̂, \\ B̃(ρ_{j}, ρt̂) + B̃(ρt̂, ρk̂), j ∈ (arg max t), \\ B̃(ρ_{j}, ρk̂), j ∈ \{k̂ + 1, …, M\} \ (arg max t), \end{cases}

where the chained bound B̃ is defined from the bound of Theorem 2.2.2 (page 89) by equation (2.12, page 90). In the mean time, for any j such that t̂≤ j < k̂, t(j) < t̂= max t, because j ∉ (arg max t). Thus

ρk̂(R) ≤ ρ_{t(j)}(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρ_{t(j)})]
while ρ_{t(j)}(r) ≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρ_{t(j)}),

where the function Ξ is defined by equation (2.11, page 89) and S_{λ} is defined by equation (2.13, page 91). For any j ∈ (arg max t), (including notably k̂),

B(ρt̂, ρ_{j}) ≥ B̃(ρt̂, ρ_{j}) > 0,
B(ρ_{j}, ρt̂) ≥ B̃(ρ_{j}, ρt̂) > 0,

so in this case

ρk̂(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [S_{λ}(ρ_{j}, ρt̂) + S_{λ}(ρt̂, ρk̂) + S_{λ}(ρ_{j}, ρk̂)],
while ρt̂(r) ≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρt̂),

ρk̂(r) ≤ ρt̂(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρt̂, ρk̂),
and ρt̂(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρt̂)].

Finally in the case when j ∈ {k̂ + 1, …, M} \ (arg max t), due to the fact that in particular j ∉ (arg max t),

B(ρk̂, ρ_{j}) ≥ B̃(ρk̂, ρ_{j}) > 0.

Thus in this last case

ρk̂(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρk̂)],
while ρk̂(r) ≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρk̂).

Thus for any j = 1, …, M, ρk̂(R) − ρ_{j}(R) is bounded from above by an empirical quantity involving only variance and entropy terms of posterior distributions ρ_{ℓ} such that ℓ ≤ j, and therefore such that 𝒞(ρ_{ℓ}) ≤ 𝒞(ρ_{j}). Moreover, these distributions ρ_{ℓ} are such that ρ_{ℓ}(r)−ρ_{j}(r) and ρ_{ℓ}(R)−ρ_{j}(R) have an empirical upper bound of the same order as the bound stated for ρk̂(R) − ρ_{j}(R) — namely the bound for ρ_{ℓ}(r) − ρ_{j}(r) is in all circumstances not greater than Ξ−1_{\frac{λ}{N}} applied to the bound stated for ρk̂(R) − ρ_{j}(R), whereas

the bound for ρ_{ℓ}(R)−ρ_{j}(R) is always smaller than two times the bound stated for ρk̂(R) − ρ_{j}(R). This shows that variance terms are between posterior distributions whose empirical as well as expected error rates cannot be much larger than those of ρ_{j}.

Let us remark that the estimation scheme described in this theorem is very general, the same method can be used as soon as some confidence interval for the relative expected risks

−B(ρ_{2}, ρ_{1}) ≤ ρ_{2}(R) − ρ_{1}(R) ≤ B(ρ_{1}, ρ_{2}) with ℙ probability at least 1 − ϵ,

is available. The definition of the complexity is arbitrary, and could in an abstract context be chosen as

𝒞(ρ_{1}) = inf_{ρ_{2}≠ρ_{1}} B(ρ_{1}, ρ_{2}) + B(ρ_{2}, ρ_{1}).

Proof. The case when 1 ≤ j < t̂ is straightforward from the definitions: when j < t̂, B̃(ρ_{j}, ρk̂) ≤ 0 and therefore ρk̂(R) ≤ ρ_{j}(R).

In the second case, that is when t̂ ≤ j < k̂, j cannot be in arg max t, because of the special choice of k̂ in arg max t. Thus t(j) < t̂ and we deduce from the first case that

ρk̂(R) ≤ ρ_{t(j)}(R) ≤ ρ_{j}(R) + B̃(ρ_{j}, ρ_{t(j)}).

Moreover, we see from the defintion of t that B̃(ρ_{t(j)}, ρ_{j}) > 0, implying

ρ_{t(j)}(r) ≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρ_{t(j)}),

and therefore that

ρk̂(R) ≤ ρ_{j}(R) + inf_{λ} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρ_{t(j)})].

In the third case j belongs to arg max t. In this case, we are not sure that B̃(ρk̂, ρ_{j}) > 0, and it is appropriate to involve t̂, which is the index of the first posterior distribution which cannot be improved by ρk̂, implying notably that B̃(ρt̂, ρ_{k}) > 0 for any k ∈ arg max t. On the other hand, ρt̂ cannot either improve any posterior distribution ρ_{k} with k ∈ (arg max t), because this would imply for any ℓ < t̂ that B̃(ρ_{ℓ}, ρt̂) ≤ B̃(ρ_{ℓ}, ρ_{k}) + B̃(ρ_{k}, ρt̂) ≤ 0, and therefore that t(t̂) ≥ t̂+ 1, in contradiction of the fact that t̂ = max t. Thus B̃(ρ_{k}, ρt̂) > 0, and these two remarks imply that

ρt̂(r) ≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρt̂),
ρk̂(r) ≤ ρt̂(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρt̂, ρk̂)

≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρt̂) + inf_{λ∈ℝ_{+}} S_{λ}(ρt̂, ρk̂),

and consequently also that

ρk̂(R) ≤ ρ_{j}(R) + B̃(ρ_{j}, ρk̂)

≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [S_{λ}(ρ_{j}, ρt̂) + S_{λ}(ρt̂, ρk̂) + S_{λ}(ρ_{j}, ρk̂)]

and that

ρt̂(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρt̂)] ≤ ρ_{j}(R) + 2 inf_{λ∈ℝ_{+}} 2Ξ_{\frac{λ}{N}} [S_{λ}(ρ_{j}, ρt̂)],

the last inequality being due to the fact that Ξ_{\frac{λ}{N}} is a concave function. Let us

notice that it may be the case that k̂ < t̂, but that only the case when j ≥ t̂ is to be considered, since otherwise we already know that ρk̂(R) ≤ ρ_{j}(R).

In the fourth case, j is greater than k̂, and the complexity of ρ_{j} is larger than the complexity of ρk̂. Moreover, j is not in arg max t, and thus B̃(ρk̂, ρ_{j}) > 0, because otherwise, the sub-additivity of B̃ would imply that B̃(ρ_{ℓ}, ρ_{j}) ≤ 0 for any ℓ ≤ t̂ and therefore that t(j) ≥ t̂= max t. Therefore

ρk̂(r) ≤ ρ_{j}(r) + inf_{λ∈ℝ_{+}} S_{λ}(ρ_{j}, ρk̂),

and

ρk̂(R) ≤ ρ_{j}(R) + B̃(ρ_{j}, ρk̂) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρk̂)].

□ 2.2.3. Analysis of relative bounds. Let us start our investigation of the theoretical properties of the algorithm described in Theorem 2.2.4 (page 93) by computing some non-random upper bounds for B(ρ, ρ′), the bound of Theorem 2.2.2 (page 89), and 𝒞(ρ), the complexity factor defined by equation (2.14, page 92), for any ρ, ρ′ ∈ 𝒫.

This analysis will be done in the case when

𝒫 = {π_{exp(−βr)}^{i} : ν(β) > 0, μ(i) > 0},

in which it will be possible to get some control on the randomness of any ρ ∈ 𝒫, in addition to controlling the other random expressions appearing in the definition of B(ρ, ρ′), ρ, ρ′ ∈ 𝒫. We will also use a simpler choice of complexity function, removing from equation (2.14 page 92) the optimization in i and β and using instead the definition

𝒞(π_{exp(−βr)}^{i}) =^{def} (1 − ζ^{−1})^{−1} log{π_{exp(−βr)}^{i}[

exp\{\frac{N}{ζ} log[cosh(\frac{ζβ}{N} )]π_{exp(−βr)}^{i}(m′)\}]\}

+ \frac{ζ + 1}{ζ − 1} log[ν(β)μ(i)]. (2.18)

With this definition,

S_{λ}(π_{exp(−βr)}^{i}, π_{exp(−β′r)}^{j}) ≤ \frac{N}{λ} log[cosh(\frac{λ}{N})]π_{exp(−βr)}^{i} ⊗ π_{exp(−β′r)}^{j}(m′)

+ \frac{𝒞[π_{exp(−βr)}^{i}] + 𝒞[π_{exp(−β′r)}^{j}]}{λ}

+ \frac{(ζ + 1)}{(ζ − 1)λ} log[3^{−1}ν(λ)ϵ],

where S_{λ} is defined by equation (2.13, page 91), so that

B[π_{exp(−βr)}^{i}, π_{exp(−β′r)}^{j}] = inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} \{π_{exp(−β′r)}^{j}(r) − π_{exp(−βr)}^{i}(r)

+ S_{λ}[π_{exp(−βr)}^{i}, π_{exp(−βr)}^{j}]}.

Let us successively bound the various random factors entering into the definition of B[π_{exp(−βr)}^{i}, π_{exp(−β′r)}^{j}]. The quantity π_{exp(−β′r)}^{j}(r) − π_{exp(−βr)}^{i}(r) can be bounded using a slight adaptation of Proposition 2.1.11 (page 77).

Proposition 2.2.5. For any positive real constants λ, λ′ and γ, with ℙ probability at least 1 − η, for any positive real constants β, β′ such that β < λ\frac{γ}{N} sinh(\frac{γ}{N})^{−1} and β′ > λ′ \frac{γ}{N} sinh(\frac{γ}{N})^{−1}, π_{exp(−β′r)}^{j}(r) − π_{exp(−βr)}^{i}(r)

≤ π_{exp(−λ′R)}^{j} ⊗ π_{exp(−λR)}^{i}[Ψ_{− \frac{γ}{N}} (R′, M′)]

+ \frac{log(\frac{3}{η})}{γ} + \frac{C^{j}(λ′, γ) + log(\frac{3}{η})}{\frac{Nβ′}{λ′} sinh(\frac{γ}{N}) − γ} + \frac{C^{i}(λ, γ) + log(\frac{3}{η})}{γ − \frac{Nβ}{λ} sinh(\frac{γ}{N})},

where

C^{i}(λ, γ) =^{def} log\{∫_{Θ} exp[−γ ∫_{Θ}\{Ψ_{\frac{γ}{N}} [R′(θ_{1}, θ_{2}), M′(θ_{1}, θ_{2})]

− \frac{N}{γ} sinh(\frac{γ}{N})R′(θ_{1}, θ_{2})\}π_{exp(−λR)}^{i}(dθ_{2})]π_{exp(−λR)}^{i}(dθ_{1})\}

≤ log\{π_{exp(−λR)}^{i}[exp\{2N sinh(\frac{γ}{2N} )^{2}π_{exp(−λR)}^{i}(M′)\}]\}.

As for π_{exp(−βr)}^{i} ⊗ π_{exp(−β′r)}^{j}(m′), we can write with ℙ probability at least 1 − η, for any posterior distributions ρ and ρ′ : Ω → ℳ_{+}^{1}(Θ),

γρ ⊗ ρ′(m′) ≤ log[π_{exp(−λR)}^{i} ⊗ π_{exp(−λ′R)}^{j}\{exp[γΦ_{− \frac{γ}{N}} (M′)]\}]

+ 𝒦[ρ, π_{exp(−λR)}^{i}] + 𝒦[ρ′, π_{exp(−λ′R)}^{j}] − log(η).

We can then replace λ with β \frac{N}{λ} sinh(\frac{λ}{N}) and use Theorem 2.1.12 (page 79) to get

Proposition 2.2.6. For any positive real constants γ, λ, λ′, β and β′, with ℙ probability 1 − η,

γρ ⊗ ρ′(m′)

≤ log[π_{exp[−β \frac{N}{λ} sinh( \frac{λ}{N})R]}^{i} ⊗ π_{exp[−β′ \frac{N}{λ′} sinh( \frac{λ′}{N})R]}^{j}\{exp[γΦ_{− \frac{γ}{N}} (M′)]\}]

+ \frac{𝒦[ρ, π_{exp(−βr)}^{i}]}{1 − \frac{β}{λ}} + \frac{C^{i}[β \frac{N}{λ} sinh(\frac{λ}{N}), λ] − log(\frac{η}{3})}{\frac{λ}{β} − 1}

+ \frac{𝒦[ρ′, π_{exp(−β′r)}^{j}]}{1 − \frac{β′}{λ′}} + \frac{C^{j}[β′ \frac{N}{λ′} sinh(\frac{λ′}{N}), λ′] − log(\frac{η}{3})}{\frac{λ}{β′} − 1} − log(\frac{η}{3}).

The last random factor in B(ρ, ρ′) that we need to upper bound is

log\{π_{exp(−βr)}^{i}[exp\{β \frac{N}{γ} log[cosh(\frac{γ}{N})]π_{exp(−βr)}^{i}(m′)\}]\}.

A slight adaptation of Proposition 2.1.13 (page 79) shows that with ℙ probability at least 1 − η, log\{π_{exp(−βr)}^{i}[exp\{β \frac{N}{γ} log[cosh(\frac{γ}{N})]π_{exp(−βr)}^{i}(m′)\}]\}

≤ \frac{2β}{γ} C^{i}[\frac{Nβ}{γ} sinh(\frac{γ}{N}), γ] + (1 − \frac{β}{γ} ) log\{(π_{exp[− \frac{Nβ}{γ} sinh( \frac{γ}{N})R]}^{i})^{⊗2}[

exp(\frac{N log[cosh(\frac{γ}{N})]}{\frac{γ}{β} − 1}Φ_{− \frac{log[cosh( \frac{γ}{N} )]}{\frac{γ}{β} −1}} ∘ M′)]\}

+ (1 + \frac{β}{γ} ) log(\frac{2}{η}),

where as usual Φ is the function defined by equation (1.1, page 15). This leads us to define for any i, j ∈ ℕ, any β, β′ ∈ ℝ_{+},

\overline{𝒞}(i, β) =^{def} \frac{2}{ζ − 1}C^{i}[\frac{N}{ζ} sinh(\frac{ζβ}{N}), ζβ]

+ log\{(π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i})^{⊗2}[

exp(\frac{N log[cosh(\frac{ζβ}{N})]}{ζ − 1}Φ_{−^{\frac{log[cosh( \frac{ζβ}{N} )]}{ζ−1}}} ∘ M′)]\}

− \frac{ζ + 1}{ζ − 1}\{2 log[ν(β)μ(i)] + log(\frac{η}{2})\}. (2.19)

Recall that the definition of C^{i}(λ, γ) is to be found in Proposition 2.2.5, page 96. Let us remark that, since

exp[NaΦ_{−a}(p)] = exp{N log[1 + [exp(a) − 1]p]}

≤ exp{N[exp(a) − 1]p}, p ∈ (0, 1), a ∈ ℝ,

we have

\overline{𝒞}(i, β) ≤ \frac{2}{ζ − 1} log\{π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i}[

exp\{2N sinh(\frac{ζβ}{2N} )^{2}π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i}(M′)\}]\}
+ log\{(π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i})^{⊗2}[
exp\{N[exp\{(ζ − 1)^{−1} log[cosh(\frac{ζβ}{N} )]\} − 1]M′\}]\}

− \frac{ζ + 1}{ζ − 1}\{2 log[ν(β)μ(i)] + log(\frac{η}{2})\}.

Let us put

\overline{S}_{λ}[(i, β), (j, β′)] =^{def} \frac{N}{λ} log[cosh(\frac{λ}{N})] inf_{γ∈ℝ_{+}} γ^{−1}\{

2.2. Playing with two posterior and two local prior distributions  99

log[(π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i} ⊗ π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ′}{N})R]}^{j})\{exp[γΦ_{− \frac{γ}{N}} (M′)]\}]

+ \frac{C^{i}[\frac{N}{ζ} sinh(\frac{ζβ}{N}), ζβ] − log(\frac{\overline{η}}{3})}{ζ − 1}

+ \frac{C^{j}[\frac{N}{ζ} sinh(\frac{ζβ′}{N}), ζβ′] − log(\frac{\overline{η}}{3})}{ζ − 1} − log(\frac{\overline{η}}{3})\}

+ \frac{1}{λ}[\overline{𝒞}(i, β) + \overline{𝒞}(j, β′) − \frac{ζ + 1}{ζ − 1} log[3^{−1}ν(λ)ϵ]],

where

\overline{η} = ν(γ)ν(β)ν(β′)μ(i)μ(j)η.

Let us remark that

\overline{S}_{λ}[(i, β), (j, β′)] ≤ inf_{γ∈ℝ_{+}}

\frac{λ}{2Nγ} log[(π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i} ⊗ π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ′}{N})R]}^{j})\{

exp[N[exp(\frac{γ}{N} ) − 1]M′]\}]

+ (\frac{λ}{2Nγ(ζ − 1)} + \frac{2}{λ(ζ − 1)}) log\{π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i}[

exp\{2N sinh(\frac{ζβ}{2N} )^{2}π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i}(M′)\}]\}

+ λ^{−1} log\{(π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ}{N})R]}^{i})^{⊗2}[

exp\{N[exp\{(ζ − 1)^{−1} log[cosh(\frac{ζβ}{N} )]\} − 1]M′\}]\}

+ (\frac{λ}{2Nγ(ζ − 1)} + \frac{2}{λ(ζ − 1)}) log\{π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ′}{N})R]}^{j}[

exp\{2N sinh(\frac{ζβ′}{2N} )^{2}π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ′}{N})R]}^{j}(M′)\}]\}

+ λ^{−1} log\{(π_{exp[− \frac{N}{ζ} sinh( \frac{ζβ′}{N})R]}^{j})^{⊗2}[

exp\{N[exp\{(ζ − 1)^{−1} log[cosh(\frac{ζβ′}{N} )]\} − 1]M′\}]\}

− \frac{(ζ + 1)λ}{2N(ζ − 1)γ} log[3^{−1}ν(γ)ν(β)ν(β′)μ(i)μ(j)η]

− \frac{(ζ + 1)}{(ζ − 1)λ}(2 log[2^{−1}ν(β)ν(β′)μ(i)μ(j)η] + log[3^{−1}ν(λ)ϵ]).

Let us define accordingly

\overline{B}[(i, β), (j, β′)] =^{def}

inf_{λ} Ξ_{\frac{λ}{N}} \{ inf_{α,γ,α′,γ′}[π_{exp(−α′R)}^{j} ⊗ π_{exp(−αR)}^{i}[Ψ_{− \frac{λ}{N}} (R′, M′)]

− \frac{log(\frac{η̃}{3})}{λ} + \frac{C^{j}(α′, γ′) − log(\frac{η̃}{3})}{\frac{Nβ′}{α′} sinh(\frac{γ′}{N}) − γ′} + \frac{C^{i}(α, γ) − log(\frac{η̃}{3})}{γ − \frac{Nβ}{α} sinh(\frac{γ}{N})}]

+ \overline{S}_{λ}[(i, β), (j, β′)]\},

where

η̃ = ν(λ)ν(α)ν(γ)ν(β)ν(α′)ν(γ′)ν(β′)μ(i)μ(j)η.
Proposition 2.2.7.

• With ℙ probability at least 1 − η, for any β ∈ ℝ_{+} and i ∈ ℕ, 𝒞(π_{exp(−βr)}^{i}) ≤ \overline{𝒞}(i, β);

• With ℙ probability at least 1 − 3η, for any λ, β, β′ ∈ ℝ_{+}, any i, j ∈ ℕ, S_{λ}[(i, β), (j, β′)] ≤ \overline{S}_{λ}[(i, β), (j, β′)];

• With ℙ probability at least 1 − 4η, for any i, j ∈ ℕ, any β, β′ ∈ ℝ_{+}, B(π_{exp(−βr)}^{i}, π_{exp(−β′r)}^{j}) ≤ \overline{B}[(i, β), (j, β′)].

It is also interesting to find a non-random lower bound for 𝒞(π_{exp(−βr)}^{i}). Let us start from the fact that with ℙ probability at least 1 − η,

π_{exp(−αR)}^{i} ⊗ π_{exp(−αR)}^{i}[Φ_{\frac{γ′}{N}} (M′)] ≤ π_{exp(−αR)}^{i} ⊗ π_{exp(−αR)}^{i}(m′) − \frac{log(η)}{γ′}.

On the other hand, we already proved that with ℙ probability at least 1−η,

0 ≤ (1 − \frac{α}{N tanh(\frac{λ}{N})})𝒦[ρ, π_{exp(−αR)}^{i}]

≤ \frac{α}{N tanh(\frac{λ}{N})}\{λ[ρ(r) − π_{exp(αR)}^{i}(r)]

+ N log[cosh(\frac{λ}{N})]ρ ⊗ π_{exp(−αR)}^{i}(m′) − log(η)\}

+ 𝒦(ρ, π^{i}) − 𝒦(π_{exp(−αR)}^{i}, π^{i}).

Thus for any ξ > 0, putting β = _{\frac{αλ}{N tanh( \frac{λ}{N})}}, with ℙ probability at least 1 − η,

ξπ_{exp(−αR)}^{i} ⊗ π_{exp(−αR)}^{i}[Φ_{\frac{γ′}{N}} (M′)]

≤ π_{exp(−αR)}^{i}{log[π_{exp(−βr)}^{i}{

exp[β \frac{N}{λ} log[cosh(\frac{λ}{N})]π_{exp(−βr)}^{i}(m′) + ξm′]\}]\}

2.2. Playing with two posterior and two local prior distributions  101

− (\frac{β}{λ} + \frac{ξ}{γ′} ) log(\frac{η}{2})

≤ log\{π_{exp(−βr)}^{i}[exp\{β \frac{N}{λ} log[cosh(\frac{λ}{N})]π_{exp(−βr)}^{i}(m′)\}

× π_{exp(−βr)}^{i}\{exp[β \frac{N}{λ} log[cosh(\frac{λ}{N})]π_{exp(−βr)}^{i}(m′) + ξm′]\}]\}

− (2\frac{β}{λ} + \frac{ξ}{γ′} ) log(\frac{η}{2})

≤ 2 log\{π_{exp(−βr)}^{i}[exp\{[ξ + β \frac{N}{λ} log[cosh(\frac{λ}{N})]]π_{exp(−βr)}^{i}(m′)\}]\}

− (2\frac{β}{λ} + \frac{ξ}{γ′} ) log(\frac{η}{2})

≤ 2 log\{π_{exp(−βr)}^{i}[exp\{[ξ + \frac{βλ}{2N} ]π_{exp(−βr)}^{i}(m′)\}]\}

− (\frac{2β}{λ} + \frac{ξ}{γ′} ) log(\frac{η}{2}).

Taking ξ = \frac{βλ}{2N}, we get with ℙ probability at least 1 − η

\frac{βλ}{4N} (π_{exp[−β \frac{N}{λ} tanh( \frac{λ}{N})R]}^{i})^{⊗2}[Φ_{\frac{γ′}{N}} (M′)]

≤ log\{π_{exp(−βr)}^{i}[exp\{\frac{βλ}{N}π_{exp(−βr)}^{i}(m′)\}]\}

− (\frac{2β}{λ} + \frac{βλ}{2Nγ′} ) log(\frac{η}{2}).

Putting

λ = \frac{N^{2}}{γ} log[cosh(\frac{γ}{N})]
and Υ(γ) =^{def} \frac{γ tanh\{\frac{N}{γ} log[cosh(\frac{γ}{N})]\}}{N log[cosh(\frac{γ}{N})]} ∼_{γ→0} 1,

this can be rewritten as

\frac{βN}{4γ} log[cosh(\frac{γ}{N})](π_{exp(−βΥ(γ)R)}^{i})^{⊗2}[Φ_{\frac{γ′}{N}} (M′)]

≤ log\{π_{exp(−βr)}^{i}[exp\{β \frac{N}{γ} log[cosh(\frac{γ}{N})]π_{exp(−βr)}^{i}(m′)\}]\}

− (\frac{2βγ}{N^{2} log[cosh(\frac{γ}{N})]} + \frac{βN log[cosh(\frac{γ}{N})]}{2γγ′}) log(\frac{η}{2}).

It is now tempting to simplify the picture a little bit by setting γ′ = γ, leading to Proposition 2.2.8. With ℙ probability at least 1 − η, for any i ∈ ℕ, any β ∈ ℝ_{+},

𝒞[π_{exp(−βr)}^{i}] ≥ \underline{𝒞}(i, β)

=^{def} \frac{1}{ζ − 1}\{\frac{N}{4} log[cosh(\frac{ζβ}{N})](π_{exp(−βΥ(ζβ)R)}^{i})^{⊗2}[Φ_{\frac{ζβ}{N}} (M′)]

+ (\frac{2ζ^{2}β^{2}}{N^{2} log[cosh(\frac{ζβ}{N})]} + \frac{N log[cosh(\frac{ζβ}{N})]}{2ζβ}) log[2^{−1}ν(β)μ(i)η]

− (ζ + 1){log[ν(β)μ(i)] + 2^{−1} log(3^{−1}ϵ)}},

where 𝒞[π_{exp(−βr)}^{i}] is defined by equation (2.18, page 96).

We are now going to analyse Theorem 2.2.4 (page 93). For this, we will also need an upper bound for S_{λ}(ρ, ρ′), defined by equation (2.13, page 91), using M′ and empirical complexities, because of the special relations between empirical complexities induced by the selection algorithm. To this purpose, a useful alternative to Proposition 2.2.6 (page 97) is to write, with ℙ probability at least 1 − η,

γρ ⊗ ρ′(m′) ≤ γρ ⊗ ρ′[Φ_{− \frac{γ}{N}} (M′)]

+ 𝒦[ρ, π_{exp(−λR)}^{i}] + 𝒦[ρ′, π_{exp(−λ′R)}^{j}] − log(η),

and thus at least with ℙ probability 1 − 3η,

γρ ⊗ ρ′(m′) ≤ γρ ⊗ ρ′[Φ_{− \frac{γ}{N}} (M′)]

+ (1 − ζ^{−1})^{−1}{𝒦[ρ, π_{exp(−βr)}^{i}]
+ log\{π_{exp(−βr)}^{i}[exp\{\frac{N}{ζ} log[cosh(\frac{ζβ}{N} )]ρ(m′)\}]\} − ζ^{−1} log(η)\}
+ (1 − ζ^{−1})^{−1}{𝒦[ρ, π_{exp(−β′r)}^{j}]
+ log\{π_{exp(−β′r)}^{j}[exp\{\frac{N}{ζ} log[cosh(\frac{ζβ′}{N} )]ρ(m′)\}]\} − ζ^{−1} log(η)\}
− log(η).

When ρ = π_{exp(−βr)}^{i} and ρ′ = π_{exp(−β′r)}^{j}, we get with ℙ probability at least 1 − η, for any β, β′, γ ∈ ℝ_{+}, any i, j ∈ ℕ,

γρ ⊗ ρ′(m′) ≤ γρ ⊗ ρ′[Φ_{− \frac{γ}{N}} [(M′)]

+ 𝒞(ρ) + 𝒞(ρ′) − \frac{ζ + 1}{ζ − 1}[log[3^{−1}ν(γ)η]].

Proposition 2.2.9. With ℙ probability at least 1−η, for any ρ = π_{exp(−βr)}^{i}, any ρ′ = π_{exp(−β′r)}^{j} ∈ 𝒫,

S_{λ}(ρ, ρ′) ≤ \frac{N}{λ} log[cosh(\frac{λ}{N})]ρ ⊗ ρ′[Φ_{− \frac{γ}{N}} (M′)]

+ \frac{1 + \frac{N}{γ} log[cosh(\frac{λ}{N})]}{λ}[𝒞(ρ) + 𝒞(ρ′)]

− \frac{(ζ + 1)}{(ζ − 1)λ}\{log[3^{−1}ν(λ)ϵ] + \frac{N}{γ} log[cosh(\frac{λ}{N} )] log[3^{−1}ν(γ)η]\}.

In order to analyse Theorem 2.2.4 (page 93), we need to index 𝒫 = {ρ_{1}, …, ρ_{M}} in order of increasing empirical complexity 𝒞(ρ). To deal in a convenient way with this indexation, we will write \overline{𝒞}(i, β) as \overline{𝒞}[π_{exp(−βr)}^{i}], \underline{𝒞}(i, β) as \underline{𝒞}[π_{exp(−βr)}^{i}], and \overline{S}[(i, β), (j, β′)] as \overline{S}[π_{exp(−βr)}^{i}, π_{exp(−β′r)}^{j}].

With ℙ probability at least 1 − ϵ, when t̂≤ j < k̂, as we already saw,

ρk̂(R) ≤ ρ_{i}(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} [2S_{λ}(ρ_{j}, ρ_{i})],

where i = t(j) < t̂. Therefore, with ℙ probability at least 1 − ϵ − η,

ρ_{i}(R) ≤ ρ_{j}(R) + inf_{λ∈ℝ_{+}} Ξ_{\frac{λ}{N}} \{2\frac{N}{λ} log[cosh(\frac{λ}{N} )]ρ_{j} ⊗ ρ_{i}[Φ_{− \frac{γ}{N}} (M′)]

+ 4\frac{1 + \frac{N}{γ} log[cosh(\frac{λ}{N} )]}{λ}𝒞(ρ_{j})

− \frac{(ζ + 1)}{(ζ − 1)λ}\{log[3^{−1}ν(λ)ϵ] + \frac{N}{γ} log[cosh(\frac{λ}{N} )] log[3^{−1}ν(γ)η]\}\}.

We can now remark that

Ξ_{a}(p + q) ≤ Ξ_{a}(p) + qΞ′_{a}(p)q ≤ Ξ_{a}(p) + Ξ′_{a}(0)q = Ξ_{a}(p) + \frac{a}{tanh(a)}q

and that

Φ_{−a}(p + q) ≤ Φ_{−a}(p) + Φ′_{−a}(0)q = Φ_{−a}(p) + \frac{exp(a) − 1}{a}q.

Moreover, assuming as usual without substantial loss of generality that there exists θ̃ ∈ arg min_{Θ} R, we can split M′(θ, θ′) ≤ M′(θ, θ̃) + M′(θ̃, θ′). Let us then consider the expected margin function defined by

φ(y) = sup_{θ∈Θ} M′(θ, θ̃) − yR′(θ, θ̃), y ∈ ℝ_{+},

and let us write for any y ∈ ℝ_{+}, ρ_{j} ⊗ ρ_{i}[Φ_{− \frac{λ}{N}} (M′)] ≤ ρ_{j} ⊗ ρ_{i}\{Φ_{− \frac{γ}{N}} [M′(., θ̃) + yR′(., θ̃) + φ(y)]\}

≤ ρ_{j}\{Φ_{− \frac{λ}{N}} [M′(., θ̃) + φ(y)]\} + \frac{Ny[exp(\frac{γ}{N}) − 1]}{γ}[ρ_{i}(R) − R(θ̃)]

and

(1 − \frac{2yN[exp(\frac{γ}{N}) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )})[ρ_{i}(R) − R(θ̃)]

≤ [ρ_{j}(R) − R(θ̃)] + Ξ_{\frac{λ}{N}} \{\frac{2N}{λ} log[cosh(\frac{λ}{N} )]ρ_{j}\{Φ_{− \frac{γ}{N}} [M′(., θ̃) + φ(y)]\}

+ 4\frac{1 + \frac{N}{γ} log[cosh(\frac{λ}{N} )]}{λ}𝒞(ρ_{j})

− \frac{2(ζ + 1)}{(ζ − 1)λ}\{log[3^{−1}ν(λ)ϵ] + \frac{N}{γ} log[cosh(\frac{λ}{N} )] log[3^{−1}ν(γ)η]\}\}.

With ℙ probability at least 1 − ϵ − η, for any λ, γ, x, y ∈ ℝ_{+}, any j ∈ {t̂, …, k̂ − 1},

ρk̂(R) − R(θ̃) ≤ ρ_{i}(R) − R(θ̃)

≤ (1 − \frac{2yN[exp(\frac{γ}{N}) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )})^{−1}\{

(1 + \frac{2xN[exp(\frac{γ}{N}) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )})[ρ_{j}(R) − R(θ̃)]

+ Ξ_{\frac{λ}{N}} \{\frac{2N}{λ} log[cosh(\frac{λ}{N} )]Φ_{− \frac{γ}{N}} [φ(x) + φ(y)]

+ 4\frac{1 + \frac{N}{γ} log[cosh(\frac{λ}{N} )]}{λ}\overline{𝒞}(ρ_{j})

− \frac{2(ζ + 1)}{(ζ − 1)λ}\{log[3^{−1}ν(λ)ϵ] + \frac{N}{γ} log[cosh(\frac{λ}{N} )] log[3^{−1}ν(γ)η]\}\}\}.

Now we have to get an upper bound for ρ_{j}(R). We can write ρ_{j} = π_{exp(−β′r)}^{ℓ}, as we assumed that all the posterior distributions in 𝒫 are of this special form. Moreover, we already know from Theorem 2.1.8 (page 77) that with ℙ probability at least 1 − η,

[N sinh(\frac{β′}{N} ) − β′ζ^{−1}][π_{exp(−β′r)}^{ℓ}(R) − π_{exp(−β′ζ^{−1}R)}^{ℓ}(R)]

≤ C^{ℓ}(β′ζ^{−1}, β′) − log[ν(β′)μ(ℓ)η].

This proves that with ℙ probability at least 1 − ϵ − 2η, ρk̂(R) ≤ R(θ̃)

+ (1 − \frac{2yN[exp(\frac{γ}{N} ) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )})^{−1}\{

(1 + \frac{2xN[exp(\frac{γ}{N} ) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )})

× (π_{exp(−ζ^{−1}β′R)}^{ℓ}(R) − R(θ̃) + \frac{C^{ℓ}(ζ^{−1}β′, β′) − log[ν(β′)μ(ℓ)η]}{N sinh(\frac{β′}{N}) − ζ^{−1}β′})

+ Ξ_{\frac{λ}{N}} \{\frac{2N}{λ} log[cosh(\frac{λ}{N} )]Φ_{− \frac{γ}{N}} [φ(x) + φ(y)]

+ 4\frac{1 + \frac{N}{γ} log[cosh(\frac{λ}{N} )]}{λ}\overline{𝒞}(ℓ, β′)

− \frac{2(ζ + 1)}{(ζ − 1)λ}\{log[3^{−1}ν(λ)ϵ] + \frac{N}{γ} log[cosh(\frac{λ}{N} )] log[3^{−1}ν(γ)η]\}\}\}.

The case when j ∈ {k̂ + 1, …, M} \ (arg max t) is dealt with exactly in the same way, with i = t(j) replaced directly with k̂ itself, leading to the same inequality.

The case when j ∈ (arg max t) is dealt with bounding first ρk̂(R) − R(θ̃) in terms of ρt̂(R) − R(θ̃), and this latter in terms of ρ_{j}(R) − R(θ̃). Let us put

A(λ, γ) = (1 − \frac{2xN[exp(\frac{γ}{N} ) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )}),

B(λ, γ) = 1 + \frac{2yN[exp(\frac{γ}{N} ) − 1] log[cosh(\frac{λ}{N} )]}{γ tanh(\frac{λ}{N} )},
D(λ, γ, ρ_{j}) = Ξ_{\frac{λ}{N}} \{\frac{2N}{λ} log[cosh(\frac{λ}{N} )]Φ_{− \frac{γ}{N}} [φ(x) + φ(y)] +4\frac{1 + \frac{N}{γ} log[cosh(\frac{λ}{N} )]}{λ}\overline{𝒞}(ρ_{j}) −\frac{2(ζ + 1)}{(ζ − 1)λ}\{log[3^{−1}ν(λ)ϵ] +\frac{N}{γ} log[cosh(\frac{λ}{N} )] log[3^{−1}ν(γ)η]\}\}, (2.20)

where \overline{𝒞}(ρ_{j}) = \overline{𝒞}(ℓ, β′) is defined, when ρ_{j} = π_{exp(−β′r)}^{ℓ}, by equation (2.19, page 98). We obtain, still with ℙ probability 1 − ϵ − 2η,

ρk̂(R) − R(θ̃) ≤ \frac{B(λ, γ)}{A(λ, γ)}[ρt̂(R) − R(θ̃)] + \frac{D(λ, γ, ρ_{j})}{A(λ, γ)},
ρt̂(R) − R(θ̃) ≤ \frac{B(λ, γ)}{A(λ, γ)}[ρ_{j}(R) − R(θ̃)] + \frac{D(λ, γ, ρ_{j})}{A(λ, γ)}.

The use of the factor D(λ, γ, ρ_{j}) in the first of these two inequalities, instead of D(λ, γ, ρt̂), is justified by the fact that 𝒞(ρt̂) ≤ 𝒞(ρ_{j}). Combining the two we get

ρk̂(R) ≤ R(θ̃) + \frac{B(λ, γ)^{2}}{A(λ, γ)^{2}} [ρ_{j}(R) − R(θ̃)] + [\frac{B(λ, γ)}{A(λ, γ)} + 1]\frac{D(λ, γ, ρ_{j})}{A(λ, γ)}.

Since it is the worst bound of all cases, it holds for any value of j, proving

Theorem 2.2.10. With ℙ probability at least 1 − ϵ − 2η,

ρk̂(R) ≤ R(θ̃) + inf_{i,β,λ,γ,x,y}{

\frac{B(λ, γ)^{2}}{A(λ, γ)^{2}} [π_{exp(−βr)}^{i}(R) − R(θ̃)] + [\frac{B(λ, γ)}{A(λ, γ)} + 1]\frac{D(λ, γ, π_{exp(−βr)}^{i})}{A(λ, γ)}\}

≤ R(θ̃) + inf_{i,β,λ,γ,x,y}{

\frac{B(λ, γ)^{2}}{A(λ, γ)^{2}} (π_{exp(−ζ^{−1}βR)}^{i}(R) − R(θ̃) + \frac{C^{i}(ζ^{−1}β, β) − log[ν(β)μ(i)η]}{N sinh(\frac{β}{N} ) − ζ^{−1}β})

+ [\frac{B(λ, γ)}{A(λ, γ)} + 1]\frac{D(λ, γ, π_{exp(−βr)}^{i})}{A(λ, γ)}\},

where the notation A(λ, γ), B(λ, γ) and D(λ, γ, ρ) is defined by equation (2.20 page 105) and where the notation C^{i}(β, γ) is defined in Proposition 2.2.5 (page 96).

The bound is a little involved, but as we will prove next, it gives the same rate as Theorem 2.1.15 (page 85) and its corollaries, when we work with a single model (meaning that the support of μ is reduced to one point) and the goal is to choose adaptively the temperature of the Gibbs posterior, except for the appearance of the union bound factor − log[ν(β)] which can be made of order log[log(N)] without spoiling the order of magnitude of the bound.

We will encompass the case when one must choose between possibly several parametric models. Let us assume that each π^{i} is supported by some measurable parameter subset Θ_{i} ( meaning that π^{i}(Θ_{i}) = 1), let us also assume that the behaviour of π^{i} is parametric in the sense that there exists a dimension d_{i} ∈ ℝ_{+} such that

sup_{β∈ℝ_{+}} β[π_{exp(−βR)}^{i}(R) − inf_{Θ_{i}} R] ≤ d_{i}.  (2.21)

Then

C^{i}(λ, γ) ≤ log\{π_{exp(−λR)}^{i}[exp\{2N sinh(\frac{γ}{2N} )^{2}M′(., θ̃)\}]\}

2.2. Playing with two posterior and two local prior distributions  107
+ 2N sinh(\frac{γ}{2N} )^{2}π_{exp(−λR)}^{i}[M′(., θ̃)]
≤ log\{π_{exp(−λR)}^{i}[exp 2xN sinh(\frac{γ}{2N} )^{2}[R − R(θ̃)]\}]\}
+ 2xN sinh(\frac{γ}{2N} )^{2}π_{exp(−λR)}^{i}[R − R(θ̃)]
+ 4N sinh(\frac{γ}{2N} )^{2}φ(x)
≤ 2xN sinh(\frac{γ}{2N} )^{2}π_{exp\{−[λ−2xN sinh( \frac{γ}{2N})^{2}]R\}}^{i}[R − R(θ̃)]
+ 2xN sinh(\frac{γ}{2N} )^{2}π_{exp(−λR)}^{i}[R − R(θ̃)]
+ 4N sinh(\frac{γ}{2N} )^{2}φ(x).

Thus

C^{i}(λ, γ) ≤ 4N sinh(\frac{γ}{2N} )^{2}(x[inf_{Θ_{i}} R − R(θ̃)] + φ(x)

+ \frac{xd_{i}}{2λ} + \frac{xd_{i}}{2λ − 4xN sinh(\frac{γ}{2N} )^{2}} ).

In the same way,

\overline{𝒞}(i, β) ≤ \frac{8N}{ζ−1} sinh(\frac{ζβ}{2N} )^{2}[x[inf_{Θ_{i}} R − R(θ̃)] + φ(x)

+ \frac{ζxd_{i}}{2N sinh(\frac{ζβ}{N} )}(1 + \frac{1}{1 − xζ tanh(\frac{ζβ}{2N} )})]

+ 2N[exp(\frac{ζ^{2}β^{2}}{2N^{2}(ζ−1)}) − 1](φ(x) + x[inf_{Θ_{i}} R − R(θ̃)]

+ \frac{xζd_{i}}{N sinh(\frac{ζβ}{N} ) − xζN[exp(\frac{ζ^{2}β^{2}}{2N^{2}(ζ−1)}) − 1]})

− \frac{(ζ + 1)}{(ζ − 1)}[2 log[ν(β)μ(i)] + log(\frac{η}{2})].

In order to keep the right order of magnitude while simplifying the bound, let us consider

C_{1} = max\{ζ − 1, (\frac{2N}{ζβ_{max}} )^{2} sinh(\frac{ζβ_{max}}{2N} )^{2},

\frac{2N^{2}(ζ−1)}{ζ^{2}β_{max}^{2}}[exp(\frac{ζ^{2}β_{max}^{2}}{2N^{2}(ζ−1)}) − 1]\}. (2.22)

Then, for any β ∈ (0, β_{max}),

\overline{𝒞}(i, β) ≤ inf_{y∈ℝ_{+}} \frac{3C_{1}ζ^{2}β^{2}}{(ζ − 1)N} [y[inf_{Θ_{i}} R − R(θ̃)] + φ(y) + \frac{yd_{i}}{β[1 − \frac{yC_{1}ζ^{2}β}{2(ζ−1)N} ]}]

− \frac{(ζ + 1)}{(ζ − 1)}[2 log[ν(β)μ(i)] + log(\frac{η}{2})].

Thus

D[λ, γ, π_{exp(−βr)}^{i}] ≤ \frac{λ}{N tanh(\frac{λ}{N} )}\{\frac{λ[exp(\frac{γ}{N} ) − 1]}{γ}[φ(x) + φ(y)]

+ 4\frac{1 + \frac{λ^{2}}{2Nγ}}{λ} [\frac{3C_{1}ζ^{2}β^{2}}{(ζ − 1)N} (z[inf_{Θ_{i}} R − R(θ̃)] + φ(z) + \frac{zd_{i}}{β[1 − \frac{zC_{1}ζ^{2}β}{2(ζ−1)N} ]})

− \frac{(ζ + 1)}{(ζ − 1)}[2 log[ν(β)μ(i)] + log(\frac{η}{2})]]

− \frac{2(ζ + 1)}{(ζ − 1)λ}[log[3^{−1}ν(λ)ϵ] + \frac{λ^{2}}{2Nγ} log[3^{−1}ν(γ)η]]\}

If we are not seeking tight constants, we can take for the sake of simplicity λ = γ = β, x = y and ζ = 2.

Let us put

C_{2} = max\{C_{1}, \frac{N[exp(\frac{β_{max}}{N} ) − 1]}{β_{max}},

\frac{2N log[cosh(\frac{β_{max}}{N} )]}{β_{max} tanh(\frac{β_{max}}{N} )}, \frac{β_{max}}{N tanh(\frac{β_{max}}{N} )}\}, (2.23)

so that

A(β, β)^{−1} ≤ (1 − \frac{C_{2}xβ}{N} )^{−1},
B(β, β) ≤ 1 + \frac{C_{2}xβ}{N} ,

D[β, β, π_{exp(−βr)}^{i}] ≤ C_{2}^{2} \frac{2β}{N} φ(x)

+ (4 + \frac{2β}{N} )\frac{C_{2}}{β} [\frac{12C_{1}β^{2}}{N} (z[inf_{Θ_{i}} R − R(θ̃)] + φ(z) + \frac{zd_{i}}{β[1 − \frac{2zC_{1}β}{N} ]})

− 6 log[ν(β)μ(i)] − 3 log(\frac{η}{2})]

− \frac{6C_{2}}{β} [log[3^{−1}ν(β)ϵ] + \frac{β}{2N} log[3^{−1}ν(β)η]]

and

C^{i}(ζ^{−1}β, β) ≤ \frac{C_{1}β^{2}}{N} (x[inf_{Θ_{i}} R − R(θ̃)] + φ(x) + \frac{2xd_{i}}{β[1 − \frac{xβ}{N} ]}).

This leads to

ρk̂(R) ≤ R(θ̃) + inf_{i,β}(\frac{1 + \frac{C_{2}xβ}{N}}{1 − \frac{C_{2}xβ}{N}} )^{2}\{\frac{2d_{i}}{β} + inf_{Θ_{i}} R − R(θ̃)

+ \frac{2}{β} [\frac{C_{1}β^{2}}{N} (x[inf_{Θ_{i}} R − R(θ̃] + φ(x) + \frac{2xd_{i}}{β(1 − \frac{xβ}{N} )})

− log[ν(β)μ(i)η]]}

+ \frac{2}{(1 − \frac{C_{2}xβ}{N} )^{2}} \{C_{2}^{2} \frac{2β}{N} φ(x)

+ (4 + \frac{2β}{N} )\frac{C_{2}}{β} [\frac{12C_{1}β^{2}}{N} (x[inf_{Θ_{i}} R − R(θ̃)] + φ(x) + \frac{xd_{i}}{β[1 − \frac{2xC_{1}β}{N} ]})

− 6 log[ν(β)μ(i)] − 3 log(\frac{η}{2})]

− \frac{6C_{2}}{β} [log[3^{−1}ν(β)ϵ] + \frac{β}{2N} log[3^{−1}ν(β)η]]\}.

We see in this expression that, in order to balance the various factors depending on x it is advisable to choose x such that

inf_{Θ_{i}} R − R(θ̃) = \frac{φ(x)}{x},

as long as x ≤ \frac{N}{4C_{2}β}.

Following Mammen and Tsybakov, let us assume that the usual margin assumption holds: for some real constants c > 0 and κ ≥ 1,

R(θ) − R(θ̃) ≥ c[D(θ, θ̃)]^{κ}.

As D(θ, θ̃) ≥ M′(θ, θ̃), this also implies the weaker assumption

R(θ) − R(θ̃) ≥ c[M′(θ, θ̃)]^{κ}, θ ∈ Θ,

which we will really need and use. Let us take β_{max} = N and

ν = \frac{1}{⌈log_{2}(N)⌉} ∑_{k=1}^{⌈log_{2}(N)⌉} δ_{2^{k}}.

110  Chapter 2. Comparing posterior distributions to Gibbs priors

Then, as we have already seen, φ(x) ≤ (1 − κ^{−1})(κcx)^{− \frac{1}{κ−1}} . Thus φ(x)/x ≤ bx^{− \frac{κ}{κ−1}} , where b = (1 − κ^{−1})(κc)^{− \frac{1}{κ−1}} . Let us choose accordingly

x = min\{x_{1} =^{def} (\frac{inf_{Θ_{i}} R − R(θ̃)}{b})^{− \frac{κ−1}{κ}} , x_{2} =^{def} \frac{N}{4C_{2}β} \}.

Using the fact that when r ∈ (0, \frac{1}{2}), (\frac{1+r}{1−r} )^{2} ≤ 1 + 16r ≤ 9, we get with ℙ probability at least 1 − ϵ, for any β ∈ supp ν, in the case when x = x_{1} ≤ x_{2},

ρk̂(R) ≤ inf_{Θ_{i}} R + 538 C_{2}^{2} \frac{β}{N} b^{\frac{κ−1}{κ}} [inf_{Θ_{i}} R − R(θ̃)]^{\frac{1}{κ}}

+ \frac{C_{2}}{β} [138 d_{i} + 166 log[1 + log_{2}(N)] − 134 log[μ(i)] − 102 log(ϵ) + 724], and in the case when x = x_{2} ≤ x_{1},

ρk̂(R) ≤ inf_{Θ_{i}} R + 68C_{1}[inf_{Θ_{i}} R − R(θ̃)] + 269 C_{2}^{2} \frac{β}{N} φ(x)

+ \frac{C_{2}}{β} [138 d_{i} + 166 log[1 + log_{2}(N)] − 134 log[μ(i)] − 102 log(ϵ) + 724]

≤ inf_{Θ_{i}} R + 541C_{2}^{2} \frac{β}{N} φ(x)

+ \frac{C_{2}}{β} [138 d_{i} + 166 log[1 + log_{2}(N)] − 134 log[μ(i)] − 102 log(ϵ) + 724]. Thus with ℙ probability at least 1 − ϵ,

ρk̂(R) ≤ inf_{Θ_{i}} R + inf_{β∈(1,N)} 1082 C_{2}^{2} \frac{β}{N} max\{b^{\frac{κ−1}{κ}} [inf_{Θ_{i}} R − R(θ̃)]^{\frac{1}{κ}} ,

b(\frac{4C_{2}β}{N} )^{\frac{1}{κ−1}} \}

+ \frac{C_{2}}{β} [138 d_{i} + 166 log[1 + log_{2}(N)]

− 134 log[μ(i)] − 102 log(ϵ) + 724].

Theorem 2.2.11. With probability at least 1 − ϵ, for any i ∈ ℕ,

ρk̂(R) ≤ inf_{Θ_{i}} R

+ max \{ 847C2^{\frac{3}{2}} \sqrt{\frac{b^{\frac{κ−1}{κ}} [inf_{Θ_{i}} R − R(θ̃)]^{\frac{1}{κ}} \{d_{i} + log(\frac{1+log_{2}(N)}{ϵμ(i)}) + 5\}}{N}},

2C_{2}[1082 b]^{\frac{κ−1}{2κ−1}} 4^{\frac{1}{2κ−1}} \{\frac{166C_{2}[d_{i} + log(\frac{1+log_{2}(N)}{ϵμ(i)}) + 5]}{N}\}^{\frac{κ}{2κ−1}} \} ,

where C_{2}, given by equation (2.23 page 108), will in most cases be close to 1, and in any case less than 3.2.

This result gives a bound of the same form as that given in Theorem 2.1.15 (page 85) in the special case when there is only one model — that is when μ is a Dirac mass, for instance μ(1) = 1, implying that R(θ̃_{1}) − R(θ̃) = 0. Morover the parametric complexity assumption we made for this theorem, given by equation (2.21 page 106), is weaker than the one used in Theorem 2.1.15 and described by equation (2.8, page 82). When there is more than one model, the bound shows that the estimator makes a trade-off between model accuracy, represented by inf_{Θ_{i}} R − R(θ̃), and dimension, represented by d_{i}, and that for optimal parametric sub-models, meaning those for which inf_{Θ_{i}} R = inf_{Θ} R, the estimator does at least as well as the minimax optimal convergence speed in the best of these.

Another point is that we obtain more explicit constants than in Theorem 2.1.15. It is also clear that a more careful choice of parameters could have brought some improvement in the value of these constants.

These results show that the selection scheme described in this section is a good candidate to perform temperature selection of a Gibbs posterior distribution built within a single parametric model in a rate optimal way, as well as a proposal with proven performance bound for model selection.

2.3. Two step localization

2.3.1. Two step localization of bounds relative to a Gibbs prior. Let us reconsider the case where we want to choose adaptively among a family of parametric models. Let us thus assume that the parameter set is a disjoint union of measurable sub-models, so that we can write Θ = ⊔_{m∈M}Θ_{m}, where M is some measurable index set. Let us choose some prior probability distribution on the index set μ ∈ ℳ_{+}^{1}(M), and some regular conditional prior distribution π : M → ℳ_{+}^{1}(Θ), such that π(i, Θ_{i}) = 1, i ∈ M. Let us then study some arbitrary posterior distributions ν : Ω → ℳ_{+}^{1}(M) and ρ : Ω × M :→ ℳ_{+}^{1}(Θ), such that ρ(ω, i, Θ_{i}) = 1, ω ∈ Ω, i ∈ M. We would like to compare νρ(R) with some doubly localized prior distribution μ_{exp[− \frac{β}{1+ζ_{2}} π_{exp(−βR)}(R)]}[π_{exp(−βR)}](R) (where ζ_{2} is a positive parameter to be

set as needed later on). To ease notation we will define two prior distributions (one being more precisely a conditional distribution) depending on the positive real parameters β and ζ_{2}, putting

\overline{π} = π_{exp(−βR)} and \overline{μ} = μ_{exp[− \frac{β}{1+ζ_{2}} \overline{π}(R)]}.  (2.24)

Similarly to Theorem 1.4.3 on page 53 we can write for any positive real constants β and γ

ℙ\{(\overline{μ} \overline{π}) ⊗ (\overline{μ} \overline{π})[exp[−N log[1 − tanh(\frac{γ}{N})R′]

− γr′ − N log[cosh(\frac{γ}{N})]m′]]\} ≤ 1,

and deduce, using Lemma 1.1.3 on page 16, that

ℙ\{exp[ sup_{ν∈ℳ_{+}^{1}(M)} sup_{ρ:M→ℳ_{+}^{1}(Θ)} \{−N log[1 − tanh(\frac{γ}{N})(νρ − \overline{μ} \overline{π})(R)]

− γ(νρ − \overline{μ} \overline{π})(r) − N log[cosh(\frac{γ}{N})](νρ) ⊗ (\overline{μ} \overline{π})(m′)

− 𝒦(ν, \overline{μ}) − ν[𝒦(ρ, \overline{π})]\}]\} ≤ 1. (2.25)

This will be our starting point in comparing νρ(R) with \overline{μ} \overline{π}(R). However, obtaining an empirical bound will require some supplementary efforts. For each index of the model index set M, we can write in the same way

ℙ\{\overline{π} ⊗ \overline{π}[exp[−N log[1 − tanh(\frac{γ}{N})R′] − γr′ − N log[cosh(\frac{γ}{N})]m′]]\} ≤ 1.

Integrating this inequality with respect to \overline{μ} and using Fubini’s lemma for positive functions, we get

ℙ\{\overline{μ}(\overline{π} ⊗\overline{π})[exp[−N log[1−tanh(\frac{γ}{N})R′]−γr′ −N log[cosh(\frac{γ}{N})]m′]]\} ≤ 1.

Note that \overline{μ}(\overline{π} ⊗ \overline{π}) is a probability measure on M × Θ × Θ, whereas (\overline{μ} \overline{π}) ⊗ (\overline{μ} \overline{π}) considered previously is a probability measure on (M × Θ) × (M × Θ). We get as previously

ℙ\{exp[ sup_{ν∈ℳ_{+}^{1}(M)} sup_{ρ:M→ℳ_{+}^{1}(Θ)} \{−N log[1 − tanh(\frac{γ}{N})ν(ρ − \overline{π})(R)]

− γν(ρ − \overline{π})(r) − N log[cosh(\frac{γ}{N})]ν(ρ ⊗ \overline{π})(m′)
− 𝒦(ν, \overline{μ}) − ν[𝒦(ρ, \overline{π})]\}]\} ≤ 1. (2.26)

Let us finally recall that

𝒦(ν, \overline{μ}) = \frac{β}{1+ζ_{2}} (ν − \overline{μ})\overline{π}(R) + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ),  (2.27)
𝒦(ρ, \overline{π}) = β(ρ − \overline{π})(R) + 𝒦(ρ, π) − 𝒦(\overline{π}, π).  (2.28)

From equations (2.25), (2.26) and (2.28) we deduce Proposition 2.3.1. For any positive real constants β, γ and ζ_{2}, with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M) and any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

− N log[1 − tanh(\frac{γ}{N})(νρ − \overline{μ} \overline{π})(R)] − βν(ρ − \overline{π})(R)

≤ γ(νρ − \overline{μ} \overline{π})(r) + N log[cosh(\frac{γ}{N})](νρ) ⊗ (\overline{μ} \overline{π})(m′)

+ 𝒦(ν, \overline{μ}) + ν[𝒦(ρ, π)] − ν[𝒦(\overline{π}, π)] + log(\frac{2}{ϵ}).

and

− N log[1 − tanh(\frac{γ}{N})ν(ρ − \overline{π})(R)]

≤ γν(ρ − \overline{π})(r) + N log[cosh(\frac{γ}{N})]ν(ρ ⊗ \overline{π})(m′)

+ 𝒦(ν, \overline{μ}) + ν[𝒦(ρ, \overline{π})] + log(\frac{2}{ϵ}),

where the prior distribution \overline{μ} \overline{π} is defined by equation (2.24) on page 111 and depends on β and ζ_{2}.

Let us put for short

T = tanh(\frac{γ}{N}) and C = N log[cosh(\frac{γ}{N})].

We will use an entropy compensation strategy for which we need a couple of entropy bounds. We have according to Proposition 2.3.1, with ℙ probability at least 1 − ϵ,

ν[𝒦(ρ, \overline{π})] = βν(ρ − \overline{π})(R) + ν[𝒦(ρ, π) − 𝒦(\overline{π}, π)]

≤ \frac{β}{NT} [γν(ρ − \overline{π})(r) + Cν(ρ ⊗ \overline{π})(m′)

+ 𝒦(ν, \overline{μ}) + ν[𝒦(ρ, \overline{π})] + log(\frac{2}{ϵ})]
+ ν[𝒦(ρ, π) − 𝒦(\overline{π}, π)].

Similarly

𝒦(ν, \overline{μ}) = \frac{β}{1 + ζ_{2}} (ν − \overline{μ})\overline{π}(R) + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ)

≤ \frac{β}{(1 + ζ_{2})NT} [γ(ν − \overline{μ})\overline{π}(r) + C(ν\overline{π}) ⊗ (\overline{μ} \overline{π})(m′)

+ 𝒦(ν, \overline{μ}) + log(\frac{2}{ϵ})] + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ).

Thus, for any positive real constants β, γ and ζ_{i}, i = 1, …, 5, with ℙ probability at least 1 − ϵ, for any posterior distributions ν, ν_{3} : Ω → ℳ_{+}^{1}(Θ), any posterior conditional distributions ρ, ρ_{1}, ρ_{2}, ρ_{4}, ρ_{5} : Ω × M → ℳ_{+}^{1}(Θ),

114  Chapter 2. Comparing posterior distributions to Gibbs priors

− N log[1 − T (νρ − \overline{μ} \overline{π})(R)] − βν(ρ − \overline{π})(R)

≤ γ(νρ − \overline{μ} \overline{π})(r) + C(νρ) ⊗ (\overline{μ} \overline{π})(m′)

+ 𝒦(ν, \overline{μ}) + ν[𝒦(ρ, π) − 𝒦(\overline{π}, π)] + log(\frac{2}{ϵ}),

ζ_{1} \frac{NT}{β} \overline{μ}[𝒦(ρ_{1}, \overline{π})] ≤ ζ_{1}γ\overline{μ}(ρ_{1} − \overline{π})(r) + ζ_{1}C\overline{μ}(ρ_{1} ⊗ \overline{π})(m′)

+ ζ_{1}\overline{μ}[𝒦(ρ_{1}, \overline{π})] + ζ_{1} log(\frac{2}{ϵ}) + ζ_{1} \frac{NT}{β} \overline{μ}[𝒦(ρ_{1}, π) − 𝒦(\overline{π}, π)],

ζ_{2} \frac{NT}{β} ν[𝒦(ρ_{2}, \overline{π})] ≤ ζ_{2}γν(ρ_{2} − \overline{π})(r) + ζ_{2}Cν(ρ_{2} ⊗ \overline{π})(m′)

+ ζ_{2}𝒦(ν, \overline{μ}) + ζ_{2}ν[𝒦(ρ_{2}, \overline{π})] + ζ_{2} log(\frac{2}{ϵ})

+ ζ_{2} \frac{NT}{β} ν[𝒦(ρ_{2}, π) − 𝒦(\overline{π}, π)],

ζ_{3}(1 + ζ_{2})\frac{NT}{β} 𝒦(ν_{3}, \overline{μ}) ≤ ζ_{3}γ(ν_{3} − \overline{μ})\overline{π}(r)

+ ζ_{3}C[(ν_{3}\overline{π}) ⊗ (ν_{3}ρ_{1}) + (ν_{3}ρ_{1}) ⊗ (\overline{μ} \overline{π})](m′) + ζ_{3}𝒦(ν_{3}, \overline{μ}) + ζ_{3} log(\frac{2}{ϵ})

+ ζ_{3}(1 + ζ_{2})\frac{NT}{β} [𝒦(ν_{3}, μ) − 𝒦(\overline{μ}, μ)],

ζ_{4} \frac{NT}{β} ν_{3}[𝒦(ρ_{4}, \overline{π})] ≤ ζ_{4}γν_{3}(ρ_{4} − \overline{π})(r)

+ ζ_{4}Cν_{3}(ρ_{4} ⊗ \overline{π})(m′) + ζ_{4}𝒦(ν_{3}, \overline{μ}) + ζ_{4}ν_{3}[𝒦(ρ_{4}, \overline{π})] + ζ_{4} log(\frac{2}{ϵ})

+ ζ_{4} \frac{NT}{β} ν_{3}[𝒦(ρ_{4}, π) − 𝒦(\overline{π}, π)],

ζ_{5} \frac{NT}{β} \overline{μ}[𝒦(ρ_{5}, \overline{π})] ≤ ζ_{5}γ\overline{μ}(ρ_{5} − \overline{π})(r) + ζ_{5}C\overline{μ}(ρ_{5} ⊗ \overline{π})(m′)

+ ζ_{5}\overline{μ}[𝒦(ρ_{5}, \overline{π})] + ζ_{5} log(\frac{2}{ϵ}) + ζ_{5} \frac{NT}{β} \overline{μ}[𝒦(ρ_{5}, π) − 𝒦(\overline{π}, π)].

Adding these six inequalities and assuming that

ζ_{4} ≤ ζ_{3}[(1 + ζ_{2})\frac{NT}{β} − 1],  (2.29)

we find

− N log[1 − T (νρ − \overline{μ} \overline{π})(R)] − β(νρ − \overline{μ} \overline{π})(R)

≤ −N log[1 − T (νρ − \overline{μ} \overline{π})(R)] − β(νρ − \overline{μ} \overline{π})(R)

+ ζ_{1}(\frac{NT}{β} − 1)\overline{μ}[𝒦(ρ_{1}, \overline{π})] + ζ_{2}(\frac{NT}{β} − 1)ν[𝒦(ρ_{2}, \overline{π})]
+ [ζ_{3}(1 + ζ_{2})\frac{NT}{β} − ζ_{3} − ζ_{4}]𝒦(ν_{3}, \overline{μ})

+ ζ_{4}(\frac{NT}{β} − 1)ν_{3}[𝒦(ρ_{4}, \overline{π})] + ζ_{5}(\frac{NT}{β} − 1)\overline{μ}[𝒦(ρ_{5}, \overline{π})]

≤ γ(νρ − \overline{μ} \overline{π})(r) + ζ_{1}γ\overline{μ}(ρ_{1} − \overline{π})(r) + ζ_{2}γν(ρ_{2} − \overline{π})(r)

+ ζ_{3}γ(ν_{3} − \overline{μ})\overline{π}(r) + ζ_{4}γν_{3}(ρ_{4} − \overline{π})(r) + ζ_{5}γ\overline{μ}(ρ_{5} − \overline{π})(r)

+ C[(νρ) ⊗ (\overline{μ} \overline{π}) + ζ_{1}\overline{μ}(ρ_{1} ⊗ \overline{π}) + ζ_{2}ν(ρ_{2} ⊗ \overline{π})

+ ζ_{3}(ν_{3}\overline{π}) ⊗ (ν_{3}ρ_{1}) + ζ_{3}(ν_{3}ρ_{1}) ⊗ (\overline{μ} \overline{π})

+ ζ_{4}ν_{3}(ρ_{4} ⊗ \overline{π}) + ζ_{5}\overline{μ}(ρ_{5} ⊗ \overline{π})](m′)

+ (1 + ζ_{2})[𝒦(ν, μ) − 𝒦(\overline{μ}, μ)] + ν[𝒦(ρ, π) − 𝒦(\overline{π}, π)]

+ ζ_{1} \frac{NT}{β} \overline{μ}[𝒦(ρ_{1}, π) − 𝒦(\overline{π}, π)] + ζ_{2} \frac{NT}{β} ν[𝒦(ρ_{2}, π) − 𝒦(\overline{π}, π)]

+ ζ_{3}(1 + ζ_{2})\frac{NT}{β} [𝒦(ν_{3}, μ) − 𝒦(\overline{μ}, μ)] + ζ_{4} \frac{NT}{β} ν_{3}[𝒦(ρ_{4}, π) − 𝒦(\overline{π}, π)]

+ ζ_{5} \frac{NT}{β} \overline{μ}[𝒦(ρ_{5}, π) − 𝒦(\overline{π}, π)] + (1 + ζ_{1} + ζ_{2} + ζ_{3} + ζ_{4} + ζ_{5}) log(\frac{2}{ϵ}),

where we have also used the fact (concerning the 11th line of the preceding inequalities) that

− β(νρ − \overline{μ} \overline{π})(R) + 𝒦(ν, \overline{μ}) + ν[𝒦(ρ, \overline{π})]

≤ −β(νρ − \overline{μ} \overline{π})(R) + (1 + ζ_{2})𝒦(ν, \overline{μ}) + ν[𝒦(ρ, \overline{π})] = (1 + ζ_{2})[𝒦(ν, μ) − 𝒦(\overline{μ}, μ)] + ν[𝒦(ρ, π) − 𝒦(\overline{π}, π)].

Let us now apply to \overline{π} (we shall later do the same with \overline{μ}) the following inequalities, holding for any random functions of the sample and the parameters h : Ω × Θ → ℝ and g : Ω × Θ → ℝ,

\overline{π}(g − h) − 𝒦(\overline{π}, π) ≤ sup_{ρ:Ω×M→ℳ_{+}^{1}(Θ)} ρ(g − h) − 𝒦(ρ, π)

= log{π[exp(g − h)]}
= log{π[exp(−h)]} + log{π_{exp(−h)}[exp(g)]}

= −π_{exp(−h)}(h) − 𝒦(π_{exp(−h)}, π) + log{π_{exp(−h)}[exp(g)]}.

When h and g are observable, and h is not too far from βr ≃ βR, this gives a way to replace \overline{π} with a satisfactory empirical approximation. We will apply this method, choosing ρ_{1} and ρ_{5} such that \overline{μ} \overline{π} is replaced either with \overline{μ}ρ_{1}, when it comes from the first two inequalities or with \overline{μ}ρ_{5} otherwise, choosing ρ_{2} such that ν\overline{π} is replaced with νρ_{2} and ρ_{4} such that ν_{3}\overline{π} is replaced with ν_{3}ρ_{4}. We will do so because it leads to a lot of helpful cancellations. For those to happen, we need to choose ρ_{i} = π_{exp(−λ_{i}r)}, i = 1, 2, 4, where λ_{1}, λ_{2} and λ_{4} are such that

(1 + ζ_{1})γ = ζ_{1} \frac{NT}{β} λ_{1},  (2.30)
ζ_{2}γ = (1 + ζ_{2} \frac{NT}{β} )λ_{2},  (2.31)
(ζ_{4} − ζ_{3})γ = ζ_{4} \frac{NT}{β} λ_{4},  (2.32)
ζ_{3}γ = ζ_{5} \frac{NT}{β} λ_{5},  (2.33)

and to assume that

ζ_{4} > ζ_{3}.  (2.34)

We obtain that with ℙ probability at least 1 − ϵ, − N log[1 − T (μρ − \overline{μ} \overline{π})(R)] − β(νρ − \overline{μ} \overline{π})(R)

≤ γ(νρ − \overline{μ} ρ_{1})(r) + ζ_{3}γ(ν_{3}ρ_{4} − \overline{μ}ρ_{5})(r)

+ ζ_{1} \frac{NT}{β} \overline{μ}\{log[ρ_{1}\{exp[C \frac{β}{NT ζ_{1}} [νρ + ζ_{1}ρ_{1}](m′)]\}]\}

+ (1 + ζ_{2} \frac{NT}{β} )ν\{log\{ρ_{2}\{exp[_{\frac{C}{1+ζ_{2} \frac{NT}{β}}} ζ_{2}ρ_{2}(m′)]\}]\}

+ ζ_{4} \frac{NT}{β} ν_{3}\{log[ρ_{4}\{exp[C \frac{β}{NT ζ_{4}} [ζ_{3}ν_{3}ρ_{1} + ζ_{4}ρ_{4}](m′)]\}]\}

+ ζ_{5} \frac{NT}{β} \overline{μ}\{log[ρ_{5}\{exp[C \frac{β}{NT ζ_{5}} [ζ_{3}ν_{3}ρ_{1} + ζ_{5}ρ_{5}](m′)]\}]\}
+ (1 + ζ_{2})[𝒦(ν, μ) − 𝒦(\overline{μ}, μ)] + ν[𝒦(ρ, π) − 𝒦(ρ_{2}, π)]
+ ζ_{3}(1 + ζ_{2})\frac{NT}{β} [𝒦(ν_{3}, μ) − 𝒦(\overline{μ}, μ)]

+ (1 + ∑_{i=1}^{5} ζ_{i}) log(\frac{2}{ϵ}).

In order to obtain more cancellations while replacing \overline{μ} by some posterior distribution, we will choose the constants such that λ_{5} = λ_{4}, which can be done by choosing

ζ_{5} = \frac{ζ_{3}ζ_{4}}{ζ_{4} − ζ_{3}} .  (2.35)

We can now replace \overline{μ} with μ_{exp −ξ_{1}ρ_{1}(r)−ξ_{4}ρ_{4}(r)}, where

ξ_{1} = \frac{γ}{(1 + ζ_{2})(1 + \frac{NT}{β} ζ_{3})},  (2.36)
ξ_{4} = \frac{γζ_{3}}{(1 + ζ_{2})(1 + \frac{NT}{β} ζ_{3})}.  (2.37)

Choosing moreover ν_{3} = μ_{exp −ξ_{1}ρ_{1}(r)−ξ_{4}ρ_{4}(r)}, to induce some more cancellations, we get

Theorem 2.3.2. Let us use the notation introduced above. For any positive real constants satisfying equations (2.29, page 114), (2.30, page 115), (2.31, page 115), (2.32, page 115), (2.33, page 115), (2.34, page 115), (2.35, page 116), (2.36, page 116), (2.37, page 116), with ℙ probability at least 1−ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M) and any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

− N log[1 − T (νρ − \overline{μ} \overline{π})(R)] − β(νρ − \overline{μ} \overline{π})(R) ≤ B(ν, ρ, β),

where B(ν, ρ, β) =^{def} γ(νρ − ν_{3}ρ_{1})(r)

+ (1 + ζ_{2})(1 + \frac{NT}{β} ζ_{3})

× log\{ν_{3}[ρ_{1}\{exp[C \frac{β}{NT ζ_{1}} [νρ + ζ_{1}ρ_{1}](m′)]\}^{\frac{ζ_{1}NT}{β(1+ζ_{2})(1+ \frac{NT}{β} ζ_{3})}}

× ρ_{4}\{exp[C \frac{β}{NT ζ_{5}} [ζ_{3}ν_{3}ρ_{1} + ζ_{5}ρ_{4}](m′)]\}^{\frac{ζ_{5}NT}{β(1+ζ_{2})(1+ \frac{NT}{β} ζ_{3})}} ]\}

+ (1 + ζ_{2} \frac{NT}{β} )ν\{log\{ρ_{2}\{exp[_{\frac{C}{1+ζ_{2} \frac{NT}{β}}} ζ_{2}ρ_{2}(m′)]\}]\}

+ ζ_{4} \frac{NT}{β} ν_{3}\{log[ρ_{4}\{exp[C \frac{β}{NT ζ_{4}} [ζ_{3}ν_{3}ρ_{1} + ζ_{4}ρ_{4}](m′)]\}]\}
+ (1 + ζ_{2})[𝒦(ν, μ) − 𝒦(ν_{3}, μ)]

+ ν[𝒦(ρ, π) − 𝒦(ρ_{2}, π)] + (1 + ∑_{i=1}^{5} ζ_{i}) log(\frac{2}{ϵ}).

This theorem can be used to find the largest value β̂(νρ) of β such that B(ν, ρ, β) ≤ 0, thus providing an estimator for β(νρ) defined as νρ(R) = \overline{μ}_{β(νρ)}\overline{π}_{β(νρ)}(R), where we have mentioned explicitly the dependence of \overline{μ} and \overline{π} in β, the constant ζ_{2} staying fixed. The posterior distribution νρ may then be chosen to maximize β̂(νρ) within some manageable subset of posterior distributions 𝒫, thus gaining the assurance that νρ(R) ≤ \overline{μ}_{β̂(νρ)}\overline{π}_{β̂(νρ)}(R),

with the largest parameter β̂(νρ) that this approach can provide. Maximizing β̂(νρ) is supported by the fact that lim_{β→+∞} \overline{μ}_{β}\overline{π}_{β}(R) = ess inf_{μπ} R. Anyhow, there is no assurance (to our knowledge) that β ↦ \overline{μ}_{β}\overline{π}_{β}(R) will be a decreasing function of β all the way, although this may be expected to be the case in many practical situations.

We can make the bound more explicit in several ways. One point of view is to put forward the optimal values of ρ and ν. We can thus remark that

ν[γρ(r) + 𝒦(ρ, π) − 𝒦(ρ_{2}, π)] + (1 + ζ_{2})𝒦(ν, μ)

= ν[𝒦[ρ, π_{exp(−γr)}] + λ_{2}ρ_{2}(r) + ∫_{λ^{2}}^{γ} π_{exp(−αr)}(r)dα] + (1 + ζ_{2})𝒦(ν, μ)

= ν\{𝒦[ρ, π_{exp(−γr)}]\} + (1 + ζ_{2})𝒦[ν, μ_{exp(− \frac{λ_{2}ρ_{2}(r)}{1+ζ_{2}} − \frac{1}{1+ζ_{2}} ∫_{λ_{2}}^{γ} π_{exp(−αr)}(r)dα)}]

− (1 + ζ_{2}) log\{μ[exp\{−\frac{λ_{2}}{1 + ζ_{2}} ρ_{2}(r) − \frac{1}{1 + ζ_{2}} ∫_{λ_{2}}^{γ} π_{exp(−αr)}(r)dα\}]\}.

Thus

B(ν, ρ, β) = (1 + ζ_{2})[ξ_{1}ν_{3}ρ_{1}(r) + ξ_{4}ν_{3}ρ_{4}(r)

+ log{μ[exp(−ξ_{1}ρ_{1}(r) − ξ_{4}ρ_{4}(r))]}]

− (1 + ζ_{2}) log\{μ[exp\{−\frac{λ_{2}}{1 + ζ_{2}} ρ_{2}(r) − \frac{1}{1 + ζ_{2}} ∫_{λ_{2}}^{γ} π_{exp(−αr)}(r)dα\}]\}

− γν_{3}ρ_{1}(r) + (1 + ζ_{2})(1 + \frac{NT}{β} ζ_{3})
× log\{ν_{3}[ρ_{1}\{exp[C \frac{β}{NT ζ_{1}} [νρ + ζ_{1}ρ_{1}](m′)]\}^{\frac{ζ_{1}NT}{β(1+ζ_{2})(1+ \frac{NT}{β} ζ_{3})}}

× ρ_{4}\{exp[C \frac{β}{NT ζ_{5}} [ζ_{3}ν_{3}ρ_{1} + ζ_{5}ρ_{4}](m′)]\}^{\frac{ζ_{5}NT}{β(1+ζ_{2})(1+ \frac{NT}{β} ζ_{3})}} ]\}

+ (1 + ζ_{2} \frac{NT}{β} )ν\{log\{ρ_{2}\{exp[_{\frac{C}{1+ζ_{2} \frac{NT}{β}}} ζ_{2}ρ_{2}(m′)]\}]\}

+ ζ_{4} \frac{NT}{β} ν_{3}\{log[ρ_{4}\{exp[C \frac{β}{NT ζ_{4}} [ζ_{3}ν_{3}ρ_{1} + ζ_{4}ρ_{4}](m′)]\}]\}
+ ν{𝒦[ρ, π_{exp(−γr)}]}
+ (1 + ζ_{2})𝒦[ν, μ_{exp(− \frac{λ_{2}ρ_{2}(r)}{1+ζ_{2}} − \frac{1}{1+ζ_{2}} ∫_{λ_{2}}^{γ} π_{exp(−αr)}(r)dα)}]

+ (1 + ∑_{i=1}^{5} ζ_{i}) log(\frac{2}{ϵ}).

This formula is better understood when thinking about the following upper bound for the two first lines in the expression of B(ν, ρ, β):

(1 + ζ_{2})[ξ_{1}ν_{3}ρ_{1}(r) + ξ_{4}ν_{3}ρ_{4}(r) + log{μ[exp(−ξ_{1}ρ_{1}(r) − ξ_{4}ρ_{4}(r))]}]

− (1 + ζ_{2}) log\{μ[exp\{−\frac{λ_{2}}{1 + ζ_{2}} ρ_{2}(r)

− \frac{1}{1 + ζ_{2}} ∫_{λ_{2}}^{γ} π_{exp(−αr)}(r)dα\}]\} − γν_{3}ρ_{1}(r)

≤ ν_{3}[λ_{2}ρ_{2}(r) + ∫_{λ_{2}}^{γ} π_{exp(−αr)}(r)dα − γρ_{1}(r)].

Another approach to understanding Theorem 2.3.2 is to put forward ρ_{0} = π_{exp(−λ_{0}r)}, for some positive real constant λ_{0} < γ, noticing that

ν[𝒦(ρ_{0}, π) − 𝒦(ρ_{2}, π)] = λ_{0}ν(ρ_{2} − ρ_{0})(r) − ν[𝒦(ρ_{2}, ρ_{0})].

Thus

B(ν, ρ_{0}, β) ≤ ν_{3}[(γ − λ_{0})(ρ_{0} − ρ_{1})(r) + λ_{0}(ρ_{2} − ρ_{1})(r)]

+ (1 + ζ_{2})(1 + \frac{NT}{β} ζ_{3})

× log\{ν_{3}[ρ_{1}\{exp[C \frac{β}{NT ζ_{1}} [νρ_{0} + ζ_{1}ρ_{1}](m′)]\}^{\frac{ζ_{1}NT}{β(1+ζ_{2})(1+ \frac{NT}{β} ζ_{3})}}

× ρ_{4}\{exp[C \frac{β}{NT ζ_{5}} [ζ_{3}ν_{3}ρ_{1} + ζ_{5}ρ_{4}](m′)]\}^{\frac{ζ_{5}NT}{β(1+ζ_{2})(1+ \frac{NT}{β} ζ_{3})}} ]\}

+ (1 + ζ_{2} \frac{NT}{β} )ν\{log\{ρ_{2}\{exp[_{\frac{C}{1+ζ_{2} \frac{NT}{β}}} ζ_{2}ρ_{2}(m′)]\}]\}

+ ζ_{4} \frac{NT}{β} ν_{3}\{log[ρ_{4}\{exp[C \frac{β}{NT ζ_{4}} [ζ_{3}ν_{3}ρ_{1} + ζ_{4}ρ_{4}](m′)]\}]\}
+ (1 + ζ_{2})𝒦[ν, μ_{exp(− \frac{(γ−λ_{0})ρ_{0}(r)+λ_{0}ρ_{2}(r)}{1+ζ_{2}})}]

− ν[𝒦(ρ_{2}, ρ_{0})] + (1 + ∑_{i=1}^{5} ζ_{i}) log(\frac{2}{ϵ}).

In the case when we want to select a single model m̂(ω), and therefore to set ν = δm̂, the previous inequality engages us to take

m̂ ∈ arg min_{m∈M}(γ − λ_{0})ρ_{0}(m, r) + λ_{0}ρ_{2}(m, r).

In parametric situations where

π_{exp(−λr)}(r) ≃ r^{⋆}(m) + \frac{d_{e}(m)}{λ},

we get

(γ − λ_{0})ρ_{0}(m, r) − λ_{0}ρ_{2}(m, r) ≃ γ[r^{⋆}(m) + d_{e}(m)(\frac{1}{λ_{0}} + \frac{λ_{0}−λ_{2}}{γλ_{2}} )],

resulting in a linear penalization of the empirical dimension of the models.

2.3.2. Analysis of two step bounds relative to a Gibbs prior. We will not state a formal result, but will nevertheless give some hints about how to establish one. This is a rather technical section, which can be skipped at a first reading , since it will not be used below. We should start from Theorem 1.4.2 (page 52), which gives a deterministic variance term. From Theorem 1.4.2, after a change of prior distribution, we obtain for any positive constants α_{1} and α_{2}, any prior distributions μ̃_{1} and μ̃_{2} ∈ ℳ_{+}^{1}(M), for any prior conditional distributions π̃_{1} and π̃_{2} : M → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − η, for any posterior distributions ν_{1}ρ_{1} and ν_{2}ρ_{2},

α_{1}(ν_{1}ρ_{1} − ν_{2}ρ_{2})(R) ≤ α_{2}(ν_{1}ρ_{1} − ν_{2}ρ_{2})(r)

+ 𝒦[(ν_{1}ρ_{1}) ⊗ (ν_{2}ρ_{2}), (μ̃_{1} π̃_{1}) ⊗ (μ̃_{2} π̃_{2})]

+ log\{(μ̃_{1} π̃_{1}) ⊗ (μ̃_{2} π̃_{2})[exp\{−α_{2}Ψ_{\frac{α_{2}}{N}} (R′, M′) + α_{1}R′\}]\} − log(η).

Applying this to α_{1} = 0, we get that

(νρ − ν_{3}ρ_{1})(r) ≤ \frac{1}{α_{2}} [𝒦[(νρ) ⊗ (ν_{3}ρ_{1}), (μ̃ π̃) ⊗ (μ̃_{3} π̃_{1})]

+ log\{(μ̃ ν̃) ⊗ (μ̃_{3} π̃_{1})[exp\{α_{2}Ψ_{− \frac{α_{2}}{N}} (R′, M′)\}]\} − log(η)].

In the same way, to bound quantities of the form

log{ν_{3}[ρ_{1}{exp[C_{1}(νρ + ζ_{1}ρ_{1})(m′)]}^{p_{1}}

× ρ_{4}{exp[C_{2}[ζ_{3}ν_{3}ρ_{1} + ζ_{5}ρ_{4}](m′)]}^{p_{2}}]}

= sup_{ν_{5}}{p_{1} sup_{ρ_{5}}{C_{1}[(νρ) ⊗ (ν_{5}ρ_{5}) + ζ_{1}ν_{5}(ρ_{1} ⊗ ρ_{5})](m′) − 𝒦(ρ_{5}, ρ_{1})}

+ p_{2} sup_{ρ_{6}}{C_{2}[ζ_{3}(ν_{3}ρ_{1}) ⊗ (ν_{5}ρ_{6})

+ ζ_{5}ν_{5}(ρ_{4} ⊗ ρ_{6})](m′) − 𝒦(ρ_{6}, ρ_{4})} − 𝒦(ν_{5}, ν_{3})},

where C_{1}, C_{2}, p_{1} and p_{2} are positive constants, and similar terms, we need to use inequalities of the type: for any prior distributions μ̃_{i} π̃_{i}, i = 1, 2, with ℙ probability at least 1 − η, for any posterior distributions ν_{i}ρ_{i}, i = 1, 2,

α_{3}(ν_{1}ρ_{1}) ⊗ (ν_{2}ρ_{2})(m′) ≤ log\{(μ̃_{1} π̃_{1}) ⊗ (μ̃_{2} π̃_{2}) exp[α_{3}Φ_{\frac{−α_{3}}{N}} (M′)]\}

+ 𝒦[(ν_{1}ρ_{1}) ⊗ (ν_{2}ρ_{2}), (μ̃_{1} π̃_{1}) ⊗ (μ̃_{2} π̃_{2})] − log(η).

We need also the variant: with ℙ probability at least 1 − η, for any posterior distribution ν_{1} : Ω → ℳ_{+}^{1}(M) and any conditional posterior distributions ρ_{1}, ρ_{2} : Ω × M → ℳ_{+}^{1}(Θ),

α_{3}ν_{1}(ρ_{1} ⊗ ρ_{2})(m′) ≤ log\{μ̃_{1}(π̃_{1} ⊗ π̃_{2}) exp[α_{3}Φ_{− \frac{α_{3}}{N}} (M′)]\}

+ 𝒦(ν_{1}, μ̃_{1}) + ν_{1}{𝒦[ρ_{1} ⊗ ρ_{2}, π̃_{1} ⊗ π̃_{2}]} − log(η).

We deduce that

log{ν_{3}[ρ_{1}{exp[C_{1}(νρ + ζ_{1}ρ_{1})(m′)]}^{p_{1}}

× ρ_{4}{exp[C_{2}[ζ_{3}ν_{3}ρ_{1} + ζ_{5}ρ_{4}](m′)]}^{p_{2}}]}

≤ sup_{ν_{5}}\{p_{1} sup_{ρ_{5}}[\frac{C_{1}}{α_{3}} \{log\{(μ̃ π̃) ⊗ (μ̃_{5} π̃_{5}) exp[α_{3}Φ_{− \frac{α_{3}}{N}} (M′)]\}

+ 𝒦[(νρ) ⊗ (ν_{5}ρ_{5}), (μ̃ π̃ ⊗ (μ̃_{5} π̃_{5})] + log(\frac{2}{η})
+ ζ_{1}[log\{μ̃_{5}(π̃_{1} ⊗ π̃_{5}) exp[α_{3}Φ_{− \frac{α_{3}}{N}} (M′)]\}

+ 𝒦(ν_{5}, μ̃_{5}) + ν_{5}\{𝒦[ρ_{1} ⊗ ρ_{5}, π̃_{1} ⊗ π̃_{5}]\} + log(\frac{2}{η})]\} − 𝒦(ρ_{5}, ρ_{1})]

+ p_{2} sup_{ρ_{6}}[\frac{C_{1}}{α_{3}} \{log\{(μ̃_{3} π̃_{1}) ⊗ (μ̃_{5} π̃_{6}) exp[α_{3}Φ_{− \frac{α_{3}}{N}} (M′)]\}

+ 𝒦[(ν_{3}ρ_{1}) ⊗ (ν_{5}ρ_{6}), (μ̃_{3} π̃_{1} ⊗ (μ̃_{5} π̃_{6})] + log(\frac{2}{η})
+ ζ_{1}[log\{μ̃_{5}(π̃_{4} ⊗ π̃_{6}) exp[α_{3}Φ_{− \frac{α_{3}}{N}} (M′)]\}
+ 𝒦(ν_{5}, μ̃_{5}) + ν_{5}\{𝒦[ρ_{4} ⊗ ρ_{6}, π̃_{4} ⊗ π̃_{6}]\} + log(\frac{2}{η})]\}

− 𝒦(ρ_{6}, ρ_{4})] − 𝒦(ν_{5}, ν_{3})}.

We are then left with the need to bound entropy terms like 𝒦(ν_{3}ρ_{1}, μ̃_{3}π̃_{1}), where we have the choice of μ̃_{3} and π̃_{1}, to obtain a useful bound. As could be expected, we decompose it into

𝒦(ν_{3}ρ_{1}, μ̃_{3}π̃_{1}) = 𝒦(ν_{3}, μ̃_{3}) + ν_{3}[𝒦(ρ_{1}, π̃_{1})].

Let us look after the second term first, choosing π̃_{1} = π_{exp(−β_{1}R)}:

ν_{3}[𝒦(ρ_{1}, π̃_{1})] = ν_{3}[β_{1}(ρ_{1} − π̃_{1})(R) + 𝒦(ρ_{1}, π) − 𝒦(π̃_{1}, π)]

≤ \frac{β_{1}}{α_{1}} [α_{2}ν_{3}(ρ_{1} − π̃_{1})(r) + 𝒦(ν_{3}, μ̃_{3}) + ν_{3}[𝒦(ρ_{1}, π̃_{1})]

+ log\{μ̃_{3}(π_{1}^{⊗2})[exp\{−α_{2}Ψ_{\frac{α_{2}}{N}} (R′, M′) + α_{1}R′\}]\} − log(η)]
+ ν_{3}[𝒦(ρ_{1}, π) − 𝒦(π̃_{1}, π)]

≤ \frac{β_{1}}{α_{1}} [𝒦(ν_{3}, μ̃_{3}) + ν_{3}[𝒦(ρ_{1}, π̃_{1})]

+ log\{μ̃_{3}(π_{1}^{⊗2})[exp\{−α_{2}Ψ_{\frac{α_{2}}{N}} (R′, M′) + α_{1}R′\}]\} − log(η)]
+ ν_{3}\{𝒦[ρ_{1}, π_{exp(− \frac{β_{1}α_{2}}{α_{1}} r)}]\}.

Thus, when the constraint λ_{1} = \frac{β_{1}α_{2}}{α_{1}} is satisfied,

ν_{3}[𝒦(ρ_{1}, π̃_{1})] ≤ (1 − \frac{β_{1}}{α_{1}} )^{−1} \frac{β_{1}}{α_{1}} [𝒦(ν_{3}, μ̃_{3})

+ log\{μ̃_{3}(π_{1}^{⊗2})[exp\{−α_{2}Ψ_{\frac{α_{2}}{N}} (R′, M′) + α_{1}R′\}]\} − log(η)].

We can further specialize the constants, choosing α_{1} = N sinh(\frac{α_{2}}{N}), so that

−α_{2}Ψ_{\frac{α_{2}}{N}} (R′, M′) + α_{1}R′ ≤ 2N sinh(\frac{α_{2}}{2N} )^{2}M′.

We can for instance choose α_{2} = γ, α_{1} = N sinh(\frac{γ}{N}) and β_{1} = λ_{1} \frac{N}{γ} sinh(\frac{γ}{N}), leading to Proposition 2.3.3. With the notation of Theorem 2.3.2, the constants being set as explained above, putting π̃_{1} = π_{exp(−λ_{1} \frac{N}{γ} sinh( \frac{γ}{N})R)}, with ℙ probability at least 1 − η,

ν_{3}[𝒦(ρ_{1}, π̃_{1})] ≤ (1 − \frac{λ_{1}}{γ} )^{−1} \frac{λ_{1}}{γ} [𝒦(ν_{3}, μ̃_{3})

+ log\{μ̃_{3}(π_{1}^{⊗2})[exp\{2N sinh(\frac{γ}{2N})^{2}M′\}]\} − log(η)].

More generally

ν_{3}[𝒦(ρ, π̃_{1})] ≤ (1 − \frac{λ_{1}}{γ} )^{−1} \frac{λ_{1}}{γ} [𝒦(ν_{3}, μ̃_{3})

+ log\{μ̃_{3}(π_{1}^{⊗2})[exp\{2N sinh(\frac{γ}{2N})^{2}M′\}]\} − log(η)]

+ (1 − \frac{λ_{1}}{γ} )^{−1}ν_{3}[𝒦(ρ, ρ_{1})].

In a similar way, let us now choose μ̃_{3} = μ_{exp[−α_{3}\overline{π}(R)]}. We can write

𝒦(ν, μ̃_{3}) = α_{3}(ν − μ̃_{3})\overline{π}(R) + 𝒦(ν, μ) − 𝒦(μ̃_{3}, μ)

≤ \frac{α_{3}}{α_{1}} [α_{2}(ν − μ̃_{3})\overline{π}(r) + 𝒦(ν, μ̃_{3})

+ log\{(μ̃_{3}\overline{π}) ⊗ (μ̃_{3}\overline{π})[exp\{−α_{2}Ψ_{\frac{α_{2}}{N}} (R′, M′) + α_{1}R′\}]\} − log(η)]

+ 𝒦(ν, μ) − 𝒦(μ̃_{3}, μ).

Let us choose α_{2} = γ, α_{1} = N sinh(\frac{γ}{N}), and let us add some other entropy inequalities to get rid of \overline{π} in a suitable way, the approach of entropy compensation being the same as that used to obtain the empirical bound of Theorem 2.3.2 (page 116). This results with ℙ probability at least 1 − η in

(1 − \frac{α_{3}}{α_{1}} )𝒦(ν, μ̃_{3}) ≤ \frac{α_{3}}{α_{1}} [γ(ν − μ̃_{3})\overline{π}(r)

+ log\{(μ̃_{3}\overline{π}) ⊗ (μ̃_{3}\overline{π})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\} + log(\frac{2}{η})]
+ 𝒦(ν, μ) − 𝒦(μ̃_{3}, μ),

ζ_{6}(1 − \frac{β}{α_{1}} )μ̃_{3}[𝒦(ρ_{6}, \overline{π})] ≤ ζ_{6} \frac{β}{α_{1}} [γμ̃_{3}(ρ_{6} − \overline{π})(r)

+ log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\} + log(\frac{2}{η})]

+ ζ_{6}μ̃_{3}[𝒦(ρ_{6}, π) − 𝒦(\overline{π}, π)],

ζ_{7}(1 − \frac{β}{α_{1}} )μ̃_{3}[𝒦(ρ_{7}, \overline{π})] ≤ ζ_{7} \frac{β}{α_{1}} [γμ̃_{3}(ρ_{7} − \overline{π})(r)

+ log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\} + log(\frac{2}{η})]

+ ζ_{7}μ̃_{3}[𝒦(ρ_{7}, π) − 𝒦(\overline{π}, π)],

ζ_{8}(1 − \frac{β}{α_{1}} )ν[𝒦(ρ_{8}, \overline{π})] ≤ ζ_{8} \frac{β}{α_{1}} [γν(ρ_{8} − \overline{π})(r) + 𝒦(ν, μ̃_{3})

+ log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\} + log(\frac{2}{η})]

+ ζ_{8}ν[𝒦(ρ_{8}, π) − 𝒦(\overline{π}, π)],

ζ_{9}(1 − \frac{β}{α_{1}} )ν[𝒦(ρ_{9}, \overline{π})] ≤ ζ_{9} \frac{β}{α_{1}} [γν(ρ_{9} − \overline{π})(r) + 𝒦(ν, μ̃_{3})

+ log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\} + log(\frac{2}{η})]

+ ζ_{9}ν[𝒦(ρ_{9}, π) − 𝒦(\overline{π}, π)],

where we have introduced a bunch of constants, assumed to be positive, that we will more precisely set to

x_{8} + x_{9} = 1,
(ζ_{6}β + x_{8}α_{3})\frac{γ}{α_{1}} = λ_{6},
(ζ_{7}β + x_{9}α_{3})\frac{γ}{α_{1}} = λ_{7},
(ζ_{8}β − x_{8}α_{3})\frac{γ}{α_{1}} = λ_{8},
(ζ_{9}β − x_{9}α_{3})\frac{γ}{α_{1}} = λ_{9}.

We get with ℙ probability at least 1 − η,

(1 − \frac{α_{3}}{α_{1}} − (ζ_{8} + ζ_{9})\frac{β}{α_{1}} )𝒦(ν, μ̃_{3}) ≤

\frac{α_{3}}{α_{1}} [γ[ν(x_{8}ρ_{8} + x_{9}ρ_{9})(r) − μ̃_{3}(x_{8}ρ_{6} + x_{9}ρ_{7})(r)]

+ \frac{α_{3}}{α_{1}} log\{(μ̃_{3}\overline{π}) ⊗ (μ̃_{3}\overline{π})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ 𝒦(ν, μ) − 𝒦(μ̃_{3}, μ) + (\frac{α_{3}}{α_{1}} + (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} ) log(\frac{2}{η}).

Let us choose the constants so that λ_{1} = λ_{7} = λ_{9}, λ_{4} = λ_{6} = λ_{8}, α_{3}x_{9} \frac{γ}{α_{1}} = ξ_{1} and α_{3}x_{8} \frac{γ}{α_{1}} = ξ_{4}. This is done by setting

x_{8} = \frac{ξ_{4}}{ξ_{1} + ξ_{4}} ,

x_{9} = \frac{ξ_{1}}{ξ_{1} + ξ_{4}} ,
α_{3} = \frac{N}{γ} sinh(\frac{γ}{N})(ξ_{1} + ξ_{4}),
ζ_{6} = \frac{N}{γ} sinh(\frac{γ}{N})\frac{(λ_{4} − ξ_{4})}{β},
ζ_{7} = \frac{N}{γ} sinh(\frac{γ}{N})\frac{(λ_{1} − ξ_{1})}{β},
ζ_{8} = \frac{N}{γ} sinh(\frac{γ}{N})\frac{(λ_{4} + ξ_{4})}{β},
ζ_{9} = \frac{N}{γ} sinh(\frac{γ}{N})\frac{(λ_{1} + ξ_{1})}{β}.

The inequality λ_{1} > ξ_{1} is always satisfied. The inequality λ_{4} > ξ_{4} is required for the above choice of constants, and will be satisfied for a suitable choice of ζ_{3} and ζ_{4}.

Under these assumptions, we obtain with ℙ probability at least 1 − η

(1 − \frac{α_{3}}{α_{1}} − (ζ_{8} + ζ_{9})\frac{β}{α_{1}} )𝒦(ν, μ̃_{3}) ≤ (ν − μ̃_{3})(ξ_{1}ρ_{1} + ξ_{4}ρ_{4})(r)

+ \frac{α_{3}}{α_{1}} log\{(μ̃_{3}\overline{π}) ⊗ (μ̃_{3}\overline{π})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ 𝒦(ν, μ) − 𝒦(μ̃_{3}, μ) + (\frac{α_{3}}{α_{1}} + (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} ) log(\frac{2}{η}).

This proves

Proposition 2.3.4. The constants being set as explained above, with ℙ probability at least 1 − η, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M),

𝒦(ν, μ̃_{3}) ≤ (1 − \frac{α_{3}}{α_{1}} − (ζ_{8} + ζ_{9})\frac{β}{α_{1}} )^{−1}[𝒦(ν, ν_{3})

+ \frac{α_{3}}{α_{1}} log\{(μ̃_{3}\overline{π}) ⊗ (μ̃_{3}\overline{π})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ (\frac{α_{3}}{α_{1}} + (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} ) log(\frac{2}{η})].

Thus

𝒦(ν_{3}ρ_{1}, μ̃_{3} π̃_{1}) ≤ \frac{1 + (1 − \frac{λ_{1}}{γ} )^{−1} \frac{λ_{1}}{γ}}{1 − \frac{α_{3}}{α_{1}} − (ζ_{8} + ζ_{9})\frac{β}{α_{1}}}

× [\frac{α_{3}}{α_{1}} log\{(μ̃_{3}\overline{π} ⊗ (μ̃_{3}\overline{π})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} log\{μ̃_{3}(\overline{π}^{⊗2})[exp\{−γΨ_{\frac{γ}{N}} (R′, M′) + α_{1}R′\}]\}

+ (\frac{α_{3}}{α_{1}} + (ζ_{6} + ζ_{7} + ζ_{8} + ζ_{9})\frac{β}{α_{1}} ) log(\frac{2}{η})]

+ (1 − \frac{λ_{1}}{γ} )^{−1} \frac{λ_{1}}{γ} [log\{μ̃_{3}(π_{1}^{⊗2})[exp\{2N sinh(\frac{γ}{2N} )^{2}M′\}]\} − log(\frac{2}{η})].

We will not go further, lest it may become tedious, but we hope we have given sufficient hints to state informally that the bound B(ν, ρ, β) of Theorem 2.3.2 (page 116) is upper bounded with ℙ probability close to one by a bound of the same flavour where the empirical quantities r and m′ have been replaced with their expectations R and M′.

2.3.3. Two step localization between posterior distributions. Here we work with a family of prior distributions described by a regular conditional prior distribution π = M → ℳ_{+}^{1}(Θ), where M is some measurable index set. This family may typically describe a countable family of parametric models. In this case M = ℕ, and each of the prior distributions π(i, .), i ∈ ℕ satisfies some parametric complexity assumption of the type

lim sup_{β→+∞} β[π_{exp(−βR)}(i, .)(R) − ess inf_{π(i,.)} R] = d_{i} < +∞, i ∈ M.

Let us consider also a prior distribution μ ∈ ℳ_{+}^{1}(M) defined on the index set M.

Our aim here will be to compare the performance of two given posterior distributions ν_{1}ρ_{1} and ν_{2}ρ_{2}, where ν_{1}, ν_{2} : Ω → ℳ_{+}^{1}(M), and where ρ_{1}, ρ_{2} : Ω × M → ℳ_{+}^{1}(Θ). More precisely, we would like to establish a bound for (ν_{1}ρ_{1} − ν_{2}ρ_{2})(R) which could be a starting point to implement a selection method similar to the one described in Theorem 2.2.4 (page 93). To this purpose, we can start with Theorem 2.2.1 (page 88), which says that with ℙ probability at least 1 − ϵ,

− N log\{1 − tanh(\frac{λ}{N})(ν_{1}ρ_{1} − ν_{2}ρ_{2})(R)\} ≤ λ(ν_{1}ρ_{1} − ν_{2}ρ_{2})(r)

+ N log[cosh(\frac{λ}{N})](ν_{1}ρ_{1}) ⊗ (ν_{2}ρ_{2})(m′) + 𝒦(ν_{1}, μ̃) + 𝒦(ν_{2}, μ̃)

+ ν_{1}[𝒦(ρ_{1}, π̃)] + ν_{2}[𝒦(ρ_{2}, π̃)] − log(ϵ),

where μ̃ ∈ ℳ_{+}^{1}(M) and π̃ : M → ℳ_{+}^{1}(Θ) are suitably localized prior distributions to be chosen later on. To use these localized prior distributions, we need empirical bounds for the entropy terms 𝒦(ν_{i}, μ̃) and ν_{i}[𝒦(ρ_{i}, π̃)], i = 1, 2.

Bounding ν[𝒦(ρ, π̃)] can be done using the following generalization of Corollary 2.1.19 page 88:

Corollary 2.3.5. For any positive real constants γ and λ such that γ < λ, for any prior distribution \overline{μ} ∈ ℳ_{+}^{1}(M) and any conditional prior distribution π : M → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M), and any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

ν\{𝒦[ρ, π_{exp[−N \frac{γ}{λ} tanh( \frac{λ}{N})R]}]\} ≤ K′(ν, ρ, γ, λ, ϵ) + \frac{1}{\frac{λ}{γ} − 1}𝒦(ν, \overline{μ}),

where

K′(ν, ρ, γ, λ, ϵ) =^{def} (1 − \frac{γ}{λ})^{−1}\{ν[𝒦(ρ, π_{exp(−γr)}]

− \frac{γ}{λ} log(ϵ) + ν\{log[π_{exp(−γr)}(exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]ρ(m′)\})]\}\}.

To apply this corollary to our case, we have to set

π̃ = π_{exp[−N \frac{γ}{λ} tanh( \frac{λ}{N})R]}.

Let us also consider for some positive real constant β the conditional prior distribution

\overline{π} = π_{exp(−βR)}

and the prior distribution

\overline{μ} = μ_{exp[−α\overline{π}(R)]}.

Let us see how we can bound, given any posterior distribution ν : Ω → ℳ_{+}^{1}(M), the divergence 𝒦(ν, \overline{μ}). We can see that

𝒦(ν, \overline{μ}) = α(ν − \overline{μ})\overline{π}(R) + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ).

Now, let us introduce the conditional posterior distribution

π̂ = π_{exp(−γr)}

and let us decompose

(ν − \overline{μ})[\overline{π}(R)] = ν[\overline{π}(R) − π̂(R)] + (ν − \overline{μ})[π̂(R)] + \overline{μ}[π̂(R) − \overline{π}(R)].

Starting from the exponential inequality

ℙ[\overline{μ}[\overline{π} ⊗ \overline{π}] exp\{−N log[1 − tanh(\frac{γ}{N})R′] − γr′ − N log[cosh(\frac{γ}{N})]m′\}] ≤ 1,

and reasoning in the same way that led to Theorem 2.1.1 (page 71) in the simple case when we take in this theorem λ = γ, we get with ℙ probability at least 1 − ϵ, that

− N log\{1 − tanh(\frac{γ}{N})ν(\overline{π} − π̂)(R)\} + βν(\overline{π} − π̂)(R)

≤ ν[log\{π̂[exp\{N log[cosh(\frac{γ}{N})π̂(m′)\}]\}] + 𝒦(ν, \overline{μ}) − log(ϵ).

− N log\{1 − tanh(\frac{γ}{N})\overline{μ}(π̂ − \overline{π})(R)\} − β\overline{μ}(π̂ − \overline{π})(R)

≤ \overline{μ}[log\{π̂[exp\{N log[cosh(\frac{γ}{N})π̂(m′)\}]\}] − log(ϵ).

In the meantime, using Theorem 2.2.1 (page 88) and Corollary 2.3.5 above, we see that with ℙ probability at least 1 − 2ϵ, for any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

− N log\{1 − tanh(\frac{λ}{N})(ν − \overline{μ})ρ(R)\} ≤ λ(ν − \overline{μ})ρ(r)

+ N log[cosh(\frac{λ}{N})](νρ) ⊗ (\overline{μ}ρ)(m′) + (ν + \overline{μ})𝒦(ρ, π̃) + 𝒦(ν, \overline{μ}) − log(ϵ)

≤ λ(ν − \overline{μ})ρ(r) + N log[cosh(\frac{λ}{N})](νρ) ⊗ (\overline{μ}ρ)(m′) + 𝒦(ν, \overline{μ}) − log(ϵ)

+ (1 − \frac{γ}{λ})^{−1}(ν + \overline{μ})\{𝒦(ρ, π̂) + log\{π̂[exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]ρ(m′)\}]\}\}

+ (\frac{λ}{γ} − 1)^{−1}[𝒦(ν, \overline{μ}) − 2 log(ϵ)].

Putting all this together, we see that with ℙ probability at least 1 − 3ϵ, for any posterior distribution ν ∈ ℳ_{+}^{1}(M),

[1 − \frac{α}{N tanh(\frac{γ}{N}) + β} − \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}]𝒦(ν, \overline{μ}) ≤

α[N tanh(\frac{γ}{N}) + β]^{−1}\{ν[log\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] − log(ϵ)\}

+α[N tanh(\frac{γ}{N})−β]^{−1}\{\overline{μ}[log\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}]−log(ϵ)\}

+ α[N tanh(\frac{λ}{N})]^{−1}\{
λ(ν − \overline{μ})π̂(r) + N log[cosh(\frac{λ}{N})](νπ̂) ⊗ (\overline{μ}π̂)(m′)
+ (1 − \frac{γ}{λ})^{−1}(ν + \overline{μ})[log\{π̂[exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]π̂(m′)\}]\}]

− \frac{1 + \frac{γ}{λ}}{1 − \frac{γ}{λ}} log(ϵ)\} + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ).

Replacing in the right-hand side of this inequality the unobserved prior distribution \overline{μ} with the worst possible posterior distribution, we obtain

Theorem 2.3.6. For any positive real constants α, β, γ and λ, using the notation,

\overline{π} = π_{exp(−βR)},

\overline{μ} = μ_{exp[−α\overline{π}(R)]},
π̂ = π_{exp(−γr)},
μ̂ = μ_{exp[−α \frac{λ}{N} tanh( \frac{λ}{N})^{−1}π̂(r)]},

with ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M),

[1 − \frac{α}{N tanh(\frac{γ}{N}) + β} − \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}]𝒦(ν, \overline{μ}) ≤ 𝒦(ν, μ̂)

+ \frac{α}{N tanh(\frac{γ}{N}) + β} \{ν[log\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}]\}

+ \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}\{ν[log\{π̂[exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]π̂(m′)\}]\}]\}

+ log\{μ̂[[π̂\{exp[N log[cosh(\frac{γ}{N})]π̂(m′)]\}]^{\frac{α}{N tanh( \frac{γ}{N} )−β}}

× [π̂\{exp[N \frac{γ}{λ} log[cosh(\frac{λ}{N})]π̂(m′)]\}]^{\frac{α}{N tanh( \frac{λ}{N} )(1− \frac{γ}{λ})}}

× exp[\frac{α log[cosh(\frac{λ}{N})]}{tanh(\frac{λ}{N})}(νπ̂) ⊗ π̂(m′)]]\}

+ [\frac{1}{N tanh(\frac{γ}{N}) + β} + \frac{1}{N tanh(\frac{γ}{N}) − β} + \frac{1 + \frac{γ}{λ}}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}] log(\frac{3}{ϵ}).

This result is satisfactory, but in the same time hints at some possible improvement in the choice of the localized prior \overline{μ}, which is here somewhat lacking a variance term. We will consider in the remainder of this section the use of

\overline{μ} = μ_{exp[−α\overline{π}(R)−ξπ̃⊗π̃(M′)},  (2.38)

where ξ is some positive real constant and π̃ = π_{exp(−β̃R)} is some appropriate

conditional prior distribution with positive real parameter β̃. With this new choice

𝒦(ν, \overline{μ}) = α(ν − \overline{μ})\overline{π}(R) + ξ(ν − \overline{μ})(π̃ ⊗ π̃)(M′) + 𝒦(ν, μ) − 𝒦(\overline{μ}, μ).

We already know how to deal with the first factor α(ν − \overline{μ})\overline{π}(R), since the computations we made to give it an empirical upper bound were valid for any choice of the localized prior distribution \overline{μ}. Let us now deal with ξ(ν−\overline{μ})(π̃⊗π̃)(M′). Since m′(θ, θ′) is a sum of independent Bernoulli random variables, we can easily generalize the result of Theorem 1.1.4 (page 17) to prove that with ℙ probability at least 1 − ϵ

N[1 − exp(−\frac{ζ}{N})]ν(π̃ ⊗ π̃)(M′)

2.3. Two step localization  129

≤ ζΦ_{\frac{ζ}{N}} [ν(π̃ ⊗ π̃)(M′)] ≤ ζν(π̃ ⊗ π̃)(m′) + 𝒦(ν, \overline{μ}) − log(ϵ).

In the same way, with ℙ probability at least 1 − ϵ,

− N[exp(\frac{ζ}{N}) − 1]\overline{μ}(π̃ ⊗ π̃)(M′)

≤ −ζΦ_{− \frac{ζ}{N}} [\overline{μ}(π̃ ⊗ π̃)(M′)] ≤ −ζ\overline{μ}(π̃ ⊗ π̃)(m′) − log(ϵ).

We would like now to replace (π̃ ⊗ π̃)(m′) with an empirical quantity. In order to do this, we will use an entropy bound. Indeed for any conditional posterior distribution ρ : Ω × M → ℳ_{+}^{1}(Θ),

ν[𝒦(ρ, π̃)] = β̃ν(ρ − π̃)(R) + ν[𝒦(ρ, π) − 𝒦(π̃, π)]

≤ \frac{β̃}{N tanh(\frac{γ}{N})}\{γν(ρ − π̃)(r) + N log[cosh(\frac{γ}{N})]ν(ρ ⊗ π̃)(m′)

+ 𝒦(ν, \overline{μ}) + ν[𝒦(ρ, π̃)] − log(ϵ)\} + ν[𝒦(ρ, π) − 𝒦(π̃, π)].

Thus choosing β̃ = N tanh(\frac{γ}{N}),

γν(π̃ − ρ)(r) + ν[𝒦(π̃, π) − 𝒦(ρ, π)]

≤ N log[cosh(\frac{γ}{N})]ν(ρ ⊗ π̃)(m′) + 𝒦(ν, \overline{μ}) − log(ϵ).

Choosing ρ = π̂, we get

ν[𝒦(π̃, π̂)] ≤ N log[cosh(\frac{γ}{N})]ν(π̂ ⊗ π̃)(m′) + 𝒦(ν, \overline{μ}) − log(ϵ). This implies that

ξν(π̂ ⊗ π̃)(m′) = ν{π̃[ξπ̂(m′)] − 𝒦(π̃, π̂)} + ν[𝒦(π̃, π̂)] ≤ ν{log[π̂{exp[ξπ̂(m′)]}]}

+ N log[cosh(\frac{γ}{N})]ν(π̂ ⊗ π̃)(m′) + 𝒦(ν, \overline{μ}) − log(ϵ).

Thus

\{ξ − N log[cosh(\frac{γ}{N})]\}ν(π̂ ⊗ π̃)(m′)

≤ ν\{log[π̂\{exp[ξπ̂(m′)]\}]\} + 𝒦(ν, \overline{μ}) − log(ϵ)

and

ν[𝒦(π̃, π̂)] ≤ (\frac{ξ}{N log[cosh(\frac{γ}{N})]} − 1)^{−1}[ν\{log[π̂\{exp[ξπ̂(m′)]\}]\}

+ 𝒦(ν, \overline{μ}) − log(ϵ)] + 𝒦(ν, \overline{μ}) − log(ϵ).

130  Chapter 2. Comparing posterior distributions to Gibbs priors

Taking for simplicity ξ = 2N log[cosh(\frac{γ}{N})] and noticing that

2N log[cosh(\frac{γ}{N})] = −N log(1 − \frac{β̃^{2}}{N^{2}} ),

we get Theorem 2.3.7. Let us put π̃ = π_{exp(−β̃R)} and π̂ = π_{exp(−γr)}, where γ is some arbitrary positive real constant and β̃ = N tanh(\frac{γ}{N}), so that γ = \frac{N}{2} log(\frac{1+ \frac{β̃}{N}}{1− \frac{β̃}{N}} ). With ℙ probability at least 1 − ϵ,

ν[𝒦(π̃, π̂)] ≤ ν[log\{π̂[exp\{2N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}]+2[𝒦(ν, \overline{μ})−log(ϵ)]. As a consequence

ζν(π̃ ⊗ π̃)(m′) = ζν(π̃ ⊗ π̃)(m′) − ν[𝒦(π̃ ⊗ π̃, π̂ ⊗ π̂)] + 2ν[𝒦(π̃, π̂)]

≤ ν{log[π̂ ⊗ π̂[exp(ζm′)]]} + 2ν[log\{π̂[exp\{2N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] + 4[𝒦(ν, \overline{μ}) − log(ϵ)].

Let us take for the sake of simplicity ζ = 2N log[cosh(\frac{γ}{N})], to get

ζν(π̃ ⊗ π̃)(m′) ≤ 3ν\{log[π̂ ⊗ π̂[exp(ζm′)]]\} + 4[𝒦(ν, \overline{μ}) − log(ϵ)]. This proves Proposition 2.3.8. Let us consider some arbitrary prior distribution \overline{μ} ∈ ℳ_{+}^{1}(M) and some arbitrary conditional prior distribution π : M → ℳ_{+}^{1}(Θ). Let β̃ < N be some positive real constant. Let us put π̃ = π_{exp(−β̃R)} and π̂ = π_{exp(−γr)}, with β̃ = N tanh(\frac{γ}{N}). Moreover let us put ζ = 2N log[cosh(\frac{γ}{N})]. With ℙ probability at least 1−2ϵ, for any posterior distribution ν ∈ ℳ_{+}^{1}(M),

ν(π̃ ⊗ π̃)(M′) ≤ \frac{3ν\{log[π̂ ⊗ π̂[exp(ζm′)]]\} + 5[𝒦(ν, \overline{μ}) − log(ϵ)]}{N[1 − exp(−\frac{ζ}{N})]}

= \frac{1}{N tanh(\frac{γ}{N})^{2}} \{3ν[log\{π̂ ⊗ π̂[exp\{2N log[cosh(\frac{γ}{N})]m′\}]\}]

+ 5[𝒦(ν, \overline{μ}) − log(ϵ)]\}.

In the same way, − ζ\overline{μ}(π̃ ⊗ π̃)(m′) ≤ \overline{μ}\{log[π̂ ⊗ π̂[exp(−ζm′)]]\}

+ 2\overline{μ}[log\{π̂[exp\{2N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] − 4 log(ϵ)

and thus

− \overline{μ}(π̃ ⊗ π̃)(M′) ≤ \frac{1}{N[exp(\frac{ζ}{N}) − 1]}\{\overline{μ}\{log[π̂ ⊗ π̂[exp(−ζm′)]]\}

+ 2\overline{μ}[log\{π̂[exp\{2N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] − 5 log(ϵ)\}.

Here we have purposely kept ζ as an arbitrary positive real constant, to be tuned later (in order to be able to strengthen more or less the compensation of variance terms).

We are now properly equipped to estimate the divergence with respect to \overline{μ}, the choice of prior distribution made in equation (2.38, page 128). Indeed we can now write

[1 − \frac{α}{N tanh(\frac{γ}{N}) + β} − \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})} − \frac{5ξ}{N tanh(\frac{γ}{N})^{2}} ]𝒦(ν, \overline{μ})

≤ \frac{α}{N tanh(\frac{γ}{N}) + β} \{ν[log\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] − log(ϵ)\}

+ \frac{α}{N tanh(\frac{γ}{N}) − β} \{\overline{μ}[log\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] − log(ϵ)\}

+ \frac{α}{N tanh(\frac{λ}{N})}\{

λ(ν − \overline{μ})π̂(r) + N log[cosh(\frac{λ}{N})](νπ̂) ⊗ (\overline{μ}π̂)(m′)

+ (1 − \frac{γ}{λ})^{−1}(ν + \overline{μ})[log\{π̂[exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]π̂(m′)\}]\}]

− \frac{1 + \frac{γ}{N}}{1 − \frac{γ}{N}} log(ϵ)\}

+ \frac{ξ}{N tanh(\frac{γ}{N})^{2}} \{3ν[log\{π̂ ⊗ π̂[exp\{2N log[cosh(\frac{γ}{N})]m′\}]\}] − 5 log(ϵ)\}

+ \frac{ξ}{N[exp(\frac{ζ}{N}) − 1]}\{\overline{μ}\{log[π̂ ⊗ π̂[exp(−ζm′)]]\}

+ 2\overline{μ}[log\{π̂[exp\{2N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}] − 5 log(ϵ)\}.

+ 𝒦(ν, μ) − 𝒦(\overline{μ}, μ).

It remains now only to replace in the right-hand side of this inequality \overline{μ} with the worst possible posterior distribution to obtain

Theorem 2.3.9. Let λ > γ > β, ζ, α and ξ be arbitrary positive real constants. Let us use the notation \overline{π} = π_{exp(−βR)}, π̃ = π_{exp(−N tanh( \frac{γ}{N})R)}, π̂ = π_{exp(−γr)}, \overline{μ} = μ_{exp[−α\overline{π}(R)−ξπ̃⊗π̃(M′)]} and let us define the posterior distribution μ̂ : Ω → ℳ_{+}^{1}(M) by

\frac{dμ̂}{dμ} ∼ exp\{−\frac{αλ}{N tanh(\frac{λ}{N})}π̂(r)

+ \frac{ξ}{N[exp(\frac{ζ}{N}) − 1]} log\{π̂ ⊗ π̂[exp(−ζm′)]\}\}.

Let us assume moreover that

\frac{α}{N tanh(\frac{γ}{N}) + β} + \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})} + \frac{5ξ}{N tanh(\frac{γ}{N})^{2}} < 1.

With ℙ probability at least 1 − ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M),

𝒦(ν, \overline{μ}) ≤ [1 − \frac{α}{N tanh(\frac{γ}{N}) + β}

− \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})} − \frac{5ξ}{N tanh(\frac{γ}{N})^{2}} ]^{−1}\{𝒦(ν, μ̂)

+ \frac{α}{N tanh(\frac{γ}{N}) + β} \{ν[log\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}]\}

+ \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}\{ν[log\{π̂[exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]π̂(m′)\}]\}]\}

+ \frac{ξ}{N tanh(\frac{γ}{N})^{2}} \{3ν[log\{π̂ ⊗ π̂[exp\{2N log[cosh(\frac{γ}{N})]m′\}]\}]\}

+ \frac{ξ}{N[exp(\frac{ζ}{N}) − 1]}\{ν\{log[π̂ ⊗ π̂[exp(−ζm′)]]\}\}

+ log\{μ̂[\{π̂[exp\{N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}^{\frac{α}{N tanh( \frac{γ}{N} )−β}}

× \{π̂[exp\{N \frac{γ}{λ} log[cosh(\frac{λ}{N})]π̂(m′)\}]\}^{\frac{α}{N tanh( \frac{λ}{N} )(1− \frac{γ}{λ})}}

× \{π̂[exp\{2N log[cosh(\frac{γ}{N})]π̂(m′)\}]\}^{\frac{2ξ}{N[exp( \frac{ζ}{N} )−1]}}

× exp\{N log[cosh(\frac{λ}{N})][(νπ̂) ⊗ π̂](m′)\}]\}

+ [\frac{α}{N tanh(\frac{γ}{N}) + β} + \frac{α}{N tanh(\frac{γ}{N}) − β} + \frac{2α(1 + \frac{γ}{N} )}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}

+ \frac{5ξ}{N tanh(\frac{γ}{N})^{2}} + \frac{5ξ}{N[exp(\frac{ζ}{N}) − 1]}] log(\frac{5}{ϵ})\}.

The interest of this theorem lies in the presence of a variance term in the localized posterior distribution μ̂, which with a suitable choice of parameters seems to be an interesting option in the case when there are nested models: in this situation there may be a need to prevent integration with respect to μ̂ in the right-hand side to put weight on wild oversized models with large variance terms. Moreover, the right-hand side being empirical, parameters can be, as usual, optimized from data using a union bound on a grid of candidate values.

If one is only interested in the general shape of the result, a simplified inequality as the one below may suffice:

Corollary 2.3.10. For any positive real constants λ > γ > β, ζ, α and ξ, let us use the same notation as in Theorem 2.3.9 (page 131). Let us put moreover

A_{1} = \frac{α}{N tanh(\frac{γ}{N}) + β} + \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})} + \frac{5ξ}{N tanh(\frac{γ}{N})^{2}} ,
A_{2} = \frac{α}{N tanh(\frac{γ}{N}) + β} + \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})} + \frac{3ξ}{N tanh(\frac{γ}{N})^{2}}
A_{3} = \frac{ξ}{N[exp(\frac{ζ}{N} ) − 1]}
A_{4} = \frac{α}{N tanh(\frac{γ}{N}) − β} + \frac{α}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})} + \frac{2ξ}{N[exp(\frac{ζ}{N}) − 1]},

A_{5} = \frac{α}{N tanh(\frac{γ}{N}) + β} + \frac{α}{N tanh(\frac{γ}{N}) − β} + \frac{2α(1 + \frac{γ}{N} )}{N tanh(\frac{λ}{N})(1 − \frac{γ}{λ})}

+ \frac{5ξ}{N tanh(\frac{γ}{N})^{2}} + \frac{5ξ}{N[exp(\frac{ζ}{N}) − 1]},

C_{1} = 2N log[cosh(\frac{λ}{N} )],
C_{2} = N log[cosh(\frac{λ}{N} )].

Let us assume that A_{1} < 1. With ℙ probability at least 1−ϵ, for any posterior distribution ν : Ω → ℳ_{+}^{1}(M),

𝒦(ν, \overline{μ}) ≤ K(ν, α, β, γ, λ, ξ, ζ, ϵ) =^{def} (1 − A_{1})^{−1}\{𝒦(ν, μ̂)

+ A_{2}ν[log(π̂ ⊗ π̂[exp(C_{1}m′)])] + A_{3}ν[log(π̂ ⊗ π̂[exp(−ζm′ )])]

+ log{μ̂[[π̂(exp[C_{1}π̂(m′)])]^{A_{4}} exp(C_{2}[(νπ̂) ⊗ π̂](m′))]}

+ A_{5} log(\frac{5}{ϵ})\}.

Putting this corollary together with Corollary 2.3.5 (page 125), we obtain Theorem 2.3.11. Let us consider the notation introduced in Corollary 2.3.5 (page 125) and in Theorem 2.3.9 (page 131) and its Corollary 2.3.10 (page 133). Let us consider real positive parameters λ, γ′_{1} < λ′_{1} and γ′_{2} < λ′_{2}. Let us consider also two sets of parameters α_{i}, β_{i}, γ_{i}, λ_{i}, ξ_{i}, ζ_{i},where i = 1, 2, both satisfying the conditions stated in Corollary 2.3.10 (page 133). With ℙ probability at least 1 − ϵ, for any posterior distributions ν_{1}, ν_{2} : Ω → ℳ_{+}^{1}(M), any conditional posterior distributions ρ_{1}, ρ_{2} : Ω × M → ℳ_{+}^{1}(Θ),

− N log\{1 − tanh(\frac{λ}{N} )(ν_{1}ρ_{1} − ν_{2}ρ_{2})(R)\} ≤ λ(ν_{1}ρ_{1} − ν_{2}ρ_{2})(r)

+ N log[cosh(\frac{λ}{N} )](ν_{1}ρ_{1}) ⊗ (ν_{2}ρ_{2})(m′ ) + K′(ν_{1}, ρ_{1}, γ′_{1}, λ′_{1}, \frac{ϵ}{5}) + K′(ν_{2}, ρ_{2}, γ′_{2}, λ′_{2}, \frac{ϵ}{5})

+ \frac{1}{1 − \frac{γ′_{1}}{λ′_{1}}} K(ν_{1}, α_{1}, β_{1}, γ_{1}, λ_{1}, ξ_{1}, ζ_{1}, \frac{ϵ}{5})

+ \frac{1}{1 − \frac{γ′_{2}}{λ′_{2}}} K(ν_{2}, α_{2}, β_{2}, γ_{2}, λ_{2}, ξ_{2}, ζ_{2}, \frac{ϵ}{5}) − log(\frac{ϵ}{5}).

This theorem provides, using a union bound argument to further optimize the parameters, an empirical bound for ν_{1}ρ_{1}(R) − ν_{2}ρ_{2}(R), which can serve to build a selection algorithm exactly in the same way as what was done in Theorem 2.2.4 (page 93). This represents the highest degree of sophistication that we will achieve in this monograph, as far as model selection is concerned: this theorem shows that it is indeed possible to derive a selection scheme in which localization is performed in two steps and in which the localization of the model selection itself, as opposed to the localization of the estimation in each model, includes a variance term as well as a bias term, so that it should be possible to localize the choice of nested models, something that would not have been feasible with the localization techniques exposed in the previous sections of this study. We should point out however that more sophisticated does not necessarily mean more efficient: as the reader may have noticed, sophistication comes at a price, in terms of the complexity of the estimation schemes, with some possible loss of accuracy in the constants that can mar the benefits of using an asymptotically more efficient method for small sample sizes.

We will do the hurried reader a favour: we will not launch into a study of the theoretical properties of this selection algorithm, although it is clear that all the tools needed are at hand !

We would like as a conclusion to this chapter, to put forward a simple idea: this approach of model selection revolves around entropy estimates concerned with the divergence of posterior distributions with respect to localized prior distributions. Moreover, this localization of the prior distribution is more effectively done in several steps in some situations, and it is worth mentioning that these situations include the typical case of selection from a family of parametric models. Finally, the whole story relies upon estimating the relative generalization error rate of one posterior distribution with respect to some local prior distribution as well as with respect to another posterior distribution, because these relative rates can be estimated more accurately than absolute generalization error rates, at least as soon as no classification model of reasonable size provides a good match to the training sample, meaning that the classification problem is either difficult or noisy.

136  Chapter 2. Comparing posterior distributions to Gibbs priors

Transductive PAC-Bayesian learning

3.1. Basic inequalities

3.1.1. The transductive setting. In this chapter the observed sample (X_{i}, Y_{i})_{i=1}^{N} will be supplemented with a test or shadow sample (X_{i}, Y_{i})_{i=N+1}^{(k+1)N}. This point of view, called transductive classification, has been introduced by V. Vapnik. It may be justified in different ways.

On the practical side, one interest of the transductive setting is that it is often a lot easier to collect examples than it is to label them, so that it is not unrealistic to assume that we indeed have two training samples, one labelled and one unlabelled. It also covers the case when a batch of patterns is to be classified and we are allowed to observe the whole batch before issuing the classification.

On the mathematical side, considering a shadow sample proves technically fruitful. Indeed, when introducing the Vapnik–Cervonenkis entropy and Vapnik–Cervonenkis dimension concepts, as well as when dealing with compression schemes, albeit the inductive setting is our final concern, the transductive setting is a useful detour. In this second scenario, intermediate technical results involving the shadow sample are integrated with respect to unobserved random variables in a second stage of the proofs.

Let us describe now the changes to be made to previous notation to adapt them to the transductive setting. The distribution ℙ will be a probability measure on the canonical space Ω = (𝒳×𝒴)^{(k+1)N}, and (X_{i}, Y_{i})_{i=1}^{(k+1)N} will be the canonical process on this space (that is the coordinate process). Unless explicitly mentioned, the parameter k indicating the size of the shadow sample will remain fixed. Assuming the shadow sample size is a multiple of the training sample size is convenient without significantly restricting generality. For a while, we will use a weaker assumption than independence, assuming that ℙ is partially exchangeable, since this is all we need in the proofs.

Definition 3.1.1. For i = 1, …, N, let τ_{i} : Ω → Ω be defined for any ω = (ω_{j})_{j=1}^{(k+1)N} ∈ Ω by

\begin{cases}τ_{i}(ω)_{i+jN} = ω_{i+(j−1)N}, & j = 1, …, k, \\ τ_{i}(ω)_{i} = ω_{i+kN}, \\ and τ_{i}(ω)_{m+jN} = ω_{m+jN}, & m ≠ i, m = 1, …, N, j = 0, … k. \end{cases}

Clearly, if we arrange the (k+1)N samples in a N ×(k+1) array, τ_{i} performs a circular permutation of k +1 entries on the ith row, leaving the other rows unchanged. Moreover, all the circular permutations of the ith row have the form τ_{i}^{j}, j ranging from 0 to k.

The probability distribution ℙ is said to be partially exchangeable if for any i = 1, …, N, ℙ ∘ τ_{i}^{−1} = ℙ.

This means equivalently that for any bounded measurable function h : Ω → ℝ, ℙ(h ∘ τ_{i}) = ℙ(h).

In the same way a function h defined on Ω will be said to be partially exchangeable if h ∘ τ_{i} = h for any i = 1, …, N. Accordingly a posterior distribution ρ : Ω → ℳ_{+}^{1}(Θ, 𝒯) will be said to be partially exchangeable when ρ(ω, A) = ρ[τ_{i}(ω), A], for any ω ∈ Ω, any i = 1, …, N and any A ∈ 𝒯.

For any bounded measurable function h, let us define T_{i}(h) = \frac{1}{k+1} ∑_{j=0}^{k} h ∘ τ_{i}^{j}. Let T (h) = T_{N} ∘ ⋯ ∘ T_{1}(h). For any partially exchangeable probability distribution ℙ, and for any bounded measurable function h, ℙ[T (h)] = ℙ(h). Let us put

σ_{i}(θ) = 𝟙[f_{θ}(X_{i}) ≠ Y_{i}], indicating the success or failure of f_{θ} to predict Y_{i} from X_{i},

r_{1}(θ) = \frac{1}{N} ∑_{i=1}^{N} σ_{i}(θ), the empirical error rate of f_{θ} on the observed sample,

r_{2}(θ) = \frac{1}{kN} ∑_{i=N+1}^{(k+1)N} σ_{i}(θ), the error rate of f_{θ} on the shadow sample,

\overline{r}(θ) = \frac{r_{1}(θ) + kr_{2}(θ)}{k + 1} = \frac{1}{(k + 1)N} ∑_{i=1}^{(k+1)N} σ_{i}(θ), the global error rate of f_{θ},
R_{i}(θ) = ℙ[f_{θ}(X_{i}) ≠ Y_{i}], the expected error rate of f_{θ} on the ith input,

R(θ) = \frac{1}{N} ∑_{i=1}^{N} R_{i}(θ) = ℙ[r_{1}(θ)] = ℙ[r_{2}(θ)], the average expected
error rate of f_{θ} on all inputs.

We will allow for posterior distributions ρ : Ω → ℳ_{+}^{1}(Θ) depending on the shadow sample. The most interesting ones will anyhow be independent of the shadow labels Y_{N+1}, …, Y_{(k+1)N}. We will be interested in the conditional expected error rate of the randomized classification rule described by ρ on the shadow sample, given the observed sample, that is, ℙ[ρ(r_{2})|(X_{i}, Y_{i})_{i=1}^{N}]. This is a natural extension of the notion of generalization error rate: this is indeed the error rate to be expected when the randomized classification rule described by the posterior distribution ρ is applied to the shadow sample (which should in this case more purposefully be called the test sample).

To see the connection with the previously defined generalization error rate, let us comment on the case when ℙ is invariant by any permutation of any row, meaning that

ℙ[h(ω ∘ s)] = ℙ[h(ω)] for all s ∈ 𝔖({i + jN; j = 0, …, k}) and all i = 1, …, N, where 𝔖(A) is the set of permutations of A, extended to {1, …, (k + 1)N} so as to be the identity outside of A. In other words, ℙ is assumed to be invariant under any permutation which keeps the rows unchanged. In this case, if ρ is invariant by any permutation of any row of the shadow sample, meaning that ρ(ω ∘ s) = ρ(ω) ∈ ℳ_{+}^{1}(Θ), s ∈ 𝔖({i + jN; j = 1, …, k}), i = 1, …, N, then ℙ[ρ(r_{2})|(X_{i}, Y_{i})_{i=1}^{N}] = \frac{1}{N} ∑_{i=1}^{N} ℙ[ρ(σ_{i+N})|(X_{i}, Y_{i})_{i=1}^{N}], meaning that the expectation can be taken on a restricted shadow sample of the same size as the observed sample. If moreover the rows are equidistributed, meaning that their marginal distributions are equal, then

ℙ[ρ(r_{2})|(X_{i}, Y_{i})_{i=1}^{N}] = ℙ[ρ(σ_{N+1})|(X_{i}, Y_{i})_{i=1}^{N}].

This means that under these quite commonly fulfilled assumptions, the expectation can be taken on a single new object to be classified, our study thus covers the case when only one of the patterns from the shadow sample is to be labelled and one is interested in the expected error rate of this single labelling. Of course, in the case when ℙ is i.i.d. and ρ depends only on the training sample (X_{i}, Y_{i})_{i=1}^{N}, we fall back on the usual criterion of performance ℙ[ρ(r_{2})|(Z_{i})_{i=1}^{N}] = ρ(R) = ρ(R_{1}).

3.1.2. Absolute bound. Using an obvious factorization, and considering for the moment a fixed value of θ and any partially exchangeable positive real measurable function λ : Ω → ℝ_{+}, we can compute the log-Laplace transform of r_{1} under T, which acts like a conditional probability distribution:

log\{T [exp(−λr_{1})]\} = ∑_{i=1}^{N} log\{T_{i}[exp(−\frac{λ}{N} σ_{i})]\}

≤ N log\{\frac{1}{N} ∑_{i=1}^{N} T_{i}[exp(−\frac{λ}{N} σ_{i})]\} = −λΦ_{\frac{λ}{N}} (\overline{r}),

where the function Φ_{\frac{λ}{N}} was defined by equation (1.1, page 15). Remarking

that T \{exp[λ[Φ_{\frac{λ}{N}} (\overline{r}) − r_{1}]]\} = exp[λΦ_{\frac{λ}{N}} (\overline{r})]T [exp(−λr_{1})] we obtain Lemma 3.1.1. For any θ ∈ Θ and any partially exchangeable positive real measurable function λ : Ω → ℝ_{+},

T \{exp[λ\{Φ_{\frac{λ}{N}} [\overline{r}(θ)] − r_{1}(θ)\}]\} ≤ 1.

We deduce from this lemma a result analogous to the inductive case:

Theorem 3.1.2. For any partially exchangeable positive real measurable function λ : Ω × Θ → ℝ_{+}, for any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} ρ[λ[Φ_{\frac{λ}{N}} (\overline{r}) − r_{1}]] − 𝒦(ρ, π)]\} ≤ 1.

The proof is deduced from the previous lemma, using the fact that π is partially exchangeable:

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} ρ[λ[Φ_{\frac{λ}{N}} (\overline{r}) − r_{1}]] − 𝒦(ρ, π)]\}

= ℙ\{π\{exp[λ[Φ_{\frac{λ}{N}} (\overline{r}) − r_{1}]]\}\} = ℙ\{T π\{exp[λ[Φ_{\frac{λ}{N}} (\overline{r}) − r_{1}]]\}\}

= ℙ\{π\{T exp[λ[Φ_{\frac{λ}{N}} (\overline{r}) − r_{1}]]\}\} ≤ 1.

3.1.3. Relative bounds. Introducing in the same way

m′(θ, θ′) = \frac{1}{N} ∑_{i=1}^{N}|𝟙[f_{θ}(X_{i}) ≠ Y_{i}] − 𝟙[f_{θ′}(X_{i}) ≠ Y_{i}]|

and \overline{m}(θ, θ′) = \frac{1}{(k + 1)N} ∑_{i=1}^{(k+1)N}|𝟙[f_{θ}(X_{i}) ≠ Y_{i}] − 𝟙[f_{θ′}(X_{i}) ≠ Y_{i}]|,

we could prove along the same line of reasoning

Theorem 3.1.3. For any real parameter λ, any θ̃ ∈ Θ, any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} λ[ρ\{Ψ_{\frac{λ}{N}} [\overline{r}(·) − \overline{r}(θ̃), \overline{m}(·, θ̃)]\}

− [ρ(r_{1}) − r_{1}(θ̃)]] − 𝒦(ρ, π)]} ≤ 1,

where the function Ψ_{\frac{λ}{N}} was defined by equation (1.21, page 51).

Theorem 3.1.4. For any real constant γ, for any θ̃ ∈ Θ, for any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} \{−Nρ\{log[1 − tanh(\frac{γ}{N} )[\overline{r}(·) − \overline{r}(θ̃)]]\}

− γ[ρ(r_{1}) − r_{1}(θ̃)] − N log[cosh(\frac{γ}{N} )]ρ[m′(·, θ̃)] − 𝒦(ρ, π)\}]\} ≤ 1.

This last theorem can be generalized to give

Theorem 3.1.5. For any real constant γ, for any partially exchangeable posterior distributions π^{1}, π^{2} : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ_{1},ρ_{2}∈ℳ_{+}^{1}(Θ)} \{−N log\{1 − tanh(\frac{γ}{N} )[ρ_{1}(\overline{r}) − ρ_{2}(\overline{r})]\}

− γ[ρ_{1}(r_{1}) − ρ_{2}(r_{1})] − N log[cosh(\frac{γ}{N} )]ρ_{1} ⊗ ρ_{2}(m′)

− 𝒦(ρ_{1}, π^{1}) − 𝒦(ρ_{2}, π^{2})}]} ≤ 1.

To conclude this section, we see that the basic theorems of transductive PAC-Bayesian classification have exactly the same form as the basic inequalities of inductive classification, Theorems 1.1.4 (page 17), 1.4.2 (page 52) and 1.4.3 (page 53) with R(θ) replaced with \overline{r}(θ), r(θ) replaced with r_{1}(θ) and M′(θ, θ̃) replaced with \overline{m}(θ, θ̃).

Thus all the results of the first two chapters remain true under the hypotheses of transductive classification, with R(θ) replaced with \overline{r}(θ), r(θ) replaced with r_{1}(θ) and M′(θ, θ̃) replaced with \overline{m}(θ, θ̃).

Consequently, in the case when the unlabelled shadow sample is observed, it is possible to improve on the Vapnik bounds to be discussed hereafter by using an explicit partially exchangeable posterior distribution π and resorting to localized or to relative bounds (in the case at least of unlimited computing resources, which of course may still be unrealistic in many real world situations, and with the caveat, to be recalled in the conclusion of this study, that for small sample sizes and comparatively complex classification models, the improvement may not be so decisive).

Let us notice also that the transductive setting when experimentally available, has the advantage that

\overline{d}(θ, θ′) = \frac{1}{(k + 1)N} ∑_{i=1}^{(k+1)N} 𝟙[f_{θ′}(X_{i}) ≠ f_{θ}(X_{i})]

≥ \overline{m}(θ, θ′) ≥ \overline{r}(θ) − \overline{r}(θ′), θ, θ′ ∈ Θ, is observable in this context, providing an empirical upper bound for the difference \overline{r}(θ̂) − ρ(\overline{r}) for any non-randomized estimator θ̂ and any posterior distribution ρ, namely

\overline{r}(θ̂) ≤ ρ(\overline{r}) + ρ[ \overline{d}(·, θ̂)].

Thus in the setting of transductive statistical experiments, the PAC-Bayesian framework provides fully empirical bounds for the error rate of non-randomized estimators θ̂: Ω → Θ, even when using a non-atomic prior π (or more generally a non-atomic partially exchangeable posterior distribution π), even when Θ is not a vector space and even when θ ↦ R(θ) cannot be proved to be convex on the support of some useful posterior distribution ρ.

3.2. Vapnik bounds for transductive classification

In this section, we will stick to plain unlocalized non-relative bounds. As we have already mentioned, (and as it was put forward by Vapnik himself in his seminal works), these bounds are not always superseded by the asymptotically better ones when the sample is of small size: they deserve all our attention for this reason. We will start with the general case of a shadow sample of arbitrary size. We will then discuss the case of a shadow sample of equal size to the training set and the case of a fully exchangeable sample distribution, showing how they can be taken advantage of to sharpen inequalities.

3.2.1. With a shadow sample of arbitrary size. The great thing with the transductive setting is that we are manipulating only r_{1} and \overline{r} which can take only a finite number of values and therefore are piecewise constant on Θ. This makes it possible to derive inequalities that will hold uniformly for any value of the parameter θ ∈ Θ. To this purpose, let us consider for any value θ ∈ Θ of the parameter the subset Δ(θ) ⊂ Θ of parameters θ′ such that the classification rule f_{θ′} answers the same on the extended sample (X_{i})_{i=1}^{(k+1)N} as f_{θ}. Namely, let us put for any θ ∈ Θ

Δ(θ) = {θ′ ∈ Θ; f_{θ′}(X_{i}) = f_{θ}(X_{i}), i = 1, …, (k + 1)N}.

We see immediately that Δ(θ) is an exchangeable parameter subset on which r_{1} and r_{2} and therefore also \overline{r} take constant values. Thus for any θ ∈ Θ we may consider the posterior ρ_{θ} defined by

\frac{dρ_{θ}}{dπ} (θ′) = 𝟙[θ′ ∈ Δ(θ)]π[Δ(θ)]^{−1},

and use the fact that ρ_{θ}(r_{1}) = r_{1}(θ) and ρ_{θ}(\overline{r}) = \overline{r}(θ), to prove that Lemma 3.2.1. For any partially exchangeable positive real measurable function λ : Ω × Θ → ℝ such that

λ(ω, θ′) = λ(ω, θ), θ ∈ Θ, θ′ ∈ Δ(θ), ω ∈ Ω,  (3.1)

and any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

Φ_{\frac{λ}{N}} [\overline{r}(θ)] + \frac{log\{ϵπ[Δ(θ)]\}}{λ(θ)} ≤ r_{1}(θ).

We can then remark that for any value of λ independent of ω, the lefthand side of the previous inequality is a partially exchangeable function of ω ∈ Ω. Thus this left-hand side is maximized by some partially exchangeable function λ, namely

arg max_{λ}\{Φ_{\frac{λ}{N}} [\overline{r}(θ)] + \frac{log\{ϵπ[Δ(θ)]\}}{λ}\}

is partially exchangeable as depending only on partially exchangeable quantities. Moreover this choice of λ(ω, θ) satisfies also condition (3.1) stated in the previous lemma of being constant on Δ(θ), proving

Lemma 3.2.2. For any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ and any λ ∈ ℝ_{+},

Φ_{\frac{λ}{N}} [\overline{r}(θ)] + \frac{log\{ϵπ[Δ(θ)]\}}{λ} ≤ r_{1}(θ).

Writing \overline{r} = \frac{r_{1}+kr_{2}}{k+1} and rearranging terms we obtain

Theorem 3.2.3. For any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

r_{2}(θ) ≤ \frac{k + 1}{k} inf_{λ∈ℝ_{+}} \frac{1 − exp (−\frac{λ}{N} r_{1}(θ) + \frac{log\{ϵπ[Δ(θ)]\}}{N})}{1 − exp(−\frac{λ}{N} )} − \frac{r_{1}(θ)}{k}.

If we have a set of binary classification rules {f_{θ}; θ ∈ Θ} whose Vapnik– Cervonenkis dimension is not greater than h, we can choose π such that

π[Δ(θ)] is independent of θ and not less than (\frac{h}{e(k + 1)N} )^{h}, as will be

proved further on in Theorem 4.2.2 (page 174).

Another important setting where the complexity term − log{π[Δ(θ)]} can easily be controlled is the case of compression schemes, introduced by Little et al. (1986). It goes as follows: we are given for each labelled subsample (X_{i}, Y_{i})_{i∈J}, J ⊂ {1, …, N}, an estimator of the parameter

θ̂[(X_{i}, Y_{i})_{i∈J}] = θ̂_{J}, J ⊂ {1, …, N}, |J| ≤ h,

where

θ̂: ⨆_{k=1}^{N}(𝒳 × 𝒴)^{k} → Θ

is an exchangeable function providing estimators for sub-samples of arbitrary size. Let us assume that θ̂is exchangeable, meaning that for any k = 1, …, N and any permutation σ of {1, …, k}

θ̂[(x_{i}, y_{i})_{i=1}^{k}] = θ̂[(x_{σ(i)}, y_{σ(i)})_{i=1}^{k}], (x_{i}, y_{i})_{i=1}^{k} ∈ (𝒳 × 𝒴)^{k}.

In this situation, we can introduce the exchangeable subset

{θ̂_{J}; J ⊂ {1, …, (k + 1)N}, |J| ≤ h} ⊂ Θ,

which is seen to contain at most

∑_{j=0}^{h} \binom{(k + 1)N}{j} ≤ (\frac{e(k + 1)N}{h} )^{h}

classification rules — as will be proved later on in Theorem 4.2.3 (page 174). Note that we had to extend the range of J to all the subsets of the extended sample, although we will use for estimation only those of the training sample, on which the labels are observed. Thus in this case also we can find a partially exchangeable posterior distribution π such that

π[Δ(θ̂_{J})] ≥ (\frac{h}{e(k + 1)N} )^{h} .

We see that the size of the compression scheme plays the same role in this complexity bound as the Vapnik–Cervonenkis dimension for Vapnik– Cervonenkis classes.

In these two cases of binary classification with Vapnik–Cervonenkis dimension not greater than h and compression schemes depending on a compression set with at most h points, we get a bound of

r_{2}(θ) ≤ \frac{k + 1}{k} inf_{λ∈ℝ_{+}} \frac{1 − exp (−\frac{λ}{N} r_{1}(θ) − \frac{h log (\frac{e(k+1)N}{h} ) − log(ϵ)}{N})}{1 − exp(−\frac{λ}{N} )}

− \frac{r_{1}(θ)}{k}.

Let us make some numerical application: when N = 1000, h = 10, ϵ = 0.01, and inf_{Θ} r_{1} = r_{1}(θ̂) = 0.2, we find that r_{2}(θ̂) ≤ 0.4093, for k between 15 and 17, and values of λ equal respectively to 965, 968 and 971. For k = 1, we find only r_{2}(θ̂) ≤ 0.539, showing the interest of allowing k to be larger than 1.

3.2.2. When the shadow sample has the same size as the training sample. In the case when k = 1, we can improve Theorem 3.1.2 by taking advantage of the fact that T_{i}(σ_{i}) can take only 3 values, namely 0, 0.5 and 1. We see thus that T_{i}(σ_{i}) − Φ_{\frac{λ}{N}} [T_{i}(σ_{i})] can take only two values, 0 and \frac{1}{2} − Φ_{\frac{λ}{N}} (\frac{1}{2}), because Φ_{\frac{λ}{N}} (0) = 0 and Φ_{\frac{λ}{N}} (1) = 1. Thus

T_{i}(σ_{i}) − Φ_{\frac{λ}{N}} [T_{i}(σ_{i})] = [1 − |1 − 2T_{i}(σ_{i})|][\frac{1}{2} − Φ_{\frac{λ}{N}} (\frac{1}{2})].

This shows that in the case when k = 1,

log\{T [exp(−λr_{1})]\} = −λ\overline{r} + \frac{λ}{N} ∑_{i=1}^{N} T_{i}(σ_{i}) − Φ_{\frac{λ}{N}} [T_{i}(σ_{i})]

= −λ\overline{r} + \frac{λ}{N} ∑_{i=1}^{N}[1 − |1 − 2T_{i}(σ_{i})|][\frac{1}{2} − Φ_{\frac{λ}{N}} (\frac{1}{2})]

≤ −λ\overline{r} + λ[\frac{1}{2} − Φ_{\frac{λ}{N}} (\frac{1}{2})][1 − |1 − 2\overline{r}|].

Noticing that \frac{1}{2} − Φ_{\frac{λ}{N}} (\frac{1}{2}) = \frac{N}{λ} log[cosh(\frac{λ}{2N})], we obtain

Theorem 3.2.4. For any partially exchangeable function λ : Ω × Θ → ℝ_{+}, for any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} ρ[λ(\overline{r} − r_{1})

− N log[cosh(\frac{λ}{2N})](1 − |1 − 2\overline{r}|)] − 𝒦(ρ, π)]\} ≤ 1.

As a consequence, reasoning as previously, we deduce

Theorem 3.2.5. In the case when k = 1, for any partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ and any λ ∈ ℝ_{+},

\overline{r}(θ) − \frac{N}{λ} log[cosh(\frac{λ}{2N})](1 − |1 − 2\overline{r}(θ)|) + \frac{log\{ϵπ[Δ(θ)]\}}{λ} ≤ r_{1}(θ);

and consequently for any θ ∈ Θ,

r_{2}(θ) ≤ 2 inf_{λ∈ℝ_{+}} \frac{r_{1}(θ) − \frac{log\{ϵπ[Δ(θ)]\}}{λ}}{1 − \frac{2N}{λ} log[cosh(\frac{λ}{2N})]} − r_{1}(θ).

In the case of binary classification using a Vapnik–Cervonenkis class of Vapnik–Cervonenkis dimension not greater than h, we can choose π such that − log\{π[Δ(θ)]\} ≤ h log(\frac{2eN}{h}) and obtain the following numerical illustration of this theorem: for N = 1000, h = 10, ϵ = 0.01 and inf_{Θ} r_{1} = r_{1}(θ̂) = 0.2, we find an upper bound r_{2}(θ̂) ≤ 0.5033, which improves on Theorem 3.2.3 but still is not under the significance level \frac{1}{2} (achieved by blind random classification). This indicates that considering shadow samples of arbitrary sizes some noisy situations yields a significant improvement on bounds obtained with a shadow sample of the same size as the training sample.

3.2.3. When moreover the distribution of the augmented sample is exchangeable. When k = 1 and ℙ is exchangeable meaning that for any bounded measurable function h : Ω → ℝ and any permutation s ∈ 𝔖({1, …, 2N}) ℙ[h(ω ∘ s)] = ℙ[h(ω)], then we can still improve the bound as follows. Let

T′(h) = \frac{1}{N!} ∑ ; s∈𝔖(\{N+1,...,2N\})h(ω ∘ s).

Then we can write

1 − |1 − 2T_{i}(σ_{i})| = (σ_{i} − σ_{i+N})^{2} = σ_{i} + σ_{i+N} − 2σ_{i}σ_{i+N}.

Using this identity, we get for any exchangeable function λ : Ω × Θ → ℝ_{+},

T \{exp[λ(\overline{r} − r_{1}) − log[cosh(\frac{λ}{2N})] ∑_{i=1}^{N}(σ_{i} + σ_{i+N} − 2σ_{i}σ_{i+N})]\}≤ 1.

Let us put

A(λ) = \frac{2N}{λ} log[cosh(\frac{λ}{2N})],  (3.2)
v(θ) = \frac{1}{2N} ∑_{i=1}^{N}(σ_{i} + σ_{i+N} − 2σ_{i}σ_{i+N}).  (3.3)

With this notation

T \{exp\{λ[\overline{r} − r_{1} − A(λ)v]\}\} ≤ 1.

Let us notice now that

T′[v(θ)] = \overline{r}(θ) − r_{1}(θ)r_{2}(θ).

Let π : Ω → ℳ_{+}^{1}(Θ) be any given exchangeable posterior distribution. Using the exchangeability of ℙ and π and the exchangeability of the exponential function, we get

ℙ\{π[exp\{λ[\overline{r} − r_{1} − A(\overline{r} − r_{1}r_{2})]\}]\} = ℙ\{π[exp\{λ[\overline{r} − r_{1} − AT′(v)]\}]\}

≤ ℙ\{π[T′ exp\{λ[\overline{r} − r_{1} − Av]\}]\} = ℙ\{T′π[exp\{λ[\overline{r} − r_{1} − Av]\}]\}

= ℙ\{π[exp\{λ[\overline{r} − r_{1} − Av]\}]\} = ℙ\{T π[exp\{λ[\overline{r} − r_{1} − Av]\}]\}

= ℙ\{π[T exp\{λ[\overline{r} − r_{1} − Av]\}]\} ≤ 1.

We are thus ready to state

Theorem 3.2.6. In the case when k = 1, for any exchangeable probability distribution ℙ, for any exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), for any exchangeable function λ : Ω × Θ → ℝ_{+},

ℙ\{exp[ sup_{ρ∈ℳ_{+}^{1}(Θ)} ρ\{λ[\overline{r} − r_{1} − A(λ)(\overline{r} − r_{1}r_{2})]\} − 𝒦(ρ, π)]\} ≤ 1,

where A(λ) is defined by equation (3.2, page 146).

We then deduce as previously

Corollary 3.2.7. For any exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), for any exchangeable probability measure ℙ ∈ ℳ_{+}^{1}(Ω), for any measurable exchangeable function λ : Ω × Θ → ℝ_{+}, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

\overline{r}(θ) ≤ r_{1}(θ) + A(λ)[\overline{r}(θ) − r_{1}(θ)r_{2}(θ)] − \frac{log\{ϵπ[Δ(θ)]\}}{λ},

where A(λ) is defined by equation (3.2, page 146).

In order to deduce an empirical bound from this theorem, we have to make some choice for λ(ω, θ). Fortunately, it is easy to show that the bound holds uniformly in λ, because the inequality can be rewritten as a function of only one non-exchangeable quantity, namely r_{1}(θ). Indeed, since r_{2} = 2\overline{r} − r_{1}, we see that the inequality can be written as

\overline{r}(θ) ≤ r_{1}(θ) + A(λ)[\overline{r}(θ) − 2\overline{r}(θ)r_{1}(θ) + r_{1}(θ)^{2}] − \frac{log\{ϵπ[Δ(θ)]}{λ}.

It can be solved in r_{1}(θ), to get

r_{1}(θ) ≥ f(λ, \overline{r}(θ), − log\{ϵπ[Δ(θ)]\}),

where

f(λ, \overline{r}, d) = [2A(λ)]^{−1}\{2\overline{r}A(λ) − 1

+ \sqrt{[1 − 2\overline{r}A(λ)]^{2} + 4A(λ)\{\overline{r}[1 − A(λ)] − \frac{d}{λ}\}}\}.

Thus we can find some exchangeable function λ(ω, θ), such that

f(λ(ω, θ), \overline{r}(θ), − log\{ϵπ[Δ(θ)]\}) = sup_{β∈ℝ_{+}} f(β, \overline{r}(θ), − log\{ϵπ[Δ(θ)]\}).

Applying Corollary 3.2.7 (page 147) to that choice of λ, we see that

Theorem 3.2.8. For any exchangeable probability measure ℙ ∈ ℳ_{+}^{1}(Ω), for any exchangeable posterior probability distribution π : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ, for any λ ∈ ℝ_{+},

\overline{r}(θ) ≤ r_{1}(θ) + A(λ)[\overline{r}(θ) − r_{1}(θ)r_{2}(θ)] − \frac{log\{ϵπ[Δ(θ)]\}}{λ},

where A(λ) is defined by equation (3.2, page 146).

Solving the previous inequality in r_{2}(θ), we get

Corollary 3.2.9. Under the same assumptions as in the previous theorem, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

r_{2}(θ) ≤ inf_{λ∈ℝ_{+}} \frac{r_{1}(θ)\{1 + \frac{2N}{λ} log[cosh(\frac{λ}{2N})]\} − \frac{2 log\{ϵπ[Δ(θ)]\}}{λ}}{1 − \frac{2N}{λ} log[cosh(\frac{λ}{2N})][1 − 2r_{1}(θ)]} .

Applying this to our usual numerical example of a binary classification model with Vapnik–Cervonenkis dimension not greater than h = 10, when N = 1000, inf_{Θ} r_{1} = r_{1}(θ̂) = 10 and ϵ = 0.01, we obtain that r_{2}(θ̂) ≤ 0.4450.

3.3. Vapnik bounds for inductive classification

3.3.1. Arbitrary shadow sample size. We assume in this section that

ℙ = (⨂_{i=1}^{N} P_{i})^{⊗ ∞} ∈ ℳ_{+}^{1}{[(𝒳 × 𝒴)^{N}]^{ℕ} },

where P_{i} ∈ ℳ_{+}^{1}(𝒳×𝒴): we consider an infinite i.i.d. sequence of independent non-identically distributed samples of size N, the first one only being observed. More precisely, under ℙ each sample (X_{i+jN}, Y_{i+jN})_{i=1}^{N} is distributed according to ⨂_{i=1}^{N} P_{i}, and they are all independent from each other. Only the first sample (X_{i}, Y_{i})_{i=1}^{N} is assumed to be observed. The shadow samples will only appear in the proofs. The aim of this section is to prove better Vapnik bounds, generalizing them in the same time to the independent non-i.i.d. setting, which to our knowledge has not been done before.

Let us introduce the notation ℙ′[h(ω)] = ℙ[h(ω) | (X_{i}, Y_{i})_{i=1}^{N}], where h may be any suitable (e.g. bounded) random variable, let us also put Ω = [(𝒳 × 𝒴)^{N}]^{ℕ} .

Definition 3.3.1. For any subset A ⊂ ℕ of integers, let ℭ(A) be the set of circular permutations of the totally ordered set A, extended to a permutation of ℕ by taking it to be the identity on the complement ℕ \ A of A. We will say that a random function h : Ω → ℝ is k-partially exchangeable if

h(ω ∘ s) = h(ω), s ∈ ℭ({i + jN ; j = 0, …, k}), i = 1, …, N.

In the same way, we will say that a posterior distribution π : Ω → ℳ_{+}^{1}(Θ) is k-partially exchangeable if

π(ω ∘ s) = π(ω) ∈ ℳ_{+}^{1}(Θ), s ∈ ℭ({i + jN ; j = 0, …, k}), i = 1, …, N.

Note that ℙ itself is k-partially exchangeable for any k in the sense that for any bounded measurable function h : Ω → ℝ

ℙ[h(ω ∘ s)] = ℙ[h(ω)], s ∈ ℭ({i + jN ; j = 0, …, k}), i = 1, …, N.

Let Δ_{k}(θ) = {θ′ ∈ Θ ; [f_{θ′}(X_{i})]_{i=1}^{(k+1)N} = [f_{θ}(X_{i})]_{i=1}^{(k+1)N}}, θ ∈ Θ, k ∈ ℕ^{∗},

and let also \overline{r}_{k}(θ) = \frac{1}{(k + 1)N} ∑_{i=1}^{(k+1)N} 𝟙[f_{θ}(X_{i}) ≠ Y_{i}]. Theorem 3.1.2 shows

that for any positive real parameter λ and any k-partially exchangeable posterior distribution π_{k} : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[sup_{θ∈Θ} λ[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1}] + log\{ϵπ_{k}[Δ_{k}(θ)]\}]\} ≤ ϵ.

Using the general fact that

ℙ[exp(h)] = ℙ{ℙ′[exp(h)]} ≥ ℙ{exp[ℙ′(h)]},

and the fact that the expectation of a supremum is larger than the supremum of an expectation, we see that with ℙ probability at most 1−ϵ, for any θ ∈ Θ,

ℙ′\{Φ_{\frac{λ}{N}} [\overline{r}_{k}(θ)]\} ≤ r_{1}(θ) − \frac{ℙ′\{log\{ϵπ_{k}[Δ_{k}(θ)]\}\}}{λ}.

For short let us put

d̄_{k}(θ) = − log{ϵπ_{k}[Δ_{k}(θ)]},
d′_{k}(θ) = −ℙ′{log{ϵπ_{k}[Δ_{k}(θ)]}},
d_{k}(θ) = −ℙ{log{ϵπ_{k}[Δ_{k}(θ)]}}.

We can use the convexity of Φ_{\frac{λ}{N}} and the fact that ℙ′(\overline{r}_{k}) = \frac{r_{1}+kR}{k+1}, to establish that

ℙ′\{Φ_{\frac{λ}{N}} [\overline{r}_{k}(θ)]\} ≥ Φ_{\frac{λ}{N}} [\frac{r_{1}(θ) + kR(θ)}{k + 1}] .

Theorem 3.3.1. Using the above hypotheses and notation, for any sequence π_{k} : Ω → ℳ_{+}^{1}(Θ), where π_{k} is a k-partially exchangeable posterior distribution, for any positive real constant λ, any positive integer k, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

Φ_{\frac{λ}{N}} [\frac{r_{1}(θ) + kR(θ)}{k + 1}] ≤ r_{1}(θ) + \frac{d′_{k}(θ)}{λ}.

We can make as we did with Theorem 1.2.6 (page 25) the result of this theorem uniform in λ ∈ {α^{j} ; j ∈ ℕ^{∗}} and k ∈ ℕ^{∗} (considering on k the prior \frac{1}{k(k+1)} and on j the prior \frac{1}{j(j+1)}), and obtain

Theorem 3.3.2. For any real parameter α > 1, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

R(θ) ≤

inf_{k∈ℕ^{∗},j∈ℕ^{∗}} \frac{1 − exp\{−\frac{α^{j}}{N} r_{1}(θ) − \frac{1}{N} \{d′_{k}(θ) + log[k(k + 1)j(j + 1)]\}\}}{\frac{k}{k+1} [1 − exp (−\frac{α^{j}}{N} )]}

− \frac{r_{1}(θ)}{k}.

As a special case we can choose π_{k} such that log{π_{k}[Δ_{k}(θ)]} is independent of θ and equal to log(𝔑_{k}), where

𝔑_{k} = |{[f_{θ}(X_{i})]_{i=1}^{(k+1)N} ; θ ∈ Θ}|

is the size of the trace of the classification model on the extended sample of size (k + 1)N. With this choice, we obtain a bound involving a new flavour of conditional Vapnik entropy, namely

d′_{k}(θ) = ℙ[log(𝔑_{k}) |(Z_{i})_{i=1}^{N}] − log(ϵ).

In the case of binary classification using a Vapnik–Cervonenkis class of Vapnik–Cervonenkis dimension not greater than h = 10, when N = 1000, inf_{Θ} r_{1} = r_{1}(θ̂) = 0.2 and ϵ = 0.01, choosing α = 1.1, we obtain R(θ̂) ≤ 0.4271 (for an optimal value of λ = 1071.8, and an optimal value of k = 16).

3.3.2. A better minimization with respect to the exponential parameter. If we are not pleased with optimizing λ on a discrete subset of the real line, we can use a slightly different approach. From Theorem 3.1.2 (page 140), we see that for any positive integer k, for any k-partially exchangeable positive real measurable function λ : Ω × Θ → ℝ_{+} satisfying equation (3.1, page 143) — with Δ(θ) replaced with Δ_{k}(θ) — for any ϵ ∈)0, 1) and η ∈)0, 1),

ℙ\{ℙ′[exp[sup_{θ} λ[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1}] + log\{ϵηπ_{k}[Δ_{k}(θ)]\}]\} ≤ ϵη,

therefore with ℙ probability at least 1 − ϵ,

ℙ′\{exp[sup_{θ} λ[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1}] + log\{ϵηπ_{k}[Δ_{k}(θ)]\}]\} ≤ η,

and consequently, with ℙ probability at least 1 − ϵ, with ℙ′ probability at least 1 − η, for any θ ∈ Θ,

Φ_{\frac{λ}{N}} (\overline{r}_{k}) + \frac{log\{ϵηπ_{k}[Δ_{k}(θ)]\}}{λ} ≤ r_{1}.

Now we are entitled to choose

λ(ω, θ) ∈ arg max_{λ′∈ℝ_{+}} Φ_{\frac{λ′}{N}} (\overline{r}_{k}) + \frac{log\{ϵηπ_{k}[Δ_{k}(θ)]\}}{λ′}.

This shows that with ℙ probability at least 1 − ϵ, with ℙ′ probability at least 1 − η, for any θ ∈ Θ,

sup_{λ∈ℝ_{+}} Φ_{\frac{λ}{N}} (\overline{r}_{k}) − \frac{d̄_{k}(θ) − log(η)}{λ} ≤ r_{1},

which can also be written

Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1} − \frac{d̄_{k}(θ)}{λ} ≤ −\frac{log(η)}{λ}, λ ∈ ℝ_{+}.

Thus with ℙ probability at least 1 − ϵ, for any θ ∈ Θ, any λ ∈ ℝ_{+},

ℙ′[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1} − \frac{d̄_{k}(θ)}{λ}] ≤ −\frac{log(η)}{λ} + [1 − r_{1} + \frac{log(η)}{λ}]η.

On the other hand, Φ_{\frac{λ}{N}} being a convex function,

ℙ′[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1} − \frac{d̄_{k}(θ)}{λ}] ≥ Φ_{\frac{λ}{N}} [ℙ′(\overline{r}_{k})] − r_{1} − \frac{d′_{k}}{λ}

= Φ_{\frac{λ}{N}} (\frac{kR + r_{1}}{k + 1} ) − r_{1} − \frac{d′_{k}}{λ} .

Thus with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

\frac{kR + r_{1}}{k + 1} ≤ inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}}[r_{1}(1 − η) + η + \frac{d′_{k} − log(η)(1 − η)}{λ}].

We can generalize this approach by considering a finite decreasing sequence η_{0} = 1 > η_{1} > η_{2} > ⋯ > η_{J} > η_{J+1} = 0, and the corresponding sequence of levels

L_{j} = −\frac{log(η_{j})}{λ}, 0 ≤ j ≤ J,

L_{J+1} = 1 − r_{1} − \frac{log(J) − log(ϵ)}{λ}.

Taking a union bound in j, we see that with ℙ probability at least 1 − ϵ, for any θ ∈ Θ, for any λ ∈ ℝ_{+},

ℙ′[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1} − \frac{d̄_{k} + log(J)}{λ} ≥ L_{j}] ≤ η_{j}, j = 0, …, J + 1,

and consequently

ℙ′[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1} − \frac{d̄_{k} + log(J)}{λ}]

≤ ∫_{0}^{L_{J+1}} ℙ′[Φ_{\frac{λ}{N}} (\overline{r}_{k}) − r_{1} − \frac{d̄_{k} + log(J)}{λ} ≥ α]dα ≤ ∑_{j=1}^{J+1} η_{j−1}(L_{j} − L_{j−1})

= η_{J}[1 − r_{1} − \frac{log(J) − log(ϵ) − log(η_{J})}{λ}] − \frac{log(η_{1})}{λ} + ∑_{j=1}^{J−1} \frac{η_{j}}{λ} log(\frac{η_{j}}{η_{j+1}} ).

Let us put

d′′_{k}[θ, (η_{j})_{j=1}^{J}] = d′_{k}(θ) + log(J) − log(η_{1})

+ ∑_{j=1}^{J−1} η_{j} log (\frac{η_{j}}{η_{j+1}} ) + log (\frac{ϵη_{J}}{J} ) η_{J}.

We have proved that for any decreasing sequence (η_{j})_{j=1}^{J}, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

\frac{kR + r_{1}}{k + 1} ≤ inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}}[r_{1}(1 − η_{J}) + η_{J} + \frac{d′′_{k}[θ, (η_{j})_{j=1}^{J}]}{λ}].

Remark 3.3.1. We can for instance choose J = 2, η_{2} = \frac{1}{10N}, η_{1} = \frac{1}{log(10N)}, resulting in

d′′_{k} = d′_{k} + log(2) + log log(10N) + 1 − \frac{log log(10N)}{log(10N)} − \frac{log (\frac{20N}{ϵ} )}{10N}.

In the case where N = 1000 and for any ϵ ∈)0, 1), we get d′′_{k} ≤ d′_{k} + 3.7, in the case where N = 10^{6}, we get d′′_{k} ≤ d′_{k} + 4.4, and in the case N = 10^{9}, we get d′′_{k} ≤ d′_{k} + 4.7.

Therefore, for any practical purpose we could take d′′_{k} = d′_{k} + 4.7 and η_{J} = \frac{1}{10N} in the above inequality.

Taking moreover a weighted union bound in k, we get Theorem 3.3.3. For any ϵ ∈)0, 1), any sequence 1 > η_{1} > ⋯ > η_{J} > 0, any sequence π_{k} : Ω → ℳ_{+}^{1}(Θ), where π_{k} is a k-partially exchangeable posterior distribution, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

R(θ) ≤ inf_{k∈ℕ^{∗}} \frac{k + 1}{k} inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}}[r_{1}(θ) + η_{J}[1 − r_{1}(θ)]

+ \frac{d′′_{k}[θ, (η_{j})_{j=1}^{J}] + log[k(k + 1)]}{λ}] − \frac{r_{1}(θ)}{k}.

Corollary 3.3.4. For any ϵ ∈)0, 1), for any N ≤ 10^{9}, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

R(θ) ≤ inf_{k∈ℕ^{∗}} inf_{λ∈ℝ_{+}} \frac{k + 1}{k}[1 − exp(−\frac{λ}{N})]^{−1}\{1 − exp[−\frac{λ}{N} [r_{1}(θ) + \frac{1}{10N} ]

− \frac{ℙ′[log(𝔑_{k}) | (Z_{i})_{i=1}^{N}] − log(ϵ) + log[k(k + 1)] + 4.7}{N}]\} − \frac{r_{1}(θ)}{k}.

Let us end this section with a numerical example: in the case of binary classification with a Vapnik–Cervonenkis class of dimension not greater than 10, when N = 1000, inf_{Θ} r_{1} = r_{1}(θ̂) = 0.2 and ϵ = 0.01, we get a bound R(θ̂) ≤ 0.4211 (for optimal values of k = 15 and of λ = 1010).

3.3.3. Equal shadow and training sample sizes. In the case when k = 1, we can use Theorem 3.2.5 (page 145) and replace Φ−1_{\frac{λ}{N}}(q) with \{1 −

\frac{2N}{λ} log[cosh(\frac{λ}{2N})]\}^{−1}q, resulting in

Theorem 3.3.5. For any ϵ ∈)0, 1), any N ≤ 10^{9}, any one-partially exchangeable posterior distribution π_{1} : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

R(θ) ≤ inf_{λ∈ℝ_{+}} \frac{\{1 + \frac{2N}{λ} log[cosh(\frac{λ}{2N})]\}r_{1}(θ) + \frac{1}{5N} + 2\frac{d′_{1}(θ) + 4.7}{λ}}{1 − \frac{2N}{λ} log[cosh(\frac{λ}{2N})]} .

3.3.4. Improvement on the equal sample size bound in the i.i.d. case. Finally, in the case when ℙ is i.i.d., meaning that all the P_{i} are equal, we can improve the previous bound. For any partially exchangeable function λ : Ω × Θ → ℝ_{+}, we saw in the discussion preceding Theorem 3.2.6 (page 147) that

T [exp[λ(\overline{r}_{k} − r_{1}) − A(λ)v]] ≤ 1,

with the notation introduced therein. Thus for any partially exchangeable positive real measurable function λ : Ω × Θ → ℝ_{+} satisfying equation (3.1, page 143), any one-partially exchangeable posterior distribution π_{1} : Ω → ℳ_{+}^{1}(Θ),

ℙ\{exp[sup_{θ∈Θ} λ[\overline{r}_{k}(θ) − r_{1}(θ) − A(λ)v(θ)] + log[ϵπ_{1}[Δ(θ)]]\} ≤ 1.

Therefore with ℙ probability at least 1 − ϵ, with ℙ′ probability 1 − η,

\overline{r}_{k}(θ) ≤ r_{1}(θ) + A(λ)v(θ) + \frac{1}{λ}[d̄_{1}(θ) − log(η)].

We can then choose λ(ω, θ) ∈ arg min_{λ′∈ℝ_{+}} A(λ′)v(θ)+ \frac{d̄_{1}(θ) − log(η)]}{λ′}, which

satisfies the required conditions, to show that with ℙ probability at least 1 − ϵ, for any θ ∈ Θ, with ℙ′ probability at least 1 − η, for any λ ∈ ℝ_{+},

\overline{r}_{k}(θ) ≤ r_{1}(θ) + A(λ)v(θ) + \frac{d̄_{1}(θ) − log(η)}{λ}.

We can then take a union bound on a decreasing sequence of J values η_{1} ≥ ⋯ ≥ η_{J} of η. Weakening the order of quantifiers a little, we then obtain the following statement: with ℙ probability at least 1 − ϵ, for any θ ∈ Θ, for any λ ∈ ℝ_{+}, for any j = 1, …, J

ℙ′[\overline{r}_{k}(θ) − r_{1}(θ) − A(λ)v(θ) − \frac{d̄_{1}(θ) + log(J)}{λ} ≥ −\frac{log(η_{j})}{λ}] ≤ η_{j}.

Consequently for any λ ∈ ℝ_{+},

ℙ′[\overline{r}_{k}(θ) − r_{1}(θ) − A(λ)v(θ) − \frac{d̄_{1}(θ) + log(J)}{λ}]

≤ −\frac{log(η_{1})}{λ} + η_{J}[1 − r_{1}(θ) − \frac{log(J) − log(ϵ) − log(η_{J})}{λ}]

+ ∑_{j=1}^{J−1} \frac{η_{j}}{λ} log (\frac{η_{j}}{η_{j+1}} ) .

Moreover ℙ′[v(θ)] = \frac{r_{1}+R}{2} − r_{1}R, (this is where we need equidistribution) thus proving that

\frac{R − r_{1}}{2} ≤ \frac{A(λ)}{2}[R + r_{1} − 2r_{1}R] + \frac{d′′_{1}[θ, (η_{j})_{j=1}^{J}]}{λ} + η_{J}[1 − r_{1}(θ)].

Keeping track of quantifiers, we obtain

Theorem 3.3.6. For any decreasing sequence (η_{j})_{j=1}^{J}, any ϵ ∈)0, 1), any one-partially exchangeable posterior distribution π : Ω → ℳ_{+}^{1}(Θ), with ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

R(θ) ≤ inf_{λ∈ℝ_{+}}

\frac{\{1 + \frac{2N}{λ} log[cosh(\frac{λ}{2N})]\}r_{1}(θ) + \frac{2d′′_{1}[θ, (η_{j})_{j=1}^{J}]}{λ} + 2η_{J}[1 − r_{1}(θ)]}{1 − \frac{2N}{λ} log[cosh(\frac{λ}{2N})][1 − 2r_{1}(θ)]}.

3.4. Gaussian approximation in Vapnik bounds

3.4.1. Gaussian upper bounds of variance terms. To obtain formulas which could be easily compared with original Vapnik bounds, we may replace p − Φ_{a}(p) with a Gaussian upper bound:

Lemma 3.4.1. For any p ∈ (0, \frac{1}{2}), any a ∈ ℝ_{+},

p − Φ_{a}(p) ≤ \frac{a}{2}p(1 − p).

For any p ∈ (\frac{1}{2}, 1),

p − Φ_{a}(p) ≤ \frac{a}{8}.

Proof. Let us notice that for any p ∈ (0, 1),

\frac{∂}{∂a}[−aΦ_{a}(p)] = −\frac{p exp(−a)}{1 − p + p exp(−a)},
\frac{∂^{2}}{∂^{2}a}[−aΦ_{a}(p)] = \frac{p exp(−a)}{1 − p + p exp(−a)} (1 − \frac{p exp(−a)}{1 − p + p exp(−a)})

≤ \begin{cases}p(1 − p) & p ∈ (0, \frac{1}{2}), \\ \frac{1}{4} & p ∈ (\frac{1}{2}, 1). \end{cases}

Thus taking a Taylor expansion of order one with integral remainder:

−aΦ(a) ≤ \begin{cases}\begin{matrix}−ap + ∫_{0}^{a} p(1 − p) & (a − b)db \\ = −ap + \frac{a^{2}}{2} p(1 − p), \end{matrix} & p ∈ (0, \frac{1}{2}), \\ −ap + ∫_{0}^{a} \frac{1}{4}(a − b)db = −ap + \frac{a^{2}}{8} , & p ∈ (\frac{1}{2}, 1). \end{cases}

This ends the proof of our lemma. □

Lemma 3.4.2. Let us consider the bound

B(q, d) = (1 + \frac{2d}{N})^{−1} [q + \frac{d}{N} + \sqrt{\frac{2dq(1 − q)}{N} + \frac{d^{2}}{N^{2}}} ], q ∈ ℝ_{+}, d ∈ ℝ_{+}.

Let us also put

B̄(q, d) = \begin{cases}B(q, d) & B(q, d) ≤ \frac{1}{2}, \\ q + \sqrt{\frac{d}{2N}} & otherwise. \end{cases}

For any positive real parameters q and d

inf_{λ∈ℝ_{+}} Φ−1_{\frac{λ}{N}}(q + \frac{d}{λ}) ≤ B̄(q, d).

Proof. Let p = inf_{λ} Φ−1_{\frac{λ}{N}}(q + \frac{d}{λ} ). For any λ ∈ ℝ_{+},

p − \frac{λ}{2N} (p ∧ \frac{1}{2})[1 − (p ∧ \frac{1}{2})] ≤ Φ_{\frac{λ}{N}} (p) ≤ q + \frac{d}{λ}.

Thus

p ≤ q + inf_{λ∈ℝ_{+}} \frac{λ}{2N} (p ∧ \frac{1}{2})[1 − (p ∧ \frac{1}{2})] + \frac{d}{λ}

= q + \sqrt{\frac{2d(p ∧ \frac{1}{2})[1 − (p ∧ \frac{1}{2})]}{N}} ≤ q + \sqrt{\frac{d}{2N}} .

Then let us remark that B(q, d) = sup \{p′ ∈ ℝ_{+} ; p′ ≤ q + \sqrt{\frac{2dp′(1 − p′)}{N}}\} .

If moreover \frac{1}{2} ≥ B(q, d), then according to this remark \frac{1}{2} ≥ q + \sqrt{\frac{d}{2N}} ≥ p.

Therefore p ≤ \frac{1}{2}, and consequently p ≤ q + \sqrt{\frac{2dp(1−p)}{N}}, implying that p ≤ B(q, d). □

3.4.2. Arbitrary shadow sample size. The previous lemma combined with Corollary 3.3.4 (page 153) implies

Corollary 3.4.3. Let us use the notation introduced in Lemma 3.4.2 (page 155). For any ϵ ∈)0, 1), any integer N ≤ 10^{9}, with ℙ probability at least 1−ϵ, for any θ ∈ Θ,

R(θ) ≤ inf_{k∈ℕ^{∗}} \frac{k + 1}{k}\{B̄[r_{1}(θ) + \frac{1}{10N} , d′_{k}(θ) + log[k(k + 1)] + 4.7]\} − \frac{r_{1}(θ)}{k}.

3.4.3. Equal sample sizes in the i.i.d. case. To make a link with Vapnik’s result, it is useful to state the Gaussian approximation to Theorem 3.3.6 (page 154). Indeed, using the upper bound A(λ) ≤ \frac{λ}{4N}, where A(λ) is defined by equation (3.2) on page 146, we get with ℙ probability at least 1 − ϵ

R − r_{1} − 2η_{J} ≤ inf_{λ∈ℝ_{+}} \frac{λ}{4N} [R + r_{1} − 2r_{1}R] + \frac{2d′′_{1}}{λ} = \sqrt{\frac{2d′′_{1}(R + r_{1} − 2r_{1}R)}{N}},

which can be solved in R to obtain

Corollary 3.4.4. With ℙ probability at least 1 − ϵ, for any θ ∈ Θ,

R(θ) ≤ r_{1}(θ) + \frac{d′′_{1}(θ)}{N}[1 − 2r_{1}(θ)] + 2η_{J}

+\sqrt{\frac{4d′′_{1}(θ)[1 − r_{1}(θ)]r_{1}(θ)}{N} + \frac{d′′_{1}(θ)^{2}}{N^{2}} [1 − 2r_{1}(θ)]^{2} + \frac{4d′′_{1}(θ)}{N}[1 − 2r_{1}(θ)]η_{J}}.

This is to be compared with Vapnik’s result, as proved in Vapnik (1998, page 138):

Theorem 3.4.5 (Vapnik). For any i.i.d. probability distribution ℙ, with ℙ probability at least 1 − ϵ, for any θ ∈ Θ, putting

d_{V} = log[ℙ(𝔑_{1})] + log(4/ϵ),

R(θ) ≤ r_{1}(θ) + \frac{2d_{V}}{N} + \sqrt{\frac{4d_{V} r_{1}(θ)}{N} + \frac{4d_{V}^{2}}{N^{2}}} .

Recalling that we can choose (η_{j})_{j=1}^{2} such that η_{J} = η_{2} = \frac{1}{10N} (which brings a negligible contribution to the bound) and such that for any N ≤ 10^{9},

d′′_{1}(θ) ≤ ℙ[log(𝔑_{1}) | (Z_{i})_{i=1}^{N}] − log(ϵ) + 4.7,

we see that our complexity term is somehow more satisfactory than Vapnik’s, since it is integrated outside the logarithm, with a slightly larger additional constant (remember that log 4 ≃ 1.4, which is better than our 4.7, which could presumably be improved by working out a better sequence η_{j}, but not down to log(4)). Our variance term is better, since we get r_{1}(1−r_{1}), instead

of r_{1}. We also have \frac{d′′_{1}}{N} instead of 2\frac{d_{V}}{N} , because we use no symmetrization trick.

Let us illustrate these bounds on a numerical example, corresponding to a situation where the sample is noisy or the classification model is weak. Let us assume that N = 1000, inf_{Θ} r_{1} = r_{1}(θ̂) = 0.2, that we are performing binary classification with a model with Vapnik–Cervonenkis dimension not greater than h = 10, and that we work at confidence level ϵ = 0.01. Vapnik’s theorem provides an upper bound for R(θ̂) not smaller than 0.610, whereas Corollary 3.4.4 gives R(θ̂) ≤ 0.461 (using the bound d′′_{1} ≤ d′_{1} + 3.7 when N = 1000). Now if we go for Theorem 3.3.6 and do not make a Gaussian approximation, we get R(θ̂) ≤ 0.453. It is interesting to remark that this bound is achieved for λ = 1195 > N = 1000. This explains why the Gaussian approximation in Vapnik’s bound can be improved: for such a large value of λ, λr_{1}(θ) does not behave like a Gaussian random variable.

Let us recall in conclusion that the best bound is provided by Theorem 3.3.3 (page 153), giving R(θ̂) ≤ 0.4211, (that is approximately 2/3 of Vapnik’s bound), for optimal values of k = 15, and of λ = 1010. This bound can be seen to take advantage of the fact that Bernoulli random variables are not Gaussian (its Gaussian approximation, Corollary 3.4.3, gives a bound R(θ) ≃ 0.4325, still with an optimal k = 15), and of the fact that the optimal size of the shadow sample is significantly larger than the size of the observed sample. Moreover, Theorem 3.3.3 does not assume that the sample is i.i.d., but only that it is independent, thus generalizing Vapnik’s bounds to inhomogeneous data (this will presumably be the case when data are collected from different places where the experimental conditions may not be the same, although they may reasonably be assumed to be independent).

Our little numerical example was chosen to illustrate the case when it is non-trivial to decide whether the chosen classifier does better than the 0.5 error rate of blind random classification. This case is of interest to choose “weak learners” to be aggregated or combined in some appropriate way in a second stage to reach a better classification rate. This stage of feature selection is unavoidable in many real world classification tasks. Our little computations are meant to exemplify the fact that Vapnik’s bounds, although asymptotically suboptimal, as is obvious by comparison with the first two chapters, can do the job when dealing with moderate sample sizes.

Support Vector Machines

4.1. How to build them

4.1.1. The canonical hyperplane. Support Vector Machines, of wide use and renown, were conceived by V. Vapkik (Vapnik, 1998). Before introducing them, we will study as a prerequisite the separation of points by hyperplanes in a finite dimensional Euclidean space. Support Vector Machines perform the same kind of linear separation after an implicit change of pattern space. The preceding PAC-Bayesian results provide a fit framework to analyse their generalization properties.

In this section we deal with the classification of points in ℝ^{d} in two classes. Let Z = (x_{i}, y_{i})_{i=1}^{N} ∈ (ℝ^{d} × {−1, +1})^{N} be some set of labelled examples (called the training set hereafter). Let us split the set of indices I = {1, …, N} according to the labels into two subsets

I_{+} = {i ∈ I : y_{i} = +1},
I_{−} = {i ∈ I : y_{i} = −1}.

Let us then consider the set of admissible separating directions

A_{Z} = {w ∈ ℝ^{d} : sup_{b∈ℝ} inf_{i∈I}(⟨w, x_{i}⟩ − b)y_{i} ≥ 1},

which can also be written as

A_{Z} = {w ∈ ℝ^{d} : max_{i∈I_{−}}⟨w, x_{i}⟩ + 2 ≤ min_{i∈I_{+}}⟨w, x_{i}⟩}.

As it is easily seen, the optimal value of b for a fixed value of w, in other words the value of b which maximizes inf_{i∈I}(⟨w, x_{i}⟩ − b)y_{i}, is equal to

b_{w} = \frac{1}{2}[max_{i∈I_{−}}⟨w, x_{i}⟩ + min_{i∈I_{+}}⟨w, x_{i}⟩].

Lemma 4.1.1. When A_{Z} ≠ ∅, inf{∥w∥^{2} : w ∈ A_{Z}} is reached for only one value w_{Z} of w.

Proof. Let w_{0} ∈ A_{Z}. The set A_{Z} ∩ {w ∈ ℝ^{d} : ∥w∥ ≤ ∥w_{0}∥} is a compact convex set and w ↦ ∥w∥^{2} is strictly convex and therefore has a unique minimum on this set, which is also obviously its minimum on A_{Z}. □

Definition 4.1.1. When A_{Z} ≠ ∅, the training set Z is said to be linearly separable. The hyperplane

H = {x ∈ ℝ^{d} : ⟨w_{Z}, x⟩ − b_{Z} = 0},

where

w_{Z} = arg min{∥w∥ : w ∈ A_{Z}},
b_{Z} = b_{w_{Z}},

is called the canonical separating hyperplane of the training set Z. The quantity ∥w_{Z}∥^{−1} is called the margin of the canonical hyperplane.

As min_{i∈I_{+}}⟨w_{Z}, x_{i}⟩ − max_{i∈I_{−}}⟨w_{Z}, x_{i}⟩ = 2, the margin is also equal to half the distance between the projections on the direction w_{Z} of the positive and negative patterns.

4.1.2. Computation of the canonical hyperplane. Let us consider the convex hulls X_{+} and X_{−} of the positive and negative patterns:

𝒳_{+} = {∑_{i∈I_{+}} λ_{i}x_{i} : (λ_{i})_{i∈I_{+}} ∈ ℝ_{+}^{I_{+}}, ∑_{i∈I_{+}} λ_{i} = 1},
𝒳_{−} = {∑_{i∈I_{−}} λ_{i}x_{i} : (λ_{i})_{i∈I_{−}} ∈ ℝ_{+}^{I_{−}}, ∑_{i∈I_{−}} λ_{i} = 1}.

Let us introduce the closed convex set

𝒱 = 𝒳_{+} − 𝒳_{−} = {x_{+} − x_{−} : x_{+} ∈ 𝒳_{+}, x_{−} ∈ 𝒳_{−}}.

As v ↦ ∥v∥^{2} is strictly convex, with compact lower level sets, there is a unique vector v^{∗} such that

∥v^{∗}∥^{2} = inf_{v∈𝒱}{∥v∥^{2} : v ∈ 𝒱}.

Lemma 4.1.2. The set A_{Z} is non-empty (i.e. the training set Z is linearly separable) if and only if v^{∗} ≠ 0. In this case

w_{Z} = \frac{2}{∥v^{∗}∥^{2}} v^{∗},

and the margin of the canonical hyperplane is equal to \frac{1}{2}∥v^{∗}∥.

This lemma proves that the distance between the convex hulls of the positive and negative patterns is equal to twice the margin of the canonical hyperplane.

Proof. Let us assume first that v^{∗} = 0, or equivalently that 𝒳_{+}∩𝒳_{−} ≠ ∅. For any vector w ∈ ℝ^{d},

min_{i∈I_{+}}⟨w, x_{i}⟩ = min_{x∈𝒳_{+}} ⟨w, x⟩,
max_{i∈I_{−}}⟨w, x_{i}⟩ = max_{x∈𝒳_{−}}⟨w, x⟩,

so min_{i∈I_{+}}⟨w, x_{i}⟩ − max_{i∈I_{−}}⟨w, x_{i}⟩ ≤ 0, which shows that w cannot be in A_{Z} and therefore that A_{Z} is empty.

Let us assume now that v^{∗} ≠ 0, or equivalently that 𝒳_{+} ∩ 𝒳_{−} = ∅. Let us put w^{∗} = 2v^{∗}/∥v^{∗}∥^{2}. Let us remark first that

min_{i∈I_{+}}⟨w^{∗}, x_{i}⟩ − max_{i∈I_{−}}⟨w^{∗}, x_{i}⟩ = inf_{x∈𝒳_{+}} ⟨w^{∗}, x⟩ − sup_{x∈𝒳_{−}} ⟨w^{∗}, x⟩

= inf_{x_{+}∈𝒳_{+},x_{−}∈𝒳_{−}} ⟨w^{∗}, x_{+} − x_{−}⟩

= \frac{2}{∥v^{∗}∥^{2}} inf_{v∈𝒱}⟨v^{∗}, v⟩.

Let us now prove that inf_{v∈𝒱} ⟨v^{∗}, v⟩ = ∥v^{∗}∥^{2}. Some arbitrary v ∈ 𝒱 being fixed, consider the function

β ↦ ∥βv + (1 − β)v^{∗}∥^{2} : [0, 1] → ℝ.

By definition of v^{∗}, it reaches its minimum value for β = 0, and therefore has a non-negative derivative at this point. Computing this derivative, we find that ⟨v − v^{∗}, v^{∗}⟩ ≥ 0, as claimed. We have proved that

min_{i∈I_{+}}⟨w^{∗}, x_{i}⟩ − max_{i∈I_{−}}⟨w^{∗}, x_{i}⟩ = 2,

and therefore that w^{∗} ∈ A_{Z}. On the other hand, any w ∈ A_{Z} is such that

2 ≤ min_{i∈I_{+}}⟨w, x_{i}⟩ − max_{i∈I_{−}}⟨w, x_{i}⟩ = inf_{v∈𝒱}⟨w, v⟩ ≤ ∥w∥ inf_{v∈𝒱}∥v∥ = ∥w∥ ∥v^{∗}∥.

This proves that ∥w^{∗}∥ = inf{∥w∥ : w ∈ A_{Z}}, and therefore that w^{∗} = w_{Z} as claimed. □

One way to compute w_{Z} would therefore be to compute v^{∗} by minimizing

{∥∑_{i∈I} λ_{i}y_{i}x_{i}∥^{2} : (λ_{i})_{i∈I} ∈ ℝ_{+}^{I}, ∑_{i∈I} λ_{i} = 2, ∑_{i∈I} y_{i}λ_{i} = 0}.

Although this is a tractable quadratic programming problem, a direct computation of w_{Z} through the following proposition is usually preferred.

Proposition 4.1.3. The canonical direction w_{Z} can be expressed as

w_{Z} = ∑_{i=1}^{N} α_{i}^{∗}y_{i}x_{i},

where (α_{i}^{∗})_{i=1}^{N} is obtained by minimizing

inf{F (α) : α ∈ 𝒜}

where

𝒜 = {(α_{i})_{i∈I} ∈ ℝ_{+}^{I}, ∑_{i∈I} α_{i}y_{i} = 0},

and

F (α) = ∥∑_{i∈I} α_{i}y_{i}x_{i}∥^{2} − 2 ∑_{i∈I} α_{i}.

Proof. Let w(α) = ∑_{i∈I} α_{i}y_{i}x_{i} and let S(α) = \frac{1}{2} ∑_{i∈I} α_{i}. We can express the function F (α) as F (α) = ∥w(α)∥^{2} − 4S(α). Moreover it is important to notice that for any s ∈ ℝ_{+}, {w(α) : α ∈ 𝒜, S(α) = s} = s𝒱. This shows that for any s ∈ ℝ_{+}, inf{F (α) : α ∈ 𝒜, S(α) = s} is reached and that for any α_{s} ∈ {α ∈ 𝒜 : S(α) = s} reaching this infimum, w(α_{s}) = sv^{∗}. As s ↦ s^{2}∥v^{∗}∥^{2} − 4s : ℝ_{+} → ℝ reaches its infimum for only one value s^{∗} of s, namely at s^{∗} = \frac{2}{∥v^{∗}∥^{2}} , this shows that F (α) reaches its infimum on 𝒜, and that for any α^{∗} ∈ 𝒜 such that F (α^{∗}) = inf{F (α) : α ∈ 𝒜}, w(α^{∗}) = \frac{2}{∥v^{∗}∥^{2}} v^{∗} = w_{Z}. □

4.1.3. Support vectors.

Definition 4.1.2. The set of support vectors 𝒮 is defined by

𝒮 = {x_{i} : ⟨w_{Z}, x_{i}⟩ − b_{Z} = y_{i}}.

Proposition 4.1.4. Any α^{∗} minimizing F (α) on 𝒜 is such that

{x_{i} : α_{i}^{∗} > 0} ⊂ 𝒮.

This implies that the representation w_{Z} = w(α^{∗}) involves in general only a limited number of non-zero coefficients and that w_{Z} = w_{Z′}, where Z′ = {(x_{i}, y_{i}) : x_{i} ∈ 𝒮}.

Proof. Let us consider any given i ∈ I_{+} and j ∈ I_{−}, such that α_{i}^{∗} > 0 and α_{j}^{∗} > 0. There exists at least one such index in each set I_{−} and I_{+}, since the sum of the components of α^{∗} on each of these sets are equal and since ∑_{k∈I} α_{k}^{∗} > 0. For any t ∈ ℝ, consider

α_{k}(t) = α_{k}^{∗} + t𝟙(k ∈ {i, j}), k ∈ I.

The vector α(t) is in 𝒜 for any value of t in some neighbourhood of 0, therefore \frac{∂}{∂t}_{|t=0}F [α(t)] = 0. Computing this derivative, we find that

y_{i}⟨w(α^{∗}), x_{i}⟩ + y_{j}⟨w(α^{∗}), x_{j}⟩ = 2.

As y_{i} = −y_{j}, this can also be written as

y_{i}[⟨w(α^{∗}), x_{i}⟩ − b_{Z}] + y_{j}[⟨w(α^{∗}), x_{j}⟩ − b_{Z}] = 2.

As w(α^{∗}) ∈ A_{Z},

y_{k}[⟨w(α^{∗}), x_{k}⟩ − b_{Z}] ≥ 1, k ∈ I,

which implies necessarily as claimed that

y_{i}[⟨w(α^{∗}), x_{i}⟩ − b_{Z}] = y_{j}[⟨w(α^{∗}), x_{j}⟩ − b_{Z}] = 1.

□

4.1.4. The non-separable case. In the case when the training set Z = (x_{i}, y_{i})_{i=1}^{N} is not linearly separable, we can define a noisy canonical hyperplane as follows: we can choose w ∈ ℝ^{d} and b ∈ ℝ to minimize

C(w, b) = ∑_{i=1}^{N}[1 − (⟨w, x_{i}⟩ − b)y_{i}]_{+} + \frac{1}{2}∥w∥^{2},  (4.1)

where for any real number r, r_{+} = max{r, 0} is the positive part of r.

Theorem 4.1.5. Let us introduce the dual criterion

F (α) = ∑_{i=1}^{N} α_{i} − \frac{1}{2}∥∑_{i=1}^{N} y_{i}α_{i}x_{i}∥^{2}

and the domain 𝒜′ = {α ∈ ℝ_{+}^{N} : α_{i} ≤ 1, i = 1, …, N, ∑_{i=1}^{N} y_{i}α_{i} = 0}. Let

α^{∗} ∈ 𝒜′ be such that F (α^{∗}) = sup_{α∈𝒜′} F (α). Let w^{∗} = ∑_{i=1}^{N} y_{i}α_{i}^{∗}x_{i}. There is a threshold b^{∗} (whose construction will be detailed in the proof), such that

C(w^{∗}, b^{∗}) = inf_{w∈ℝ^{d},b∈ℝ} C(w, b).

Corollary 4.1.6. (scaled criterion) For any positive real parameter λ let us consider the criterion

C_{λ}(w, b) = λ^{2} ∑_{i=1}^{N}[1 − (⟨w, x_{i}⟩ − b)y_{i}]_{+} + \frac{1}{2}∥w∥^{2}

and the domain 𝒜′_{λ} = {α ∈ ℝ_{+}^{N} : α_{i} ≤ λ^{2}, i = 1, …, N, ∑_{i=1}^{N} y_{i}α_{i} = 0}. For

any solution α^{∗} of the minimization problem F (α^{∗}) = sup_{α∈𝒜′_{λ}} F (α), the vector w^{∗} = ∑_{i=1}^{N} y_{i}α_{i}^{∗}x_{i} is such that

inf_{b∈ℝ} C_{λ}(w^{∗}, b) = inf_{w∈ℝ^{d},b∈ℝ} C_{λ}(w, b).

In the separable case, the scaled criterion is minimized by the canonical hyperplane for λ large enough. This extension of the canonical hyperplane computation in dual space is often called the box constraint, for obvious reasons. Proof. The corollary is a straightforward consequence of the scale property C_{λ}(w, b, x) = λ^{2}C(λ^{−1}w, b, λx), where we have made the dependence of the criterion in x ∈ ℝ^{dN} explicit. Let us come now to the proof of the theorem.

The minimization of C(w, b) can be performed in dual space extending the couple of parameters (w, b) to \overline{w} = (w, b, γ) ∈ ℝ^{d} × ℝ × ℝ_{+}^{N} and introducing the dual multipliers α ∈ ℝ_{+}^{N} and the criterion

G(α, \overline{w}) = ∑_{i=1}^{N} γ_{i} + ∑_{i=1}^{N} α_{i}\{[1 − (⟨w, x_{i}⟩ − b)y_{i}] − γ_{i}\} + \frac{1}{2}∥w∥^{2}.

We see that

C(w, b) = inf_{γ∈ℝ_{+}^{N}} sup_{α∈ℝ_{+}^{N}} G[α, (w, b, γ)],

and therefore, putting \overline{𝒲} = \{(w, b, γ) : w ∈ ℝ^{d}, b ∈ ℝ, γ ∈ ℝ_{+}^{N}\}, we are led to solve the minimization problem

G(α_{∗}, \overline{w}_{∗}) = inf_{\overline{w}∈\overline{𝒲}} sup_{α∈ℝ_{+}^{N}} G(α, \overline{w}),

whose solution \overline{w}_{∗} = (w_{∗}, b_{∗}, γ_{∗}) is such that C(\overline{w}_{∗}, b_{∗}) = inf_{(w,b)∈ℝ^{d+1}} C(w, b), according to the preceding identity. As for any value of α′ ∈ ℝ_{+}^{N},

inf_{\overline{w}∈\overline{𝒲}} sup_{α∈ℝ_{+}^{N}} G(α, \overline{w}) ≥ inf_{\overline{w}∈\overline{𝒲}} G(α′, \overline{w}),

it is immediately seen that

inf_{\overline{w}∈\overline{𝒲}} sup_{α∈ℝ_{+}^{N}} G(α, \overline{w}) ≥ sup_{α∈ℝ_{+}^{N}} inf_{\overline{w}∈\overline{𝒲}} G(α, \overline{w}).

We are going to show that there is no duality gap, meaning that this inequality is indeed an equality. More importantly, we will do so by exhibiting a saddle point, which, solving the dual minimization problem will also solve the original one.

Let us first make explicit the solution of the dual problem (the interest of this dual problem precisely lies in the fact that it can more easily be solved explicitly). Introducing the admissible set of values of α,

𝒜′ = {α ∈ ℝ^{N} : 0 ≤ α_{i} ≤ 1, i = 1, …, N, ∑_{i=1}^{N} y_{i}α_{i} = 0},

it is elementary to check that

inf_{\overline{w}∈\overline{𝒲}} G(α, \overline{w}) = \begin{cases}inf_{w∈ℝ^{d}} G[α, (w, 0, 0)], & α ∈ 𝒜′, \\ −∞, & otherwise. \end{cases}

As

G[α, (w, 0, 0)] = \frac{1}{2}∥w∥^{2} + ∑_{i=1}^{N} α_{i}(1 − ⟨w, x_{i}⟩y_{i}),

we see that inf_{w∈ℝ^{d}} G[α, (w, 0, 0)] is reached at

w_{α} = ∑_{i=1}^{N} y_{i}α_{i}x_{i}.

This proves that

inf_{\overline{w}∈\overline{𝒲}} G(α, \overline{w}) = F (α).

The continuous map α ↦ inf_{\overline{w}∈\overline{𝒲}} G(α, \overline{w}) reaches a maximum α^{∗}, not necessarily unique, on the compact convex set 𝒜′. We are now going to exhibit a choice of \overline{w}^{∗} ∈ \overline{𝒲} such that (α^{∗}, \overline{w}^{∗}) is a saddle point. This means that we are going to show that

G(α^{∗}, \overline{w}^{∗}) = inf_{\overline{w}∈\overline{𝒲}} G(α^{∗}, \overline{w}) = sup_{α∈ℝ_{+}^{N}} G(α, \overline{w}^{∗}).

It will imply that

inf_{\overline{w}∈\overline{𝒲}} sup_{α∈ℝ_{+}^{d}} G(α, \overline{w}) ≤ sup_{α∈ℝ_{+}^{N}} G(α, \overline{w}^{∗}) = G(α^{∗}, \overline{w}^{∗})

on the one hand and that

inf_{\overline{w}∈\overline{𝒲}} sup_{α∈ℝ_{+}^{d}} G(α, \overline{w}) ≥ inf_{\overline{w}∈\overline{𝒲}} G(α^{∗}, \overline{w}) = G(α^{∗}, \overline{w}^{∗})

on the other hand, proving that

G(α^{∗}, \overline{w}^{∗}) = inf_{\overline{w}∈\overline{𝒲}} sup_{α∈ℝ_{+}^{N}} G(α, \overline{w})

as required. Construction of \overline{w}^{∗}.

• Let us put w^{∗} = w_{α^{∗}}.

• If there is j ∈ {1, …, N} such that 0 < α_{j}^{∗} < 1, let us put

b^{∗} = ⟨x_{j}, w^{∗}⟩ − y_{j}.

Otherwise, let us put

b^{∗} = sup{⟨x_{i}, w^{∗}⟩ − 1 : α_{i}^{∗} > 0, y_{i} = +1, i = 1, …, N}.

• Let us then put

γ_{i}^{∗} = \begin{cases}0, & α_{i}^{∗} < 1, \\ 1 − (⟨w^{∗}, x_{i}⟩ − b^{∗})y_{i}, & α_{i}^{∗} = 1. \end{cases}

If we can prove that

1 − (⟨w^{∗}, x_{i}⟩ − b^{∗})y_{i} \begin{cases}≤ 0, & α_{i}^{∗} = 0, \\ = 0, & 0 < α_{i}^{∗} < 1, \\ ≥ 0, & α_{i}^{∗} = 1, \end{cases}  (4.2)

it will show that γ^{∗} ∈ ℝ_{+}^{N} and therefore that \overline{w}^{∗} = (w^{∗}, b^{∗}, γ^{∗}) ∈ \overline{𝒲}. It will also show that

G(α, \overline{w}^{∗}) = ∑_{i=1}^{N} γ_{i}^{∗} + ∑_{i,α_{i}^{∗}=0} α_{i}[1 − (⟨\overline{w}^{∗}, x_{i}⟩ − b^{∗})y_{i}] + \frac{1}{2}∥\overline{w}^{∗}∥^{2},

proving that G(α^{∗}, \overline{w}^{∗}) = sup_{α∈ℝ_{+}^{N}} G(α, \overline{w}^{∗}). As obviously G(α^{∗}, \overline{w}^{∗}) = G[α^{∗}, (w^{∗}, 0, 0)], we already know that G(α^{∗}, \overline{w}^{∗}) = inf_{\overline{w}∈\overline{𝒲}} G(α^{∗}, \overline{w}). This will show that (α^{∗}, \overline{w}^{∗}) is the saddle point we were looking for, thus ending the proof of the theorem. Proof of equation (4.2). Let us deal first with the case when there is j ∈ {1, …, N} such that 0 < α_{j}^{∗} < 1.

For any i ∈ {1, …, N} such that 0 < α_{i}^{∗} < 1, there is ϵ > 0 such that for any t ∈ (−ϵ, ϵ), α^{∗} + ty_{i}e_{i} − ty_{j}e_{j} ∈ 𝒜′, where (e_{k})_{k=1}^{N} is the canonical base of ℝ^{N}. Thus \frac{∂}{∂t}_{|t=0}F (α^{∗} + ty_{i}e_{i} − ty_{j}e_{j}) = 0. Computing this derivative, we obtain

\frac{∂}{∂t}_{|t=0}F (α^{∗} + ty_{i}e_{i} − ty_{j}e_{j}) = y_{i} − ⟨w^{∗}, x_{i}⟩ + ⟨w^{∗}, x_{j}⟩ − y_{j}
= y_{i}[1 − (⟨w, x_{i}⟩ − b^{∗})y_{i}].

Thus 1−(⟨w, x_{i}⟩−b^{∗})y_{i} = 0, as required. This shows also that the definition of b^{∗} does not depend on the choice of j such that 0 < α_{j}^{∗} < 1.

For any i ∈ {1, …, N} such that α_{i}^{∗} = 0, there is ϵ > 0 such that for any t ∈ (0, ϵ), α^{∗} + te_{i} − ty_{i}y_{j}e_{j} ∈ 𝒜′. Thus \frac{∂}{∂t}_{|t=0}F (α^{∗} + te_{i} − ty_{i}y_{j}e_{j}) ≤ 0, showing that 1 − (⟨w^{∗}, x_{i}⟩ − b^{∗})y_{i} ≤ 0 as required.

For any i ∈ {1, …, N} such that α_{i}^{∗} = 1, there is ϵ > 0 such that α^{∗} − te_{i} + ty_{i}y_{j}e_{j} ∈ 𝒜′. Thus \frac{∂}{∂t}_{|t=0}F (α^{∗} − te_{i} + ty_{i}y_{j}e_{j}) ≤ 0, showing that 1 − (⟨w^{∗}, x_{i}⟩ − b^{∗})y_{i} ≥ 0 as required. This shows that (α^{∗}, \overline{w}^{∗}) is a saddle point in this case.

Let us deal now with the case where α^{∗} ∈ {0, 1}^{N}. If we are not in the trivial case where the vector (y_{i})_{i=1}^{N} is constant, the case α^{∗} = 0 is ruled out. Indeed, in this case, considering α^{∗} + te_{i} + te_{j}, where y_{i}y_{j} = −1, we would get the contradiction 2 = \frac{∂}{∂t}_{|t=0}F (α^{∗} + te_{i} + te_{j}) ≤ 0.

Thus there are values of j such that α_{j}^{∗} = 1, and since ∑_{i=1}^{N} α_{i}y_{i} = 0, both classes are present in the set {j : α_{j}^{∗} = 1}.

Now for any i, j ∈ {1, …, N} such that α_{i}^{∗} = α_{j}^{∗} = 1 and such that y_{i} = +1 and y_{j} = −1, \frac{∂}{∂t}_{|t=0}F (α^{∗} − te_{i} − te_{j}) = −2 + ⟨w^{∗}, x_{i}⟩ − ⟨w^{∗}, x_{j}⟩ ≤ 0. Thus

sup{⟨w^{∗}, x_{i}⟩ − 1 : α_{i}^{∗} = 1, y_{i} = +1} ≤ inf{⟨w^{∗}, x_{j}⟩ + 1 : α_{j}^{∗} = 1, y_{j} = −1},

showing that

1 − (⟨w^{∗}, x_{k}⟩ − b^{∗})y_{k} ≥ 0, α_{k}^{∗} = 1.

Finally, for any i such that α_{i}^{∗} = 0, for any j such that α_{j}^{∗} = 1 and y_{j} = y_{i}, we have

\frac{∂}{∂t}_{|t=0}F (α^{∗} + te_{i} − te_{j}) = y_{i}⟨w^{∗}, x_{i} − x_{j}⟩ ≤ 0,

showing that 1 − (⟨w^{∗}, x_{i}⟩ − b^{∗})y_{i} ≤ 0. This shows that (α^{∗}, \overline{w}^{∗}) is always a saddle point.

4.1.5. Support Vector Machines.

Definition 4.1.3. The symmetric measurable kernel K : 𝒳×𝒳 → ℝ is said to be positive (or more precisely positive semi-definite) if for any n ∈ ℕ, any (x_{i})_{i=1}^{n} ∈ 𝒳^{n},

inf_{α∈ℝ^{n}} ∑_{i=1}^{n} ∑_{j=1}^{n} α_{i}K(x_{i}, x_{j})α_{j} ≥ 0.

Let Z = (x_{i}, y_{i})_{i=1}^{N} be some training set. Let us consider as previously

𝒜 = {α ∈ ℝ_{+}^{N} : ∑_{i=1}^{N} α_{i}y_{i} = 0}.

Let

F (α) = ∑_{i=1}^{N} ∑_{j=1}^{N} α_{i}y_{i}K(x_{i}, x_{j})y_{j}α_{j} − 2 ∑_{i=1}^{N} α_{i}.

Definition 4.1.4. Let K be a positive symmetric kernel. The training set Z is said to be K-separable if

inf{F (α) : α ∈ 𝒜} > −∞.

Lemma 4.1.7. When Z is K-separable, inf{F (α) : α ∈ 𝒜} is reached.

Proof. Consider the training set Z′ = (x′_{i}, y_{i})_{i=1}^{N}, where

x′_{i} = {[{K(x_{k}, x_{ℓ})}_{k=1,ℓ=1}^{N N}]^{1/2}(i, j)}_{j=1}^{N} ∈ ℝ^{N}.

We see that F (α) = ∥∑_{i=1}^{N} α_{i}y_{i}x′_{i}∥^{2} − 2 ∑_{i=1}^{N} α_{i}. We proved in the previous section that Z′ is linearly separable if and only if inf{F (α) : α ∈ 𝒜} > −∞, and that the infimum is reached in this case. □

Proposition 4.1.8. Let K be a symmetric positive kernel and let Z = (x_{i}, y_{i})_{i=1}^{N} be some K-separable training set. Let α^{∗} ∈ 𝒜 be such that F (α^{∗}) = inf{F (α) : α ∈ 𝒜}. Let

I_{−}^{∗} = {i ∈ ℕ : 1 ≤ i ≤ N, y_{i} = −1, α_{i}^{∗} > 0}
I_{+}^{∗} = {i ∈ ℕ : 1 ≤ i ≤ N, y_{i} = +1, α_{i}^{∗} > 0}
b^{∗} = \frac{1}{2}\{∑_{j=1}^{N} α_{j}^{∗}y_{j}K(x_{j}, x_{i_{−}}) + ∑_{j=1}^{N} α_{j}^{∗}y_{j}K(x_{j}, x_{i_{+}})\},  i_{−} ∈ I_{−}^{∗}, i_{+} ∈ I_{+}^{∗},

where the value of b^{∗} does not depend on the choice of i_{−} and i_{+}. The classification rule f : 𝒳 → 𝒴 defined by the formula

f(x) = sign (∑_{i=1}^{N} α_{i}^{∗}y_{i}K(x_{i}, x) − b^{∗})

is independent of the choice of α^{∗} and is called the support vector machine defined by K and Z. The set 𝒮 = {x_{j} : ∑_{i=1}^{N} α_{i}^{∗}y_{i}K(x_{i}, x_{j}) − b^{∗} = y_{j}} is called the set of support vectors. For any choice of α^{∗}, {x_{i} : α_{i}^{∗} > 0} ⊂ 𝒮.

An important consequence of this proposition is that the support vector machine defined by K and Z is also the support vector machine defined by K and Z′ = {(x_{i}, y_{i}) : α_{i}^{∗} > 0, 1 ≤ i ≤ N}, since this restriction of the index set contains the value α^{∗} where the minimum of F is reached.

Proof. The independence of the choice of α^{∗}, which is not necessarily unique, is seen as follows. Let (x_{i})_{i=1}^{N} and x ∈ 𝒳 be fixed. Let us put for ease of notation x_{N+1} = x. Let M be the (N+1)×(N+1) symmetric semi-definite matrix defined by M(i, j) = K(x_{i}, x_{j}), i = 1, …, N + 1, j = 1, …, N + 1. Let us consider the mapping Ψ : {x_{i} : i = 1, …, N + 1} → ℝ^{N+1} defined by

Ψ(x_{i}) = [M^{1/2}(i, j)]_{j=1}^{N+1} ∈ ℝ^{N+1}.  (4.3)

Let us consider the training set Z′ = [Ψ(x_{i}), y_{i}]_{i=1}^{N}. Then Z′ is linearly separable,

F (α) = ∥∑_{i=1}^{N} α_{i}y_{i}Ψ(x_{i})∥^{2} − 2 ∑_{i=1}^{N} α_{i},

and we have proved that for any choice of α^{∗} ∈ 𝒜 minimizing F (α), w_{Z′} = ∑_{i=1}^{N} α_{i}^{∗}y_{i}Ψ(x_{i}). Thus the support vector machine defined by K and Z can also be expressed by the formula

f(x) = sign[⟨w_{Z′}, Ψ(x)⟩ − b_{Z′}]

which does not depend on α^{∗}. The definition of 𝒮 is such that Ψ(𝒮) is the set of support vectors defined in the linear case, where its stated property has already been proved. □

We can in the same way use the box constraint and show that any solution α^{∗} ∈ arg min{F (α) : α ∈ 𝒜, α_{i} ≤ λ^{2}, i = 1, …, N} minimizes

inf_{b∈ℝ} λ^{2} ∑_{i=1}^{N}[1 − (∑_{j=1}^{N} y_{j}α_{j}K(x_{j}, x_{i}) − b)y_{i}]_{+}

+ \frac{1}{2} ∑_{i=1}^{N} ∑_{j=1}^{N} α_{i}α_{j}y_{i}y_{j}K(x_{i}, x_{j}). (4.4)

4.1.6. Building kernels. Except the last, the results of this section are drawn from Cristianini et al. (2000). We have no reference for the last proposition of this section, although we believe it is well known. We include them for the convenience of the reader.

Proposition 4.1.9. Let K_{1} and K_{2} be positive symmetric kernels on 𝒳. Then for any a ∈ ℝ_{+}

(aK_{1} + K_{2})(x, x′) =^{def} aK_{1}(x, x′) + K_{2}(x, x′)
and (K_{1} · K_{2})(x, x′) =^{def} K_{1}(x, x′)K_{2}(x, x′)

are also positive symmetric kernels. Moreover, for any measurable function g : 𝒳 → ℝ, K_{g}(x, x′) =^{def} g(x)g(x′) is also a positive symmetric kernel.

Proof. It is enough to prove the proposition in the case when 𝒳 is finite and kernels are just ordinary symmetric matrices. Thus we can assume without loss of generality that 𝒳 = {1, …, n}. Then for any α ∈ ℝ^{N}, using usual matrix notation,

⟨α, (aK_{1} + K_{2})α⟩ = a⟨α, K_{1}α⟩ + ⟨α, K_{2}α⟩ ≥ 0,
⟨α, (K_{1} · K_{2})α⟩ = ∑_{i,j} α_{i}K_{1}(i, j)K_{2}(i, j)α_{j}

= ∑_{i,j,k} α_{i}K_{1}^{1/2}(i, k)K_{1}^{1/2}(k, j)K_{2}(i, j)α_{j}

= ∑_{k} \underbrace{∑_{i,j}[K_{1}^{1/2}(k, i)α_{i}]K_{2}(i, j)[K_{1}^{1/2}(k, j)α_{j}]}_{≥0} ≥ 0,

⟨α, K_{g}α⟩ = ∑_{i,j} α_{i}g(i)g(j)α_{j} = (∑_{i} α_{i}g(i))^{2} ≥ 0.

□

Proposition 4.1.10. Let K be some positive symmetric kernel on 𝒳. Let p : ℝ → ℝ be a polynomial with positive coefficients. Let g : 𝒳 → ℝ^{d} be a measurable function. Then

p(K)(x, x′) =^{def} p[K(x, x′)],
exp(K)(x, x′) =^{def} exp[K(x, x′)]
and G_{g}(x, x′) =^{def} exp(−∥g(x) − g(x′)∥^{2})

are all positive symmetric kernels.

Proof. The first assertion is a direct consequence of the previous proposition. The second comes from the fact that the exponential function is the pointwise limit of a sequence of polynomial functions with positive coefficients. The third is seen from the second and the decomposition

G_{g}(x, x′) = [exp(−∥g(x)∥^{2}) exp(−∥g(x′)∥^{2})] exp[2⟨g(x), g(x′)⟩]

□

Proposition 4.1.11. With the notation of the previous proposition, any training set Z = (x_{i}, y_{i})_{i=1}^{N} ∈ (𝒳 × {−1, +1})^{N} is G_{g}-separable as soon as g(x_{i}), i = 1, …, N are distinct points of ℝ^{d}.

Proof. It is clearly enough to prove the case when 𝒳 = ℝ^{d} and g is the identity. Let us consider some other generic point x_{N+1} ∈ ℝ^{d} and define Ψ as in (4.3). It is enough to prove that Ψ(x_{1}), …, Ψ(x_{N}) are affine independent, since the simplex, and therefore any affine independent set of points, can be split in any arbitrary way by affine half-spaces. Let us assume that (x_{1}, …, x_{N}) are affine dependent; then for some (λ_{1}, …, λ_{N}) ≠ 0 such that ∑_{i=1}^{N} λ_{i} = 0,

∑_{i=1}^{N} ∑_{j=1}^{N} λ_{i}G(x_{i}, x_{j})λ_{j} = 0.

Thus, (λ_{i})_{i=1}^{N+1}, where we have put λ_{N+1} = 0 is in the kernel of the symmetric positive semi-definite matrix G(x_{i}, x_{j})_{i,j∈{1,...,N+1}}. Therefore

∑_{i=1}^{N} λ_{i}G(x_{i}, x_{N+1}) = 0,

for any x_{N+1} ∈ ℝ^{d}. This would mean that the functions x ↦ exp(−∥x−x_{i}∥^{2}) are linearly dependent, which can be easily proved to be false. Indeed, let n ∈ ℝ^{d} be such that ∥n∥ = 1 and ⟨n, x_{i}⟩, i = 1, …, N are distinct (such a vector exists, because it has to be outside the union of a finite number of hyperplanes, which is of zero Lebesgue measure on the sphere). Let us assume for a while that for some (λ_{i})_{i=1}^{N} ∈ ℝ^{N}, for any x ∈ ℝ^{d},

∑_{i=1}^{N} λ_{i} exp(−∥x − x_{i}∥^{2}) = 0.

Considering x = tn, for t ∈ ℝ, we would get

∑_{i=1}^{N} λ_{i} exp(2t⟨n, x_{i}⟩ − ∥x_{i}∥^{2}) = 0, t ∈ ℝ.

Letting t go to infinity, we see that this is only possible if λ_{i} = 0 for all values of i. □

4.2. Bounds for Support Vector Machines

4.2.1. Compression scheme bounds. We can use Support Vector Machines in the framework of compression schemes and apply Theorem 3.3.3 (page 153). More precisely, given some positive symmetric kernel K on 𝒳, we may consider for any training set Z′ = (x′_{i}, y′_{i})_{i=1}^{h} the classifier f̂_{Z′} : 𝒳 → 𝒴 which is equal to the Support Vector Machine defined by K and Z′ whenever Z′ is K-separable, and which is equal to some constant classification rule otherwise; we take this convention to stick to the framework described on page 144, we will only use f̂_{Z′} in the K-separable case, so this extension of the definition is just a matter of presentation. In the application of Theorem 3.3.3 in the case when the observed sample (X_{i}, Y_{i})_{i=1}^{N} is K-separable, a natural if perhaps sub-optimal choice of Z′ is to choose for (x′_{i}) the set of support vectors defined by Z = (X_{i}, Y_{i})_{i=1}^{N} and to choose for (y′_{i}) the corresponding values of Y. This is justified by the fact that f̂_{Z} = f̂_{Z′}, as shown in Proposition 4.1.8 (page 168). If Z is not K-separable, we can train a Support Vector Machine with the box constraint, then remove all the errors to obtain a Kseparable sub-sample Z′ = {(X_{i}, Y_{i}) : α_{i}^{∗} < λ^{2}, 1 ≤ i ≤ N}, using the same notation as in equation (4.4) on page 169, and then consider its support vectors as the compression set. Still using the notation of page 169, this means we have to compute successively α^{∗} ∈ arg min{F (α) : α ∈ 𝒜, α_{i} ≤ λ^{2}}, and α^{∗∗} ∈ arg min{F (α) : α ∈ 𝒜, α_{i} = 0 when α_{i}^{∗} = λ^{2}}, to keep the compression set indexed by J = {i : 1 ≤ i ≤ N, α_{i}^{∗∗} > 0}, and the corresponding Support Vector Machine f̂_{J}. Different values of λ can be used at this stage, producing different candidate compression sets: when λ increases, the number of errors should decrease, on the other hand when λ decreases, the margin ∥w∥^{−1} of the separable subset Z′ increases, supporting the hope for a smaller set of support vectors, thus we can use λ to monitor the number of errors on the training set we accept from the compression scheme. As we can use whatever heuristic we want while selecting the compression set, we can also try to threshold in the previous construction α_{i}^{∗∗} at different levels η ≥ 0, to produce candidate compression sets J_{η} = {i : 1 ≤ i ≤ N, α_{i}^{∗∗} > η} of various sizes.

As the size |J| of the compression set is random in this construction, we must use a version of Theorem 3.3.3 (page 153) which handles compression sets of arbitrary sizes. This is done by choosing for each k a k-partially exchangeable posterior distribution π_{k} which weights the compression sets of all dimensions. We immediately see that we can choose π_{k} such that − log[π_{k}(Δ_{k}(J))] ≤ log[|J|(|J| + 1)] + |J| log[\frac{(k+1)eN}{|J|} ].

If we observe the shadow sample patterns, and if computer resources permit, we can of course use more elaborate bounds than Theorem 3.3.3, such as the transductive equivalent for Theorem 1.3.15 (page 46) (where we may consider the submodels made of all the compression sets of the same size). Theorems based on relative bounds, such as Theorem 2.2.4 (page 93) or Theorem 2.3.9 (page 131) can also be used. Gibbs distributions can be approximated by Monte Carlo techniques, where a Markov chain with the proper invariant measure consists in appropriate local perturbations of the compression set.

Let us mention also that the use of compression schemes based on Support Vector Machines can be tailored to perform some kind of feature aggregation. Imagine that the kernel K is defined as the scalar product in L_{2}(π), where π ∈ ℳ_{+}^{1}(Θ). More precisely let us consider for some set of soft classification rules {f_{θ} : 𝒳 → ℝ; θ ∈ Θ} the kernel

K(x, x′) = ∫_{θ∈Θ} f_{θ}(x)f_{θ}(x′)π(dθ).

In this setting, the Support Vector Machine applied to the training set Z = (x_{i}, y_{i})_{i=1}^{N} has the form

f_{Z}(x) = sign (∫_{θ∈Θ} f_{θ}(x) ∑_{i=1}^{N} y_{i}α_{i}f_{θ}(x_{i})π(dθ) − b)

and, if this is too burdensome to compute, we can replace it with some finite approximation

f̃_{Z}(x) = sign (\frac{1}{m} ∑_{k=1}^{m} f_{θ_{k}}(x)w_{k} − b) ,

where the set {θ_{k}, k = 1, …, m} and the weights {w_{k}, k = 1, …, m} are computed in some suitable way from the set Z′ = (x_{i}, y_{i})_{i,α_{i}>0} of support vectors of f_{Z}. For instance, we can draw {θ_{k}, k = 1, …, m} at random according to the probability distribution proportional to

|∑_{i=1}^{N} y_{i}α_{i}f_{θ}(x_{i})| π(dθ),

define the weights w_{k} by

w_{k} = sign (∑_{i=1}^{N} y_{i}α_{i}f_{θ_{k}}(x_{i})) ∫_{θ∈Θ} |∑_{i=1}^{N} y_{i}α_{i}f_{θ}(x_{i})| π(dθ),

and choose the smallest value of m for which this approximation still classifies Z′ without errors. Let us remark that we have built f̃_{Z} in such a way that

lim_{m→+∞} f̃_{Z}(x_{i}) = f_{Z}(x_{i}) = y_{i}, a.s.

for any support index i such that α_{i} > 0.

Alternatively, given Z′, we can select a finite set of features Θ′ ⊂ Θ such that Z′ is K_{Θ′} separable, where K_{Θ′}(x, x′) = ∑_{θ∈Θ′} f_{θ}(x)f_{θ}(x′) and consider the Support Vector Machines f_{Z′} built with the kernel K_{Θ′}. As soon as Θ′ is chosen as a function of Z′ only, Theorem 3.3.3 (page 153) applies and provides some level of confidence for the risk of f_{Z′}.

4.2.2. The Vapnik–Cervonenkis dimension of a family of subsets. Let us consider some set X and some set S ⊂ {0, 1}^{X} of subsets of X. Let h(S) be the Vapnik–Cervonenkis dimension of S, defined as

h(S) = max{|A| : A ⊂ X, |A| < ∞ and A ∩ S = {0, 1}^{A}},

where by definition A∩S = {A∩B : B ∈ S} and |A| is the number of points in A. Let us notice that this definition does not depend on the choice of the reference set X. Indeed X can be chosen to be ⋃ S, the union of all the sets in S or any bigger set. Let us notice also that for any set B, h(B∩S) ≤ h(S), the reason being that A ∩ (B ∩ S) = B ∩ (A ∩ S).

This notion of Vapnik–Cervonenkis dimension is useful because, as we will see for Support Vector Machines, it can be computed in some important special cases. Let us prove here as an illustration that h(S) = d + 1 when X = ℝ^{d} and S is made of all the half spaces:

S = {A_{w,b} : w ∈ ℝ^{d}, b ∈ ℝ}, where A_{w,b} = {x ∈ X : ⟨w, x⟩ ≥ b}.

Proposition 4.2.1. With the previous notation, h(S) = d + 1.

Proof. Let (e_{i})_{i=1}^{d+1} be the canonical base of ℝ^{d+1}, and let X be the affine subspace it generates, which can be identified with ℝ^{d}. For any (ϵ_{i})_{i=1}^{d+1} ∈ {−1, +1}^{d+1}, let w = ∑_{i=1}^{d+1} ϵ_{i}e_{i} and b = 0. The half space A_{w,b} ∩ X is such that {e_{i}; i = 1, …, d + 1} ∩ (A_{w,b} ∩ X) = {e_{i}; ϵ_{i} = +1}. This proves that h(S) ≥ d + 1.

To prove that h(S) ≤ d+1, we have to show that for any set A ⊂ ℝ^{d} of size |A| = d+2, there is B ⊂ A such that B ∉ (A∩S). Obviously this will be the case if the convex hulls of B and A\B have a non-empty intersection: indeed if a hyperplane separates two sets of points, it also separates their convex hulls. As |A| > d + 1, A is affine dependent: there is (λ_{x})_{x∈A} ∈ ℝ^{d+2} \ {0} such that ∑_{x∈A} λ_{x}x = 0 and ∑_{x∈A} λ_{x} = 0. The set B = {x ∈ A : λ_{x} > 0} and its complement A \ B are non-empty, because ∑_{x∈A} λ_{x} = 0 and λ ≠ 0. Moreover ∑_{x∈B} λ_{x} = ∑_{x∈A\B} −λ_{x} > 0. The relation

\frac{1}{∑_{x∈B} λ_{x}} ∑_{x∈B} λ_{x}x = \frac{1}{∑_{x∈B} λ_{x}} ∑_{x∈A\B} −λ_{x}x

shows that the convex hulls of B and A \ B have a non-void intersection. □ Let us introduce the function of two integers

Φ_{n}^{h} = ∑_{k=0}^{h} \binom{n}{k},

which can alternatively be defined by the relations

Φ_{n}^{h} = \begin{cases}2^{n} & when n ≤ h, \\ Φ_{n−1}^{h−1} + Φ_{n−1}^{h} & when n > h. \end{cases}

Theorem 4.2.2. Whenever ⋃ S is finite,

|S| ≤ Φ (|⋃ S| , h(S)) .

Theorem 4.2.3. For any h ≤ n,

Φ_{n}^{h} ≤ exp[nH(\frac{h}{n})] ≤ exp[h(log(\frac{n}{h}) + 1)],

where H(p) = −p log(p) − (1 − p) log(1 − p) is the Shannon entropy of the Bernoulli distribution with parameter p.

Proof of theorem 4.2.2. Let us prove this theorem by induction on |⋃ S|. It is easy to check that it holds true when |⋃ S| = 1. Let X = ⋃ S, let x ∈ X and X′ = X \ {x}. Define (△ denoting the symmetric difference of two sets)

S′ = {A ∈ S : A △ {x} ∈ S},

S′′ = {A ∈ S : A △ {x} ∉ S}.

Clearly, ⊔ denoting the disjoint union, S = S′ ⊔ S′′ and S ∩ X′ = (S′ ∩ X′) ⊔ (S′′ ∩ X′). Moreover |S′| = 2|S′ ∩ X′| and |S′′| = |S′′ ∩ X′|. Thus

|S| = |S′| + |S′′| = 2|S′ ∩ X′| + |S′′| = |S ∩ X′| + |S′ ∩ X′|.

Obviously h(S ∩ X′) ≤ h(S). Moreover h(S′ ∩ X′) = h(S′) − 1, because if A ⊂ X′ is shattered by S′ (or equivalently by S′ ∩ X′), then A ∪ {x} is shattered by S′ (we say that A is shattered by S when A ∩ S = {0, 1}^{A}). Using the induction hypothesis, we then see that |S ∩X′| ≤ Φ_{|X′|}^{h(S)} +Φ_{|X′|}^{h(S)−1}.

But as |X′| = |X|−1, the right-hand side of this inequality is equal to Φ_{|X|}^{h(S)}, according to the recurrence equation satisfied by Φ.

Proof of theorem 4.2.3: This is the well-known Chernoff bound for the deviation of sums of Bernoulli random variables: let (σ_{1}, …, σ_{n}) be i.i.d. Bernoulli random variables with parameter 1/2. Let us notice that

Φ_{n}^{h} = 2^{n}ℙ (∑_{i=1}^{n} σ_{i} ≤ h) .

For any positive real number λ ,

ℙ(∑_{i=1}^{n} σ_{i} ≤ h) ≤ exp(λh)𝔼[exp(−λ ∑_{i=1}^{n} σ_{i})]
= exp{λh + n log{𝔼[exp(−λσ_{1})]}}.

Differentiating the right-hand side in λ shows that its minimal value is exp[−n𝒦(\frac{h}{n}, \frac{1}{2})], where 𝒦(p, q) = p log(\frac{p}{q}) + (1 − p) log(\frac{1−p}{1−q}) is the Kullback divergence function between two Bernoulli distributions B_{p} and B_{q} of parameters p and q. Indeed the optimal value λ^{∗} of λ is such that

h = n\frac{𝔼[σ_{1} exp(−λ^{∗}σ_{1})]}{𝔼[exp(−λ^{∗}σ_{1})]} = nB_{h/n}(σ_{1}).

Therefore, using the fact that two Bernoulli distributions with the same expectations are equal,

log\{𝔼[exp(−λ^{∗}σ_{1})]\} = −λ^{∗}B_{h/n}(σ_{1}) − 𝒦(B_{h/n}, B_{1/2}) = −λ^{∗} \frac{h}{n} − 𝒦(\frac{h}{n}, \frac{1}{2}).

The announced result then follows from the identity

H(p) = log(2) − 𝒦(p, \frac{1}{2})

= p log(p^{−1}) + (1 − p) log(1 + \frac{p}{1 − p}) ≤ p[log(p^{−1}) + 1].

4.2.3. Vapnik–Cervonenkis dimension of linear rules with margin. The proof of the following theorem was suggested to us by a similar proof presented in Cristianini et al. (2000).

Theorem 4.2.4. Consider a family of points (x_{1}, …, x_{n}) in some Euclidean vector space E and a family of affine functions

ℋ = {g_{w,b} : E → ℝ; w ∈ E, ∥w∥ = 1, b ∈ ℝ},

where

g_{w,b}(x) = ⟨w, x⟩ − b, x ∈ E.

Assume that there is a set of thresholds (b_{i})_{i=1}^{n} ∈ ℝ^{n} such that for any (y_{i})_{i=1}^{n} ∈ {−1, +1}^{n}, there is g_{w,b} ∈ ℋ such that

inf_{i=1}^{n}(g_{w,b}(x_{i}) − b_{i})y_{i} ≥ γ.

Let us also introduce the empirical variance of (x_{i})_{i=1}^{n},

𝕍ar(x_{1}, …, x_{n}) = \frac{1}{n} ∑_{i=1}^{n}∥x_{i} − \frac{1}{n} ∑_{j=1}^{n} x_{j}∥^{2}.

In this case and with this notation,

\frac{𝕍ar(x_{1}, …, x_{n})}{γ^{2}} ≥ \begin{cases}n − 1 & when n is even, \\ (n − 1)\frac{n^{2}−1}{n^{2}} & when n is odd. \end{cases}  (4.5)

Moreover, equality is reached when γ is optimal, b_{i} = 0, i = 1, …, n and (x_{1}, …, x_{n}) is a regular simplex (i.e. when 2γ is the minimum distance between the convex hulls of any two subsets of {x_{1}, …, x_{n}} and ∥x_{i} − x_{j}∥ does not depend on i ≠ j).

Proof. Let (s_{i})_{i=1}^{n} ∈ ℝ^{n} be such that ∑_{i=1}^{n} s_{i} = 0. Let σ be a uniformly distributed random variable with values in 𝔖_{n}, the set of permutations of the first n integers {1, …, n}. By assumption, for any value of σ, there is an affine function g_{w,b} ∈ ℋ such that

min_{i=1,...,n}[g_{w,b}(x_{i}) − b_{i}][2𝟙(s_{σ(i)} > 0) − 1] ≥ γ.

As a consequence

⟨∑_{i=1}^{n} s_{σ(i)}x_{i}, w⟩ = ∑_{i=1}^{n} s_{σ(i)}(⟨x_{i}, w⟩ − b − b_{i}) + ∑_{i=1}^{n} s_{σ(i)}b_{i}

≥ ∑_{i=1}^{n} γ|s_{σ(i)}| + s_{σ(i)}b_{i}.

4.2. Bounds for Support Vector Machines  177

Therefore, using the fact that the map x ↦ (max{0, x})^{2} is convex,

𝔼 (∥∑_{i=1}^{n} s_{σ(i)}x_{i}∥^{2}) ≥ 𝔼 [(max {0, ∑_{i=1}^{n} γ|s_{σ(i)}| + s_{σ(i)}b_{i}})^{2}]

≥ (max {0, ∑_{i=1}^{n} γ𝔼(|s_{σ(i)}|) + 𝔼(s_{σ(i)})b_{i}})^{2} = γ^{2} (∑_{i=1}^{n}|s_{i}|)^{2} ,

where 𝔼 is the expectation with respect to the random permutation σ. On the other hand

𝔼 (∥∑_{i=1}^{n} s_{σ(i)}x_{i}∥^{2}) = ∑_{i=1}^{n} 𝔼(s_{σ(i)}^{2})∥x_{i}∥^{2} + ∑_{i≠j} 𝔼(s_{σ(i)}s_{σ(j)})⟨x_{i}, x_{j}⟩.

Moreover

𝔼(s_{σ(i)}^{2}) = \frac{1}{n}𝔼 (∑_{i=1}^{n} s_{σ(i)}^{2}) = \frac{1}{n} ∑_{i=1}^{n} s_{i}^{2}.

In the same way, for any i ≠ j,

𝔼 (s_{σ(i)}s_{σ(j)}) = \frac{1}{n(n − 1)}𝔼 (∑_{i≠j} s_{σ(i)}s_{σ(j)})

= \frac{1}{n(n − 1)} ∑_{i≠j} s_{i}s_{j}

= \frac{1}{n(n − 1)}[(\underbrace{∑_{i=1}^{n} s_{i}}_{=0})^{2} − ∑_{i=1}^{n} s_{i}^{2}]

= −\frac{1}{n(n − 1)} ∑_{i=1}^{n} s_{i}^{2}.

Thus

𝔼 (∥∑_{i=1}^{n} s_{σ(i)}x_{i}∥^{2}) = (∑_{i=1}^{n} s_{i}^{2}) [\frac{1}{n} ∑_{i=1}^{n}∥x_{i}∥^{2} − \frac{1}{n(n − 1)} ∑_{i≠j}⟨x_{i}, x_{j}⟩]

= (∑_{i=1}^{n} s_{i}^{2}) [(\frac{1}{n} + \frac{1}{n(n − 1)}) ∑_{i=1}^{n}∥x_{i}∥^{2}

− \frac{1}{n(n − 1)}∥∑_{i=1}^{n} x_{i}∥^{2}]

= \frac{n}{n − 1} (∑_{i=1}^{n} s_{i}^{2}) 𝕍ar(x_{1}, …, x_{n}).

We have proved that

\frac{𝕍ar(x_{1}, …, x_{n})}{γ^{2}} ≥ \frac{(n − 1)(∑_{i=1}^{n}|s_{i}|)^{2}}{n ∑_{i=1}^{n} s_{i}^{2}} .

This can be used with s_{i} = 𝟙(i ≤ \frac{n}{2}) − 𝟙(i > \frac{n}{2}) in the case when n is even and s_{i} = \frac{2}{(n−1)}𝟙(i ≤ \frac{n−1}{2}) − \frac{2}{n+1}𝟙(i > \frac{n−1}{2}) in the case when n is odd, to establish the first inequality (4.5) of the theorem.

Checking that equality is reached for the simplex is an easy computation when the simplex (x_{i})_{i=1}^{n} ∈ (ℝ^{n})^{n} is parametrized in such a way that

x_{i}(j) = \begin{cases}1 & if i = j, \\ 0 & otherwise. \end{cases}

Indeed the distance between the convex hulls of any two subsets of the simplex is the distance between their mean values (i.e. centers of mass). □

4.2.4. Application to Support Vector Machines. We are going to apply Theorem 4.2.4 (page 176) to Support Vector Machines in the transductive case. Let (X_{i}, Y_{i})_{i=1}^{(k+1)N} be distributed according to some partially exchangeable distribution ℙ and assume that (X_{i})_{i=1}^{(k+1)N} and (Y_{i})_{i=1}^{N} are observed. Let us consider some positive kernel K on 𝒳. For any K-separable training set of the form Z′ = (X_{i}, y′_{i})_{i=1}^{(k+1)N}, where (y′_{i})_{i=1}^{(k+1)N} ∈ 𝒴^{(k+1)N}, let f̂_{Z′} be the Support Vector Machine defined by K and Z′ and let γ(Z′) be its margin. Let

R^{2} = max_{i=1,...,(k+1)N} K(X_{i}, X_{i}) + \frac{1}{(k + 1)^{2}N^{2}} ∑_{j=1}^{(k+1)N} ∑_{k=1}^{(k+1)N} K(X_{j}, X_{k})

− \frac{2}{(k + 1)N} ∑_{j=1}^{(k+1)N} K(X_{i}, X_{j}).

This is an easily computable upper-bound for the radius of some ball containing the image of (X_{1}, …, X_{(k+1)N}) in feature space.

Let us define for any integer h the margins

γ_{2h} = (2h − 1)^{−1/2} and γ_{2h+1} = [2h (1 − \frac{1}{(2h + 1)^{2}} )]^{−1/2} .  (4.6)

Let us consider for any h = 1, …, N the exchangeable model

ℛ_{h} = {f̂_{Z′} : Z′ = (X_{i}, y′_{i})_{i=1}^{(k+1)N} is K-separable and γ(Z′) ≥ Rγ_{h}}.

The family of models ℛ_{h}, h = 1, …, N is nested, and we know from Theorem 4.2.4 (page 176) and Theorems 4.2.2 (page 174) and 4.2.3 (page 174) that

log(|ℛ_{h}|) ≤ h log(\frac{(k+1)eN}{h} ).

We can then consider on the large model ℛ = ⨆_{h=1}^{N} ℛ_{h} (the disjoint union of the sub-models) an exchangeable prior π which is uniform on each ℛ_{h} and is such that π(ℛ_{h}) ≥ \frac{1}{h(h+1)}. Applying Theorem 3.2.3 (page 143) we get

Proposition 4.2.5. With ℙ probability at least 1−ϵ, for any h = 1, …, N, any Support Vector Machine f ∈ ℛ_{h},

r_{2}(f) ≤

\frac{k + 1}{k} inf_{λ∈ℝ_{+}} \frac{1 − exp[−\frac{λ}{N} r_{1}(f) − \frac{h}{N} log(\frac{e(k+1)N}{h} ) − \frac{log[h(h+1)]−log(ϵ)}{N}]}{1 − exp(−\frac{λ}{N})}

− \frac{r_{1}(f)}{k}.

Searching the whole model ℛ_{h} to optimize the bound may require more computer resources than are available, but any heuristic can be applied to choose f, since the bound is uniform. For instance, a Support Vector Machine f′ using a box constraint can be trained from the training set (X_{i}, Y_{i})_{i=1}^{N} and then (y′_{i})_{i=1}^{(k+1)N} can be set to y′_{i} = sign(f′(X_{i})), i = 1, …, (k + 1)N.

4.2.5. Inductive margin bounds for Support Vector Machines. In order to establish inductive margin bounds, we will need a different combinatorial lemma. It is due to Alon et al. (1997). We will reproduce their proof with some tiny improvements on the values of constants.

Let us consider the finite case when 𝒳 = {1, …, n}, 𝒴 = {1, …, b} and b ≥ 3. The question we will study would be meaningless when b ≤ 2. Assume as usual that we are dealing with a prescribed set of classification rules ℛ = {f : 𝒳 → 𝒴}. Let us say that a pair (A, s), where A ⊂ 𝒳 is a nonempty set of shapes and s : A → {2, …, b − 1} a threshold function, is shattered by the set of functions F ⊂ ℛ if for any (σ_{x})_{x∈A} ∈ {−1, +1}^{A}, there exists some f ∈ F such that min_{x∈A} σ_{x}[f(x) − s(x)] ≥ 1.

Definition 4.2.1. Let the fat shattering dimension of (𝒳, ℛ) be the maximal size |A| of the first component of the pairs which are shattered by ℛ.

Let us say that a subset of classification rules F ⊂ 𝒴^{𝒳} is separated whenever for any pair (f, g) ∈ F^{2} such that f ≠ g, ∥f − g∥_{∞} = max_{x∈𝒳} |f(x) − g(x)| ≥ 2. Let 𝔐(ℛ) be the maximum size |F | of separated subsets F of ℛ. Note that if F is a separated subset of ℛ such that |F | = 𝔐(ℛ), then it is a 1-net for the ℒ_{∞} distance: for any function f ∈ ℛ there exists g ∈ F such that ∥f − g∥_{∞} ≤ 1 (otherwise f could be added to F to create a larger separated set).

Lemma 4.2.6. With the above notation, whenever the fat shattering dimension of (𝒳, ℛ) is not greater than h,

log[𝔐(ℛ)] < log[(b − 1)(b − 2)n]\{\frac{log[∑_{i=1}^{h} \binom{n}{i}(b − 2)^{i}]}{log(2)} + 1\} + log(2)

≤ log[(b − 1)(b − 2)n]\{[log[\frac{(b−2)n}{h}] + 1]\frac{h}{log(2)} + 1\} + log(2).

Proof. For any set of functions F ⊂ 𝒴^{𝒳}, let t(F) be the number of pairs (A, s) shattered by F. Let t(m, n) be the minimum of t(F) over all separated sets of functions F ⊂ 𝒴^{𝒳} of size |F | = m (n is here to recall that the shape space 𝒳 is made of n shapes). For any m such that t(m, n) > ∑_{i=1}^{h} \binom{n}{i}(b−2)^{i}, it is clear that any separated set of functions of size |F | ≥ m shatters at least one pair (A, s) such that |A| > h. Indeed, from its definition t(m, n) is clearly a non-decreasing function of m, so that t(|F |, n) > ∑_{i=1}^{h} \binom{n}{i}(b − 2)^{i}. Moreover there are only ∑_{i=1}^{h} \binom{n}{i}(b−2)^{i} pairs (A, s) such that |A| ≤ h. As a consequence, whenever the fat shattering dimension of (𝒳, ℛ) is not greater than h we have 𝔐(ℛ) < m.

It is clear that for any n ≥ 1, t(2, n) = 1.

Lemma 4.2.7. For any m ≥ 1, t[mn(b − 1)(b − 2), n] ≥ 2t[m, n − 1], and therefore t[2n(n − 1) … (n − r + 1)(b − 1)^{r}(b − 2)^{r}, n] ≥ 2^{r}.

Proof. Let F = {f_{1}, …, f_{mn(b−1)(b−2)}} be some separated set of functions of size mn(b−1)(b−2). For any pair (f_{2i−1}, f_{2i}), i = 1, …, mn(b−1)(b−2)/2, there is x_{i} ∈ 𝒳 such that |f_{2i−1}(x_{i}) − f_{2i}(x_{i})| ≥ 2. Since |𝒳| = n, there is x ∈ 𝒳 such that ∑_{i=1}^{mn(b−1)(b−2)/2} 𝟙(x_{i} = x) ≥ m(b − 1)(b − 2)/2. Let I = {i : x_{i} = x}. Since there are (b − 1)(b − 2)/2 pairs (y_{1}, y_{2}) ∈ 𝒴^{2} such that 1 ≤ y_{1} < y_{2} − 1 ≤ b − 1, there is some pair (y_{1}, y_{2}), such that 1 ≤ y_{1} < y_{2} ≤ b and such that ∑_{i∈I} 𝟙({y_{1}, y_{2}} = {f_{2i−1}(x), f_{2i}(x)}) ≥ m. Let J = {i ∈ I : {f_{2i−1}(x), f_{2i}(x)} = {y_{1}, y_{2}}}. Let

F_{1} = {f_{2i−1} : i ∈ J, f_{2i−1}(x) = y_{1}} ∪ {f_{2i} : i ∈ J, f_{2i}(x) = y_{1}},
F_{2} = {f_{2i−1} : i ∈ J, f_{2i−1}(x) = y_{2}} ∪ {f_{2i} : i ∈ J, f_{2i}(x) = y_{2}}.

Obviously |F_{1}| = |F_{2}| = |J| = m. Moreover the restrictions of the functions of F_{1} to 𝒳 \ {x} are separated, and it is the same with F_{2}. Thus F_{1} strongly shatters at least t(m, n − 1) pairs (A, s) such that A ⊂ 𝒳 \ {x} and it is the same with F_{2}. Finally, if the pair (A, s) where A ⊂ 𝒳 \ {x} is both shattered by F_{1} and F_{2}, then F_{1} ∪ F_{2} shatters also (A ∪ {x}, s′) where s′(x′) = s(x′) for any x′ ∈ A and s′(x) = ⌊\frac{y_{1}+y_{2}}{2} ⌋. Thus F_{1} ∪ F_{2}, and therefore F, shatters at least 2t(m, n − 1) pairs (A, s). □

Resuming the proof of lemma 4.2.6, let us choose for r the smallest integer such that 2^{r} > ∑_{i=1}^{h} \binom{n}{i}(b − 2)^{i}, which is no greater than

\{^{\frac{log[∑_{i=1}^{h} \binom{n}{i}(b−2)^{i}]}{log(2)}} + 1\}.

In the case when 1 ≤ n ≤ r,

log(𝔐(ℛ)) < |𝒳| log(|𝒴|) = n log(b) ≤ r log(b) ≤ r log[(b−1)(b−2)n]+log(2),

which proves the lemma. In the remaining case n > r,

t[2n^{r}(b − 1)^{r}(b − 2)^{r}, n] ≥ t[2n(n − 1) … (n − r + 1)(b − 1)^{r}(b − 2)^{r}, n]

> ∑_{i=1}^{h} \binom{n}{i}(b − 2)^{i}.

Thus |𝔐(ℛ)| < 2[(b − 2)(b − 1)n]^{r} as claimed. □

In order to apply this combinatorial lemma to Support Vector Machines, let us consider now the case of separating hyperplanes in ℝ^{d} (the generalization to Support Vector Machines being straightforward). Assume that 𝒳 = ℝ^{d} and 𝒴 = {−1, +1}. For any sample (X)_{i=1}^{(k+1)N}, let

R(X_{1}^{(k+1)N}) = max{∥X_{i}∥ : 1 ≤ i ≤ (k + 1)N}.

Let us consider the set of parameters

Θ = {(w, b) ∈ ℝ^{d} × ℝ : ∥w∥ = 1}.

For any (w, b) ∈ Θ, let g_{w,b}(x) = ⟨w, x⟩ − b. Let h be some fixed integer and let γ = R(X_{1}^{(k+1)N})γ_{h}, where γ_{h} is defined by equation (4.6, page 178).

Let us define ζ : ℝ → ℤ by

ζ(r) = \{ \begin{matrix}−5 & when & r ≤ −4γ, \\ −3 & when & −4γ < & r ≤ −2γ, \\ −1 & when & −2γ < & r ≤ 0, \\ +1 & when & 0 < & r ≤ 2γ, \\ +3 & when & 2γ < & r ≤ 4γ, \\ +5 & when & 4γ < & r. \end{matrix}

Let G_{w,b}(x) = ζ[g_{w,b}(x)]. The fat shattering dimension (as defined in 4.2.1) of

(X_{1}^{(k+1)N}, {(G_{w,b} + 7)/2 : (w, b) ∈ Θ})

is not greater than h (according to Theorem 4.2.4, page 176), therefore there is some set ℱ of functions from X_{1}^{(k+1)N} to {−5, −3, −1, +1, +3, +5} such that

log(|ℱ|) ≤ log[20(k + 1)N]\{\frac{h}{log(2)}[log (\frac{4(k + 1)N}{h} ) + 1] + 1\} + log(2).

and for any (w, b) ∈ Θ, there is f_{w,b} ∈ ℱ such that sup{|f_{w,b}(X_{i})−G_{w,b}(X_{i})| : i = 1, …, (k + 1)N} ≤ 2. Moreover, the choice of f_{w,b} may be required to depend on (X_{i})_{i=1}^{(k+1)N} in an exchangeable way. Similarly to Theorem 3.2.3 (page 143), it can be proved that for any partially exchangeable probability distribution ℙ ∈ ℳ_{+}^{1}(Ω), with ℙ probability at least 1 − ϵ, for any f_{w,b} ∈ ℱ,

\frac{1}{kN} ∑_{i=N+1}^{(k+1)N} 𝟙[f_{w,b}(X_{i})Y_{i} ≤ 1]

≤ \frac{k + 1}{k} inf_{λ∈ℝ_{+}} [1 − exp(−\frac{λ}{N})]^{−1}\{1−

exp[−\frac{λ}{N^{2}} ∑_{i=1}^{N} 𝟙[f_{w,b}(X_{i})Y_{i} ≤ 1] − \frac{log(|ℱ|) − log(ϵ)}{N}]\}

− \frac{1}{kN} ∑_{i=1}^{N} 𝟙[f_{w,b}(X_{i})Y_{i} ≤ 1].

Let us remark that

𝟙{2𝟙[g_{w,b}(X_{i}) ≥ 0] − 1 ≠ Y_{i}} = 𝟙[G_{w,b}(X_{i})Y_{i} < 0] ≤ 𝟙[f_{w,b}(X_{i})Y_{i} ≤ 1]

and

𝟙[f_{w,b}(X_{i})Y_{i} ≤ 1] ≤ 𝟙[G_{w,b}(X_{i})Y_{i} ≤ 3] ≤ 𝟙[g_{w,b}(X_{i})Y_{i} ≤ 4γ].

This proves the following theorem.

Theorem 4.2.8. Let us consider the sequence (γ_{h})_{h∈ℕ^{∗}} defined by equation (4.6, page 178). With ℙ probability at least 1 − ϵ, for any (w, b) ∈ Θ,

\frac{1}{kN} ∑_{i=N+1}^{(k+1)N} 𝟙\{2𝟙[g_{w,b}(X_{i}) ≥ 0] − 1 ≠ Y_{i}\}

≤ \frac{k + 1}{k} inf_{λ∈ℝ_{+},h∈ℕ^{∗}} [1 − exp(−\frac{λ}{N})]^{−1}\{1−

exp[−\frac{λ}{N^{2}} ∑_{i=1}^{N} 𝟙[g_{w,b}(X_{i})Y_{i} ≤ 4Rγ_{h}]

−\frac{log[20(k + 1)N]\{\frac{h}{log(2)} log(\frac{4e(k+1)N}{h} ) + 1\} + log[\frac{2h(h+1)}{ϵ}]}{N}]\}

− \frac{1}{kN} ∑_{i=1}^{N} 𝟙[g_{w,b}(X_{i})Y_{i} ≤ 4Rγ_{h}].

Properly speaking this theorem is not a margin bound, but more precisely a margin quantile bound, since it covers the case where some fraction of the training sample falls within the region defined by the margin parameter γ_{h} which optimizes the bound.

As a consequence though, we get a true (weaker) margin bound: with ℙ probability at least 1 − ϵ, for any (w, b) ∈ Θ such that

γ = min_{i=1,...,N} g_{w,b}(X_{i})Y_{i} > 0,

\frac{1}{kN} ∑_{i=N+1}^{(k+1)N} 𝟙[g_{w,b}(X_{i})Y_{i} < 0]

≤ \frac{k+1}{k}\{1 − exp[−^{\frac{log[20(k+1)N]}{N}}\{\frac{16R^{2}+2γ^{2}}{log(2)γ^{2}} log(\frac{e(k+1)Nγ^{2}}{4R^{2}} ) + 1\}

+ \frac{1}{N} log(\frac{ϵ}{2})]\}.

This inequality compares favourably with similar inequalities in Cristianini et al. (2000), which moreover do not extend to the margin quantile case as this one.

Let us also mention that it is easy to circumvent the fact that R is not observed when the test set X_{N+1}^{(k+1)N} is not observed.

Indeed, we can consider the sample obtained by projecting X_{1}^{(k+1)N} on some ball of fixed radius R_{max}, putting

t_{R_{max}}(X_{i}) = min \{1, \frac{R_{max}}{∥X_{i}∥} \} X_{i}.

We can further consider an atomic prior distribution ν ∈ ℳ_{+}^{1}(ℝ_{+}) bearing on R_{max}, to obtain a uniform result through a union bound. As a consequence of the previous theorem, we have

Corollary 4.2.9. For any atomic prior ν ∈ ℳ_{+}^{1}(ℝ_{+}), for any partially exchangeable probability measure ℙ ∈ ℳ_{+}^{1}(Ω), with ℙ probability at least 1 − ϵ, for any (w, b) ∈ Θ, any R_{max} ∈ ℝ_{+},

\frac{1}{kN} ∑_{i=N+1}^{(k+1)N} 𝟙\{2𝟙[g_{w,b} ∘ t_{R_{max}}(X_{i}) ≥ 0] − 1 ≠ Y_{i}\}

≤ \frac{k + 1}{k} inf_{λ∈ℝ_{+},h∈ℕ^{∗}} [1 − exp(−\frac{λ}{N})]^{−1}\{1−

exp[−\frac{λ}{N^{2}} ∑_{i=1}^{N} 𝟙[g_{w,b} ∘ t_{R_{max}}(X_{i})Y_{i} ≤ 4R_{max}γ_{h}]

−\frac{log[20(k + 1)N]\{\frac{h}{log(2)} log(\frac{4e(k+1)N}{h} ) + 1\} + log[\frac{2h(h+1)}{ϵν(R_{max})}]}{N}]\}

− \frac{1}{kN} ∑_{i=1}^{N} 𝟙[g_{w,b} ∘ t_{R_{max}}(X_{i})Y_{i} ≤ 4R_{max}γ_{h}].

Let us remark that t_{R_{max}}(X_{i}) = X_{i}, i = N + 1, …, (k + 1)N, as soon as we consider only the values of R_{max} not smaller than max_{i=N+1,...,(k+1)N}∥X_{i}∥ in this corollary. Thus we obtain a bound on the transductive generalization error of the unthresholded classification rule 2𝟙[g_{w,b}(X_{i}) ≥ 0] − 1, as well as some incitation to replace it with a thresholded rule when the value of R_{max} minimizing the bound falls below max_{i=N+1,...,(k+1)N}∥X_{i}∥.

Appendix: classification by thresholding

In this appendix, we show how the bounds given in the first section of this monograph can be computed in practice on a simple example: the case when the classification is performed by comparing a series of measurements to threshold values. Let us mention that our description covers the case when the same measurement is compared to several thresholds, since it is enough to repeat a measurement in the list of measurements describing a pattern to cover this case.

5.1. Description of the model

Let us assume that the patterns we want to classify are described through h real valued measurements normalized in the range (0, 1). In this setting the pattern space can thus be defined as 𝒳 = (0, 1)^{h}.

Consider the threshold set 𝒯 = (0, 1)^{h} and the response set ℛ = 𝒴^{{0,1}^{h}}. For any t ∈ (0, 1)^{h} and any a : {0, 1}^{h} → 𝒴, let

f_{(t,a)}(x) = a{[𝟙(x^{j} ≥ t_{j})]_{j=1}^{h}}, x ∈ 𝒳,

where x^{j} is the jth coordinate of x ∈ 𝒳. Thus our parameter set here is Θ = 𝒯 × ℛ. Let us consider the Lebesgue measure L on 𝒯 and the uniform probability distribution U on ℛ. Let our prior distribution be π = L ⊗ U. Let us define for any threshold sequence t ∈ 𝒯

Δ_{t} = \{t′ ∈ 𝒯 : \overline{(t′_{j}, t_{j})} ∩ \{X_{i}^{j}; i = 1, …, N\} = ∅, j = 1, …, h\},

where X_{i}^{j} is the jth coordinate of the sample pattern X_{i}, and where the interval \overline{(t′_{j}, t_{j})} of the real line is defined as the convex hull of the two point set {t′_{j}, t_{j}}, whether t′_{j} ≤ t_{j} or not. We see that Δ_{t} is the set of thresholds giving the same response as t on the training patterns. Let us consider for any t ∈ 𝒯 the middle

m(Δ_{t}) = \frac{∫_{Δ_{t}} t′L(dt′)}{L(Δ_{t})}

of Δ_{t}. The set Δ_{t} being a product of intervals, its middle is the point whose coordinates are the middle of these intervals. Let us introduce the finite set T composed of the middles of the cells Δ_{t}, which can be defined as

T = {t ∈ 𝒯 : t = m(Δ_{t})}.

It is easy to see that |T | ≤ (N + 1)^{h} and that |ℛ| = |𝒴|^{2^{h}}.

5.2. Computation of inductive bounds

For any parameter (t, a) ∈ 𝒯 × ℛ = Θ, let us consider the posterior distribution defined by its density

\frac{dρ_{(t,a)}}{dπ} (t′, a′) = \frac{𝟙(t′ ∈ Δ_{t})𝟙(a′ = a)}{π(Δ_{t} × \{a\})}.

In fact we are considering a finite number of posterior distributions, since ρ_{(t,a)} = ρ_{(m(Δ_{t}),a)}, where m(Δ_{t}) ∈ T. Moreover, for any exchangeable sample distribution ℙ ∈ ℳ_{+}^{1}[(𝒳 × 𝒴)^{N+1}] and any thresholds t ∈ 𝒯,

ℙ[ \overline{(X_{N+1}^{j}, t_{j})} ∩ \{X_{i}^{j}, i = 1, …, N\} = ∅] ≤ \frac{2}{N + 1}.

Thus, for any (t, a) ∈ Θ,

ℙ\{ρ_{(t,a)}[f_{.}(X_{N+1})] ≠ f_{(t,a)}(X_{N+1})\} ≤ \frac{2h}{N + 1},

showing that the classification produced by ρ_{(t,a)} on new examples is typically non-random; this result is only indicative, since it is concerned with a non-random choice of (t, a).

Let us compute the various quantities needed to apply the results of the first section, focussing our attention on Theorem 2.1.3 (page 72).

First note that ρ_{(t,a)}(r) = r[(t, a)]. The entropy term is such that

𝒦(ρ_{t,a}, π) = − log[π(Δ_{t} × {r})] = − log[L(Δ_{t})] + 2^{h} log(|𝒴|).

Let us notice accordingly that

min_{(t,a)∈Θ} 𝒦(ρ_{(t,a)}, π) ≤ h log(N + 1) + 2^{h} log(|𝒴|).

Let us introduce the counters

b_{y}^{t}(c) = \frac{1}{N} ∑_{i=1}^{N} 𝟙\{Y_{i} = y and [𝟙(X_{i}^{j} ≥ t_{j})]_{j=1}^{h} = c\},
t ∈ T, c ∈ {0, 1}^{h}, y ∈ 𝒴,

b^{t}(c) = ∑_{y∈𝒴}b_{y}^{t}(c) = \frac{1}{N} ∑_{i=1}^{N} 𝟙\{[𝟙(X_{i}^{j} ≥ t_{j})]_{j=1}^{h} = c\},  t ∈ T, c ∈ \{0, 1\}^{h}.

Since

r[(t, a)] = ∑_{c∈{0,1}^{h}}[b^{t}(c) − b_{a(c)}^{t}(c)],

the partition function of the Gibbs estimator can be computed as

π[exp(−λr)] = ∑_{t∈T} L(Δ_{t}) ∑_{a∈ℛ} \frac{1}{|𝒴|^{2^{h}}} exp[−λ ∑_{i=1}^{N} 𝟙[Y_{i} ≠ f_{(t,a)}(X_{i})]]

= ∑_{t∈T} L(Δ_{t}) ∑_{a∈ℛ} \frac{1}{|𝒴|^{2^{h}}} exp[−λ ∑_{c∈\{0,1\}^{h}}[b^{t}(c) − b_{a(c)}^{t}(c)]]

= ∑_{t∈T} L(Δ_{t}) ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)])].

We see that the number of operations needed to compute π[exp(−λr)] is proportional to |T | × 2^{h} × |𝒴| ≤ (N + 1)^{h}2^{h}|𝒴|. An exact computation will therefore be feasible only for small values of N and h. For higher values, a Monte Carlo approximation of this sum will have to be performed instead.

If we want to compute the bound provided by Theorem 2.1.3 (page 72) or by Theorem 2.2.2 (page 89), we need also to compute, for any fixed parameter θ ∈ Θ, quantities of the type

π_{exp(−λr)}{exp[ξm′(·, θ)]} = π_{exp(−λr)}{exp[ξρ_{θ}(m′)]}, λ, ξ ∈ ℝ_{+}.

We need to introduce

\overline{b}_{y}^{t}(θ, c) = \frac{1}{N} ∑_{i=1}^{N}|𝟙[f_{θ}(X_{i}) ≠ Y_{i}] − 𝟙(y ≠ Y_{i})|𝟙\{[𝟙(X_{i}^{j} ≥ t_{j})]_{j=1}^{h} = c\}.

Similarly to what has been done previously, we obtain

π{exp[−λr + ξm′(·, θ)]}

= ∑_{t∈T} L(Δ_{t}) ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)] + ξ\overline{b}_{y}^{t}(θ, c))].

We can then compute

π_{exp(−λr)}(r) = −\frac{∂}{∂λ} log\{π[exp(−λr)]\},
π_{exp(−λr)}\{exp[ξρ_{θ}(m′)]\} = \frac{π\{exp[−λr + ξm′(·, θ)]\}}{π[exp(−λr)]},

π_{exp(−λr)}[m′(·, θ)] = \frac{∂}{∂ξ}_{|ξ=0} log[π\{exp[−λr + ξm′(·, θ)]\}].

This is all we need to compute B(ρ_{θ}, β, γ) (and also B(π_{exp(−λr)}, β, γ)) in Theorem 2.1.3 (page 72), using the approximation

log{π_{exp(−λ_{1}r)}[exp{ξπ_{exp(−λ_{2}r)}(m′)}]}

≤ log{π_{exp(−λ_{1}r)}[exp{ξm′(·, θ)}]} + ξπ_{exp(−λ_{2}r)}[m′(·, θ)], ξ ≥ 0.

Let us also explain how to apply the posterior distribution ρ_{(t,a)}, in other words our randomized estimated classification rule, to a new pattern X_{N+1}:

ρ_{(t,a)}[f_{·}(X_{N+1}) = y] = L(Δ_{t})^{−1} ∫_{Δ_{t}} 𝟙[a{[𝟙(X_{N+1}^{j} ≥ t′_{j})]_{j=1}^{h}} = y]L(dt′)

= L(Δ_{t})^{−1} ∑_{c∈{0,1}^{h}} L({t′ ∈ Δ_{t} : [𝟙(X_{N+1}^{j} ≥ t′_{j})]_{j=1}^{h} = c})𝟙[a(c) = y].

Let us define for short

Δ_{t}(c) = {t′ ∈ Δ_{t} : [𝟙(X_{N+1}^{j} ≥ t′_{j})]_{j=1}^{h} = c}, c ∈ {0, 1}^{h}.

With this notation

ρ_{(t,a)}[f_{.}(X_{N+1}) = y] = L(Δ_{t})^{−1} ∑_{c∈{0,1}^{h}} L[Δ_{t}(c)]𝟙[a(c) = y].

We can compute in the same way the probabilities for the label of the new pattern under the Gibbs posterior distribution:

π_{exp(−λr)}[f_{·}(X_{N+1}) = y′]

= \{∑_{t∈T} ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)])]

× ∑_{c∈\{0,1\}^{h}} L[Δ_{t}(c)]\frac{∑_{y∈𝒴} 𝟙(y = y′) exp\{−λ[b^{t}(c) − b_{y}^{t}(c)]\}}{∑_{y∈𝒴} exp\{−λ[b^{t}(x) − b_{y}^{t}(c)]\}}\}

× \{∑_{t∈T} L(Δ_{t}) ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)])]\}^{−1}.

5.3. Transductive bounds

In the case when we observe the patterns of a shadow sample (X_{i})_{i=N+1}^{(k+1)N} on top of the training sample (X_{i}, Y_{i})_{i=1}^{N}, we can introduce the set of thresholds responding as t on the extended sample (X_{i})_{i=1}^{(k+1)N}

\overline{Δ}_{t} = \{t′ ∈ 𝒯 : \overline{(t′_{j}, t_{j})} ∩ \{X_{i}^{j}; i = 1, …, (k + 1)N\} = ∅, j = 1, …, h\}, consider the set

\overline{T} = \{t ∈ 𝒯 : t = m(\overline{Δ}_{t})\},

of the middle points of the cells \overline{Δ}_{t}, t ∈ 𝒯, and replace the Lebesgue measure L ∈ ℳ_{+}^{1}[(0, 1)^{h}] of the previous section with the uniform probability measure \overline{L} on \overline{T}. We can then consider π = \overline{L} ⊗ U, where U is as previously the uniform probability measure on ℛ. This gives obviously an exchangeable posterior distribution and therefore qualifies π for transductive bounds. Let us notice that |\overline{T} | ≤ [(k + 1)N + 1]^{h}, and therefore that π(t, a) ≥ [(k + 1)N + 1]^{−h}|𝒴|^{−2^{h}}, for any (t, a) ∈ \overline{T} × ℛ.

For any (t, a) ∈ 𝒯 × ℛ we may similarly to the inductive case consider the posterior distribution ρ_{(t,a)} defined by

\frac{dρ_{(t,a)}}{dπ} (t′, a′) = \frac{𝟙(t′ ∈ Δ_{t})𝟙(a′ = a)}{π(Δ_{t} × \{a\})},

but we may also consider δ_{(m(\overline{Δ}_{t}),a)}, which is such that r_{i}\{[m(\overline{Δ}_{t}), a]\} = r_{i}[(t, a)], i = 1, 2, whereas only ρ_{(t,a)}(r_{1}) = r_{1}[(t, a)], while

ρ_{(t,a)}(r_{2}) = \frac{1}{|\overline{T} ∩ Δ_{t}|} ∑_{t′∈\overline{T} ∩Δ_{t}} r_{2}[(t′, a)].

We get

𝒦(ρ_{(t,a)}, π) = − log[ \overline{L}(Δ_{t})] + 2^{h} log(|𝒴|)

≤ log(|\overline{T} |) + 2^{h} log(|𝒴|) = 𝒦(δ_{[m(\overline{Δ}_{t}),a]}, π)

≤ h log[(k + 1)N + 1] + 2^{h} log(|𝒴|),

whereas we had no such uniform bound in the inductive case. Similarly to the inductive case

π[exp(−λr_{1})] = ∑_{t∈T} \overline{L}(Δ_{t}) ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)])].

Moreover, for any θ ∈ Θ,

π{exp[−λr_{1} + ξρ_{θ}(m′)]} = π{exp[−λr_{1} + ξm′(·, θ)]}

= ∑_{t∈T} \overline{L}(Δ_{t}) ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)] + ξ\overline{b}(θ, c))].

The bound for the transductive counterpart to Theorems 2.1.3 (page 72) or 2.2.2 (page 89), obtained as explained page 141, can be computed as in the inductive case, from these two partition functions and the above entropy computation.

Let us mention finally that, using the same notation as in the inductive case,

π_{exp(−λr_{1})}[f_{·}(X_{N+1}) = y′]

= \{∑_{t∈T} ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)])]

× ∑_{c∈\{0,1\}^{h}} \overline{L}[Δ_{t}(c)]\frac{∑_{y∈𝒴} 𝟙(y = y′) exp\{−λ[b^{t}(c) − b_{y}^{t}(c)]\}}{∑_{y∈𝒴} exp\{−λ[b^{t}(x) − b_{y}^{t}(c)]\}}\}

× \{∑_{t∈T} \overline{L}(Δ_{t}) ∏_{c∈\{0,1\}^{h}}[\frac{1}{|𝒴|} ∑_{y∈𝒴}exp(−λ[b^{t}(c) − b_{y}^{t}(c)])]\}^{−1}.

To conclude this appendix on classification by thresholding, note that similar factorized computations are feasible in the important case of classification trees. This can be achieved using some variant of the context tree weighting method discovered by Willems et al. (1995) and successfully used in lossless compression theory. The interested reader can find a description of this algorithm applied to classification trees in Catoni (2004, page 62).

Bibliography

N. Alon, S. Ben-David, N. Cesa-Bianchi and D. Haussler, (1997) Scale sensitive dimensions, uniform convergence and learnability, Journal of the ACM, 44: 615-631.

J.-Y. Audibert, (2004a) Aggregated estimators and empirical complexity for least square regression, Annales de l’Institut Henri Poincaré (B) Probabilités et Statistiques, 40: 685-736.

J. Y. Audibert, (2004b) PAC-Bayesian Statistical Learning Theory, PhD,

University Paris 6, June, 1–178, http://cermics.enpc.fr/~audibert/.

A. Barron (1987) Are Bayes rules consistent in information ? in Open Problems in Communication and Computation, T. M. Cover and B. Gopinath (Editors), Springer-Verlag, New York, 85-91.

A. Barron and Y. Yang (1999) Information-theoretic determination of minimax rates of convergence, Annals of Statistics 27: 1564–1599.

A. Barron, L. Birgé and P. Massart, (1999) Risk bounds for model selection by penalization, Probabability Theory and Related Fields, 113: 301–413.

G. Blanchard, (1999) The “progressive mixture” estimator for regression trees, Annales de l’Institut Henri Poincaré (B) Probabilités et Statistiques, 35: 793–820.

G. Blanchard, (2001) Mixture and Aggregation of Estimators for Pattern Recognition. Application to Decision Trees PhD, Université Paris 13,

January, 1–158, http://ida.first.fraunhofer.de/~blanchard/.

G. Blanchard, (2004) Un algorithme accéléré d’échantillonnage Bayésien pour le modèle CART, Revue d’Intelligence artificielle, 18: 383–410.

L. Birgé and P. Massart, (1997) From model selection to adaptive estimation, in Festschrift for Lucien Le Cam, D. Pollard (Editor), Springer-Verlag, New York, 55-87.

L. Birgé and P. Massart, (1998) Minimum contrast estimators on sieves, Bernoulli 4: 329–375.

L. Birgé and P. Massart, (2001a) A generalized C_{p} criterion for Gaussian model selection, preprint, 1–39,

http://www.math.u-psud.fr/~massart/.

L. Birgé and P. Massart, (2001b) Gaussian model selection, Journal of the European Mathematical Society, 3: 203-268.

A. Blum and J. Langford, (2003) PAC-MDL bounds, in Computational Learning Theory and Kernel Machines, 16th Annual Conference on Computational Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003, Proceedings, Lecture Notes in Computer Science 2777, Springer, 344-357.

O. Catoni, (2002) Data compression and adaptive histograms, in Foundations of Computational Mathematics, Proceedings of the Smalefest 2000, F. Cucker and J.M. Rojas (Editors), World Scientific, 35–60.

O. Catoni, (2003) Laplace transform estimates and deviation inequalities, Annales de l’Institut Henri Poincaré (B) Probabilités et Statistiques, 39: 1–26.

O. Catoni, (2004) Statistical Learning Theory and Stochastic Optimization, Ecole d’Été de Probabilités de Saint-Flour XXXI - 2001, Lecture Notes in Mathematics, 1851, Springer, 1–270.

N. Cristianini and J. Shawe Taylor, (2000) An Introduction to Support Vector Machines and Other Kernel Based Learning Methods, Cambridge University Press, 1–189.

M. Feder and N. Merhav, (1996) Hierarchical universal coding, IEEE Transactions on Informormation Theory, 42: 1354–1364.

T. Hastie, R. Tibshirani and J. Friedman, (2001) The Elements of Statistical Learning, Springer-Verlag, New York, 1–536.

J. Langford and D. McAllester, (2004) Computable shell decomposition bounds, Journal of Machine Learning Research, 5: 529-547, (communicated at COLT 2000).

J. Langford and M. Seeger, (2001a) Bounds for averaging classifiers, technical report CMU-CS-01-102, Carnegie Mellon University, 1–13,

http://www.cs.cmu.edu/~jcl.

J. Langford, M. Seeger and N. Megiddo, (2001b) An improved predictive accuracy bound for averaging classifiers, International Conference on Machine Learning, 18, 290-297.

N. Littlestone and M. Warmuth, (1986) Relating data compression and learnability, technical report, University of California, Santa Cruz, 1–13,

http://www.soe.ucsc.edu/~manfred/pubs.html.

D. A. McAllester, (1998) Some PAC-Bayesian theorems, Proceedings of the Eleventh Annual Conference on Computational Learning Theory (Madison, WI, 1998), ACM, New York, 230–234 (electronic).

D. A. McAllester, (1999) PAC-Bayesian Model Averaging, Proceedings of the Twelfth Annual Conference on Computational Learning Theory (Santa Cruz, CA, 1999), ACM, New York, 164–170 (electronic).

C. McDiarmid, (1998) Concentration, in Probabilistic Methods for Algorithmic Discrete Mathematics, M. Habib , C. McDiarmid and B. Reed (Editors), Springer, 195–248.

E. Mammen and A. Tsybakov, (1999) Smooth discrimination analysis. Annals of Statistics 27: 1808–1829.

B. Y. Ryabko, (1984) Twice-universal coding, Problems of Information Transmission, 20: 24-28.

M. Seeger, (2002) PAC-Bayesian generalization error bounds for Gaussian process classification, Journal of Machine Learning Research 3: 233–269.

J. Shawe-Taylor, P. L. Bartlett, R. C. Williamson and M. Anthony, (1998) Structural risk minimization over data-dependent hierarchies, IEEE Transactions on Information Theory, 44: 1926–1940.

J. Shawe-Taylor and N. Cristianini, (2002) On the generalization of soft margin algorithms, IEEE Transactions on Information Theory, 48: 2721–2735.

A. Tsybakov, (2004) Optimal aggregation of classifiers in statistical learning, Annals of Statistics 32: 135–166.

A. Tsybakov and S. Van de Geer, (2005) Square root penalty: adaptation to the margin in classification and in edge estimation, Annals of Statistics, 33: 1203-1224.

S. Van de Geer, (2000) Applications of Empirical Process Theory. Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press, Cambridge, 1-286.

V. N. Vapnik, (1998) Statistical Learning Theory, Wiley, New York, 1–736.

J.-P. Vert, (2000) Double mixture and universal inference, preprint, 1–27,

http://cbio.ensmp.fr/~vert/publi/.

J.-P. Vert, (2001a) Adaptive context trees and text clustering, IEEE Transactions on Information Theory, 47: 1884–1901.

J.-P. Vert, (2001b) Text categorization using adaptive context trees, in Proceedings of the CICLing-2001 conference, A. Gelbukh (Editor), Lecture Notes in Computer Science 2004, Springer-Verlag, 423–436.

F. M. J. Willems, Y. M. Shtarkov and T. J. Tjalkens, (1995) The context-tree weighting method: basic properties, IEEE Transactions on Information Theory, 41: 653–664.

F. M. J. Willems, Y. M. Shtarkov and T. J. Tjalkens, (1996) Context weighting for general finite-context sources, IEEE Transactions on Informormation Theory, 42: 1514–1520.

T. Zhang, (2006a) From epsilon-entropy to KL-entropy: analysis of minimum information complexity density estimation, Annals of Statistics, to appear, 1–37,

http://www-cs-students.stanford.edu/~tzhang/pubs.html.

T. Zhang, (2006b) Information theoretical upper and lower bounds for statistical estimation, IEEE Transactions on Information Theory, to appear,

1–18, http://www-cs-students.stanford.edu/~tzhang/pubs.html.