Proceedings of Symposia in Pure Mathematics Volume 68, 2000

Bruhat filtrations and Whittaker vectors for real groups

William Casselman, Henryk Hecht, and Dragan Miličić

Abstract. In this paper we continue earlier work of Bruhat, Jacquet, and Shalika on Whittaker models for smooth representations of real groups. We prove exactness in a certain category of such smooth representations, and also give a new proof of holomorphicity as a function of the inducing parameter of principal series. This makes the theory of Whittaker models for real groups almost as simple to formulate as that of p-adic groups.

Contents

1. Introduction 151 2. Schwartz spaces and tempered distributions 157 3. Filtrations attached to a stratification 163 4. Bruhat filtration 165 5. Lie algebra homology of the twisted regular representation on the

Schwartz space 168

6. Whittaker vectors for smooth principal series 175 7. Holomorphic continuation of Jacquet integrals 178 8. The Whittaker functor 182 9. Linear quasisplit groups 184 Appendix A. Some results on nuclear spaces 184 Appendix B. Holomorphic vector valued functions 188 References 189

1. Introduction

Suppose G to be a real reductive Lie group of Harish-Chandra class [15], P a minimal parabolic subgroup of G and N the unipotent radical of P. Let K be a maximal compact subgroup of G. Denote by 𝔤_{0}, 𝔫_{0} and 𝔨_{0} the Lie algebras of G, N and K respectively. Let 𝔤_{0} = 𝔨_{0} ⊕ 𝔮_{0} be the Cartan decomposition of 𝔤_{0}. Denote by 𝔞_{0} a maximal abelian Lie subalgebra of 𝔮_{0}. Let 𝔤, 𝔨, 𝔫 and 𝔞 be the complexifications of 𝔤_{0}, 𝔨_{0}, 𝔫_{0} and 𝔞_{0} respectively. Let R be the (restricted) root

\begin{matrix}c \\ ◯ \end{matrix}2000 American Mathematical Society

1991 Mathematics Subject Classification. Primary 22E46, Secondary 11F70.

Key words and phrases. reductive Lie groups, smooth representations, Whittaker vectors.

system in 𝔞^{∗} of the pair (𝔤, 𝔞). Then 𝔫 is the direct sum of root subspaces 𝔤_{α} for roots α in a set R^{+} of positive roots in R. Denote by B the corresponding set of simple roots in R^{+}. Let ψ be a unitary character of N, and η its differential. Then η is completely determined by its restrictions to the root subspaces 𝔤_{α} corresponding to the roots α ∈ B. The character ψ is said to be non-degenerate if all of these restrictions are non-trivial. Denote by ℂ_{η} the one-dimensional representation of N with action given by ψ. If (π, V) is a representation of 𝔤, a Whittaker functional on π is an 𝔫-covariant map from V to ℂ_{η}. If π is the restriction to 𝔤 of a smooth representation of G, then a Whittaker functional corresponds to an N-covariant map to ℂ_{η} as well, therefore a G-covariant map from V into the space of smooth functions f on G such that

f(ng) = ψ(n)f(g)

for all g ∈ G, n ∈ N. Functions in the image are called Whittaker functions, because in simple cases they essentially coincide with confluent hypergeometric functions treated extensively in Whittaker’s classic text. They, or rather their analogues for certain finite groups, were apparently introduced into representation theory by I. M. Gelfand and colleagues. The most important role they play is probably in constructing archimedean factors of L-functions associated to certain representations of adèle groups, and in particular to automorphic forms. There have been many interesting papers about them, notably Jacquet’s thesis [18], an early paper by Shalika [25], a paper of Kostant [21], and the thesis of Tze-Ming To [26]. There remain so far unanswered, however, a few interesting technical but important questions which we propose to answer here. We will also offer new proofs of known results, and along the way explain a fundamental result in the representation theory which as so far been only weakly exploited, what we call the Bruhat filtration of the smooth principal series. This is a refinement of much earlier work of Bruhat [6].

The main results of this paper depend strongly on the Bruhat filtration, and include (1) a proof of the exactness of what we call the Whittaker functor, (2) uniqueness of Whittaker functionals for suitable smooth irreducible representations of quasisplit groups, and (3) holomorphicity of Whittaker functionals for the smooth principal series and other analytic families of induced representations. Many results in this paper have appeared before in various work of others, but not in as useful or as general a form as we would like (see Matumoto [22], Wallach [29]). The Bruhat filtration can be used to give more elegant proofs of many other results in the representation theory of G. It might serve, in fact, as the cornerstone of a reasonably elegant exposition of the subject from its beginnings. We hope to return to these ideas in a subsequent paper.

It is well known that Harish-Chandra was much concerned with these matters right up to his untimely death, and we therefore think it is not inappropriate to dedicate this paper to his memory. We would like to thank Hervé Jacquet, who pointed out to us a long time ago that there is some truth to the claim that nearly all of analysis reduces to integration by parts. One of us (D.M.) would also like to thank the Department of Mathematics, Harvard University, for their support during the period when the final draft of the paper was written.

1.1. The simplest case. The main part of this paper is tough and abstract, and we want to explain the themes here in the simplest case, when the arguments can be laid out in very classical terms. Let G be the group SL(2, ℝ) of unimodular two-by-two real matrices, P the subgroup of upper triangular matrices and N the unipotent radical of P.

Further, let

ψ: N → ℂ, [\begin{matrix}1 & x \\ 0 & 1 \end{matrix}] ↦ e^{ηx}

be a unitary character of N, with η purely imaginary. At first we shall make no assumption on η, but eventually we shall assume η ≠ 0.

For s ∈ ℂ we define define the smooth principal series representation

I_{s} = {f ∈ C^{∞}(G) | f(nag) = |y|^{s+1}f(g)}

for g ∈ G, n ∈ N, and

a = [\begin{matrix}y & 0 \\ 0 & y^{−1} \end{matrix}] .

The group G is the disjoint union of two subsets P wN and P, where

w = [\begin{matrix}0 & −1 \\ 1 & 0 \end{matrix}] .

The space I_{s} is that of smooth sections of a suitable real analytic line bundle on ℙ^{1}(ℝ). It contains the subspace I_{s,w} of all sections vanishing of infinite order at ∞, and has as quotient the space I_{s,1} of formal sections (Taylor series) of the analytic line bundle at ∞. For example, if s = −1 the space I_{s} is just C^{∞}(ℙ^{1}(ℝ)), which is non-canonically the same as the space of smooth functions on the circle, I_{s,w} is the subspace of functions vanishing of infinite order at ∞, and the quotient may be identified with ℂ[[\frac{1}{x}]]. The complement of ∞ on ℙ^{1}(ℝ) is a single orbit under N, so that functions in I_{s,w} may be identified with functions on N. It is elementary in this case that this allows us to identify I_{s,w} as an N-module with S(N), the Schwartz space of the additive group. This is essentially a special case of the well known observation of Laurent Schwartz that the Schwartz space of ℝ^{n} may be identified with the space of all smooth functions on the unit n-dimensional sphere S^{n} vanishing of infinite order at the north pole, via stereographic projection. The exact sequence

0 → I_{s,w} → I_{s} → I_{s,1} → 0

is what we call the Bruhat filtration of I_{s}. The last map is surjective because of the well known result of Émile Borel. Note that these spaces are all stable under N and 𝔤. The dual of the space I_{s,1}, a space of distributions with support at ∞, may be identified with a certain Verma module over 𝔤.

For a function f ∈ I_{s,w}, the integral

Ω_{ψ,s}(f) = ∫_{N} ψ^{−1}(n)f(wn) dn

converges absolutely, and defines a ψ-covariant functional on I_{s,w} holomorphic in s. The basic question we are going to investigate here is this: to what extent can this functional be extended covariantly to one on all of I_{s}? It turns out that the integral defining Ω_{ψ,s} is convergent for s in a certain half-plane of ℂ, and that it possesses a meromorphic continuation for all s, holomorphic in the case η ≠ 0. It is the holomorphicity that we are really concerned with here.

These results can be formulated in entirely classical terms.

1.2. Some elementary functional analysis. In this section we shall investigate some problems which have no obvious relevance to representation theory, but in the next we shall explain the connection.

Suppose f to be in S(ℝ), s ∈ ℂ, η ∈ iℝ. The integral

∫_{−∞}^{∞} e^{− \frac{η}{x}} |x|^{s}f(x)dx

converges for Re(s) > −1 and defines in this region a tempered distribution on ℝ. As long as f vanishes of infinite order at 0 this integral is defined and analytic for all values of s. Does it have for all s a natural extension to all of S(ℝ)?

If η = 0 the answer is negative, but the situation is perfectly well understood. Integration by parts gives us

∫_{−∞}^{∞} |x|^{s}f(x) dx = − \frac{1}{s + 1} ∫_{−∞}^{∞} |x|^{s}xf′(x) dx

and this allows us to continue the distribution at least into the region where Re(s) > −2. If we multiply by s + 1 and take the limit as s → −1 we get

− ∫_{−∞}^{∞} sgn(x)f′(x) dx = ∫_{−∞}^{0} f′(x) dx − ∫_{0}^{∞} f′(x) dx = 2f(0) .

In other words, integration by parts allows to extend the distribution meromorphically to the region Re(s) > −2, with a residue at s = −1 equal to 2δ_{0}. Of course we can again integrate by parts and eventually we see that we have a meromorphic map from ℂ to the space of tempered distributions on ℝ, with simple poles at −1, −3, −5, … and residues that are proportional to derivatives of δ_{0}.

Now suppose that η ≠ 0. Since η is purely imaginary, the function e^{− \frac{η}{x}} oscillates wildly near 0, and it is reasonable to expect that the integral

∫_{−∞}^{∞} e^{− \frac{η}{x}} |x|^{s}f(x) dx

can be holomorphically extended. A different kind of integration by parts will do the trick. The function

φ_{s}(x) = |x|^{s}e^{− \frac{η}{x}}

satisfies the differential equation

x^{2}φ′_{s}(x) − (sx + η)φ_{s}(x) = 0

Therefore the integral becomes

∫_{−∞}^{∞} φ_{s}(x)f(x) dx = \frac{1}{η} ∫_{−∞}^{∞} x^{2}φ′_{s}(x)f(x) dx − \frac{1}{η} ∫_{−∞}^{∞} sxφ_{s}(x)f(x) dx

= −\frac{1}{η} ∫_{−∞}^{∞} x^{2}φ_{s}(x)f′(x) dx − \frac{1}{η} ∫_{−∞}^{∞}(s + 2)xφ_{s}(x)f(x) dx

= \frac{1}{η} ∫_{−∞}^{∞} φ_{s}(x) (−x^{2}f′(x) − (s + 2)xf(x)) dx.

This converges for Re(s) > −2, since the function x ⟼ −x^{2}f′(x) − (s + 2)xf(x) vanishes at 0. As before, we can repeat this procedure, eventually obtaining a distribution holomorphic as a function of s.

What is relevant for the rest of this paper is that the second argument reduces in essence to this observation: The differential operator

f(x) ⟼ −x^{2}f′(x) − (s + 2)xf(x)

induces a topologically nilpotent map of the space of formal power series ℂ[[x]] into itself.

1.3. Relevance to representation theory. Return to the notation of the earlier section on SL(2, ℝ). The group G is the disjoint union of P wN and P, and the overlapping union of open subsets P wN and P N̄, where N̄ is the subgroup of lower unipotent matrices. A function in I_{s} may therefore be expressed as a sum of functions f_{w} and f_{1}, where f_{w} has compact support modulo P on P wN and f_{1} has compact support modulo P on P N̄. In considering how to extend the functional Ω_{ψ,s}, we may assume f = f_{1}.

Therefore suppose now that f has support on P N̄. How do we evaluate Ω_{ψ,s}(f)?

We have

Ω_{ψ,s}(f) = ∫_{N} ψ^{−1}(n)f(wn) dn

and also f(pn̄) = |c|^{s+1}f(n̄) if p ∈ P, n̄ ∈ N̄, where the restriction of f to N̄ may be identified with a function φ(y) of compact support. Here

p = [\begin{matrix}c & x \\ 0 & c^{−1} \end{matrix}] ,  n̄ = [\begin{matrix}1 & 0 \\ y & 1 \end{matrix}] .

We can write

[\begin{matrix}0 & −1 \\ 1 & 0 \end{matrix}] [\begin{matrix}1 & x \\ 0 & 1 \end{matrix}] = [\begin{matrix}0 & −1 \\ 1 & x \end{matrix}] = [\begin{matrix}1 & −x^{−1} \\ 0 & 1 \end{matrix}] [\begin{matrix}x^{−1} & 0 \\ 0 & x \end{matrix}] [\begin{matrix}1 & 0 \\ x^{−1} & 1 \end{matrix}]

and therefore the integral becomes

∫_{−∞}^{∞} |x|^{−s−1}φ(x^{−1})e^{−ηx} dx = − ∫_{−∞}^{∞} |z|^{s−1}e^{− \frac{η}{z}} φ(z) dz

which of course leads us back to the calculation in the previous section.

This is admittedly elementary, but not entirely satisfactory. The explicit calculation does not really tell us what is going on in terms of representations of SL(2, ℝ). So we shall now reformulate it. Consider the exact sequence

0 → I_{s,w} → I_{s} → I_{s,1} → 0

as a sequence of modules over 𝔫, where 𝔫 is the Lie algebra of N. The integral defining the functional Ω_{ψ,s} induces, as we have seen, an 𝔫-covariant map from I_{s,w} to ℂ_{η}, or equivalently an 𝔫-covariant map from ℂ_{−η} to the strong dual I_{s,w}′ of I_{s,w}, or again equivalently an 𝔫-invariant element of I_{s,w}′ ⊗ ℂ_{η}. We can now, however, look at the long Lie algebra cohomology sequence derived from the exact sequence above:

0 → H^{0}(𝔫, I_{s,1}′ ⊗ ℂ_{η}) → H^{0}(𝔫, I_{s}′ ⊗ ℂ_{η}) → H^{0}(𝔫, I_{s,w}′ ⊗ ℂ_{η})

→ H^{1}(𝔫, I_{s,1}′ ⊗ ℂ_{η}) → H^{1}(𝔫, I_{s}′ ⊗ ℂ_{η}) → H^{1}(𝔫, I_{s,w}′ ⊗ ℂ_{η}) → 0.

We have already remarked that I_{s,1}′ is a Verma module, which means that 𝔫 acts locally nilpotently on it, and this implies in turn that each nonzero element of 𝔫 acts bijectively on I_{s,1}′ ⊗ℂ_{η}, as long as ψ is not trivial. Therefore the 𝔫-cohomology of I_{s,1}′ vanishes completely, and the map

H^{0}(𝔫, I_{s}′ ⊗ ℂ_{η}) → H^{0}(𝔫, I_{s,w}′ ⊗ ℂ_{η})

is an isomorphism. This guarantees the canonical extension of Ω_{ψ,s} from I_{s,w} to all of I_{s}. Holomorphicity in s follows, among other reasons, from a slightly more technical argument about various topological spaces of holomorphic functions, as will be shown in the main part of this paper.

How does this argument relate to the simple one in the previous section? The Lie algebra 𝔫 of N is spanned by the matrix ( \begin{matrix}0 & 1 \\ 0 & 0 \end{matrix} ). The action of the corresponding differential operator on functions in I_{s} restricted to N̄ is

−y^{2} \frac{d}{dy} − (s + 1)y

if we identify N̄ with ℝ as before. The Verma module I_{s,1}′ corresponds to the module of distributions with the support at 0. Therefore, the local nilpotency we mentioned before is just the statement dual to the observation from the end of last subsection.

1.4. What happens for more general groups. In other words, the simple argument involving integration by parts can be interpreted, essentially, in terms of cohomology. This is not necessary for SL(2, ℝ), as we have seen, but already extremely convenient for arbitrary groups of rank one. For groups of higher rank the Bruhat filtration is indexed by elements of the Weyl group W of R, with closure among closed N-stable subsets mirrored by a well known combinatorial ordering of W. The filtration itself is more complicated in that the various modules in the associated graded module are in some sense a mixture of Schwartz spaces and Verma modules. A weak description of this associated graded module was already known to Bruhat, of course, but using well known results originating with Whitney’s general theory of smooth functions on singular varieties allows a stronger statement. For these groups, it is impossible even to contemplate carrying out the calculations necessary to reduce everything to integration by parts. Instead, we introduce a rather complicated formal machine utilizing Lie algebra cohomology and functional analysis. Various simple but technical results allow one to see that the 𝔫-cohomology groups associated to non-degenerate characters of N of all of the quotients of the associated graded module except the bottom one vanish completely, and that only H^{0} of the bottom one is non-trivial.

If G is linear quasisplit then every smooth principal series has essentially a unique Whittaker functional (in the literature, commonly called a Whittaker vector). Further cohomological arguments allow us to conclude that every irreducible smooth Harish-Chandra representation of G (in the sense of [9]) has at most one. This is an important fact, on which depends the local functional equation of Whittaker functionals. For p-adic groups, this is an elementary result and has been known for a long time. Shalika’s results were perhaps the first to suggest the strong statement we prove here. But for a long time the situation for real groups has been confused by the intriguing fact that uniqueness is true only for a certain category of smooth representations of G. For algebraic Harish-Chandra modules, the dimension of the space of Whittaker functionals is equal to the cardinality of the Weyl group, a result first pointed out in Kostant’s remarkable paper [21]. Already in the book by Jacquet and Langlands [19] uniqueness was tied to the asymptotic growth at ∞ of certain N-covariant functions (the classical Whittaker functions, in fact), and the impressive thesis of To [26] analyzed growth conditions of Whittaker functions for groups of higher rank. We believe, however that the context in which we prove uniqueness here is the most natural one. It might also be of historical interest for us to mention that the construction of the useful category of smooth representations found in Casselman [9] and Wallach [28] was initiated by investigations about uniqueness of Whittaker functionals.

1.5. Plan of the rest of the paper. The body of the paper is organized as follows. In the second section we discuss the definition of Schwartz spaces and spaces of tempered distributions in the context of vector bundles on compact analytic manifolds. We prove some technical results about these spaces which are simple consequences of deep results originating in the work of Whitney and Łojasiewicz. In our exposition these results are hidden in the formalism of subanalytic sets. In the next section we study some natural filtrations on spaces of smooth sections of analytic vector bundles attached to finite stratifications of the base spaces by closed subanalytic sets. These filtrations together with the corresponding filtrations on the spaces of distributions are of fundamental importance in our paper. Up to that point, our discussion is quite general and devoid of any representation theoretic context. The example critical for our representation theoretic applications is that of a homogeneous vector bundle on a compact homogeneous manifold X for a Lie group G and the orbit stratification corresponding to a Lie subgroup N ⊂ G acting on X with finitely many orbits. It is discussed at the end of the third section.

In the fourth section we specialize this example even further to a reductive Lie group G acting on its real flag manifold X. The group N is the nilpotent radical of a minimal parabolic subgroup. The corresponding filtration of the space of sections of a homogeneous vector bundle is the Bruhat filtration. In the fifth section we compute Lie algebra (co)homology of twisted regular representations of a nilpotent Lie group N on Schwartz spaces of functions on homogeneous spaces of N and their strong duals. The calculation is based on a simple Fourier transform argument in the abelian case, and an inductive argument using the Hochschild-Serre spectral sequence in general. The main results of the paper are proved in the sixth and seventh section where we study the existence of Whittaker functionals for smooth principal series representations of reductive Lie groups and their construction as holomorphic continuation of Jacquet integrals (which are a generalization of the integral Ω_{ψ,s} from §1.3). Using the Bruhat filtration and some standard cohomological arguments, the existence follows from the results on Lie algebra cohomology in the fifth section. In the eight section we reformulate our main results for arbitrary smooth representations. Finally, in the ninth section we specialize our results to linear quasisplit groups.

Throughout the paper we freely use a number of results from the theory of nuclear spaces. These results are notoriously hard to find in the literature so we decided to include flowcharts of their proofs in two appendices at the end of the paper.

2. Schwartz spaces and tempered distributions

Let X be a compact analytic manifold. We denote by C^{∞}(X) the algebra of complex valued smooth functions on X and by Diff(X) the algebra of differential operators with smooth coefficients on X. We equip C^{∞}(X) with the topology of uniform convergence of functions and their derivatives on X (more precisely, this is the locally convex topology given by seminorms p_{D} : f ⟼ max_{x∈X} |(Df)(x)| for D ∈ Diff(X)). It is well known that C^{∞}(X) with this topology is a nuclear Fréchet space (or an NF-space, for short)^{1}. The strong dual C^{∞}(X)′ of C^{∞}(X) is the space of distributions on X. It is a dual nuclear Fréchet space (or a DNF-space, for short). Clearly, differential operators act as continuous transformations on C^{∞}(X) and C^{∞}(X) has a natural structure of a left Diff(X)-module. For any D ∈ Diff(X), let D^{t} be the adjoint endomorphism of C^{∞}(X)′. Clearly, D^{t} is a continuous linear map on C^{∞}(X)′ and (D, T) ⟼ D^{t}T for D ∈ Diff(X) and T ∈ C^{∞}(X)′, defines a structure of a right Diff(X)-module on C^{∞}(X)′.

More generally, let E be an analytic vector bundle on X. Denote by p : E → X the natural projection. Let Diff(E) be the algebra of differential operators with smooth coefficients acting on sections of E. For m ∈ ℤ, denote by Diff_{m}(E) the subspace of all differential operators of order ≤ m. Let E^{∗} be the dual vector bundle. Let C^{∞}(E) denote the space of all smooth sections of the vector bundle E. Then, using the natural pairing C^{∞}(E) × C^{∞}(E^{∗}) → C^{∞}(X), we can define seminorms p_{D,f} : s ⟼ max_{x∈X} |f(x)((Ds)(x))| on C^{∞}(E), for any D ∈ Diff(E) and f ∈ C^{∞}(E^{∗}). They define the locally convex topology of uniform convergence of sections and their derivatives on X. With this topology C^{∞}(E) is an NF-space. As above, it also has the structure of a left Diff(E)-module. Let C^{∞}(E)′ be its strong dual. By abuse of language, we call it the space of E-distributions on X. It is a DNF-space and a right Diff(E)-module.

Let T be an E-distribution on X. Then there exist a finite family {(D_{j}, f_{j}); 1 ≤ j ≤ k}, of D_{j} ∈ Diff(E) and sections f_{j} of E^{∗} such that |T (s)| ≤ ∑_{j=1}^{k} p_{D_{j},f_{j}}(s) for any s ∈ C^{∞}(E). If the orders of differential operators D_{j} are ≤ p, we say that the order of T is ≤ p. Clearly, this defines an exhaustive increasing filtration of C^{∞}(E)′. If D ∈ Diff_{s}(E) and T is of order ≤ p, D^{t}T is of order ≤ s + p. Therefore, C^{∞}(E)′ is a filtered Diff(E)-module.

Let U be an open subset in X. For any compact set K ⊂ U, D ∈ Diff(E) and a smooth section f of E^{∗}, define a seminorm p_{K,D,f} : s ⟼ max_{x∈K} |f(x)((Ds)(x))| on the space C^{∞}(U, E) of smooth sections of E on U. Let C_{0}^{∞}(U, E) be the subspace of all compactly supported smooth sections of E on U. Also, let C_{K}^{∞}(U, E) be its subspace consisting of all sections with support in K. Clearly, as a linear space, C_{0}^{∞}(U, E) = \underrightarrow{lim}_{K⊂U}C_{K}^{∞}(U, E). We endow C_{K}^{∞}(U, E) with the locally convex topology given by the family of seminorms p_{K,D,f} where D ∈ Diff(E) and f are smooth sections of E^{∗}. Moreover, we endow C_{0}^{∞}(U, E) with the topology of the direct limit of locally convex spaces. The strong dual C_{0}^{∞}(U, E)′ of C_{0}^{∞}(U, E) is the space of E-distributions on U. If V is an open subset of U, we have an obvious continuous inclusion of C_{0}^{∞}(V, E) ⟶ C_{0}^{∞}(U, E). The adjoint of this inclusion is the restriction map res_{U,V} : C_{0}^{∞}(U, E)′ ⟶ C_{0}^{∞}(V, E)′. Clearly, this defines a sheaf of E-distributions on X.

Let T be an E-distribution on X. The support supp(T) of T is the complement of the largest open set U in X such that T |_{U} = res_{X,U}(T) = 0.

The following result is a simple generalization of a well known fact.

Lemma 2.1. Let T be an E-distribution of order ≤ m on X. Then T (s) = 0 for any smooth section s of E such that Ds vanishes on supp(T) for all D ∈ Diff_{m}(E).

1For our terminology and notation concerning locally convex spaces see Appendix A.

Proof. Let U be an open set in X such that E|U is trivial and U can be covered by one coordinate chart. If the support of T is contained in U, the result is [24, Ch. III, Th. XXVIII].

As we already remarked, if T is of order ≤ p and ϕ a smooth function on X, the distribution ϕT is of order ≤ p. Therefore, by a partition of unity argument, we can reduce the general situation to the above special case.

Fix an open set U in X, and denote by Z its complement. Following [20, §3], we call the elements of the image of res_{U} = res_{X,U} : C^{∞}(E)′ ⟶ C_{0}^{∞}(U, E)′ tempered E-distributions on U with respect to X. The kernel ker res_{U} of res_{U} consists of all E-distributions on X with the support in Z. Therefore, ker res_{U} is a closed linear subspace of C^{∞}(E)′ and we have the short exact sequence

0 ⟶ ker res_{U} ⟶ C^{∞}(E)′ ⟶ im res_{U} ⟶ 0

of right Diff(E)-modules. We equip the kernel ker res_{U} with the induced topology, and the image im res_{U} with the quotient topology. Then ker res_{U} and im res_{U} are DNF-spaces. By dualizing, we have the exact sequence

0 ⟶ (im res_{U})′ ⟶ C^{∞}(E) ⟶ (ker res_{U})′ ⟶ 0,

where (im res_{U})′ = (ker res_{U})^{⊥}, i.e., this is the space of all smooth global sections of E on which the distributions supported in Z vanish.

We say that a smooth section s ∈ C^{∞}(E) vanishes at x with all of its derivatives of order ≤ m if (Ds)(x) = 0 for all differential operators D ∈ Diff_{m}(E). Also, we say that a smooth section s ∈ C^{∞}(E) vanishes at x with all of its derivatives if (Ds)(x) = 0 for all differential operators D ∈ Diff(E).

Let x ∈ X. As in [24, Ch. III, Th. XXXV], one can easily check that any E-distribution supported in {x} has the form s ⟼ f((Ds)(x)) for some differential operator D ∈ Diff(E) and a linear form f on the fiber E_{x} of E at x. Therefore, a smooth section of E vanishes at x with all of its derivatives if and only if all E-distributions with support in {x} vanish on s.

If x ∈ Z, all distributions supported in {x} vanish on sections in (im res_{U})′. Therefore, this space is contained in the subspace of C^{∞}(E) consisting of all sections vanishing with all of their derivatives on Z. The converse follows from 2.1.

Lemma 2.2. The linear subspace of all global sections in C^{∞}(E) vanishing with all of their derivatives on Z is equal to (im res_{U})′.

We denote this space by S(U, E) and call it the (relative) Schwartz space of sections of E on U with respect to X. It is equipped with the natural topology induced by C^{∞}(E). Hence, S(U, E) has a natural structure of an NF-space. Its strong dual S(U, E)′ is the space im res_{U} of all tempered E-distributions on U.

Example 2.3. Let X = S^{n} be an n-dimensional sphere, p a point in S^{n} and π : S^{n} ⟶ ℝ^{n} the stereographical projection of U = S^{n} − {p} onto ℝ^{n} with respect to the pole p. The closed subspace of the space C^{∞}(S^{n}) consisting of all functions vanishing with all of their derivatives at the pole p can be identified, via the diffeomorphism π, with the “classical” Schwartz space S(ℝ^{n}) [24, Ch. VII, Th. II]. Also, this identifies the space of all restrictions to U of the distributions on the sphere S^{n} with the space S(ℝ^{n})′ of “classical” tempered distributions on ℝ^{n}. The kernel of the restriction map is the space of all distributions supported in p. As it is well known [24, Ch. III, Th. XXXV], this is the space of all derivatives of the Dirac δ-distribution in p. Therefore, it is isomorphic to the space of polynomials ℂ[∂_{1}, ∂_{2}, …, ∂_{n}] in partial derivatives, equipped with the topology of the inductive limit of finite-dimensional spaces of polynomials of degree ≤ p, p ∈ ℤ_{+}. Its strong dual is the space of formal power series ℂ[[X_{1}, X_{2}, …, X_{n}]] with its natural topology. Therefore, the above short exact sequence corresponds to the exact sequence

0 ⟶ S(ℝ^{n}) ⟶ C^{∞}(S^{n}) ⟶^{TS} ℂ[[X_{1}, X_{2}, …, X_{n}]] ⟶ 0,

where TS is the Taylor series map which attaches to a smooth function on the sphere S^{n} its Taylor series at the point p. The surjectivity of this map is the classic theorem of Émile Borel.

As remarked in [20, 3.2], temperedness is a a local property. More precisely an E-distribution T on an open set U is tempered if and only if for any point x ∈ X there exists a neighborhood V of x and a tempered E-distribution S on V such that T |_{U∩V} = S|_{U∩V}. This remark, combined with local triviality of E, implies the following variant of the “gluing” result of [20, 3.6].

Lemma 2.4. Let U be a subanalytic open set in X, and (U_{i}; 1 ≤ i ≤ n) a covering of U by subanalytic open sets. Then, an E-distribution T on U is tempered if and only if its restrictions T |_{U_{i}} are tempered for all 1 ≤ i ≤ n.

Proof. Clearly, if T is tempered, T |_{U_{i}} are also tempered for 1 ≤ i ≤ n.

Assume that all T |_{U_{i}}, 1 ≤ i ≤ n, are tempered. Since any point x in X has a fundamental system of subanalytic open neighborhoods, we can pick a subanalytic open neighborhood V such that E|V is trivial vector bundle over V. Now, T |_{U_{i}∩V} are tempered for 1 ≤ i ≤ n. By [20, 3.6], T |_{U∩V} is a restriction of an E-distribution on V. By the above remark, it follows that T is tempered.

Let Y be a submanifold^{2} of X. Then there exists an open subset U in X such that Y is closed in U. Let U′ be another such open set contained in U. The kernel of the restriction map res_{U,U′} consists of distributions on U supported in the closed subset U − U′ of U. Therefore, this map induces a bijection of the space of E-distributions on U with support in Y onto the space of E-distributions on U′ supported in Y. The inverse of this map is “the extension by zero”. Since the restriction of a tempered distribution on U is obviously tempered distribution on U′, we see that the restriction map res_{U,U′} induces an injection of the space of tempered distributions on U with support in Y into the space of all tempered distributions on U′ with support in Y.

Assume that Y is in addition a subanalytic set in X. Let U be an open subset of X containing Y which is subanalytic in X. We say that U is a subanalytic neighborhood of Z in X if Z is a closed subset of U.

Lemma 2.5. Let Y be a submanifold of X which is also a subanalytic set in X. Then Y has a subanalytic neighborhood in X.

Proof. Since the closure Ȳ of Y in X is compact, there exists a finite covering (U_{i}; 1 ≤ i ≤ n) of Y consisting of subanalytic open sets in X. Since ∂Y = Ȳ − Y is a closed subanalytic subset in X, we also see that U = (⋃_{i=1}^{n} U_{i}) − ∂Y is an open subanalytic set in X. Clearly, Y is a closed subset of U.

2We follow [2] in our terminology. In particular, our submanifolds are locally closed.

Subanalytic neighborhoods of Y have the following property.

Lemma 2.6. Let Y be a submanifold of X which is also a subanalytic set in X. Let U and U′ be two subanalytic neighborhoods of Y such that U′ ⊂ U. Then the restriction res_{U,U′} induces an isomorphism of the space of tempered E-distributions on U with support in Y onto the space of all tempered E-distributions on U′ with support in Y.

Proof. By the preceding discussion, it is enough to show that the extension by zero of a tempered distribution on U′ with support in Z is tempered on U. Since U and Y are subanalytic in X, their difference U − Y is an open subanalytic set in X. Therefore, the assertion follows from 2.4.

If U and U′ are two arbitrary subanalytic neighborhoods of Y in X, their intersection U ∩ U′ is also a subanalytic neighborhood of Y in X. The restriction maps res_{U,U∩U′} and res_{U′,U∩U′} induce isomorphisms of the spaces of tempered Edistributions on U, resp. U′, with support in Y into the space of all tempered E-distributions on U ∩U′ with support in Y. It follows that the spaces of tempered E-distributions with support in Y on any subanalytic neighborhood of Y are naturally isomorphic. Therefore, by abuse of language, we call these distributions the tempered E-distributions supported in Y. We denote by T (Y, E) the space of all tempered E-distributions supported in Y. Clearly, T (Y, E), as a closed subspace of S(U, E)′ for an subanalytic neighborhood U of Y, has a natural structure of a DNF-space. This structure doesn’t depend on the choice of U by the open mapping theorem. Also, T (Y, E) is a right Diff(E)-module.

Now we want to define a natural exhaustive filtration on T (Y, E). First, fix a subanalytic neighborhood U of Y and consider T (Y, E) as a subspace of S(U, E)′. For p ∈ ℤ, let M_{p} denote the closed subspace of S(U, E) consisting of all sections s which vanish along Y with their derivatives of order ≤ p. Clearly, (M_{p}; p ∈ ℤ) is a decreasing filtration of S(U, E). We put F_{p} T (Y, E) = M_{p}^{⊥}, i.e., F_{p} T (Y, E) consists of all tempered E-distributions supported in Y which vanish on the sections in M_{p}. Then F T (Y, E) is an increasing filtration of T (Y, E) by closed subspaces.

First, we remark that this filtration is exhaustive.

Lemma 2.7. The filtration F T (Y, E) of T (Y, E) is exhaustive.

Proof. Fix a subanalytic neighborhood U of Y. Let Z = X − U. Let T be a distribution in T (Y, E). Then it is a restriction of an E-distribution T̄ on X supported in Y ∪ Z. Assume that the order of T̄ is ≤ p. Then T (s) = 0 for all smooth sections s of S(U, E) such that Ds vanishes on Y for all D ∈ Diff_{p}(E), i.e., for all s ∈ M_{p}. Therefore, T is in F_{p} T (Y, E).

If we consider S(U, E)′ as a filtered right Diff(E)-module with respect to the filtration induced by the order filtration, the above result means that the natural inclusion of T (Y, E) into S(U, E)′ is compatible with the filtrations.

Since F_{p} T (Y, E) are closed subspaces of T (Y, E), they are DNF-spaces with the induced topology. Therefore, by 2.7 and the open mapping theorem, we have the following result.

Proposition 2.8. As a locally convex space, T (Y, E) = \underrightarrow{lim}_{p→∞} F_{p} T (Y, E).

Let M_{0p} be the subspace of M_{p} consisting of sections with compact support contained in U.

Lemma 2.9. We have M_{0p}^{⊥} = M_{p}^{⊥} or any p ∈ ℤ.

Proof. Fix p ∈ ℤ. Obviously, M_{0p} ⊂ M_{p} and we have M_{p}^{⊥} ⊂ M_{0p}^{⊥}. Therefore, it is enough to show that M_{0p}^{⊥} ⊂ M_{p}^{⊥}.

Let T be a distribution T (Y, E) which vanishes on M_{0p}. Then there exists a distribution T̄ on X such that T is the restriction of T̄ to U. The support of T̄ is contained in Y ∪ Z. Assume first that the support of T̄ is “small”, i.e., there exists an open set V in X such that supp(T̄) ⊂ V, E|V is trivial and V can be covered by a coordinate chart. If supp(T̄) ∩ Z = ∅, the support of T in U is compact and it is clear that T is perpendicular to M_{p}.

Therefore, we can assume that supp(T̄) ∩ Z is a nonempty compact set. Let s be a smooth section in S(U, E). Then s vanishes with all of its derivatives on supp(T̄) ∩ Z. Since V can be covered by a coordinate chart, we can assume that V is an open set in ℝ^{n}. Let d be the distance function on R^{n}. Also, for any x in V let d(supp(T̄) ∩ Z, x) be the distance of x from supp(T̄) ∩ Z. As in the proof of [24, Ch. III, Th. XXXV], for small ϵ > 0, one can construct a smooth compactly supported function φ_{ϵ} which satisfies: φ_{ϵ}(x) = 1 if d(supp(T̄) ∩ Z, x) < ϵ/2 and φ_{ϵ}(x) = 0 if d(supp(T̄) ∩ Z, x) > ϵ, and such that φ_{ϵ}s and all of its derivatives tend to 0 uniformly on V as ϵ → 0. By extending φ_{ϵ} by zero outside V we see that smooth sections φ_{ϵ}s converge to 0 in S(U, E) as ϵ → 0. Therefore, it follows that lim_{ϵ→0} T̄(φ_{ϵ}s) = 0. If we put ϕ_{ϵ} = 1 − φ_{ϵ}, we get

lim_{ϵ→0}(ϕ_{ϵ}T̄)(s) = T (s) − lim_{ϵ→0} T̄(φ_{ϵ}s) = T (s),

On the other hand, ϕ_{ϵ}T̄ is a distribution with support contained in Y. Therefore, its restriction T_{ϵ} to U is a tempered distribution in T (Y, E) with compact support. By the above argument T_{ϵ} tends weakly to T as ϵ → 0. Clearly, T_{ϵ} vanish on M_{0p}. Hence, by the first part of the proof, they are perpendicular to M_{p}. It follows that T is also perpendicular to M_{p}. This completes the proof for T̄ with “small” support. The general case is reduced to this situation by a partition of unity argument.

By a partition of unity argument, from 2.9, we conclude that T ∈ T (Y, E) is in F_{p} T (Y, E) if for any y ∈ Y there exists an open neighborhood V ⊂ U of y such that T vanishes on all sections of M_{p} with support in V. This immediately implies that the filtration F T (Y, E) is independent of the choice of the subanalytic neighborhood U.

If T is in F_{p} T (Y, E), we say that the transversal degree of T is ≤ p.

Lemma 2.10. The space T (Y, E) is a filtered right Diff(E)-module.

Proof. Let D ∈ Diff_{s}(E) for s ∈ ℤ_{+}. Then D(M_{p+s}) ⊂ M_{p} for any p ∈ ℤ. Hence, D^{t}(F_{p} T (Y, E)) = D^{t}(M_{p}^{⊥}) ⊂ M_{p+s}^{⊥} = F_{p+s} T (Y, E) for all p ∈ ℤ.

Example 2.11. Consider again the situation described in 2.3. Let S^{n−1} be a (n − 1)-dimensional sphere in S^{n} passing through the pole p and such that under the stereographic projection S^{n−1} − {p} corresponds to the copy Y of ℝ^{n−1} in ℝ^{n} defined by the equation x_{n} = 0. Clearly, the restriction of smooth functions on S^{n} to S^{n−1} induces the restriction of Schwartz functions on ℝ^{n} to Schwartz functions on ℝ^{n−1}. The adjoint of this map is a natural inclusion of the space of tempered distributions on ℝ^{n−1} into the space T (Y) of tempered distributions on ℝ^{n} with support in Y. The filtration F T (Y) is described by

F_{p} T (Y) = {T ∈ S(ℝ^{n})′ | x_{n}^{p+1}T = 0}

for p ∈ ℤ_{+}. In particular, the space F_{0} T (Y) is equal to the image of the inclusion of S(ℝ^{n−1})′ into S(ℝ^{n})′ . Then, as it is well-known [24, Ch. III, §10], we have

T (Y) = ⨁_{i=0}^{∞} ∂_{n}^{i} F_{0} T (Y) and F_{p} T (Y) = ⨁_{i=0}^{p} ∂_{n}^{i} F_{0} T (Y)

for any p ∈ ℤ.

3. Filtrations attached to a stratification

Let Z_{1} ⊂ Z_{2} ⊂ ⋯ ⊂ Z_{n} ⊂ X be an increasing family of closed subanalytic sets in X (to simplify the notation, we assume that Z_{p} = ∅ for p ≤ 0 and Z_{p} = X for p > n). Let U_{p} = X − Z_{p} be their complements. They form a decreasing family of open sets in X. Assume that, for each p, Z_{p} − Z_{p−1} = Z_{p} ∩ U_{p−1} is a submanifold in X.

Then we can define a natural decreasing filtration F C^{∞}(E) of C^{∞}(E) by the Schwartz subspaces

C^{∞}(E) = S(X, E) ⊃ S(U_{1}, E) ⊃ ⋯ ⊃ S(U_{n}, E) ⊃ {0},

i.e., we put F_{p} C^{∞}(E) = C^{∞}(E) for p ≤ 0, F_{p} C^{∞}(E) = S(U_{p}, E) for 1 ≤ p ≤ n and F_{p} C^{∞}(E) = {0} for p > n. We want to describe the graded object Gr C^{∞}(E) corresponding to this filtration.

The orthogonals (F_{p} C^{∞}(E))^{⊥} to the subspaces F_{p} C^{∞}(E) in the filtration are closed subspaces in the space C^{∞}(E)′ of E-distributions on X. Moreover, they form an increasing filtration of C^{∞}(E)′, i.e.,

{0} = (F_{0} C^{∞}(E))^{⊥} ⊂ (F_{1} C^{∞}(E))^{⊥} ⊂ … ⊂ (F_{n} C^{∞}(E))^{⊥} ⊂ C^{∞}(E)′.

Of course, the spaces (F_{p} C^{∞}(E))^{⊥} are just the spaces of E-distributions supported in Z_{p}. By our previous discussion, we have the following short exact sequences

0 ⟶ (F_{p} C^{∞}(E))^{⊥} ⟶ C^{∞}(E)′ ⟶ S(U_{p}, E)′ ⟶ 0.

By our assumption, Z_{p+1} − Z_{p} are closed submanifolds in U_{p}. Moreover, they are subanalytic sets in X. Therefore, for each p, U_{p} is a subanalytic neighborhood of Z_{p+1} − Z_{p} in X. Hence, the above short exact sequences give the following exact sequences

0 ⟶ (F_{p} C^{∞}(E))^{⊥} ⟶ (F_{p+1} C^{∞}(E))^{⊥} ⟶ T (Z_{p+1} − Z_{p}, E) ⟶ 0.

Consider now short exact sequences

0 ⟶ F_{p+1} C^{∞}(E) ⟶ F_{p} C^{∞}(E) ⟶ Gr^{p} C^{∞}(E) ⟶ 0.

Since we have (F_{p} C^{∞}(E))′ = C^{∞}(E)′/(F_{p} C^{∞}(E))^{⊥}, it follows that

(Gr^{p} C^{∞}(E))′ = (F_{p+1} C^{∞}(X))^{⊥}/(F_{p} C^{∞}(X))^{⊥} = T (Z_{p+1} − Z_{p}, E)

i.e., the dual of Gr^{p} C^{∞}(E) is the space of all tempered E-distributions supported in Z_{p+1} − Z_{p}. This establishes the following result.

Theorem 3.1. The graded object corresponding to the filtration F C^{∞}(E) of C^{∞}(E) is

Gr C^{∞}(E) = ⨁_{p∈ℤ} T (Z_{p} − Z_{p−1}, E)′

as a left Diff(E)-module and an NF-space.

A typical application of this result is in the following situation. Let G be a real Lie group. Let X be a compact analytic manifold with an analytic action of G. Let E be a homogeneous analytic vector bundle on X. Then G acts naturally on the space C^{∞}(E) of all smooth sections of E. Therefore, C^{∞}(E) is a representation of G. Let 𝔤 be the complexified Lie algebra of G and U(𝔤) its enveloping algebra. By differentiating the action of G on sections of E we get a homomorphism of U(𝔤) into Diff(E), i.e, C^{∞}(E) has a natural structure of a U(𝔤)-module. Moreover, the composition of the principal antiautomorphism of U(𝔤) with the natural homomorphism of U(𝔤) into Diff(E) defines a homomorphism of U(𝔤) into the opposite algebra of Diff(E). Therefore, the right Diff(E)-module of E-distributions C^{∞}(E)′ can be viewed as a left U(𝔤)-module. The action of U(𝔤) on this space is just the contragredient action.

Let N be a connected closed subgroup of G. We assume in the following that N acts on X with finitely many orbits. Therefore, the orbits of N in X are connected submanifolds [30, 5.2.4.1], and subanalytic sets in X [5].

Lemma 3.2. Let O be an N-orbit in X. The boundary ∂O = Ō − O of O is a union of finitely many N-orbits of dimension strictly smaller than dim O.

Proof. The closure Ō is a N-invariant subanalytic set in X. Therefore, it is a union of finitely many N-orbits. Since ∂O = Ō − O is a subanalytic set of dimension ≤ dim O − 1 [27, 1.16.(3)], we see that ∂O is a union of finitely many N-orbits of dimension strictly smaller than dim O.

For any p ∈ ℤ denote by Z_{p} the union of all N-orbits of dimension < p. By 3.2, it follows that Z_{p} are closed subanalytic sets in X. We define the filtration of C^{∞}(E) as in the above discussion. Clearly, the closed subspaces F_{p} C^{∞}(E) are not G-invariant. On the other hand, they are invariant for the action of U(𝔤). Therefore, F C^{∞}(E) is a decreasing filtration of C^{∞}(E) by left U(𝔤)-submodules. In addition to 3.1, we have the following result.

Lemma 3.3. The space of all tempered distributions supported in Z_{p+1} − Z_{p} is equal to

⨁_{dim O=p} T (O, E)

where O are N-orbits in X.

Proof. Let O_{1}, O_{2}, …, O_{q} be all p-dimensional N-orbits in X. Then, for any 1 ≤ i ≤ q,

U_{i} = X − (Z_{p} ∪ ⋃_{j≠i} O_{j})

is an open subanalytic set in X containing O_{i}. Moreover, by 3.2, the orbit O_{i} is closed in U_{i}, i.e., U_{i} is a subanalytic neighborhood of O_{i}. On the other hand, U_{i} ∩ O_{j} = ∅ for i ≠ j. Since the union U_{i}, 1 ≤ i ≤ q, is equal to X − Z_{p}, by 2.4, we see that

T (Z_{p+1} − Z_{p}, E) = ⨁_{i=1}^{q} T (O_{i}, E).

Denote by J(O, E) the strong dual of T (O, E). This finally implies the following result.

Theorem 3.4. The graded U(𝔤)-module corresponding to C^{∞}(E) equipped with the filtration F C^{∞}(E) is equal to

Gr C^{∞}(E) = ⨁_{p∈ℤ} ( ⨁_{dim O=p} J(O, E)) .

In particular, T (O, E) has a natural structure of a filtered left U(𝔤)-module. Finally, we have the following simple observation. We recall the definition of

the filtration by the transversal degree from §2.

Lemma 3.5. The subspaces F_{q} T (O, E), q ∈ ℤ, of T (O, E) are U(𝔫)-invariant.

Proof. Consider the space M_{p} of all smooth sections of E vanishing with their derivatives of order ≤ p along the orbit O. Then, M_{p} is obviously N-invariant. Therefore, it is invariant for the action of 𝔫. Therefore, the assertion follows from the definition of the filtration.

4. Bruhat filtration

Let G be a reductive Lie group of Harish-Chandra class [15]. Denote by 𝔤_{0} its Lie algebra. Let K be a maximal compact subgroup of G and 𝔨_{0} its Lie algebra. Let 𝔤_{0} = 𝔨_{0} ⊕ 𝔮_{0} be the corresponding Cartan decomposition. Let 𝔞_{0} be a maximal abelian Lie subalgebra of 𝔮_{0}. Let 𝔤, 𝔨 and 𝔞 be the complexifications of 𝔤_{0}, 𝔨_{0} and 𝔞_{0} respectively. Let R be the system of (restricted) roots of (𝔤, 𝔞) in 𝔞^{∗}. Let W = W (R) be the Weyl group of R. Fix a set of positive roots R^{+} in R. Let 𝔫 be the subalgebra of 𝔤 spanned by the root subspaces of 𝔤 corresponding to the roots in R^{+}. Put 𝔫_{0} = 𝔫∩𝔤_{0}. Let A be the connected abelian Lie subgroup of G with Lie algebra 𝔞_{0}. Let N be the connected nilpotent Lie subgroup of G with Lie algebra 𝔫_{0}. Then we have the Iwasawa decomposition G = KAN of G.

Let P be the minimal parabolic subgroup of G containing N and P = MAN its Langlands decomposition. Denote by X = P \G the corresponding real flag variety. The actions of groups P and N by the right translations on G induce the actions on X. The orbits of these actions are identical, and by Bruhat lemma, parametrized by the Weyl group W. The group G acts by conjugation on the set of all minimal parabolic subgroups of G and this action is transitive. Since minimal parabolic subgroups are their own normalizers, the map x ⟼ S_{x} which attaches to a point in X its stabilizer S_{x} in G is a bijection of X onto the set of all minimal parabolic subgroups in G. For w ∈ W, let 𝔫^{w} be the Lie algebra spanned by the root subspaces of 𝔤 corresponding to the roots in w(R^{+}). Let 𝔫_{0}^{w} = 𝔫^{w} ∩ 𝔤_{0}. We denote by N^{w} the connected nilpotent subgroup of G with Lie algebra 𝔫_{0}^{w}. Then P^{w} = MAN^{w} is a minimal parabolic subgroup in G which determines a point x_{w} in X. Let C(w) be the N-orbit of x_{w}. It is called the Bruhat cell attached to w.

The map W ∋ w ⟼ C(w) is the bijection between W and the set of N-orbits in X. For any p ∈ ℤ_{+}, we denote by W (p) the set of all elements w ∈ W such that dim C(w) = p. Also, as in the example of the orbit stratification in the last section, we denote by Z_{p} the union of all Bruhat cells C(w) for w ∈ W (q), q < p ∈ ℤ_{+}. Let n = dim X.

Let E be the homogeneous vector bundle on X attached to a finite-dimensional representation σ of P trivial on N. Then, the representation C^{∞}(E) of G is nothing else than the (smooth) induced representation Ind^{∞}(σ) of G. Denote by C_{p}^{∞}(σ) the space of all smooth sections of Ind^{∞}(σ) which vanish with all of their derivatives on Z_{p}. As we explained in the preceding section, then

Ind^{∞}(σ) = C_{0}^{∞}(σ) ⊃ C_{1}^{∞}(σ) ⊃ ⋯ ⊃ C_{p}^{∞}(σ) ⊃ ⋯ ⊃ C_{n+1}^{∞}(σ) = {0}.

is a finite decreasing filtration of Ind^{∞}(σ) by U(𝔤)-submodules. This filtration is called the Bruhat filtration of Ind^{∞}(σ).

As a specialization of 3.4, we get the following result (here we put J(w, σ) = J(C(w), E)).

Theorem 4.1. The graded module corresponding to the Bruhat filtration of Ind^{∞}(σ) is the U(𝔤)-module

⨁_{p∈ℤ_{+}} ( ⨁_{w∈W (p)} J(w, σ)) .

More precisely, the following sequences

0 → C_{p+1}^{∞}(σ) → C_{p}^{∞}(σ) → ⨁_{w∈W (p)} J(w, σ) → 0

are exact for p ∈ ℤ_{+}.

In the preceding section we described a natural increasing filtration of J(w, σ)′. Now we want to describe the graded modules in this filtration more precisely. This will follow from a more explicit description of T (C(w), E) which we are going to discuss now.

First, we fix a subanalytic neighborhood of C(w). Clearly, the stabilizer of x_{w} in N is N ∩ N^{w}. Let w_{0} be the longest element of W. Then w_{0}(R^{+}) = −R^{+}. Let U^{w} = N^{ww_{0}}. Then the complexified Lie algebra of U^{w} is spanned by root subspaces corresponding to the roots in −w(R^{+}). Therefore C(w) is actually the orbit of x_{w} under the action of N ∩ U^{w}. The map u ⟼ u · x_{w} is a diffeomorphism of U^{w} onto the U^{w}-orbit V^{w} of x_{w} which is open and dense in X. Therefore, V^{w} is an open subanalytic set in X. By [1, V.21.14], the isomorphism U^{w} ⟶ V^{w} maps the closed connected subgroup N ∩ U^{w} of U^{w} onto C(w), i.e., C(w) is a closed submanifold in V^{w}. Therefore, V^{w} is a subanalytic neighborhood of the Bruhat cell C(w).

In discussion of the geometry of X = P \G we can assume without any loss of generality that G is a group of real points of a complex reductive group G defined over ℝ. Then P is the group of real points of a parabolic subgroup P of G defined over ℝ. Therefore X is the space of real points of P\G. Let N be the unipotent radical of P. Then its group of real points is equal to N. The group U^{w} is the group of real points of a unipotent subgroup U^{w} of G defined over ℝ. Let 𝔲^{w} be the Lie algebra of U^{w}. The exponential map exp : 𝔲^{w} ⟼ U^{w} is an algebraic isomorphism of the Lie algebra 𝔲^{w} onto the group U^{w}. The restriction of the exponential map to the Lie algebra 𝔲_{0}^{w} of U^{w} is a diffeomorphism onto U^{w}.

In [12, §1], Du Cloux defined the notion of the Schwartz space of functions on the space of real points of a smooth algebraic variety defined over ℝ. By [12, 1.3.6.(iii)], his definition of the Schwartz space for U^{w} agrees with the “classic” one after the identification of U^{w} with 𝔲_{0}^{w} under the exponential map.

Moreover, in [12, 1.2.4.(i)], he proves that the pullback by the orbit map defines a topological isomorphism of the Schwartz space S(V^{w}) in sense of Section 2 with the Schwartz space S(U^{w}) on U^{w} we just described. Therefore, the relative Schwartz space S(V^{w}) is isomorphic to the Schwartz space on 𝔲_{0}^{w} viewed as an euclidean space.

Let E_{w} be the fiber of E at x_{w}. The stabilizer P^{w} of x_{w} in G acts on E_{w}. We denote by σ^{w} this representation of P^{w}. Let g ∈ G be an element such that x_{w} = g · x_{1}. Then P = g^{−1}P^{w}g. Since all Levi factors in P are N-conjugate, without loss of generality we can assume that g normalizes the common Levi factor MA of P and P^{w}. The representation σ^{w} is equivalent to P_{w} ∋ p ⟼ σ(g^{−1}pg).

To any section s of C^{∞}(E) we can attach the function ϕ_{s} given by V^{w} ∋ u·x_{w} ⟼ u^{−1}s(u·x_{w}) ∈ E_{w}. Clearly, ϕ_{s} can be viewed as a function in C^{∞}(V^{w})⊗_{ℂ} E_{w}. Actually, by [12, 2.2.4], the map s ⟼ ϕ_{s} is an isomorphism of the NF-space S(V^{w}, E) onto S(V^{w}) ⊗_{ℂ} E_{w}.

The adjoint of this map is an isomorphism of S(V^{w})′ ⊗_{ℂ} E_{w}^{∗} with S(V^{w}, E)′. Therefore, we get the following result.

Lemma 4.2. The space of all tempered distributions T (C(w), E) supported in the Bruhat cell C(w) is isomorphic to the tensor product of the space of all tempered distributions T (N ∩ U^{w}) on U^{w} with support in N ∩ U^{w} with E_{w}^{∗}.

The restriction of a Schwartz function on U^{w} is a Schwartz function on N ∩U^{w}. This gives a natural surjection of S(U^{w}) onto S(N ∩ U^{w}). The dual map defines an injection of S(N ∩ U^{w})′ into S(U^{w})′. Clearly, under the above isomorphism, S(N ∩U^{w})′ ⊗_{ℂ} E_{w}^{∗} corresponds to F_{0} T (C(w), E). The group MA acts on the space E_{w}^{∗} by the contragredient of σ^{w}. We can extend this action to a representation of P trivial on N. Then the above isomorphism of Gr^{0} T (C(w), E) = F_{0} T (C(w), E) with S(C(w))′ ⊗_{ℂ} E_{w}^{∗} is an isomorphism of U(𝔭)-modules.

Our final result in this section is a generalization of this module isomorphism to Gr^{p} T (C(w), E), p ∈ ℤ. For a real affine space Y we denote by R(Y) the ring of polynomials with complex coefficients on Y. Let J denote the ideal of all polynomials in R(U^{w}) vanishing along N ∩ U^{w}. Then, by the above isomorphism, the multiplication map defines a map J ⊗_{R(U_{w})} F_{p} T (C(w), E) ⟶ F_{p−1} T (C(w), E) of U(𝔭)- modules. By iteration, we get a map J^{p} ⊗_{R(U_{w})} F_{p} T (C(w), E) ⟶ F_{0} T (C(w), E). Moreover, this map factors through

J^{p}/J^{p+1} ⊗_{R(U_{w})} Gr^{p} T (C(w), E) = J^{p}/J^{p+1} ⊗_{R(N∩U^{w})} Gr^{p} T (C(w), E).

Therefore, we get a U(𝔭)-module homomorphism

J^{p}/J^{p+1} ⊗_{R(N∩U^{w})} Gr^{p} T (C(w), E) ⟶ F_{0} T (C(w), E).

This in turn leads to a U(𝔭)-module homomorphism

α_{p} : Gr^{p} T (C(w), E) ⟶ Hom_{R(N∩U^{w})}(J^{p}/J^{p+1}, F_{0} T (C(w), E)).

By a result giving the description of the space of all tempered distributions on ℝ^{n} with support in ℝ^{k} analogous to 2.11, we easily see that this map is an isomorphism, i.e., we have the following result.

We put L_{p}(w) = Hom_{R(C(w))}(J^{p}/J^{p+1}, R(C(w))). Clearly, this is an Nequivariant R(C(w))-module.

Lemma 4.3. For any p ∈ ℤ, we have a natural isomorphism

α_{p} : Gr^{p} T (C(w), E) ⟶ L_{p}(w) ⊗_{R(C(w))} S(C(w))′ ⊗_{ℂ} E_{w}^{∗}

of U(𝔭)-modules.

Clearly, L_{p}(w) is a free R(C(w))-module of finite rank. We shall need also the following observation.

Lemma 4.4. The R(C(w))-module L_{p}(w) has a finite increasing filtration by N-equivariant R(C(w))-submodules F L_{p}(w) such that the graded module is isomorphic, as an N-equivariant R(C(w))-module, to a direct sum of copies of R(C(w)).

Proof. If we consider N as the group of real points of an unipotent complex algebraic group N defined over ℝ, S^{w} = U^{w} ∩ N is the subgroup of N of real points of an algebraic subgroup S^{w} of N. The homogeneous space S^{w}\N is an affine variety [1, 21.14], and we can view R(C(w)) as the ring of regular functions on S^{w}\N. The R(C(w))-module L_{p}(w) is the module of global sections of a free O_{S^{w}\N}-module V_{p} of finite rank. This O_{S^{w}\N}-module is a sheaf V_{p} of local sections of a N-homogeneous algebraic vector bundle V_{p} on S^{w}\N. Since S^{w} is unipotent, its representation on the geometric fiber of V_{p} at the identity coset is unipotent. Therefore, V_{p} has an increasing finite filtration by N-homogeneous algebraic vector subbundles F V_{p} such that F_{q} V_{p}/ F_{q−1} V_{p} are line bundles corresponding to the trivial representation of S^{w}, i.e., trivial line bundles on S^{w}\N. The sheaves F_{q} V_{p} of local sections of F_{q} V_{p} define an increasing filtration of the O_{S^{w}\N}-module V_{p}, and we have the exact sequences

0 ⟶ F_{q−1} V_{p} ⟶ F_{q} V_{p} ⟶ O_{S^{w}\N} ⟶ 0

of locally free coherent O_{S^{w}\N}-modules. Since S^{w}\N is affine, by Serre’s theorem, this leads to the exact sequences of the corresponding global sections

0 ⟶ Γ(S^{w}\N, F_{q−1} V_{p}) ⟶ Γ(S^{w}\N, F_{q} V_{p}) ⟶ R(C(w)) ⟶ 0.

As a result, the submodules Γ(S^{w}\N, F_{q} V_{p}) of the R(C(w))-module L_{p}(w) = Γ(S^{w}\N, V_{p}) define an exhaustive increasing finite filtration F L_{p}(w) of L_{p}(w) such that

0 ⟶ F_{p−1} L_{p}(w) ⟶ F_{p} L_{p}(w) ⟶ R(C(w)) ⟶ 0

is exact. From the construction it is evident that this is a filtration by N-equivariant R(C(w))-submodules.

5. Lie algebra homology of the twisted regular representation on the Schwartz space

For any U(𝔫)-module V we denote by H_{p}(𝔫, V) (resp. H^{p}(𝔫, V)), p ∈ ℤ_{+}, the p-th Lie algebra (co)homology group with coefficients in V.

In this section we prove a vanishing theorem for the Lie algebra homology of some twisted regular representations of nilpotent Lie groups on Schwartz spaces on their homogeneous spaces.

Let N be a simply connected, connected nilpotent Lie group. Let 𝔫_{0} be its Lie algebra. We denote by 𝔫 its complexification, and by U(𝔫) the universal enveloping algebra of 𝔫. Let U be a connected Lie subgroup of N. Let U\N be the right coset space of N with respect to U. Then N acts differentiably on it by right translations. This action defines the right regular action of N on the space C^{∞}(U\N) of smooth complex valued functions on U\N. This action is clearly differentiable and leads to a U(𝔫)-module structure on C^{∞}(U\N). We say that a smooth function f on U\N is a polynomial if its annihilator in U(𝔫) contains 𝔫^{k} for sufficiently large k ∈ ℤ_{+}. Clearly, polynomials on U\N form a subalgebra of C^{∞}(U\N) stable under the action of N. The action of 𝔫 on U\N is given by a morphism τ of 𝔫 into the Lie algebra of tangent vector fields on U\N. Let Diff(U\N) be the algebra of all differential operators on U\N with smooth coefficients. Let D_{U\N} be the subalgebra of Diff(U\N) generated by the vector fields in τ(𝔫) and polynomials on U\N. We call D_{U\N} the algebra of polynomial differential operators on U\N. Clearly, D_{U\N} is stable under the action of N.

A Schwartz function on U\N is a smooth function f on U\N such that Df is bounded on U\N for any polynomial differential operator D ∈ D_{U\N} [12]. Let S(U\N) be the space of all Schwartz functions on U\N. Then, we can equip it with a locally convex topology given by the seminorms

f ⟼ ∥f∥_{D} = sup_{x∈U\N} |Df(x)|.

The following result is established in [12].

Lemma 5.1. (i) S(U\N) is an NF-space.

(ii) The action of N on S(U\N) defines a differentiable representation ρ of N. The differential of this representation is given by the natural action of 𝔫.

Continuous linear forms on S(U\N) are called tempered distributions on U\N. Let 𝔲_{0} be the Lie algebra of U. Denote by 𝔲 its complexification. Let 𝔠_{0} be a

one-dimensional central subalgebra of 𝔫_{0}. Assume that 𝔠_{0} ⊄ 𝔲_{0}. Then 𝔳_{0} = 𝔲_{0} ⊕ 𝔠_{0} is a Lie subalgebra of 𝔫_{0} and dim_{ℝ} 𝔳_{0} = dim_{ℝ} 𝔲_{0} + 1. Denote by C and V the connected Lie subgroups corresponding to 𝔠_{0} and 𝔳_{0} respectively. Clearly, C is isomorphic to ℝ. Then the fibers of natural projection p : U\N → V \N are exactly the C-orbits in U\N, i.e., V \N is the quotient of U\N under the action of C. Therefore, the map g ⟼ g ∘ p defines a homomorphism of C^{∞}(V \N) into the subalgebra of C^{∞}(U\N) of all smooth C-invariant functions on U\N.

We fix a Haar measure on C. The following results can be proved easily using simple facts on the structure of U\N (see, for example, [30, 1.1.4.1]) or deduced from the induction in stages result in [12, 2.1.6].

Lemma 5.2. Let f be a Schwartz function on U\N. Then

(i) The integral

f̄(x) = ∫_{C}(ρ(c)f)(x) dc

is finite for any x ∈ U\N. (ii) The function f̄ is a C-invariant smooth function on U\N. (iii) Let f_{C} be the unique smooth function on V \N such that f̄ = f_{C} ∘ p. Then

f_{C} is a Schwartz function.

Therefore, we have a natural linear map π : f ⟼ f_{C} of S(U\N) into S(V \N). On the other hand, let ξ be a nonzero element of 𝔠_{0}. Then it defines a polynomial differential operator τ(ξ) on U\N. Moreover, τ(ξ) is a linear endomorphism of S(U\N).

Lemma 5.3.

0 → S(U\N) ⟶^{τ(ξ)} S(U\N) ⟶^{π} S(V \N) → 0

is an exact sequence in the category of smooth representations of N.

This can be interpreted in the following way. Let 𝔠 be the complexification of the Lie algebra 𝔠_{0}.

Proposition 5.4.

H_{p}(𝔠, S(U\N)) = \begin{cases}0 & if p ≠ 0; \\ S(V \N) & if p = 0. \end{cases}

Finally, the group N is unimodular. We fix a Haar measure on N. Then, for any Schwartz function f on N, the integral ∫_{N} f(n) dn is finite. Moreover, we have the following result.

Lemma 5.5. The map f ⟼ ∫_{N} f(n) dn is a tempered distribution on N.

Proof. We prove this statement by induction in dim N. If dim N = 1, N ≅ ℝ and the statement is obvious.

Let dim N > 1. Then we can pick a one-dimensional central Lie subalgebra 𝔠_{0} of 𝔫_{0} and denote by C the corresponding connected Lie subgroup. Then, with an appropriate choice of a Haar measure on C\N, we have

∫_{N} f(n) dn = ∫_{C\N} (∫_{C} f(nc)dc) dṅ.

Hence, the statement follows from 5.2 and the induction assumption.

Since this tempered distribution is N-invariant, it vanishes on all Schwartz functions of the type ξ · f, ξ ∈ 𝔫, f ∈ S(N).

Let ψ be a character of N, i.e., a differentiable homomorphism of N into ℂ^{∗}. Let η be the differential of ψ extended to a complex valued linear form η on 𝔫. Since ψ is a character, η vanishes on [𝔫, 𝔫]. The character ψ is unitary if and only if Re η = 0.

Denote by ℂ_{η} the one-dimensional U(𝔫)-module with action η. Consider the tensor product S(U\N) ⊗_{ℂ} ℂ_{η}. This is the tensor product of the regular representation of N on S(U\N) with the one-dimensional representation given by character ψ. We call this representation a twisted regular representation of N. Clearly, it is a differentiable representation of N. Therefore, it has a natural U(𝔫)-module structure. In this section we establish the following result for the Lie algebra homology of twisted regular representations.

Theorem 5.6. (i) H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) are finite-dimensional for all p ∈

ℤ_{+}. (ii) H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) = 0 for p > dim U. (iii) If either Re η ≠ 0 or η|𝔲 ≠ 0, we have H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) = 0 for all

p ∈ ℤ_{+}.

First we consider the case of abelian N. In this situation, N = U × V for some complementary abelian Lie subgroup V. Hence, S(U\N) = S(V) and the action of U on this space is trivial. Let 𝔳_{0} be the Lie algebra of V and 𝔳 its complexification.

Assume that η|𝔲 ≠ 0. Then dim U > 0 and we can pick a one-dimensional connected Lie subgroup D of U with Lie algebra 𝔡_{0} such that η|𝔡_{0} ≠ 0. Let 𝔡 be the complexification of 𝔡_{0}. Let ξ be a generator of 𝔡. Then, H_{p}(𝔡, S(V) ⊗_{ℂ} ℂ_{η}) is the cohomology of the complex

⋯ → 0 → S(V) ⊗_{ℂ} ℂ_{η} ⟶^{η(ξ)} S(V) ⊗_{ℂ} ℂ_{η} → 0 → …,

i.e., all of them vanish. By the Hochschild-Serre spectral sequence

H_{p}(𝔫/𝔡, H_{q}(𝔡, S(V) ⊗_{ℂ} ℂ_{η})) ⇒ H_{p+q}(𝔫, S(V) ⊗_{ℂ} ℂ_{η}),

we see that H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) = 0 for all p ∈ ℤ_{+}.

Therefore, we can assume that η|𝔲 = 0. In this case, the action of U on S(U\N) ⊗_{ℂ} ℂ_{η} is trivial, and from the standard complex of Lie algebra homology we see that

H_{p}(𝔲, S(U\N) ⊗_{ℂ} ℂ_{η}) = ⋀^{p} 𝔲 ⊗_{ℂ} S(U\N) ⊗_{ℂ} ℂ_{η}

for p ∈ ℤ_{+}. To complete the argument, we have to calculate H_{p}(𝔳, S(V) ⊗_{ℂ} ℂ_{η}) for p ∈ ℤ_{+}.

Lemma 5.7. (i) If Re η ≠ 0, we have

H_{p}(𝔳, S(V) ⊗_{ℂ} ℂ_{η}) = 0

for p ∈ ℤ_{+}. (ii) If Re η = 0, we have

H_{p}(𝔳, S(V) ⊗_{ℂ} ℂ_{η}) = \begin{cases}0 & for p ≠ 0; \\ ℂ & for p = 0. \end{cases}

We prove this lemma by induction in dimension of V. Clearly, V = ℝ^{n}. Let 𝔡_{0} be a one-dimensional Lie subalgebra of 𝔳_{0} and D the corresponding connected Lie subgroup. Let 𝔡 be the complexification of 𝔡_{0}.

Lemma 5.8. (i) H_{p}(𝔡, S(V) ⊗_{ℂ} ℂ_{η}) = 0 for p > 0.

(ii) If Re η|𝔡 ≠ 0, we have

H_{0}(𝔡, S(V) ⊗_{ℂ} ℂ_{η}) = 0.

Proof. Without any loss of generality we can assume that D is the first coordinate axis in ℝ^{n}. Then the unit vector (1, 0, …, 0) in 𝔡 acts on S(ℝ^{n}) as ∂_{1} and on ℂ_{η} by the first coordinate η_{1} = η(1, 0, …, 0) of η. It follows that the Lie algebra homology is the cohomology of the complex

0 → S(ℝ^{n}) ⟶^{∂_{1}+η_{1}} S(ℝ^{n}) → 0.

If f is in the kernel of D = ∂_{1} + η_{1}, it must satisfy ∂_{1}f = −η_{1}f. By integration we see that f(x_{1}, x_{2}, …, x_{n}) = e^{−η_{1}x_{1}}f(0, x_{2}, …, x_{n}). Hence, f can be rapidly decreasing if and only if f = 0. Therefore, the kernel of D is zero, and (i) follows.

It remains to study the image of D. Consider the equation Df = g where f, g ∈ S(ℝ^{n}). Define the Fourier transform

ĥ(t_{1}, t_{2}, …, t_{n}) = \frac{1}{(2π)^{\frac{n}{2}}} ∫_{ℝ^{n}} h(x_{1}, x_{2}, …, x_{n})e^{−i(x_{1}t_{1}+x_{2}t_{2}+···+x_{n}t_{n})}dx_{1}dx_{2} … dx_{n}.

Then the Fourier transform of our equation looks like

(it_{1} + η_{1})f̂(t_{1}, t_{2}, …, t_{n}) = ĝ(t_{1}, t_{2}, …, t_{n}), (t_{1}, t_{2}, …, t_{n}) ∈ ℝ^{n}.

Assume now that Re η_{1} ≠ 0. Then the function (t_{1}, t_{2}, …, t_{n}) ⟼ \frac{1}{t_{1}−iη_{1}} is a smooth function on ℝ^{n} and all of its derivatives are bounded on ℝ^{n}. Therefore, in this case, since ĝ is in S(ℝ^{n}), the function

f̂: (t_{1}, t_{2}, …, t_{n}) ⟼ −i\frac{ĝ(t_{1}, t_{2}, …, t_{n})}{t_{1} − iη_{1}}

is a Schwartz function f̂ on ℝ^{n}. Its inverse Fourier transform is a Schwartz function f which satisfies Df = g. Hence, in this case, coker D = 0.

Assume that Re η ≠ 0. Then we can pick 𝔡_{0} so that Re η|𝔡_{0} ≠ 0. By the Hochschild-Serre spectral sequence

H_{p}(𝔳/𝔡, H_{q}(𝔡, S(V) ⊗_{ℂ} ℂ_{η})) ⇒ H_{p+q}(𝔳, S(V) ⊗_{ℂ} ℂ_{η}),

and we see that

H_{p}(𝔳, S(V) ⊗_{ℂ} ℂ_{η}) = 0, for all p ∈ ℤ_{+}.

Therefore, we can assume that Re η = 0. In this case, the function

(x_{1}, x_{2}, …, x_{n}) ⟼ e^{−(η_{1}x_{1}+η_{2}x_{2}+···+η_{n}x_{n})}

is smooth and all its partial derivatives are bounded. Hence multiplication by this function defines a linear automorphism of S(ℝ^{n}). This automorphism intertwines the representation on S(ℝ^{n})⊗_{ℂ}ℂ_{η} with the regular representation on S(ℝ^{n}). Therefore, to prove the statement we can assume that η = 0.

Now we can prove the rest of 5.7 by induction in the dimension of V. If dim V = 1, the statement follows from 5.4.

Let 𝔡_{0} be one-dimensional Lie subalgebra in 𝔳_{0} and D the corresponding connected closed Lie subgroup of V. Denote by 𝔡 the complexification of 𝔡_{0}. Then, by 5.4, we have

H_{p}(𝔡, S(V)) = \begin{cases}0 & for p ≠ 0; \\ S(V/D) & for p = 0. \end{cases}

By the Hochschild-Serre spectral sequence, we have

H_{p}(𝔳/𝔡, H_{q}(𝔡, S(V))) ⇒ H_{p+q}(𝔳, S(V)).

Hence by the above result, this spectral sequence degenerates and we have

H_{p}(𝔳/𝔡, S(V/D)) = H_{p}(𝔳, S(V)).

Moreover, V/D is a connected and simply connected abelian Lie group. Therefore, the assertion of 5.7 follows from the induction assumption.

Now we go back to the proof of the theorem for abelian N. By the Hochschild-Serre spectral sequence, we have

H_{p}(𝔳, S(V) ⊗_{ℂ} ℂ_{η}) ⊗_{ℂ} ⋀^{q} 𝔲 = H_{p}(𝔳, H_{q}(𝔲, S(U\N) ⊗_{ℂ} ℂ_{η}))

⇒ H_{p+q}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}).

Hence, if Re η ≠ 0, by 5.7, we get H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) = 0 for all p ∈ ℤ_{+}. If Re η = 0, we see that

H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) = ⋀^{p} 𝔲

for all p ∈ ℤ_{+}. This establishes 5.6 for abelian N.

Now, we prove the theorem by a downward induction in dim N. In addition, we can assume that N is not abelian.

Let 𝔡_{0} be a one-dimensional Lie subalgebra of the center of 𝔫_{0}. Since 𝔫_{0} is not abelian, we can also assume that 𝔡_{0} ⊂ [𝔫_{0}, 𝔫_{0}]. Therefore, η|𝔡_{0} = 0. Let D be the corresponding connected Lie subgroup (which is in the center of N). Denote by 𝔡 the complexification of 𝔡_{0}.

Assume first that D ⊂ U. Then the action of D on U\N is trivial. Hence, the action on S(U\N) is also trivial. As before, this implies the following result.

Lemma 5.9. Assume that 𝔡 ⊂ 𝔲. Then

H_{p}(𝔡, S(U\N) ⊗_{ℂ} ℂ_{η}) = \begin{cases}0 & for p ≠ 0, 1; \\ S(U\N) ⊗_{ℂ} ℂ_{η} & for p = 0, 1; \end{cases}

as a 𝔫/𝔡-module.

Let N′ = N/D and U′ = U/D. Denote by 𝔫′_{0} and 𝔲′_{0} their Lie algebras, and by 𝔫′ and 𝔲′ their complexifications. Since η|𝔡 = 0, it defines a linear form η′ on 𝔫′. Clearly, U\N = U′\N′. By the Hochschild-Serre spectral sequence we have

H_{p}(𝔫′, H_{q}(𝔡, S(U\N) ⊗_{ℂ} ℂ_{η})) ⇒ H_{p+q}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}).

Hence, by 5.9, we see that

H_{p}(𝔫′, H_{q}(𝔡, S(U\N) ⊗_{ℂ} ℂ_{η})) = H_{p}(𝔫′, S(U′\N′) ⊗_{ℂ} ℂ_{η′})

for q = 0, 1 and 0 otherwise.

If η doesn’t vanish on 𝔲, η′ doesn’t vanish on 𝔲′. Therefore, by the induction assumption we get that the E^{2}-term in the above spectral sequence vanishes. Hence, the vanishing theorem follows in this case.

If Re η ≠ 0, we see that Re η′ ≠ 0, and by the induction assumption the E^{2}-term in the above spectral sequence vanishes again.

If neither of these conditions is satisfied, by the induction assumption, we see that the E^{2}-terms are finite-dimensional and can be nonzero only for 0 ≤ p ≤ dim U′ and q = 0, 1. Therefore, H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) are finite-dimensional and must vanish for p > dim U′ + 1 = dim U.

This completes the induction step in this case.

Assume now that D ⊄ U. In this case, 𝔳_{0} = 𝔲_{0} + 𝔡_{0} is a Lie subalgebra of 𝔫_{0} and 𝔲_{0} is a Lie subalgebra of 𝔳_{0} of codimension 1. Put 𝔳′_{0} = 𝔳_{0}/𝔡_{0}. Then 𝔳′_{0} is a Lie subalgebra of 𝔫′_{0}. Denote by V′ the corresponding connected Lie subgroup of N′. Also, denote by 𝔳 and 𝔳′ the complexifications of 𝔳_{0} and 𝔳′_{0} respectively.

Then we have the following result.

Lemma 5.10. We have

H_{p}(𝔡, S(U\N) ⊗_{ℂ} ℂ_{η}) = 0

for p > 0 and

H_{0}(𝔡, S(U\N) ⊗_{ℂ} ℂ_{η}) = S(V′\N′) ⊗_{ℂ} ℂ_{η′}.

Proof. This is essentially 5.4, since η vanishes on 𝔡.

Therefore, by the Hochschild-Serre spectral sequence we have

H_{p}(𝔫, S(U\N) ⊗ ℂ_{η}) = H_{p}(𝔫′, S(V′\N′) ⊗ ℂ_{η′}).

Clearly, if η|𝔲 ≠ 0, we have η|𝔳 ≠ 0 and η′|𝔳′ ≠ 0. Hence, the vanishing follows by the induction assumption. The same applies in the case Re η ≠ 0.

If neither condition is satisfied, by the induction assumption H_{p}(𝔫′, S(V′\N′)⊗ ℂ_{η′}) are finite-dimensional and vanish for p > dim V′. But dim V′ = dim V − 1 = dim U. Hence, we see that H_{p}(𝔫, S(U\N) ⊗_{ℂ} ℂ_{η}) are finite-dimensional and vanish for p > dim U.

This completes the proof of 5.6.

Let S(U\N)′ be the strong dual of S(U\N). The contragredient action to the regular action on S(U\N) defines a natural structure of a U(𝔫)-module on S(U\N)′. By dualizing, 5.6 implies the following result on the Lie algebra cohomology of S(U\N)′ ⊗_{ℂ} ℂ_{η}. Its proof is based on the following observation.

Lemma 5.11. Let E be a U(𝔫)-module. Assume that E is an NF-space (or a DNF-space) and that 𝔫 acts by continuous linear transformations on E. Denote by E′ the strong dual of E equipped with the contragredient action.

(i) Assume that H_{p}(𝔫, E), p ∈ ℤ_{+}, are finite-dimensional. Then, the Lie algebra cohomology spaces H^{p}(𝔫, E′) are linear duals of H_{p}(𝔫, E) for all p ∈ ℤ_{+}.

(ii) Assume that H^{p}(𝔫, E), p ∈ ℤ_{+}, are finite-dimensional. Then, the Lie algebra homology spaces H_{p}(𝔫, E′) are linear duals of H^{p}(𝔫, E) for all p ∈ ℤ_{+}.

Proof. We prove (i), the proof of (ii) is analogous. The Lie algebra homology of E is calculated from the standard complex C_{·}(𝔫, E) with C_{p}(𝔫, E) = ⋀^{−p} 𝔫 ⊗_{ℂ} E, p ∈ ℤ. By our assumption, this is a complex of NF-spaces (or DNF-spaces), its differentials are continuous and its cohomology spaces are finite-dimensional. By A.1, the differentials in this complex are all strict. Therefore, by A.2, the cohomology of the strong dual C_{·}(𝔫, E)′ is equal to H_{·}(𝔫, E)^{∗}. On the other hand, by [7, Ch. XIII, §8], the complex C_{·}(𝔫, E)′ is equal to the complex C^{·}(𝔫, E′) with C^{p}(𝔫, E) = Hom_{ℂ}(⋀^{p} 𝔫, E′), p ∈ ℤ, which calculates Lie algebra cohomology of E′.

Therefore, 5.6 and 5.11 imply the following result.

Theorem 5.12. (i) H^{p}(𝔫, S(U\N)′ ⊗_{ℂ} ℂ_{η}) are finite-dimensional for all

p ∈ ℤ_{+}. (ii) H^{p}(𝔫, S(U\N)′ ⊗_{ℂ} ℂ_{η}) = 0 for p > dim U. (iii) If either Re η ≠ 0 or η|𝔲 ≠ 0, we have H^{p}(𝔫, S(U\N)′ ⊗_{ℂ} ℂ_{η}) = 0 for all

p ∈ ℤ_{+}.

Finally, we want to prove a nonvanishing result for Lie algebra homology of twisted regular representations. Assume that U is trivial. Then, by 5.6, we have H_{p}(𝔫, S(N) ⊗_{ℂ} ℂ_{η}) = 0 for p > 0. Moreover, H_{0}(𝔫, S(N) ⊗_{ℂ} ℂ_{η}) = 0 if Re η ≠ 0. The only remaining case is treated in the following result.

Proposition 5.13. Assume that Re η = 0. Then

H_{0}(𝔫, S(N) ⊗_{ℂ} ℂ_{η}) = ℂ.

This isomorphism is induced by the map

f ⟼ ∫_{N} f(n)ψ(n)^{−1}dn

which is a tempered distribution on N.

Proof. Since Re η = 0, ψ is a unitary character. Clearly, ψ is smooth and all its partial derivatives are bounded. Hence multiplication by this function defines a linear automorphism of S(N) with itself. This automorphism intertwines the representation on S(N) with the representation on S(N) ⊗_{ℂ} ℂ_{η}. Therefore, to prove the statement we can assume that η = 0.

If dim N = 1, N is abelian and, by 5.4, we have H_{0}(𝔫, S(N)) = ℂ.

Let 𝔡_{0} be a one-dimensional central subalgebra of 𝔫_{0}. Denote by D the corresponding connected Lie subgroup. Let 𝔡 be the complexification of 𝔡_{0}. Then, by 5.4, we have H_{0}(𝔡, S(N)) = S(N/D). In addition, we have H_{0}(𝔫/𝔡, H_{0}(𝔡, S(N))) = H_{0}(𝔫, S(N)). Therefore, it follows that H_{0}(𝔫, S(N)) = H_{0}(𝔫/𝔡, S(N/D)), and by induction in dimension of N we conclude that H_{0}(𝔫, S(N)) = ℂ.

On the other hand, by 5.5, f ⟼ ∫_{N} f(n) dn is a tempered distribution on N, i.e., a continuous linear form on S(N). As we already remarked, it vanishes on 𝔫 · S(N), i.e., it factors through H_{0}(𝔫, S(N)).

As before, using 5.11, we deduce from this the following result.

Corollary 5.14. Assume that Re η = 0. Then H^{0}(𝔫, S(N)′ ⊗_{ℂ} ℂ_{η}) is onedimensional and spanned by the tempered distribution

f ⟼ ∫_{N} f(n)ψ(n)dn.

6. Whittaker vectors for smooth principal series

In this section we prove a generalization of a result of Kostant on the space of Whittaker vectors for smooth principal series [21, Theorem 6.6.2].^{3}

We return to the setting of §4. Let η : 𝔫 → ℂ be an one-dimensional representation of 𝔫 (i.e., a complex linear form which vanishes on [𝔫, 𝔫]). Let B be the basis of the root system R corresponding to the set of positive roots R^{+}. Since the root subspaces 𝔤_{α}, α ∈ B, span a complement of [𝔫, 𝔫], η is completely determined by its restrictions to these subspaces. We say that η is nondegenerate if η|𝔤_{α} ≠ 0 for all α ∈ B.

Any η can be uniquely written as η = Re η + i Im η for some complex linear forms Re η and Im η on 𝔫 which take real values on 𝔫_{0}.

Let σ be an irreducible finite-dimensional representation of P. Denote by Ind^{∞}(σ) the corresponding smooth principal series representation of G. Let Ind^{∞}(σ)′ be the strong dual of Ind^{∞}(σ) equipped by the adjoint action of U(𝔤). We say that a distribution ϕ ∈ Ind^{∞}(σ)′ is a Whittaker vector corresponding to a one-dimensional representation η of 𝔫 if ξ · ϕ = η(ξ)ϕ for ξ ∈ 𝔫.

The main result of this section is the following theorem.

Theorem 6.1. Let η be a nondegenerate one-dimensional representation of 𝔫. Then, for arbitrary irreducible finite-dimensional representation σ of P:

(i) The space of all Whittaker vectors in Ind^{∞}(σ)′ is zero if η is not purely imaginary.

(ii) If η is purely imaginary, the dimension of the space of all Whittaker vectors in Ind^{∞}(σ)′ is equal to dim_{ℂ}(σ).

If we consider the tensor product Ind^{∞}(σ)′ ⊗_{ℂ} ℂ_{−η} as a U(𝔫)-module, the linear map ϕ ⟼ ϕ ⊗ 1 is a linear isomorphism of the space of all Whittaker vectors in Ind^{∞}(σ)′ with the space of all 𝔫-invariants in Ind^{∞}(σ)′ ⊗_{ℂ} ℂ_{−η}.

Therefore, 6.1 follows immediately from the following result (after replacing η by its negative).

Theorem 6.2. Let η be a nondegenerate one-dimensional representation of 𝔫. Then, for arbitrary irreducible finite-dimensional representation σ of P, we have:

3Compare also [22].

(i) H^{p}(𝔫, Ind^{∞}(σ)′ ⊗_{ℂ} ℂ_{η}) = 0 for all p > 0.

(ii) If η is not purely imaginary, H^{0}(𝔫, Ind^{∞}(σ)′ ⊗_{ℂ} ℂ_{η}) = 0.

(iii) If η is purely imaginary, dim_{ℂ} H^{0}(𝔫, Ind^{∞}(σ)′ ⊗_{ℂ} ℂ_{η}) = dim_{ℂ}(σ).

Let Ind^{∞}(σ) = C_{0}^{∞}(σ) ⊃ … ⊃ C_{n}^{∞}(σ) ⊃ {0} be the Bruhat filtration of Ind^{∞}(σ). The last nontrivial subspace C_{n}^{∞}(σ) is also J(w_{0}, σ) where w_{0} is the longest element in the Weyl group W. The adjoint of the inclusion J(w_{0}, σ) → Ind^{∞}(σ) is the restriction map res : Ind^{∞}(σ)′ → J(w_{0}, σ)′ of distributions to the open Bruhat cell C(w_{0}).

The main step in the proof of 6.2 is the following lemma.

Lemma 6.3. The map res ⊗1 : Ind^{∞}(σ)′ ⊗ ℂ_{η} → J(w_{0}, σ)′ ⊗ ℂ_{η} induces isomorphisms on Lie algebra cohomology, i.e.,

H^{p}(res ⊗1) : H^{p}(𝔫, Ind^{∞}(σ)′ ⊗ ℂ_{η}) → H^{p}(𝔫, J(w_{0}, σ)′ ⊗ ℂ_{η})

for p ∈ ℤ_{+}, are isomorphisms.

In particular, the restriction map on Whittaker vectors in Ind^{∞}(σ)′ is injective, i.e., the support of any nontrivial Whittaker vector is equal to X.

Assuming 6.3 for a moment, we can complete the proof of 6.2. Since C(w_{0}) is open in X, the filtration by the transversal degree on J(w_{0}, σ)′ is trivial. Therefore, by 4.3, U(𝔫)-module J(w_{0}, σ)′ is isomorphic to a direct sum of dim_{ℂ}(σ) copies of S(C(w_{0}))′. Moreover, the stabilizer in N of any point in C(w_{0}) is trivial. Hence, from 5.12, we first conclude that H^{p}(𝔫, J(w_{0}, σ)′ ⊗ ℂ_{η}) = 0 for p > 0. This implies (i) in 6.2. Moreover, if Re η ≠ 0, we have H^{0}(𝔫, J(w_{0}, σ)′ ⊗ ℂ_{η}) = 0 and (ii) in 6.2 holds. Finally, if Re η = 0, by 5.14, we have dim_{ℂ} H^{0}(𝔫, J(w_{0}, σ)′ ⊗ ℂ_{η}) = dim_{ℂ}(σ). This implies (iii) in 6.2 and completes its proof.

It remains to establish 6.3. Denote by C_{p}^{∞}(σ)^{⊥} the space off all distributions in Ind^{∞}(σ) vanishing on C_{p}^{∞}(σ), i.e., the distributions supported in the union of Bruhat cells of dimension < p. Then, we have an increasing filtration of Ind^{∞}(σ)′ given by {0} = C_{0}^{∞}(σ)^{⊥} ⊂ C_{1}^{∞}(σ)^{⊥} ⊂ … ⊂ C_{n}^{∞}(σ)^{⊥} ⊂ Ind^{∞}(σ)′. We have the obvious exact sequence

0 → C_{n}^{∞}(σ)^{⊥} → Ind^{∞}(σ)′ ⟶^{res} C_{n}^{∞}(σ)′ → 0.

By tensoring with ℂ_{η} we get the exact sequence

0 → C_{n}^{∞}(σ)^{⊥} ⊗ ℂ_{η} → Ind^{∞}(σ)′ ⊗ ℂ_{η} ⟶^{res ⊗1} C_{n}^{∞}(σ)′ ⊗ ℂ_{η} → 0.

From the long exact sequence of the Lie algebra cohomology corresponding to this short exact sequence of U(𝔫)-modules we see that 6.3 follows immediately from the following result.

Lemma 6.4. H^{p}(𝔫, C_{n}^{∞}(σ)^{⊥} ⊗ ℂ_{η}) = 0 for all p ∈ ℤ_{+}.

This is just a special case of the following assertion, which we are going to prove by induction in q:

H^{p}(𝔫, C_{q}^{∞}(σ)^{⊥} ⊗ ℂ_{η}) = 0

for all p ∈ ℤ_{+} and 0 ≤ q ≤ n.

This statement is evident for q = 0. Assume that it holds for q, n > q ≥ 0. By 4.1 and dualizing, we have the short exact sequence

0 → ⨁_{w∈W (q)} J(w, σ)′ → C_{q}^{∞}(σ)′ → C_{q+1}^{∞}(σ)′ → 0.

This leads immediately to the short exact sequence

0 → C_{q}^{∞}(σ)^{⊥} → C_{q+1}^{∞}(σ)^{⊥} → ⨁_{w∈W (q)} J(w, σ)′ → 0.

By tensoring with ℂ_{η} we get the short exact sequence

0 → C_{q}^{∞}(σ)^{⊥} ⊗ ℂ_{η} → C_{q+1}^{∞}(σ)^{⊥} ⊗ ℂ_{η} → ⨁_{w∈W (q)} J(w, σ)′ ⊗ ℂ_{η} → 0.

In addition, we have the following vanishing result, the proof of which we postpone for a moment.

Lemma 6.5. Assume that w ∈ W is not the longest element in W. Then H^{p}(𝔫, J(w, σ)′ ⊗ ℂ_{η}) = 0 for all p ∈ ℤ_{+}.

From the long exact sequence of Lie algebra cohomology corresponding to the above short exact sequence of U(𝔫)-modules, using the induction assumption and 6.5, we see that H^{p}(𝔫, C_{q+1}^{∞}(σ)^{⊥} ⊗ ℂ_{η}) = 0 for all p ∈ ℤ_{+}. This proves the above assertion and completes the proof of 6.4.

It remains to establish 6.5. The proof is based on the following elementary fact.

Lemma 6.6. Let V be a U(𝔫)-module. Assume that F V is an increasing exhaustive filtration of V by U(𝔫)-modules such that F_{p} V = {0} for sufficiently negative p ∈ ℤ.

Let H^{q}(𝔫, Gr V) = 0 for all q ∈ ℤ. Then, we have H^{q}(𝔫, V) = 0 for all q ∈ ℤ.

Proof. First, consider the short exact sequence

0 → F_{p−1} V → F_{p} V → Gr^{p} V → 0

of U(𝔫)-modules. By the assumption, H^{q}(𝔫, Gr^{p} V) = 0 for all p, q ∈ ℤ. Hence, from the long exact sequence of Lie algebra cohomology attached to the above short exact sequence, we conclude that the natural map H^{q}(𝔫, F_{p−1} V) → H^{q}(𝔫, F_{p} V) is an isomorphism for all p, q ∈ ℤ. Since F_{p}V = {0} for sufficiently negative p ∈ ℤ, obviously we have H^{q}(𝔫, F_{p} V) = 0 for all q ∈ ℤ and sufficiently negative p ∈ ℤ. By induction in p, we see that H^{q}(𝔫, F_{p} V) = 0 for all p, q ∈ ℤ.

We claim that the standard complex C^{·}(𝔫, V) is acyclic. The increasing family of submodules F_{p} V defines an increasing exhaustive filtration of the standard complex C^{·}(𝔫, V) by subcomplexes C^{·}(𝔫, F_{p} V). We just established that all these subcomplexes are acyclic.

Let u ∈ C^{q}(𝔫, V) be such that du = 0. Then, since the filtration of the standard complex is exhaustive, u ∈ C^{q}(𝔫, F_{p} V) for some large p ∈ ℤ. By the above remarks, there exists v ∈ C^{q−1}(𝔫, F_{p} V) ⊂ C^{q−1}(𝔫, V) such that u = dv. Therefore, H^{q}(𝔫, V) = 0.

As explained in §2, J(w, σ)′ has a natural increasing filtration F J(w, σ)′ by the transversal degree. This filtration is exhaustive by 2.7, and F_{p} J(w, σ)′ are U(𝔫)-modules for all p ∈ ℤ by 3.5. Therefore it induces an analogous filtration F(J(w, σ)′ ⊗ ℂ_{η}) given by F_{p}(J(w, σ)′ ⊗ ℂ_{η}) = F_{p} J(w, σ)′ ⊗ ℂ_{η} for all p ∈ ℤ. The corresponding graded object is Gr J(w, σ)′ ⊗ ℂ_{η}.

Therefore, by 6.6, to establish 6.5, it is enough to prove the following result.

Lemma 6.7. Assume that w ∈ W is not the longest element in W. Then H^{p}(𝔫, Gr^{q} J(w, σ)′ ⊗ ℂ_{η}) = 0 for all p, q ∈ ℤ_{+}.

Proof. Let w be an element of the Weyl group W. If w ≠ w_{0}, the intersection 𝔫 ∩ 𝔫^{w} is spanned by root subspaces corresponding to the roots in R^{+} ∩ w(R^{+}). Clearly, R^{+} ∩w(R^{+}) = R^{+} ∩(−ww_{0}(R^{+})), where ww_{0} is different from the identity in W. Let s_{1}s_{2} … s_{q}, q ≥ 1, be a reduced expression of w_{0}w^{−1} = (ww_{0})^{−1} by simple reflections in W. Let α_{1}, α_{2}, …, α_{q} be the simple roots corresponding to the reflections s_{1}, s_{2}, …, s_{q}. By [3, Ch. VI, §1, no. 6, Cor. 2 of Prop. 17], −w_{0}w^{−1}(α_{q}) is a positive root. Therefore, α_{q} is in R^{+}∩w(R^{+}), i.e., the root subspace corresponding to the simple root α_{q} is in 𝔫 ∩ 𝔫^{w}. It follows that the restriction of η to 𝔫 ∩ 𝔫^{w} is nonzero. Hence, by 5.12.(iii), H^{p}(𝔫, S(C(w))′ ⊗ ℂ_{η}) = 0 for all p ∈ ℤ_{+}. Now, by 4.3 and 4.4, Gr^{q} J(w, σ)′ has a finite increasing filtration F Gr^{q} J(w, σ)′ by closed U(𝔫)-modules such that the corresponding graded module is a direct sum of copies of S(C(w))′, i.e., we have exact sequences

0 ⟶ F_{s−1} Gr^{q} J(w, σ)′ ⟶ F_{s} Gr^{q} J(w, σ)′ ⟶ S(C(w))′ ⊗ E_{w}^{∗} ⟶ 0

for all p ∈ ℤ. By the above vanishing result and the long exact sequence of Lie algebra cohomology attached to this short exact sequence, we conclude that the natural morphism H^{p}(𝔫, F_{s−1} Gr^{q} J(w, σ)′ ⊗ℂ_{η}) ⟶ H^{p}(𝔫, F_{s} Gr^{q} J(w, σ)′ ⊗ℂ_{η}) is an isomorphism for all p ∈ ℤ. Since F_{s} Gr^{q} J(w, σ)′ = 0 for very negative s, we see that H^{p}(𝔫, F_{s} Gr^{q} J(w, σ)′ ⊗ ℂ_{η}) = 0 for all p, s ∈ ℤ. Therefore, since the filtration is finite, we have H^{p}(𝔫, Gr^{q} J(w, σ)′ ⊗ ℂ_{η}) = 0 for all p ∈ ℤ.

7. Holomorphic continuation of Jacquet integrals

In this section we prove a variant of 6.1. It gives a description of Whittaker vectors for smooth principal series as holomorphic continuation of certain integrals considered by Jacquet [18].

Let P = MAN be the Langlands decomposition of P. Then M is a compact group. For a linear form λ on 𝔞 we denote, by abuse of notation, by λ the one-dimensional representation a ⟼ e^{λ(log a)}. We can view it as a representation of P by man ⟼ λ(a), for m ∈ M, a ∈ A and n ∈ N. Therefore, σ determines a holomorphic family of irreducible representations σ ⊗ λ of P on 𝔞^{∗}. Since X = P \G = M\K, all representations Ind^{∞}(σ ⊗ λ) can be realized on the same space of smooth sections C^{∞}(σ) of the homogeneous vector bundle on X = M\K determined by σ restricted to M. In this way, we can view Ind^{∞}(σ ⊗ λ) as a holomorphic family of representations. The Lie algebra 𝔤 acts on C^{∞}(σ) by first order differential operators depending holomorphically on λ ∈ 𝔞^{∗}. Moreover, the continuous duals of the spaces of Ind^{∞}(σ ⊗λ) can be identified with the continuous dual C^{∞}(σ)′ of C^{∞}(σ). The contragredient action of 𝔤 on C^{∞}(σ)′ is also given by first order differential operators depending holomorphically on λ ∈ 𝔞^{∗}.

Let U be an open set in 𝔞^{∗}. Denote by H(U) the ring of all holomorphic functions on U equipped with the topology of uniform convergence on compact sets in U. Let H(U, C^{∞}(σ)′) be the space of all holomorphic functions on U with values in C^{∞}(σ)′ endowed with the topology of uniform convergence on compact sets in U. By the results discussed in the appendix, we know that H(U, C^{∞}(σ)′) = H(U) ˆ⊗C^{∞}(σ)′. The action of 𝔤 extends by linearity to an action by differential operators on H(U, C^{∞}(σ)′).

Fix a nondegenerate η ∈ 𝔫^{∗}. For each λ ∈ 𝔞^{∗}, we denote by W_{η}(σ ⊗ λ) the linear subspace of Whittaker vectors in C^{∞}(σ)′ for the representation Ind^{∞}(σ ⊗λ). By 6.1, W_{η}(σ ⊗ λ) = 0 if Re η ≠ 0. Therefore, we can assume that Re η = 0 in the following. Denote by ψ the unitary character of N with the differential η.

Finally, let W_{η}(U, σ) be the subspace of H(U, C^{∞}(σ)′) consisting of all holomorphic maps F : U → C^{∞}(σ)′ such that F (λ) ∈ W_{η}(σ ⊗ λ) for all λ ∈ U. Evidently, W_{η}(U, σ) is a module over the ring H(U) of holomorphic functions on U. Moreover, we have

W_{η}(U, σ) = H^{0}(𝔫, H(U, C^{∞}(σ)′) ⊗_{ℂ} ℂ_{−η}).

The assignment U ⟼ W_{η}(U, σ) defines a sheaf of modules W_{η}(σ) over the sheaf of holomorphic functions H on 𝔞^{∗}.

By a variant of an argument in the last section we are going to describe the structure of this sheaf. The main difference is that we have to keep track of the dependence on the parameter λ. First we have to make the above isomorphisms more explicit. Let E_{0} = E_{x_{1}} be the fiber of E at the point x_{1} ∈ X corresponding to the parabolic subgroup P. The spaces Ind^{∞}(σ ⊗ λ), λ ∈ 𝔞^{∗}, can be identified with the spaces of smooth functions F : G ⟶ E_{0} satisfying F (pg) = σ(p)λ(p)F (g) for all g ∈ G and p ∈ P. The restriction map F ⟼ F |_{K} defines the isomorphism with functions from K to E_{0} which correspond to the sections in C^{∞}(σ). The inverse map sends a function f on K to a function nak ⟼ σ(a)λ(a)f(k), n ∈ N, a ∈ A and k ∈ K, on G. In the following, we are going to view Ind^{∞}(σ ⊗ λ) and C^{∞}(σ) as these spaces of functions.

We fix representatives k_{w} ∈ K of w ∈ W. As we described in §4, the space of Schwartz sections of Ind^{∞}(σ ⊗λ) on V^{w} can be identified with S(U^{w})⊗E_{0}, i,e., the space of Schwartz functions on U^{w} with values in E_{0}. This isomorphism is given by the restriction map, i.e., it attaches to F ∈ Ind^{∞}(σ ⊗ λ) the function u ⟼ F (k_{w}u) on U^{w}. The inverse of the above map sends the function φ ∈ S(U^{w}) ⊗ E_{0} into the function on G which is obtained by extension by zero from the function namk_{w}u ⟼ σ(am)λ(a)φ(u), n ∈ N, a ∈ A, m ∈ M and u ∈ U^{w}, on the open set P wU^{w}.

For a given λ we have the natural injective linear map

S(U^{w}) ⊗ E_{0} ⟶ Ind^{∞}(σ ⊗ λ) ⟶ C^{∞}(σ).

These maps depend on λ but have the common image equal to the space of Schwartz sections on V^{w}. To see the dependence on λ explicitly, denote by h : G ⟶ A and κ : G ⟶ K the unique analytic maps such that g ∈ Nh(g)κ(g) for g ∈ G. Then the above map attaches to φ ∈ S(U^{w}) ⊗ E_{0} the function K ∋ κ(k_{w}u) ⟼ λ(h(k_{w}uk_{w^{−1}}))^{−1}φ(u). The multiplier u ⟼ λ(h(k_{w}uk_{w^{−1}})) is a function of moderate growth on U^{w} which depends holomorphically on λ. Therefore, the above linear maps depend holomorphically on λ.

In particular, the bottom part C_{n}^{∞}(σ) = J(w_{0}, σ) in the Bruhat filtration is naturally isomorphic to S(N) ⊗ E_{0} as a representation of N and this isomorphism depends holomorphically on λ. Therefore, the composition with the adjoint of the inclusion J(w_{0}, σ) ⟶ C^{∞}(σ) induces a linear map of W_{η}(σ ⊗ λ) into the space of ψ-covartiant continuous linear forms on S(N)⊗E_{0}. By 5.13, the unique ψ-covariant continuous linear form on S(N) is equal to f ⟼ ∫_{N} ψ^{−1}(n)f(n) dn. Hence, we see that this space is isomorphic to E_{0}^{∗}. Combining this with the above discussion, it follows that in this way we get a H-module morphism Φ of W_{η}(σ) into H ⊗ E_{0}^{∗}.

Theorem 7.1. Assume that η ∈ 𝔫^{∗} is nondegenerate and Re η = 0. Then, for any irreducible representation σ of P, Φ is an isomorphism of W_{η}(σ) with H ⊗ E_{0}^{∗}.

In particular, W_{η}(σ) is a free H-module of rank dim_{ℂ}(σ).

It follows that the space W_{η}(𝔞^{∗}, σ) of global sections of W_{η}(σ), i.e., of holomorphic maps from 𝔞^{∗} into C^{∞}(σ)′ with values in Whittaker vectors is a free module over the entire functions on 𝔞^{∗} of rank dim_{ℂ}(σ). By 6.1, for an arbitrary fixed λ ∈ 𝔞^{∗}, a basis of W_{η}(𝔞^{∗}, σ) (as a free H(𝔞^{∗})-module) evaluated at λ, forms a basis of the space of Whittaker vectors for Ind^{∞}(σ ⊗ λ).

Without any loss of generality, we can assume that σ is a unitary representation on a finite-dimensional inner product space E_{0}. Consider the linear map Ω_{ψ,λ} : Ind^{∞}(σ ⊗ λ) ⟶ E_{0}^{∗} given by

Ω_{ψ,λ}(f) = ∫_{N} ψ^{−1}(n)f(k_{w_{0}}n) dn.

If this integral converges and defines a continuous linear map, ω ⟼ ω ∘ Ω_{ψ,λ} is a linear map from E_{0}^{∗} into W_{η}(σ ⊗ λ). Clearly, we have

∥Ω_{ψ,λ}(f)∥ ≤ C_{1} ∫_{N} ∥f(k_{w_{0}}n)∥ dn

for some C_{1} > 0. Since n ⟼ k_{w_{0}}nk_{w_{0}}^{−1} is an isomorphism of N onto N̄, we have

∥Ω_{ψ,λ}(f)∥ ≤ C_{1} ∫N̄ ∥f(n̄k_{w_{0}})∥ dn̄

= C_{1} ∫N̄ |λ(h(n̄))| ∥f(κ(n̄)k_{w_{0}})∥ dn̄ ≤ C_{1} max_{k∈K} ∥f(k)∥ ∫N̄ |λ(h(n̄))| dn̄

with an appropriate choice of the Haar measure on N̄. Let ρ be the half-sum of positive roots (counted with their multiplicities). Denote by C_{ρ} the set of all λ ∈ 𝔞^{∗} such that Re αˇ (λ − ρ) > 0 for all dual roots αˇ of α ∈ R^{+}. By the standard arguments from the theory of intertwining operators [16, I.5], it follows that the last integral converges for λ ∈ C_{ρ}. Therefore, λ ⟼ Ω_{ψ,λ} is defined and holomorphic in C_{ρ}. Moreover, for λ ∈ C_{ρ}, ω ⟼ ω ∘ Ω_{ψ,λ} is an isomorphism of E_{0}^{∗} onto the space W_{η}(σ ⊗ λ), i.e., all Whittaker vectors are given by the Jacquet integral Ω_{ψ,λ}. Therefore, for any open set U ⊂ C_{ρ}, Jacquet integral defines the inverse of Φ(U) : W_{η}(U, σ) ⟶ H(U) ⊗ E_{0}^{∗}. To sum up, 7.1 has the following immediate consequence.

Theorem 7.2. The Jacquet integral λ ⟼ Ω_{ψ,λ} extends to a holomorphic map from 𝔞^{∗} into W_{η}(𝔞^{∗}, σ) ⊗ E_{0}.

This result was established in [29] by a different method.

To prove 7.1 it is enough to show the following local result. Let λ be a point in 𝔞^{∗}. Denote by H_{λ}(C^{∞}(σ)′) the space of all germs at λ of holomorphic maps with values in C^{∞}(σ)′. It is a module over the ring H_{λ} of germs of holomorphic functions at λ. Clearly, 𝔫 acts on this space by H_{λ}-module endomorphisms.

Lemma 7.3. For any λ ∈ 𝔞^{∗}, the natural morphism

Φ_{λ} : H^{0}(𝔫, H_{λ}(C^{∞}(σ)′) ⊗ ℂ_{η}) ⟶ H_{λ} ⊗ E_{0}^{∗}

is an isomorphism.

It remains to establish 7.3.

First, let E be a DNF-space and a U(𝔫)-module, such that 𝔫 acts on E by continuous linear transformations. Then this action defines, by linearity, a natural structure of U(𝔫)-module on H_{λ}(E).

Lemma 7.4. If the Lie algebra cohomology spaces H^{p}(𝔫, E) are finite-dimen-sional for all p ∈ ℤ_{+}, we have

H^{p}(𝔫, H_{λ}(E)) = H_{λ}(H^{p}(𝔫, E))

for all p ∈ ℤ_{+}.

Proof. The standard complex of Lie algebra cohomology C^{·}(𝔫, E) is a complex of DNF-spaces with continuous differentials. By A.1, all differentials are strict morphisms. Therefore, their kernels and images are DNF-spaces. The assertion follows by tensoring this complex by H_{λ} and using B.2.

To establish 7.3 we again use the results about the Bruhat filtration from 4.1. By dualizing, we get the exact sequence

0 → ⨁_{w∈W (p)} J(w, σ)′ → C_{p}^{∞}(σ)′ → C_{p+1}^{∞}(σ)′ → 0

for p ∈ ℤ_{+}. Since these are DNF-spaces, by B.2, we get the exact sequence

0 → ⨁_{w∈W (p)} H_{λ}(J(w, σ)′) → H_{λ}(C_{p}^{∞}(σ)′) → H_{λ}(C_{p+1}^{∞}(σ)′) → 0.

The proof of 7.3 is based on the following vanishing result.

Lemma 7.5. If η ∈ 𝔫^{∗} is nondegenerate and w ∈ W different from w_{0}, we have

H^{p}(𝔫, H_{λ}(J(w, σ)′) ⊗ ℂ_{η}) = 0

for all p ∈ ℤ_{+}.

By 7.5 and the long exact sequence of cohomology, we conclude that

H^{q}(𝔫, H_{λ}(C_{p}^{∞}(σ)′) ⊗ ℂ_{η}) = H^{q}(𝔫, H_{λ}(C_{p+1}^{∞}(σ)′) ⊗ ℂ_{η})

for all 0 ≤ p < n and q ∈ ℤ_{+}. This implies that

H^{q}(𝔫, H_{λ}(C^{∞}(σ)′) ⊗ ℂ_{η}) = H^{q}(𝔫, H_{λ}(C_{n}^{∞}(σ)′) ⊗ ℂ_{η})

for all q ∈ ℤ_{+}. As we already remarked before, C_{n}^{∞}(σ) = J(w_{0}, σ). Hence, we get the following extension result.

Lemma 7.6. Let η ∈ 𝔫^{∗} be nondegenerate. Then the restriction map C^{∞}(σ)′ → J(w_{0}, σ)′ induces an isomorphism

H^{p}(𝔫, H_{λ}(C^{∞}(σ)′) ⊗ ℂ_{η}) → H^{p}(𝔫, H_{λ}(J(w_{0}, σ)′) ⊗ ℂ_{η})

for all p ∈ ℤ_{+}.

First, we want to show that this result implies 7.3. As we already remarked, U(𝔫)-module J(w_{0}, σ)′ is isomorphic to S(N)′ ⊗ E_{0}^{∗} and this isomorphism depends holomorphically on λ. Hence, by 5.14 and 7.4, we immediately see that Φ_{λ} is an isomorphism.

It remains to establish 7.5. As we explained in §4, J(w, σ)′ has a natural filtration by closed linear subspaces F_{p} J(w, σ)′, p ∈ ℤ. Clearly, F_{p} J(w, σ)′ are DNF-spaces for all p ∈ ℤ, and J(w, σ)′ = \underrightarrow{lim}_{p∈ℤ_{+}} F_{p} J(w, σ)′ by 2.8. By B.3, we have

H_{λ}(J(w, σ)′) = \underrightarrow{lim}_{k∈ℤ_{+}}H_{λ}(F_{k} J(w, σ)′).

Moreover, by exactness of tensoring, H_{λ}(F_{k} J(w, σ)′) are closed subspaces in the space H_{λ}(J(w, σ)′), i.e., H_{λ}(F J(w, σ)′) is an exhaustive increasing filtration of H_{λ}(J(w, σ)′) by closed U(𝔫)-modules. On the other hand, by B.2, from the short exact sequences

0 ⟶ F_{p−1} J(w, σ)′ ⟶ F_{p} J(w, σ)′ ⟶ Gr^{p} J(w, σ)′ ⟶ 0

we get the short exact sequences

0 ⟶ H_{λ}(F_{p−1} J(w, σ)′) ⟶ H_{λ}(F_{p} J(w, σ)′) ⟶ H_{λ}(Gr^{p} J(w, σ)′) ⟶ 0.

Therefore, we have Gr^{p} H_{λ}(J(w, σ)′) = H_{λ}(Gr^{p} J(w, σ)′) for all p ∈ ℤ_{+}.

Now, by the discussion from the beginning of this section and 4.3, the action of 𝔫 on Gr^{p} J(w, σ)′ is isomorphic to the natural action on L_{p}(w)⊗_{R(C(w))}S(C(w))′⊗_{ℂ} E_{w}^{∗} and this isomorphism depends holomorphically on λ. Hence, by 7.4 and 6.7, we have

H^{q}(𝔫, Gr^{p} H_{λ}(J(w, σ)′) ⊗ ℂ_{η}) = H^{q}(𝔫, H_{λ}(Gr^{p} J(w, σ)′) ⊗ ℂ_{η}) = 0

for all p, q ∈ ℤ_{+}. Finally, by 6.6, this completes the proof of 7.5.

8. The Whittaker functor

Let V be a 𝔫-module and η a nondegenerate linear form on 𝔫. Denote by ℂ_{η} the one-dimensional 𝔫-module with action η. Then we define the functor V ⟼ Ψ_{η}(V) = H_{0}(𝔫, V ⊗ℂ_{η}) from the category of 𝔫-modules into the category of complex linear spaces. We call Ψ_{η} the Whittaker functor corresponding to 𝔫 and η. This functor is right exact.

The following fact is evident, since the tensoring by ℂ_{η} is an equivalence of categories.

Lemma 8.1. (i) The left cohomological dimension of the functor Ψ_{η} is ≤

dim_{ℂ} 𝔫. (ii) The left derived functors L^{−p}Ψ_{η}, p ∈ ℤ_{+}, of Ψ_{η} are given by L^{−p}Ψ_{η}(V) =

H_{p}(𝔫, V ⊗ ℂ_{η}).

Let M(𝔤, K) be the abelian category of Harish-Chandra modules for G and M_{∞}(G) the corresponding abelian category of smooth representations of moderate growth and finite length in the sense of [9]. Denote by V ⟼ V^{∞} the corresponding completion functor from M(𝔤, K) into M_{∞}(G). By [9] this functor is an equivalence of categories and its quasiinverse is the functor of K-finite vectors.

We want to establish the following result.

Theorem 8.2. (i) The functor Ψ_{η} is nonzero on the category M_{∞}(G) if

and only if Re η = 0. (ii) For any V in M_{∞}(G), Ψ_{η}(V) is finite-dimensional. (iii) The functor Ψ_{η} is exact on M_{∞}(G).

Since V is a smooth representation, Ψ_{η}(V) is the cokernel of the continuous map of the NF-space V ⊗ 𝔫 into the NF-space V. By A.1, we see that the kernel of the map V → Ψ_{η}(V) is closed, and we have the following obvious consequence.

Let V be an object in M_{∞}(G). A continuous linear form δ on V is called a Whittaker vector (with respect to 𝔫 and η) if δ spans a one-dimensional invariant subspace for the contragredient representation of 𝔫 with the action given by η.

Corollary 8.3. For any V in M_{∞}(G), the dimension of the space of Whittaker vectors with respect to 𝔫 and η is equal to dim_{ℂ} Ψ_{η}(V).

The proof of 8.2 is by reduction to the case of smooth principal series.

We start with the following observation.

Lemma 8.4. Assume that any object V in M_{∞}(G) is a subobject of an object W satisfying:

(i) W is acyclic for Ψ_{η};

(ii) Ψ_{η}(W) is finite-dimensional. Then the functor Ψ_{η} is exact on M_{∞}(G) and Ψ_{η}(V) is finite-dimensional.

Proof. By our assumption, we have a short exact sequence

0 → V → W → Q → 0

where Q is the quotient of W by V. From the long exact sequence of derived functors of Ψ_{η} it follows that L^{p−1}Ψ_{η}(Q) ≅ L^{p}Ψ_{η}(V) for p ≤ −1. Since, by 8.1, the left cohomological dimension of Ψ_{η} is ≤ dim_{ℂ} 𝔫, L^{q}Ψ_{η}(Q) = 0 for q < − dim_{ℂ} 𝔫. By upward induction in p we see that L^{p}Ψ_{η}(V) = 0 for all p < 0 and V in M_{∞}(G). It follows that the sequence

0 → Ψ_{η}(V) → Ψ_{η}(W) → Ψ_{η}(Q) → 0

is exact. Therefore dim_{ℂ} Ψ_{η}(V) ≤ dim_{ℂ} Ψ_{η}(W) < ∞.

Let V be an object in M_{∞}(G) and U the corresponding Harish-Chandra module of K-finite vectors. Then U^{∞} ≅ V. Let P be the minimal parabolic subgroup of G containing N. Let σ be a finite-dimensional representation of P. We denote by Ind^{∞}(σ) the (smooth) induced representation of G defined by σ. Let Ind(σ) be the corresponding Harish-Chandra module. Then, by [10, 8.21], there exist σ and a monomorphism U → Ind(σ). By the exactness of the completion functor, we have a monomorphism V ≅ U^{∞} → Ind(σ)^{∞} = Ind^{∞}(σ). If σ is irreducible, Ind^{∞}(σ) is a smooth principal series representation of G.

Lemma 8.5. Let σ be an irreducible finite-dimensional representation of P and Ind^{∞}(σ) the corresponding smooth principal series representation. Then

(i) L^{p}Ψ_{η}(Ind^{∞}(σ)) = 0 for p < 0; (ii) Ψ_{η}(Ind^{∞}(σ)) = 0 if Re η ≠ 0; (iii) dim_{ℂ} Ψ_{η}(Ind^{∞}(σ)) = dim_{ℂ}(σ) for Re η = 0.

Proof. As we remarked in 8.1, L^{−p}Ψ_{η}(Ind^{∞}(σ)) = H_{p}(𝔫, Ind^{∞}(σ) ⊗ ℂ_{η}) for p ∈ ℤ_{+}. Clearly, the topological dual of Ind^{∞}(σ) ⊗ ℂ_{η} is equal to Ind^{∞}(σ)′ ⊗ ℂ_{−η} as a U(𝔫)-module. By 6.2, H^{p}(𝔫, Ind^{∞}(σ)′ ⊗ ℂ_{−η}), p ∈ ℤ_{+}, are finite-dimensional. Therefore, by 5.11, we have

H_{p}(𝔫, Ind^{∞}(σ) ⊗ ℂ_{η})^{∗} = H_{p}(𝔫, (Ind^{∞}(σ)′ ⊗ ℂ_{−η})′)^{∗} = H^{p}(𝔫, Ind^{∞}(σ)′ ⊗ ℂ_{−η})

for all p ∈ ℤ_{+}. Now, the statement follows immediately from 6.2.

Now, we claim that then the statements of this lemma hold for arbitrary finitedimensional representations σ of P. We prove this by induction in the length of σ. Let

0 → σ′ → σ → σ′′ → 0

be a short exact sequence of finite-dimensional representations of P. Assume that σ′′ is irreducible. Then length(σ′) < length(σ). Moreover, by exactness of smooth induction we have an exact sequence

0 → Ind^{∞}(σ′) → Ind^{∞}(σ) → Ind^{∞}(σ′′) → 0.

By 8.5 and the induction assumption, we have L^{p}Ψ_{η}(Ind^{∞}(σ′)) = L^{p}Ψ_{η}(Ind^{∞}(σ′′)) = 0 for p < 0. Therefore, from the long exact sequence of derived functors of Ψ_{η} we conclude that L^{p}Ψ_{η}(Ind^{∞}(σ)) = 0 for all p < 0. Moreover, we have an exact sequence

0 → Ψ_{η}(Ind^{∞}(σ′)) → Ψ_{η}(Ind^{∞}(σ)) → Ψ_{η}(Ind^{∞}(σ′′)) → 0.

This clearly implies the vanishing of Ψ_{η}(Ind^{∞}(σ)) if Re η ≠ 0. If Re η = 0, we have

dim_{ℂ} Ψ_{η}(Ind^{∞}(σ)) = dim_{ℂ} Ψ_{η}(Ind^{∞}(σ′)) + dim_{ℂ} Ψ_{η}(Ind^{∞}(σ′′))

= dim_{ℂ}(σ′) + dim_{ℂ}(σ′′) = dim_{ℂ}(σ).

Therefore, it follows that 8.5 implies that Ind^{∞}(σ) satisfy the conditions (i) and (ii) in 8.4. Hence, the functor Ψ_{η} is exact on M_{∞}(G) and Ψ_{η}(V) is finitedimensional for any object V in M_{∞}(G). Moreover, if Re η ≠ 0, Ψ_{η}(Ind^{∞}(σ)) = 0 for any finite-dimensional representation σ of P. Since any object in M_{∞}(G) is a subobject of some Ind^{∞}(σ), this in turn implies that Ψ_{η} vanishes on M_{∞}(G). This completes the proof of 8.2.

9. Linear quasisplit groups

As we remarked in the introduction, from the point of view of applications to automorphic forms the most important case is of the group G of real points of a connected complex reductive algebraic group G defined over ℝ which is quasisplit, i.e., such that G contains a Borel subgroup defined over ℝ. This was the situation originally considered by Kostant.

For such group G, the Lie algebra 𝔫 is the nilpotent radical of a Borel subalgebra of 𝔤. The group of real points of the corresponding Borel subgroup of G is the minimal parabolic subgroup P containing N. If P = MAN is the Langlands decomposition of P, we conclude that M is commutative. Hence, all irreducible finite-dimensional representations of P are one-dimensional. For purely imaginary η, 6.1 implies that the space of Whittaker vectors is one-dimensional. More precisely, by combining this observation with 7.1 and 7.2, we have the following result.

Theorem 9.1. Let ψ be a nondegenerate unitary character of N. Then, for arbitrary one-dimensional representation σ of P, the space of all Whittaker vectors in Ind^{∞}(σ)′ is one-dimensional. It is spanned by the holomorphic continuation of the Jacquet integral.

In addition to this, as we remarked in the last section, any irreducible smooth representation V of G is a subrepresentation of some smooth principal series Ind^{∞}(σ). Hence, we have dim_{ℂ} Ψ_{η}(V) ≤ dim_{ℂ} Ψ_{η}(Ind^{∞}(σ)) = 1. Therefore, we have the following immediate consequence.

Theorem 9.2. Let η be a nondegenerate one-dimensional representation of 𝔫. Then, Ψ_{η}(V) is either zero or one-dimensional for any irreducible smooth representation V in M_{∞}(G).

Appendix A. Some results on nuclear spaces

In this section we include some results on nuclear spaces we use in the main text. Most of them are well known to the experts, but for some we were not able to find a suitable reference. We thank Joe Taylor for his expert tutorial on this subject.

Let E be a Fréchet space. Then it is a bornological space [4, Ch. III, §2, Prop. 2]. Therefore, its strong dual E′ is complete [4, Ch. III, §3, no. 8, Cor. 1 of Prop. 12]. Moreover, E is barreled [4, Ch. III, §4, no. 1, Cor. of Prop. 2].

A nuclear Fréchet space E is called an NF-space for short. Since, by [14, Ch. II, §2, no. 1, Cor. 1 of Lemma 3], any bounded set in E is relatively compact, we see that E is a Montel space. Therefore, E is reflexive [4, Ch. IV, §2, no. 3, Th. 2]. Its strong dual E′ is a DNF-space. By the above remarks, E′ is complete and reflexive. It is bornological and barreled [4, Ch. IV, §3, no. 4, Cor. of Prop. 4]. Also, it is a Montel space [4, Ch. IV, §2, no. 5, Prop. 9],. Moreover, it is nuclear by [14, Ch. II, §2, no. 1, Th. 7].

By [23, Ch. IV, §8, Examples 1 and 2], NF-spaces and DNF-spaces are Ptak. Therefore, if E and F are either NF-spaces or DNF-spaces, a surjective continuous linear map T : E → F is open [23, Ch. IV, §8, Cor. 1 of Th. 3].

Let T : E ⟶ F be a continuous linear map between two hausdorff locally convex spaces. Then ker T is a closed linear subspace of E. Let p : E ⟶ E/ ker T be the canonical projection. The image im T is a locally convex space with the topology induced by the topology of F. Denote by i : im T → F the canonical inclusion. The map T factors as

E ⟶^{p} E/ ker T ⟶^{τ} im T ⟶^{i} F,

where τ is continuous linear isomorphism. We say that T is a strict morphism if τ is a homeomorphism.

Let E and F be NF-spaces, resp. DNF-spaces, and T : E ⟶ F a continuous linear map. Let T^{t} : F′ ⟶ E′ be the adjoint linear map between the strong duals. Then T^{t} is continuous [4, Ch. IV, §1, no. 3, Cor. of Prop. 5].

Let E be a nuclear space. Then its subspaces and hausdorff quotient spaces of E are nuclear [14, Ch. II, §2, no. 2, Th. 9]. Therefore, closed subspaces and hausdorff quotient spaces of NF-spaces are NF-spaces.

Now we prove that closed subspaces and hausdorff quotient spaces of DNFspaces are DNF-spaces. Let E be a DNF-space. Let F be a closed subspace of E. Then F and E/F are nuclear spaces. Clearly, E = G′ for some NF-space G. Therefore, E′ = G′′ = G is an NF-space. Let F^{⊥} be the closed subspace of all linear forms in E′ vanishing on F. Then we have the exact sequence

0 → F^{⊥} ⟶^{α} E′ ⟶^{β} E′/F^{⊥} → 0.

of NF-spaces. By dualizing we get the exact sequence of strong duals

0 → (E′/F^{⊥})′ ⟶^{β^{t}} E′′ ⟶^{α^{t}} (F^{⊥})′ → 0.

The injectivity of β^{t} is evident, the surjectivity of α^{t} follows from the Hahn-Banach theorem. If f ∈ E′′ is in the kernel of α^{t}, this means that its restriction to F^{⊥} vanishes. But then it defines a continuous linear form on E′/F^{⊥}, i.e., it is in the image of β^{t}. Therefore, we have the following commutative diagram with exact rows:

\begin{aligned}0 & ⟶ & F & ⟶ & E & ⟶ & E/F & ⟶ & 0 \\ & & ↓ & & ↓ & & ↓ \\ 0 & ⟶ & (E′/F^{⊥})′ & ⟶_{β^{t}} & E′′ & ⟶_{α^{t}} & (F^{⊥})′ & ⟶ & 0 \end{aligned} .

The vertical arrows are the obvious natural maps. Since E is reflexive, the middle vertical arrow is an isomorphism, and im β^{t} corresponds to elements u ∈ E which are in the kernels of all linear forms in F^{⊥}, i.e, u ∈ F. Hence the first vertical arrow is an isomorphism. Therefore, all vertical arrows are isomorphisms of linear spaces. On the other hand, as we remarked above, NF-spaces are Montel spaces. Hence, by [4, Ch. IV, §4, no. 2, Cor. 3 of Th.1], α^{t} and β^{t} are strict morphisms. This immediately implies that all vertical arrows are isomorphisms of topological linear spaces. In particular, F and E/F are DNF-spaces.

Let E and F be either NF-spaces or DNF-spaces and T : E → F a continuous linear map. Then, by the above discussion, T a strict morphism if and only if it has closed image. If T is strict, im T = E/ ker T is complete, and therefore closed in F. Conversely, if im T is closed, τ is a homeomorphism by the open mapping theorem.

The following simple sufficient criterion for strictness is critical in our applications.

Lemma A.1. Assume that E and F are either NF-spaces or DNF-spaces. Let T : E → F be a continuous linear map. Assume that the image im T is of finite codimension in F. Then T is a strict morphism.

Proof. By replacing E with E/ ker T we can assume that T is injective. Let H be a finite-dimensional direct complement of im T in F. Then, the

natural map i : H → F is continuous. Therefore, T ⊕ i : E ⊕ H → F is a continuous bijective linear map. By the open mapping theorem, this map is also open. Therefore, it is an isomorphism of topological vector spaces. This implies that im T is closed in W.

Moreover, the next result follows easily from the above observations by reduction to short exact sequences.

Lemma A.2. Assume that (C^{·}, d) is a complex such that

(i) all C^{p}, p ∈ ℤ, are NF-spaces (or DNF-spaces); (ii) all differentials are strict morphisms.

Then the dual complex (C′^{·}, d^{t}) is a complex with differentials which are strict morphisms and

H^{p}(C′^{·}) = H^{−p}(C^{·})′.

For two locally convex spaces E and F, we equip the tensor product E ⊗ F with the finest locally convex topology such that the bilinear map E × F → E ⊗ F is continuous. If E and F are hausdorff, this topology on E ⊗ F is also hausdorff. We denote by E ˆ⊗F completion of E ⊗ F with respect to this topology. The locally convex space E ˆ⊗F is the (projective) topological tensor product of E and F.

If E and F are nuclear spaces, E ˆ⊗F is nuclear [14, Ch. II, §, no. 1, Th. 9]. If E and F are both NF-spaces (resp. DNF-spaces), E ˆ⊗F is an NF-space (resp. DNFspace) [14, Ch. I, §1, no. 3, Prop. 5]. Let E and F be NF-spaces (or DNF-spaces). Then we have a natural linear map E′ ⊗ F′ → (E ˆ⊗F)′. This map induces an isomorphism E′ ˆ⊗F′ → (E ˆ⊗F)′ of locally convex spaces [14, Ch. II, §3, no. 2, Th. 12].

Let F be an NF-space (resp. DNF-space). Then, E ⟼ E ˆ⊗F is an exact functor on the category of NF-spaces (resp. DNF-spaces). By the above remarks it is enough to establish the following result.

Lemma A.3. Let

0 → E_{1} ⟶^{i} E_{2} ⟶^{p} E_{3} → 0

be a short exact sequence of NF-spaces. Let F be an NF-space. Then

0 → E_{1} ˆ⊗F ⟶^{i⊗1_{F}} E_{2} ˆ⊗F ⟶^{p⊗1_{F}} E_{3} ˆ⊗F → 0

is also a short exact sequence.

Proof. By [14, Ch. I, §1, no. 2, Prop. 3] the map i ⊗ 1_{F} is injective and the map p ⊗ 1_{F} is surjective. Moreover, the kernel of p ⊗ 1_{F} is the closure of the subspace ker p ⊗ F = im i ⊗ F. Since i is strict, by [14, Ch. II, §3, no. 1, Cor. of Prop. 10], the map i ⊗ 1_{F} is a strict morphism. Therefore, its image is closed and it is an isomorphism of E_{1} ˆ⊗F onto the closure of im i ⊗ F.

A projective limit of nuclear spaces is nuclear [14, Ch. II, §2, no. 2, Th. 9]. Therefore, the projective limit of a countable projective system of NF-spaces is an NFspace.

Let (E_{i}; i ∈ I) be an inductive system of nuclear spaces and E = \underrightarrow{lim}_{i∈I}E_{i} the inductive limit in the category of locally convex spaces. If I is countable and E is hausdorff, it is nuclear by [14, Ch. II, §2, no. 2, Cor. 1 of Th. 9]. Moreover, if E_{i} are DNF-spaces, E is a DNF-space [13, Ch. I, no. 4, Cor. of Th. 9].

Lemma A.4. Let (E_{i}; i ∈ I) be an inductive system of DNF-spaces. Assume that I is countable and E = \underrightarrow{lim}_{i∈I}E_{i} is hausdorff. Let F be a DNF-space. Then E ˆ⊗F = \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F).

Proof. Clearly, (E_{i} ˆ⊗F; i ∈ I) is an inductive system of DNF-spaces. We denote by α_{ij} : E_{i} → E_{j} for i ≤ j in I, and α_{i} : E_{i} → E for i ∈ I, the corresponding canonical maps. Now consider the inductive system (E_{i} ˆ⊗F; i ∈ I) with the maps β_{ij} : E_{i} ˆ⊗F → E_{j} ˆ⊗F which are the tensor products of α_{ij} with the identity map on F. Denote by β_{i} the canonical continuous linear maps E_{i} ˆ⊗F → \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F).

The continuous linear maps γ_{i}, i ∈ I, which are the tensor products of α_{i} with the identity on F define a family of maps from the inductive system (E_{i} ˆ⊗F; i ∈ I) into E ˆ⊗F. Therefore, they define a continuous linear map γ : \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F) → E ˆ⊗F.

We claim that γ is injective. Assume that v ∈ \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F) is such that γ(v) = 0. Then, v = β_{i}(w) for some w ∈ E_{i} ˆ⊗F. Moreover, we have γ_{i}(w) = γ(β_{i}(w)) = 0. By [14, Ch. I, §1, no. 2, Prop. 3], this implies that w is in the closure of the subspace ker α_{i} ⊗ F in E_{i} ˆ⊗F. Since ker α_{i} is the union of ker α_{ij} for j > i, we see that w is in the union of closures of ker α_{ij} ⊗ F. Therefore, w ∈ ker β_{ij} for some j > i. This implies that v = 0 and γ is injective. Therefore, \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F) is hausdorff. By the above remark, it follows that \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F) is a DNF-space. Hence, it is complete. By [14, Ch. I, §4, no. 3, Cor. of Prop. 6], γ is a homeomorphism of \underrightarrow{lim}_{i∈I}(E_{i} ˆ⊗F) onto its image in E ˆ⊗F. Therefore, the image of γ must be closed in E ˆ⊗F. On the other hand, it is evident that the image of γ is dense in E ˆ⊗F.

If M is a compact manifold and E a vector bundle over M, we denote by C^{∞}(E) the space of all smooth sections of E equipped with the topology of uniform convergence of sections and their derivatives. In this way, C^{∞}(E) becomes an NFspace [14, Ch. II, §2, no. 3, Th. 10]. Therefore, its strong dual C^{∞}(E)′ is a DNF-space.

Appendix B. Holomorphic vector valued functions

In this section we prove some technical results on holomorphic functions with values in DNF-spaces.

Let M be a complex manifold. Denote by H the sheaf of holomorphic functions on M. For any open set U in M, we denote by H(U) the ring of holomorphic functions on U. We equip H(U) with the locally convex topology of uniform convergence on compact subsets of U. In this way H(U) becomes an NF-space [14, Ch. II, §2, no. 3, Cor. of Th. 10]. If V is an open subset of U, the restriction map from H(U) into H(V) is continuous. If K is a compact subset of M, we define H(K) as the inductive limit of H(U) over the family of all open sets U containing K. In this way, H(K) becomes a locally convex space. By [13, §1, no. 4, Exemple b], H(K) is a DNF-space.

If m is a point in M, we can put K = {m}. In this way, the space H_{m} of germs of holomorphic functions at m becomes a DNF-space.

Let E be a complete locally convex space. Analogously, we denote by H(E) the sheaf of all holomorphic functions on M with values in E. In particular, for an open set U in M, we denote by H(U, E) the space of all holomorphic functions on U with values in E. By [14, Ch. II, §3, no. 3, Exemples], we know that H(U, E) = H(U) ˆ⊗E.

For a compact set K in M, we define H(K, E) = \underrightarrow{lim}_{U⊃K}H(U, E) where the inductive limit is taken over the directed system of all open sets U containing K. In particular, for a point m ∈ M, we denote by H_{m}(E) the space of all germs at m of holomorphic maps with values in E.

Let U be an open set in M and K a compact set in U. Then we have a natural continuous linear map H(U) → H(K). This induces a natural continuous linear map H(U) ˆ⊗E → H(K) ˆ⊗E.

On the other hand, if V is an open set in M contained in K, by the definition of H(K) we have a continuous linear map H(K) → H(V) given as the inductive limit of restrictions H(U) → H(V) over the directed system of open sets U ⊃ K. It induces a continuous linear map H(K) ˆ⊗E → H(V) ˆ⊗E.

Let (U_{n}; n ∈ ℕ) be a decreasing family of open neighborhoods of a compact set K in M which is cofinal in the family of all open sets containing K, satisfying the following conditions:

(i) U_{n} are relatively compact; (ii) the closure Ū_{n} is contained in U_{n−1} for n > 1; (iii) the intersection of all U_{n}, n ∈ ℕ, is equal to K.

Also, let (K_{n}; n ∈ ℕ) be a family of compact sets such that U_{n} ⊃ K_{n} ⊃ U_{n+1}. Consider two inductive systems (H(U_{n}) ˆ⊗E; n ∈ ℕ) and (H(K_{n}) ˆ⊗E; n ∈ ℕ). As we remarked above, we have natural continuous linear maps α_{n} : H(U_{n}) ˆ⊗E → H(K_{n}) ˆ⊗E and β_{n} : H(K_{n}) ˆ⊗E → H(U_{n+1}) ˆ⊗E for n ∈ ℕ. Clearly, these maps define the maps of corresponding inductive systems, Therefore, they induce continuous linear maps α : \underrightarrow{lim}_{n→∞}(H(U_{n}) ˆ⊗E) → \underrightarrow{lim}_{n→∞}(H(K_{n}) ˆ⊗E) and β : \underrightarrow{lim}_{n→∞}(H(K_{n}) ˆ⊗E) → \underrightarrow{lim}_{n→∞}(H(U_{n}) ˆ⊗E) such that the compositions α ∘ β and β ∘ α are identities. Therefore, we have

H(K, E) = \underrightarrow{lim}_{U⊃K}H(U, E) = \underrightarrow{lim}_{U⊃K}(H(U) ˆ⊗E)

= \underrightarrow{lim}_{n→∞}(H(U_{n}) ˆ⊗E) = \underrightarrow{lim}_{n→∞}(H(K_{n}) ˆ⊗E).

In particular, for E = ℂ, we see that H(K) = \underrightarrow{lim}_{n→∞}H(K_{n}). By A.4, this implies that

H(K, E) = \underrightarrow{lim}_{n→∞}(H(K_{n}) ˆ⊗E) = (\underrightarrow{lim}_{n→∞}H(K_{n})) ˆ⊗E = H(K) ˆ⊗E.

Therefore, we proved the following result.

Lemma B.1. If E is a DNF-space, we have H(K, E) = H(K) ˆ⊗E for any compact subset K in M.

In particular, for any m ∈ M, we have H_{m}(E) = H_{m} ˆ⊗E.

This, in combination with A.4, has the following consequences.

Corollary B.2. Let

0 ⟶ E_{1} ⟶ E_{2} ⟶ E_{3} ⟶ 0

be a short exact sequence of DNF-spaces. Then, for any compact set K in M,

0 ⟶ H(K, E_{1}) ⟶ H(K, E_{2}) ⟶ H(K, E_{3}) ⟶ 0

is exact.

In particular, for any m ∈ M,

0 ⟶ H_{m}(E_{1}) ⟶ H_{m}(E_{2}) ⟶ H_{m}(E_{3}) ⟶ 0

is exact.

Corollary B.3. Let (E_{i}; i ∈ I) be a countable inductive system of DNFspaces. Assume that E = \underrightarrow{lim}_{i∈I}E_{i} is hausdorff. Then, for any compact set K in M we have H(K, E) = \underrightarrow{lim}_{i∈I}H(K, E_{i}).

In particular, for any m ∈ M, we have H_{m}(E) = \underrightarrow{lim}_{i∈I}H_{m}(E_{i}).

References

[1] Armand Borel, Linear algebraic groups, Springer-Verlag, 1991.

[2] Nicolas Bourbaki, Variétés différentielles et analytiques, Fascicule de résultats, Hermann, 1967.

[3] , Groupes et algèbres de Lie, Hermann, 1968.

[4] , Espaces vectoriels topologiques, Masson, 1981.

[5] Mladen Božičević, A property of Lie group orbits, Canad. Math. Bull., (to appear).

[6] François Bruhat, Sur les représentations induites des groupes de Lie, Bull. Soc. Math. France 84 (1956), 97–205.

[7] Henri Cartan and Samuel Eilenberg, Homological algebra, Princeton University Press, 1956.

[8] William Casselman, Letter to Harish-Chandra, November 1982.

[9] , Canonical extensions of Harish-Chandra modules to representations of G, Canad. J. Math. 41 (1989), 385–438.

[10] William Casselman and Dragan Miličić, Asymptotic behavior of matrix coefficients of admissible representations, Duke Math. J. 49 (1982), 869–930.

[11] William Casselman and Joseph Shalika, The unramified principal series of p-adic groups II. The Whittaker function, Compositio Math. 41 (1980), 207–231.

[12] Fokko Du Cloux, Sur les representations différentiables des groupes de lie algeébriques, Ann. Sci. École Norm. Sup. (4) 24 (1991), 257–318.

[13] Alexander Grothendieck, Sur les espaces (F) et (DF), Summa Brasil. Math. 3 (1954), 57–123.

[14] , Produits tensoriels topologiques et espaces nucléaires, Mem. Amer. Math. Soc., vol. 16, 1955.

[15] Harish-Chandra, Harmonic analysis on real reductive groups. I. The theory of the constant term, J. Funct. Anal. 19 (1975), 104–204.

[16] , Harmonic analysis on real reductive groups. III. The Maass-Selberg relations and the Plancherel formula, Ann. of Math. (2) 104 (1976), no. 1, 117–201.

[17] , On the theory of the Whittaker integral, handwritten notes.

[18] Hervé Jacquet, Fonctions de Whittaker associées aux groupes de Chevalley, Bull. Soc. Math. France 95 (1967), 243–309.

[19] Hervé Jacquet and Robert Langlands, Automorphic forms on GL(2), Lecture Notes in Mathematics, vol. 114, Springer-Verlag, 1970.

[20] Masaki Kashiwara, The Riemann-Hilbert problem for holonomic systems, Publ. Res. Inst. Math. Sci. 20 (1984), 319–365.

[21] Bertram Kostant, On Whittaker vectors and representation theory, Invent. Math. 48 (1978), 101–184.

[22] Hisayoshi Matumoto, C^{−∞}-Whittaker vectors for complex semisimple Lie groups, wave front sets, and Goldie rank polynomial representations, Ann. Sci. École Norm. Sup. (4) 23 (1990), 311–367.

[23] Helmut Schaefer, Topological vector spaces, Macmillan, 1966.

[24] Laurent Schwartz, Théorie des distributions, Hermann, 1966.

[25] Joseph A. Shalika, The multiplicity one theorem for GL(n), Ann. of Math. (2) 100 (1974), 171–193.

[26] Tze-Ming To, Asymptotic expansions of matrix coefficients of Whittaker vectors at irregular singularities, Acta Math. 175 (1995), 227–271.

[27] Lou Van den Dries and Chris Miller, Geometric categories and o-minimal structures, Duke Math. J. 84 (1996), 497–539.

[28] Nolan R. Wallach, Asymptotic expansions of generalized matrix entries of representations of real reductive groups, Lie group representations, I (College Park, Md., 1982/1983), Springer, Berlin, 1983, pp. 287–369.

[29] , Lie algebra cohomology and holomorphic continuation of generalized Jacquet integrals, Representations of Lie Groups, Kyoto, Hiroshima, 1986, Advanced Studies in Pure Math., vol. 14, 1988, pp. 123–151.

[30] Garth Warner, Harmonic analysis on semi-simple Lie groups I, Grundlehren der mathematischen Wissenschaften, vol. 188, Springer-Verlag, 1972.

(Casselman) Department of Mathematics, University of British Columbia, Vancouver, BC, V6T 1W5, Canada

    E-mail address: cass@math.ubc.ca
    URL: http://www.math.ubc.ca/~cass

(Hecht) Department of Mathematics, University of Utah, Salt Lake City, UT 84112, USA

    E-mail address: hecht@math.utah.edu

(Miličić) Department of Mathematics, University of Utah, Salt Lake City, UT 84112, USA

(Miličić) Department of Mathematics, Harvard University, Cambridge, MA 02138, USA

    E-mail address: milicic@math.utah.edu
    URL: http://www.math.utah.edu/~milicic