Loading menu…

Attention-based Solvers for Linear Equations that are Geometry Aware

§ School of Mathematical Sciences, Tel Aviv University

Send Message

To: Author

Attention-based Solvers for Linear Equations that are Geometry Aware

Article Fingerprint

ReserarchID

FRQ26

Attention-based Solvers for Linear Equations that are Geometry Aware Banner

Key Research Insights

Synthesized scholarly intelligence & interactive research assistant
  • English
  • Afrikaans
  • Albanian
  • Amharic
  • Arabic
  • Armenian
  • Azerbaijani
  • Basque
  • Belarusian
  • Bengali
  • Bosnian
  • Bulgarian
  • Catalan
  • Cebuano
  • Chichewa
  • Chinese (Simplified)
  • Chinese (Traditional)
  • Corsican
  • Croatian
  • Czech
  • Danish
  • Dutch
  • Esperanto
  • Estonian
  • Filipino
  • Finnish
  • French
  • Frisian
  • Galician
  • Georgian
  • German
  • Greek
  • Gujarati
  • Haitian Creole
  • Hausa
  • Hawaiian
  • Hebrew
  • Hindi
  • Hmong
  • Hungarian
  • Icelandic
  • Igbo
  • Indonesian
  • Irish
  • Italian
  • Japanese
  • Javanese
  • Kannada
  • Kazakh
  • Khmer
  • Korean
  • Kurdish (Kurmanji)
  • Kyrgyz
  • Lao
  • Latin
  • Latvian
  • Lithuanian
  • Luxembourgish
  • Macedonian
  • Malagasy
  • Malay
  • Malayalam
  • Maltese
  • Maori
  • Marathi
  • Mongolian
  • Myanmar (Burmese)
  • Nepali
  • Norwegian
  • Pashto
  • Persian
  • Polish
  • Portuguese
  • Punjabi
  • Romanian
  • Russian
  • Samoan
  • Scots Gaelic
  • Serbian
  • Sesotho
  • Shona
  • Sindhi
  • Sinhala
  • Slovak
  • Slovenian
  • Somali
  • Spanish
  • Sundanese
  • Swahili
  • Swedish
  • Tajik
  • Tamil
  • Telugu
  • Thai
  • Turkish
  • Ukrainian
  • Urdu
  • Uzbek
  • Vietnamese
  • Welsh
  • Xhosa
  • Yiddish
  • Yoruba
  • Zulu
Reading Preferences
Font Size
Line Spacing
Background
This converted HTML version may contain rendering inconsistencies. Please refer to the PDF for the authoritative version, or click here to provide feedback.

Abstract

We present a novel architecture for learning geometry-aware preconditioners for linear partial differential equations (PDEs). We show that a deep operator network (Deeponet) can be trained on a simple geometry and remain a robust preconditioner for problems defined by different geometries without further fine-tuning or additional data mining. We demonstrate our method for the Helmholtz equation, which is used to solve problems in electromagnetics and acoustics; the Helmholtz equation is not positive definite, and with absorbing boundary conditions, it is not symmetric.

1. Introduction

Solving linear PDE problems frequently involves solving a linear system of equations using iterative algorithms. A central aspect in designing such iterative solvers for a linear system of equations Au=f is finding an approximation of A−1, referred to as a preconditioner. The simplest method to utilize a preconditioner A^∼A−1 is through iterative refinement, also known as Richardson iteration or deferred correction:

(1.1)ui+1=(1−θ)ui+θA^(f−Aui),0≤θ≤1.

One step of the Richardson iteration method only requires one multiplication of a vector by the matrix A, making it suitable for large systems of equations.

The Helmholtz equation is given by

(1.2)Lu=Δu+k2u=f.

For sufficiently large k>0, the operator L is not positive definite [5]. Furthermore, if absorbing boundary conditions are imposed, L is no longer symmetric. It is well known that Richardson iteration with standard solvers like Gauss-Seidel, Jacobi, or geometric multigrid diverge when applied to non-symmetric or non-positive definite operators. In such cases, more involved techniques are required, e.g., domain decomposition methods [3, 8, 9], Krylov-type methods [5, 7] including the generalized minimal residual method (GMRES) and its variants [2, 23, 25], and GMRES with shifted Laplacian preconditioning [6]. However, computational cost and memory storage are more expensive for these methods.

A possible remedy that still allows the use of (1.6) is finding a non-linear preconditioner, N, using a neural network architecture named Deeponet [19], which has been experimentally proven to be a robust architecture for learning operators between function spaces, and in particular, for learning solutions of partial differential equations. The classical Deeponet receives as an input y∈Ω and a function f defined on Ω and returns ((Δ+k2)−1f)(y). The representation of f by finite-dimensional x∈Rn can be done most straightforwardly by evaluating f on n points in Ω. The classical Deeponet admits the following general form: Let fm:=(f(x1),…,f(xm)) be the values of f over fixed m points in Ω. Let B and T be neural networks such that

T:Rn→Rp,B:Rm→Rp.

Then

(1.3)D(y,f)=∑j=1pT(y)j⋅B(fm)j.

The network T is named the trunk and B is named the branch. The network N is trained over data samples such that the loss function

‖N(y,f)−u(y)‖L22|u(y)|2

is minimized over all samples. The structure of (1.3) is a classical structure of data-driven recommender systems [26].

One of the drawbacks of Deeponet is that the numerical accuracy of solutions for a linear system is not fully understood, and reducing the error requires improved architectures [28]. In general, solutions with errors of machine precision are not possible. To overcome the latter difficulty, a novel hybrid, iterative, numerical, and transferable solver (HINTS) has been recently developed [27]. The hybrid solver uses Deeponet as a preconditioner and is integrated inside existing numerical algorithms (e.g., Jacobi, Gauss-Seidel, multigrid) to obtain higher convergence rates for solutions of linear systems arising from discretizations of PDEs up to machine precision. Classical preconditioners reduce the residual’s high-frequency mode. HINTS balances the convergence behavior across the spectrum of eigenmodes by utilizing the spectral bias of Deeponet, resulting in a uniform convergence rate, giving rise to an efficient iterative solver. This assertion has been recently further investigated in [17], and the robustness of Deeponet-based preconditioners for iterative schemes of linear systems has been shown. We emphasize that Deeponet is trained over a low-resolution grid but provides robust preconditioners for higher resolutions without further training.

A drawback of Deeponets, and as a result also of HINTS, is that it is not geometry aware. Hence, Deeponet trained on a reference domain might not be efficient for problems on different domains, which are not only small perturbations of the reference domain [15]. The direct solution is modifying Deeponet to train using data from different geometries and multiple domains [10, 12, 13, 14, 16, 24]. Yet, as data generation for thousands of domains is time-consuming, we aim to avoid this approach, which might be overkill when using Deeponet as a preconditioner in HINTS.

In [15], a state-of-the-art (SOTA) vanilla Deeponet has been demonstrated to operate robustly when used inside HINTS on different domains by extending f to be 0 for points outside the required domain. Though this approach does not require additional training on multiple domains, it fails to converge in multiple examples we will demonstrate, and training on multiple domains is necessary.

Another potential drawback of the vanilla Deeponet is its convolutional neural networks (CNN). Though convolutional neural networks have obtained remarkable capability for studying images and functions on rectangular grids, in the problems studied by [10, 12, 13, 14, 16, 24], it becomes impractical for several 2D/3D problems and several networks excluding convolutional operations were suggested.

In this work, we strive to use HINTS, also obtained from a non-convolutional and geometry-aware Deeponet, to improve existing iterative schemes to solve problems on different domains even though the training is only on simple geometries. We avoid using CNN operations for two reasons: First, we would like to avoid any artificial extension of a function outside its domain. Second, we wish to provide a network that could be used on problems where convolutional operations are impractical.

1.1. Problem Formulation and Novelties

We consider the following problem:

(1.4){Δu+k2u=fΩ,∂u∂ν→+−1ku=0∂Ω,

where Ω⊂[0,1]2 is a piecewise smooth domain. Again, we stress that this problem is non-definite for sufficiently large k and non-symmetric. See [4] Chapter~35 for well-posedness. Standard second-order finite-difference discretization of (1.4) leads to a system of linear equations of the form

(1.5)AX=Y.

We consider the following Richardson-type iterations to solve (1.5):

(1.6)Xn+1=Xn+Pn−1rn,

where rn=−AXn+Y and Pn−1 is some preconditioner. The matrix A+A∗ with A given by (1.4) can be non-definite and, in general, cheap methods such as Jacobi/Gauss-Seidel diverge. As noted by [27], Pn−1 can be obtained using a neural network D, estimating the inverse operator of (1.4) and leading to convergence of (1.6) by using the hybrid method

(1.7){un+1=un+D(rn)nmodJ=0,un+1=Hunelse,

where J∈N and H is any classical iterative update rule (e.g., Gauss-Seidel, GMRES). Our goal is to replace D by a neural network N such that the resulting scheme (1.7) applies to families of geometries that are not necessarily “similar” to the reference geometry used in training without further fine-tuning of the neural network.

Remark 1.1. Deeponet receives as an input a function, and therefore can handle functions defined on different grids when the input function is sampled over a finite number of points. By interpolating the input function, Deeponet trained on a certain grid can still handle input data over a coarser or finer grid. When using coarser/finer grids that contain grid points used for training, the interpolation process is immediate as it requires only choosing a subset of the sample points. Therefore, when one uses Deeponet on different grids, it is less expensive to include grids which include the grid used for training.

Generating training data requires solving (1.4). So, generating data for multiple geometries can be very expensive and challenging when dealing with complex geometries. Our goal is to generate data on simple geometries only (e.g., rectangles), and then test (2.3) on different geometries without any further fine-tuning for different geometries (cf. [11]). As the neural network takes inputs of fixed sizes, one has to account for dealing with domains with different amounts of grid points. Moreover, one has to deal with how to order the points inside each domain. Another problem is how one can evaluate N on Ω by computations that “ignore” all points outside Ω, since the non-homogeneous term f in (1.4) is not defined outside Ω.

1.2. Main Novelties

We present a new architecture for learning the solutions of (1.4), which accounts for the domain’s geometry. Our model is a novel improvement to the vanilla Deeponet as it significantly improves the geometry transferability, in case the vanilla model might fail. To be more specific, our network is trained on data generated from solutions to (1.4) on simple reference geometries and still produces robust iterative schemes for different subgeometries using (2.3), without additional training or fine-tuning (cf. [11]). We examine scenarios where HINTS with vanilla Deeponet diverges while our network converges.

While common layers in Deeponet are mainly linear and convolutional layers, we also use masked self-attention layers [22]. The masked self-attention layers enable us, during evaluation on an arbitrary geometry Ω with an arbitrary number of grid points, to include only the points that belong to Ω. This approach does not require an artificial extension of f outside Ω to fit the input size of the network (cf. [15]). We observed that when narrow cracks appear in the tested geometries, the masking effect becomes significantly noticed and outperforms the vanilla model.

We compare our approaches with the vanilla Deeponet and a naive extension of the vanilla Deeponet to a geometry-aware Deeponet with CNN layers. The latter model is presented to demonstrate that our problem is not only a trivial extension of the vanilla case.

To our best knowledge, this is the first attempt to learn preconditioners using geometry-aware neural networks, and integrating them with classical iterative solvers.

1.3. Notations and Symbols

We use the following symbols and notations:

  • For v∈Rn, std⁡(v) is the statistical standard deviation.

  • For v∈Cn, std⁡(v)=std⁡(Real⁡(v))+−1std⁡(Im⁡(v)).

  • For v,u∈Rn, v⊙u∈Rn is the pointwise multiplication product of v,u.

  • h=1/N denotes the grid size in a spatial two-dimensional discretization of the unit square [0,1]2 for N2 points.

  • J is the skip factor in (2.3).

  • k is a positive constant in the Helmholtz equation (1.2).

  • GMRES corresponds to the GMRES algorithm (Algorithm 3 in [23]).

  • Hints-GS(J) is the application of (2.3) where H is a single application of the Gauss-Seidel updating rule.

  • Hints-GMRES(m,J) is the application of (2.3) where H corresponds to m iterations of the GMRES algorithm.

2. Resources and Techniques

In this section we describe in detail our proposed geometry-aware Deeponet model.

2.1. The Model

We generate a uniform grid on [0,1]2 with 15×15 points. We flatten the grid points to obtain a vector d→∈R225×2.

Model’s inputs:

  1. d→.

  2. A domain Ω⊂[0,1]2 represented by vector OΩ∈R225, where

    OiΩ=dist⁡((di0,di1),∂Ω),

    and dist is the signed distance function.

  3. A real function f:Ω→R, represented by the complex vector fΩ∈C225, where

    fiΩ={f(di0,di1),(di0,di1)∈Ω,0,otherwise.
  4. A masking matrix MΩ∈R225×225 defined as follows:

    MΩ={0,(di0,di1)∈Ω,−∞,otherwise.
  5. y∈Ω.

Model’s output: a complex number representing the solution to (1.2) evaluated at y.

2.1.1. Model’s Layers

First, we define the M-attention operation of v∈RN and MΩ to be

M-attention⁡(v,MΩ):=softmax⁡(vvT+MΩN)v

(see [22]). The function softmax:Rk→(0,1)k is defined by

softmax⁡(Z)i=exp⁡(Zi)∑i=1kexp⁡(Zi).

Notice that the M-attention does not depend on the values of f outside Ω. If the masking matrix MΩ is replaced with a zero matrix, then M-attention accounts for a trivial extension of f outside Ω and, therefore, depends on the extension one chooses for f outside Ω.

Since the output of the model should be a complex number, we first define two Deeponets, Nreal and Nimag, having the structure (1.3) with the following branch and trunk:

Branch:

B:R225×4→R80

defined by the operation of MΩ-attention on the matrix generated by concatenating fΩ,d→,OΩ and followed by a dense neural network [225,200,100,80] with a tanh activation.

Trunk:

T:R3→R80

defined by the operation of a dense neural network (NN) with layers [3,200,100,80] and sin activation function [12], on the vector generated by concatenating y and dist⁡(y,∂Ω).

We construct a neural network:

(2.1)Nmasked=Nreal+−1Nimag.

We also define a second network:

(2.2)Nnon−masked=Nreal+−1Nimag,

where the masking matrix MΩ is replaced by the zero matrix, and an input function f is extended by zeros outside its domain.

We define an updating rule for the hybrid method called HINTS, as follows:

(2.3){un+1=un+std⁡(rn)α⋅N(αrnstd⁡(rn))nmodJ=0,un+1=Hunotherwise,

where J∈N and α>0 are (non-trained) hyper-parameters to be selected by the user, and n∈N is the iteration number. The normalization factor αstd⁡(rn) ensures that the input of the network N will belong to the distribution of the training data.

Remark 2.1. We have defined our networks to consider as inputs only real functions. In order to apply Nmasked (Nnon−masked) to complex functions, one can apply the same network separately to the real and imaginary parts of f.

2.2. Model’s Training Data

We fix k2=21. Then, we discretize each of the training geometries in Figure 1 with a uniform grid of grid size h=1/14. The training data for N is obtained by solving (1.4) on the geometries in Figure 1 using a standard second-order finite difference method, with 300 different functions f, taken to be Gaussian random fields over R with zero mean and covariance kernel

(2.4)cov⁡(f(xi),f(xj))=σexp⁡(−12‖xi−xj‖2l2),

where l=σ=0.1.

A 3x3 grid of Cartesian plots showing various rectangular subdomains outlined in blue within the unit square [0, 1] x [0, 1]
Figure 1. Geometries used for training.

2.3. Network Complexity

Every J iterations of the classical iterative method is followed by one application of N. As the network’s input is a vector of fixed dimension, each evaluation of the network on a single data point is of constant computational and memory complexity. As a result, for a sparse (and band-limited) linear system of order N, the complexity of each iteration involving N in (2.3) is O(N) for both memory and computational complexity.

Tested domains Ω used in Tables 1, 2, and 3 for solving (1.2).
Figure 2. Tested domains Ω used in Tables 1, 2, and 3 for solving (1.2).

3. Results

We compare the performance of HINTS (2.3) using numerical simulations for four different models:

  1. Nmasked is given by (2.1) and was trained on geometries in Figure 1.

  2. Nnon−masked is given by (2.2) and was trained on geometries in Figure 1.

  3. Vanilla DeepONet taken from [27] (see Appendix 4) and was trained on [0,1]2.

  4. ga-vanilla is a naive implementation of geometry-aware DeepONet using CNN (see Appendix 4) that was trained on the geometries in Figure 1.

Our tested geometries for problem (1.2) are presented in Figure 2. We discretize each tested domain with a higher resolution than the resolution used for training, N=15. We solve (1.4) using (2.3) with a random function f, sampled from another distribution than the one used for training (f has been generated as a vector from a normal distribution with mean ten and variance 10). Since the neural network operates on the error terms only, the distribution of f does not affect the hybrid scheme’s robustness. We compare our method with both Gauss-Seidel iterations (see Table 1) and GMRES iterations (see Table 3). The main advantage of Gauss-Seidel (GS) is that, for sparse systems, it has linear complexity in both memory and computational time. However, GS does not always converge for the non-definite cases in our examples. We will show that Hints-GS converges even when plain Gauss-Seidel iterations diverge.

Table 1. Averaged iteration number statistics for the method (2.3) with HINTS Gauss-Seidel (H-GS) with the geometries in Figure 2, k2=21, α=0.3. The classical Gauss-Seidel iterations are not presented as they diverge.
hMethodModelIter. number
Geometry 1
1/57H-GS(J=40)Nmasked4877
1/57H-GS(J=40)ga-vanilladiverge
1/57H-GS(J=40)vanilladiverge
1/57H-GS(J=40)Nnon−maskeddiverge
Geometry 2
1/57H-GS(J=60)Nmasked12000
1/57H-GS(J=60)ga-vanilladiverge
1/57H-GS(J=60)vanilladiverge
1/57H-GS(J=60)Nnon−masked16000
Geometry 3
1/57H-GS(J=40)Nmasked2157
1/57H-GS(J=40)ga-vanilla1616
1/57H-GS(J=40)vanilla1325
1/57H-GS(J=40)Nnon−masked1381
Geometry 4
1/57H-GS(J=40)Nmasked1519
1/57H-GS(J=40)ga-vanilladiverge
1/57H-GS(J=40)vanilla813
1/57H-GS(J=40)Nnon−masked764
Geometry 5
1/57H-GS(J=40)Nmasked1620±170
1/57H-GS(J=40)ga-vanilladiverge
1/57H-GS(J=40)vanilla1064
1/57H-GS(J=40)Nnon−masked842
Geometry 6
1/57H-GS(J=40)Nmasked6300
1/57H-GS(J=40)ga-vanilladiverge
1/57H-GS(J=40)vanilla>20000
1/57H-GS(J=40)Nnon−masked7200
Geometry 7
1/57H-GS(J=40)Nmasked10800
1/57H-GS(J=40)ga-vanilladiverge
1/57H-GS(J=40)vanilladiverge
1/57H-GS(J=40)Nnon−masked8034
Geometry 8
1/57H-GS(J=40)Nmasked2028
1/57H-GS(J=40)ga-vanilla2400
1/57H-GS(J=40)vanilla1370
1/57H-GS(J=40)Nnon−masked1617
Table 2. Averaged iteration number statistics for the method (2.3) with HINTS Gauss-Seidel (H-GS) with the geometries in Figure 2, k2=21, α=0.3. The classical Gauss-Seidel iterations are not presented as they diverge.
hMethodModelIter. number
Geometry 9
1/57H-GS(J=80)Nmasked9700
1/57H-GS(J=80)ga-vanilladiverge
1/57H-GS(J=80)vanilladiverge
1/57H-GS(J=80)Nnon−masked18500
Geometry 10
1/57H-GS(J=40)Nmasked2500
1/57H-GS(J=40)ga-vanilladiverge
1/57H-GS(J=40)vanilla2450
1/57H-GS(J=40)Nnon−masked1055
Geometry 11
1/57H-GS(J=80)Nmasked2100
1/57H-GS(J=80)ga-vanilla1770
1/57H-GS(J=80)vanilla1055
1/57H-GS(J=80)Nnon−masked1040
Geometry 12
1/57H-GS(J=40)Nmasked1200
1/57H-GS(J=40)ga-vanilla1000
1/57H-GS(J=40)vanilla940
1/57H-GS(J=40)Nnon−masked560
Table 3. Averaged iteration number statistics for the method (2.3) with HINTS GMRES (H-GMRES) with the geometries in Figure 2, k2=21, α=0.3.
hMethodModelGMRES iter.DeepONet iter.
Geometry 1
1/224GMRES—27120—
1/224H-GMRES(m=30,J=10)Nmasked11400130
1/224H-GMRES(m=30,J=10)vanilla11000120
1/224H-GMRES(m=30,J=10)Nnon−masked24797273
Geometry 2
1/224GMRES—19416—
1/224H-GMRES(m=30,J=10)Nmasked850090
1/224H-GMRES(m=30,J=10)vanilla25800285
1/224H-GMRES(m=30,J=10)Nnon−masked720070
Geometry 4
1/224GMRES—21570—
1/224H-GMRES(m=30,J=10)Nmasked530057
1/224H-GMRES(m=30,J=10)vanilla382442
1/224H-GMRES(m=30,J=10)Nnon−masked449048
Geometry 6
1/224GMRES—17816—
1/224H-GMRES(m=30,J=10)Nmasked881496
1/224H-GMRES(m=30,J=10)vanilla13400147
1/224H-GMRES(m=30,J=10)Nnon−masked13079144
Geometry 7
1/224GMRES—18426—
1/224H-GMRES(m=30,J=10)Nmasked880096
1/224H-GMRES(m=30,J=10)vanilla16500190
1/224H-GMRES(m=30,J=10)Nnon−masked15300170
Geometry 9
1/224GMRES—14616—
1/224H-GMRES(m=30,J=10)Nmasked8800100
1/224H-GMRES(m=30,J=10)vanilla14600160
1/224H-GMRES(m=30,J=10)Nnon−masked9900108
Geometry 11
1/224GMRES—16257—
1/224H-GMRES(m=30,J=10)Nmasked550060
1/224H-GMRES(m=30,J=10)vanilla414045
1/224H-GMRES(m=30,J=10)Nnon−masked383042

GMRES’s main advantage is proven convergence for any non-singular matrix, while its major drawback is memory complexity. For (not necessarily sparse) linear equations with N equations (i.e., corresponding matrix A∈CN×N), the iteration complexity of GMRES after n iterations is O(Nn2) and the memory complexity is O(N). A possible remedy is the restarted GMRES method, GMRES(m), whose memory complexity reduces to O(mn) independently of the number of iterations but with a possible degradation of convergence rates. We will show that combining GMRES inside (2.3) yields a more efficient method in terms of both time and computational complexity.

It is worth noting that, even when Gauss-Seidel iterations diverge, they are still known to reduce high-frequency errors throughout the iterations. Therefore, Hints-GS highlights the capability of HINTS to effectively control low-frequency modes.

We use the following schemes for comparisons as H in (2.3): GMRES, Gauss-Seidel (GS), Hints-GS, Hints-GMRES, see Subsection 1.3. Multiple computations were performed, and we show the mean number of iterations required. k>0 is the wavenumber in the Helmholtz equation, h=dx=dy is the discretization size, and J is the skip factor in (2.3) for how often the neural network N (DeepONet) is called.

Remark 3.1. One of the highlights of [15] was the generalization capability of the vanilla DeepONet on a difficult obstacle problem. We also performed experiments of problem (1.2) on obstacle problems with Ω=[0,1]2∖[0.25,0.5]2 and Ω=[0,1]2∖[0.25,0.75]2, and observed the superiority of Nmasked over the vanilla DeepONet (see Table 4). We have also performed additional experiments on exterior domains with different shapes of obstacles (see Table 5).

Table 4. Averaged iteration number statistics for the method (2.3) with HINTS Gauss-Seidel (H-GS) and H-GMRES with the geometries in Figure 3, k2=21,α=0.3.
hMethodModelIter. number
Obstacle 1
1/57Gauss-Seidel—diverge
1/57H-GS(J=40)vanilla18000
1/57H-GS(J=40)Nmasked8000
Obstacle 2
1/57Gauss-Seidel—10600
1/57H-GS(J=100)vanilla>20000
1/57H-GS(J=100)Nmasked3425
Obstacle 1
1/224GMRES—21400
1/224H-GMRES(J=15)vanilladiverge
1/224H-GMRES(J=15)Nmasked14100
Obstacle 2
1/224GMRES—6460
1/224H-GMRES(J=20)vanilla12000
1/224H-GMRES(J=20)Nmasked8100
Tested exterior domains used in Table 4 for solving Δu + k²u = f. The boundary conditions on the inner boundary are u = 0. The boundary conditions on the outer boundary are \\frac{\\partial u}{\\partial \\vec{\\nu}} + \\sqrt{-1} k u = 0.
Figure 3. Tested exterior domains used in Table 4 for solving Δu+k2u=f. The boundary conditions on the inner boundary are u=0. The boundary conditions on the outer boundary are ∂u∂ν→+−1ku=0.

3.1. Observations from Tables 1,2,3,4,5

We have demonstrated multiple examples with complex geometries for which HINTS with the vanilla method fails, while our network succeeds. This confirms the necessity of the new models Nmasked and Nnon−masked we presented. In particular, in terms of convergence capability, our new models outperform the vanilla model. Yet, we mention that the geometry transferability of the vanilla model still performs robustly on many domains including non-rectangular ones (geometries 3, 4, 5, 8), and in many cases provides similar or slightly better convergence rates to that of our geometry-aware models.

Table 5. Averaged iteration number statistics for the method (2.3) with HINTS Gauss-Seidel (H-GS) and H-GMRES with the geometries in Figure 4, k2=21, α=0.3.
hMethodModelIter. number
1 obstacle
1/57Gauss-Seidel—diverge
1/57H-GS(J=100)vanilla16700
1/57H-GS(J=100)Nmasked8500
2 obstacles
1/57Gauss-Seidel—diverge
1/57H-GS(J=100)vanilla>40000
1/57H-GS(J=100)Nmasked14500
3 obstacles
1/57Gauss-Seidel—12500
1/57H-GS(J=200)vanilla8300
1/57H-GS(J=200)Nmasked8500
1 obstacle
1/224GMRES—32100
1/224H-GMRES(m=30,J=15)vanilla10575
1/224H-GMRES(m=30,J=15)Nmasked16480
2 obstacles
1/224GMRES—18000
1/224H-GMRES(m=30,J=15)vanilla14193
1/224H-GMRES(m=30,J=15)Nmasked15810
3 obstacles
1/224GMRES—8600
1/224H-GMRES(m=30,J=15)vanilla13343
1/224H-GMRES(m=30,J=15)Nmasked16717
Tested exterior domains used in Table 5 solving Δu + k²u = f. The boundary conditions on the inner boundaries are u = 0. The boundary conditions on the outer boundary are \\frac{\\partial u}{\\partial \\vec{\\nu}} + \\sqrt{-1} k u = 0.
Figure 4. Tested exterior domains used in Table 5 solving Δu+k2u=f. The boundary conditions on the inner boundaries are u=0. The boundary conditions on the outer boundary are ∂u∂ν→+−1ku=0.

In almost all tests ga-vanilla obtained the worst results, implying that a naive extension of the vanilla model does not work and a more sophisticated approach was required.

When the domain’s boundary contains abrupt changes (cracks/bumps/holes), such as geometries 1, 2, 6, 7, 9, Nmasked performed better than all methods. In cases for which vanilla outperformed Nmasked, Nnon−masked outperformed Nmasked, with an exception for H-GMRES for geometry 1.

In almost all cases we tested there is a correlation between HINTS’ performance with different methods (GS, GMRES). Yet, the method used inside HINTS and the corresponding parameters can violate this correlation (e.g., Table 5). In more detail, the experiments with the geometries in Figure 4 show that with GS, Nmasked outperforms the vanilla method significantly, whereas for the same problem with GMRES (J=15) the vanilla method is better. Yet, for a different parameter J=5, H-GMRES with vanilla diverged while it converged with Nmasked.

We can conclude that the idea of extending a function outside its domain to be 0, as suggested in [15], can indeed provide robust results for different domains as long as the domain does not contain abrupt changes (narrow cracks, holes). In these problematic scenarios our masked model can overcome this limit.

Based on our results, we suggest to a potential user who cannot use CNN operations to hold both Nmasked and Nnon−masked. Further study on unifying these networks is left for future research.

Remark 3.2. The running times of (2.3) can differ for different types of computers. There is room for additional improvement of (2.3) as we did not use a GPU and did not exploit parallel computation of batched data during inference. In our machine, H-GS with higher resolutions than 1/57 is either very slow or diverges, and therefore we switched to GMRES to obtain faster CPU time.

4. Conclusions, Remarks and Further Research

We have constructed a novel architecture for preconditioning linear equations, which is aware of the geometry of the problem. We showed evidence by numerical examples that our network remains a robust preconditioner for problems with different geometries without any fine-tuning procedure and outperforms existing SOTA methods for tackling similar problems. Another advantage of our proposed network is that it is non-convolutional and therefore can be generalized to non-rectangular grids and point-clouds.

We demonstrated our network on a complex Helmholtz equation with complex boundary conditions, which are hard to solve due to non-definiteness or non-symmetry, and on an obstacle problem.

In future work, we intend to generalize our network to include more attention layers and, with the availability of more advanced hardware, to train the network on a larger number of geometries.

Another possible extension is to modify the neural network so that it is invariant under domain translations, as the PDE is.

Appendix A

Appendix A.1. The Networks

For vanilla DeepONet, trained on a single domain [0,1]2, we employ the architecture in [27] page~37. The branch net is a combination of a 2D CNN (input dimension 15×15, number of channels [1,40,60,100], kernel size 3×3, stride 2) and dense NN (dimension [100,80,80]) with ReLU activation function. The dimension of the dense trunk network is [2,80,80,80] with activation of tanh.

For the naive geometry-aware DeepONet (ga-vanilla), we represent the domain as a 2D binary image and pad the values of the function f with zeros in all points outside the domain f is defined. We then use CNN as in the vanilla DeepONet whose output is concatenated and fed into a dense NN (dimension [200,80,80]), resulting in the branch. The dimension of the trunk network is [2,80,80,80] with the activation of tanh.

Appendix A.2. Data Generation and Training

We fix k2=21 in (1.4). The vanilla network is trained over 300 solutions of (1.4) in [0,1]2 discretized uniformly with h=1/15. The solutions are obtained by taking f to be 300 Gaussian random fields (2.4). The equations are solved using a standard second-order finite difference method. The network ga-vanilla is trained on the same data used to train Nmasked, see Subsection 2.2.

We trained the networks using PyTorch for 1000 epochs with a batch size of 64, and the initial learning rate 1×10−4 decreased by 0.5 at the final epochs.

References

28 Cites in Article
  1. 1. D. Alciatore,R. Miranda (1995). A winding number and point-in-polygon algorithm. Glaxo Virtual Anatomy Project Research Report
  2. 2. A.H. Baker,E.R. Jessup,T. Manteuffel (2005). A technique for accelerating the convergence of restarted GMRES. SIAM Journal on Matrix Analysis and Applications, 26(4), 962-984.
  3. 3. V. Dolean,P. Jolivet,F. Nataf (2015). An Introduction to Domain Decomposition Methods: Algorithms, Theory, and Parallel Implementation.
  4. 4. A. Ern,J.-L. Guermond (2021). Finite Elements II: Galerkin Approximation, Elliptic and Mixed PDEs.
  5. 5. O.G. Ernst,M.J. Gander (2011). Why it is difficult to solve Helmholtz problems with classical iterative methods. Numerical Analysis of Multiscale Problems, 325-363.
  6. 6. M.J. Gander,I.G. Graham,E.A. Spence (2015). Applying GMRES to the Helmholtz equation with shifted Laplacian preconditioning: what is the largest shift for which wavenumber-independent convergence is guaranteed?. Numerische Mathematik, 131(3), 567-614.
  7. 7. M.J. Gander,H. Zhang (2019). A class of iterative solvers for the Helmholtz equation: Factorizations, sweeping preconditioners, source transfer, single layer potentials, polarized traces, and optimized Schwarz methods. SIAM Review, 61(1), 3-76.
  8. 8. M.J. Gander,F. Magoules,F. Nataf (2002). Optimized Schwarz methods without overlap for the Helmholtz equation. SIAM Journal on Scientific Computing, 24(1), 38-60.
  9. 9. M.J. Gander (2006). Optimized Schwarz methods. SIAM Journal on Numerical Analysis, 44(2), 699-731.
  10. 10. R.J. Gladstone,H. Rahmani,V. Suryakumar,H. Meidani,M. D'Elia,A. Zareei (2024). Mesh-based GNN surrogates for time-independent PDEs. Scientific Reports, 14(1), 3394.
  11. 11. S. Goswami,K. Kontolati,M.D. Shields,G.E. Karniadakis (2022). Deep transfer operator learning for partial differential equations under conditional shift. Nature Machine Intelligence, 4(12), 1155-1164.
  12. 12. J. He,S. Koric,S. Kushwaha,J. Park,D. Abueidda,I. Jasiuk (2023). Novel DeepONet architecture to predict stresses in elastoplastic structures with variable complex geometries and loads. Computer Methods in Applied Mechanics and Engineering, 415, 116277.
  13. 13. J. He,S. Kushwaha,J. Park,S. Koric,D. Abueidda,I. Jasiuk (2024). Sequential Deep Operator Networks (S-DeepONet) for predicting full-field solutions under timedependent loads. Engineering Applications of Artificial Intelligence, 127, 107258.
  14. 14. J. He,S. Koric,D. Abueidda,A. Najafi,I. Jasiuk (2024). Geom-DeepONet: A Point-cloud-based Deep Operator Network for Field Predictions on 3D Parameterized Geometries. arXiv preprint
  15. 15. A. Kahana,E. Zhang,S. Goswami,G. Karniadakis,R. Ranade,J. Pathak (2023). On the geometry transferability of the hybrid iterative numerical solver for differential equations. Computational Mechanics, 72(3), 471-484.
  16. 16. A. Kashefi,D. Rempe,L.J. Guibas (2021). A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries. Physics of Fluids, 33(2)
  17. 17. A. Kopanicakova,G.E. Karniadakis (2024). DeepONet based preconditioning strategies for solving parametric linear systems of equations. arXiv preprint
  18. 18. P.L. Lions (1990). On the Schwarz alternating method. III: a variant for nonoverlapping subdomains. Third International Symposium on Domain Decomposition Methods for Partial Differential Equations, 6, 202-223.
  19. 19. L. Lu,P. Jin,G. Pang,Z. Zhang,G.E. Karniadakis (2021). Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3), 218-229.
  20. 20. N. Rahaman,A. Baratin,D. Arpit,F. Draxler,M. Lin,F. Hamprecht,Y. Bengio,A. Courville (2019). On the spectral bias of neural networks. International Conference on Machine Learning, 5301-5310.
  21. 21. S. Raza,C. Ding (2019). Progress in context-aware recommender systems---An overview. Computer Science Review, 31, 84-97.
  22. 22. A. Vaswani,N. Shazeer,N. Parmar,J. Uszkoreit,L. Jones,A.N. Gomez,L. Kaiser,I. Polosukhin (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30
  23. 23. Y. Saad,M.H. Schultz (1986). GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 7(3), 856-869.
  24. 24. M. Yin,N. Charon,R. Brody,L. Lu,N. Trayanova,M. Maggioni (2024). Dimon: Learning solution operators of partial differential equations on a diffeomorphic family of domains. arXiv preprint
  25. 25. Q. Zou (2023). GMRES algorithms over 35 years. Applied Mathematics and Computation, 445, 127869.
  26. 26. S. Zhang,L. Yao,A. Sun,Y. Tay (2019). Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR), 52(1), 1-38.
  27. 27. E. Zhang,A. Kahana,A. Kopanicakova,E. Turkel,R. Ranade,J. Pathak,G.E. Karniadakis (2023). Blending Neural Operators and Relaxation Methods in PDE Numerical Solvers. Nature Machine Intelligence
  28. 28. M. Zhu,S. Feng,Y. Lin,L. Lu (2023). Fourier-DeepONet: Fourier-enhanced deep operator networks for full waveform inversion with improved accuracy, generalizability, and robustness. arXiv preprint

Funding

The work was supported by the US–Israel Bi-national Science Foundation (BSF) under grant # 2020128.

Conflict of Interest

The authors declare no conflict of interest.

Ethical Approval

No ethics committee approval was required for this article type.

Data Availability

https://github.com/idanv87/convnet.git

How to Cite This Article

Idan Versano, Eli Turkel. 2026. "Attention-based Solvers for Linear Equations that are Geometry Aware". Global Journal of Science Frontier Research - F: Mathematics & Decision.

More Citation Formats

Download Citation

Journal Specifications

Crossref Journal DOI 10.17406/GJSFR

Print ISSN 0975-5896

e-ISSN 2249-4626

Keywords
Classification
arXiv cs.NA
arXiv cs.LG
Version of record
v1.2

Language
English
Order Article Reprint
Experience in AR

Explore published articles in an immersive Augmented Reality environment. Our platform converts research papers into interactive 3D books, allowing readers to view and interact with content using AR and VR compatible devices.

Read in 3D

Your published article is automatically converted into a realistic 3D book. Flip through pages and read research papers in a more engaging and interactive format.

Article Matrices
Total Views: 62
Total Downloads: 1
All Trends

This is the heading

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

High-quality academic research articles on global topics and journals.

Attention-based Solvers for Linear Equations that are Geometry Aware

Idan Versano
Idan Versano School of Mathematical Sciences
Eli Turkel
Eli Turkel School of Mathematical Sciences

Request Access

Please fill out the form below to request access to this research paper. Your request will be reviewed by the editorial or author team.
X