Parameter-dependent Time Sequence Prediction with Deep Learning

This project develops deep-learning (DL) methods for predicting time sequences that are parametrized with multiple variables, e.g., random initial conditions, physical/geometrical parameters, external time-varying and parameter-dependent signals, etc.

Data-Augmented Predictive Deep Neural Network: Enhancing the extrapolation capabilities of non-intrusive surrogate models

The structure of the proposed DL method.

The structure of the proposed DL method.

Numerically solving a large parametric nonlinear dynamical system is challenging due to its high complexity and the high computational costs. In recent years, machine-learning-aided surrogates are being actively researched. However, many methods fail in accurately generalizing in the entire time interval [0, T], when the training data is available only in a training time interval [0, T0], with T0<T.

Prediction at a testing parameter sample Re=800 that is outside of the training parameter range.

Prediction at a testing parameter sample Re=800 that is outside of the training parameter range.

To improve the extrapolation capabilities of the surrogate models in the entire time domain, we propose a new deep learning framework [1], where kernel dynamic mode decomposition (KDMD) is employed to evolve the dynamics of the latent space generated by the encoder part of a convolutional autoencoder (CAE). After adding the KDMD-decoder-extrapolated data into the original data set, we train the CAE along with a feed-forward deep neural network using the augmented data. The trained network can predict future states outside the training time interval at any out-of-training parameter samples.

Transformer accurately predicts multiple outputs of parametric dynamical systems with time-varying external inputs

Prediction results at randomly picked testing initial conditions.

Prediction results at randomly picked testing initial conditions.

Transformer has demonstrated to be powerful in predicting long-term sequences [3]. We explore the promising performance of a transformer model [2] in predicting outputs of parametric dynamical systems with external time-varying input signals. The outputs of such systems vary not only with physical parameters but also with external time-varying input signals. Accurately catching the dynamics of such systems is challenging for both model order reduction and machine learning. We have extended the capability of an existing transformer model [2] to predict multiple output responses of these systems. The interpretability of the attention weight matrix for a single output is naturally extended to multiple outputs. The transformer accurately predicts the sequence of multiple outputs, regardless of the nonlinearity of the system and the dimensionality of the parameter space. In contrast to many existing autoregressive methods for time sequence prediction, which generate predictions step-by-step, the transformer model is able to do multi-horizon prediction in a single prediction pass.

References

[1] Shuwen Sun, Lihong Feng, Peter Benner. Data-augmented predictive deep neural network: enhancing the extrapolation capabilities of non-intrusive surrogate models. arXiv preprint arXiv:2410.13376, 2024.

[2] B. Lim, S. O. Arık, N. Loeff, and T. Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37:1748–1764, 2021.

[3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010. Curran Associates Inc., 2017.