Numerical Analysis Reveals Automatic Differentiation Can Produce Incorrect Derivatives in Physics Simulations
By
abetusk
Solid neighbourhood-bakery energy. Trustworthy and warm.
Summary
This article discusses the numerical analysis of differentiable simulation in scientific machine learning, highlighting potential issues with automatic differentiation (AD) when applied to physics simulations. It examines how AD can produce incorrect derivatives despite its widespread use in SciML for gradient-based optimization. The content focuses on the numerical stability and robustness of AD using examples from Python libraries like Jax (diffrax) and PyTorch (torchdiffeq), challenging the simplistic notion that simulators can be easily integrated into loss functions.
Key quotes
· 4 pulledScientific machine learning (SciML) relies heavily on automatic differentiation (AD), the process of constructing gradients which include machine learning integrated into mechanistic models
While these differentiable programming approaches pitch an idea of 'simply put the simulator into a loss function and use AD', it turns out there are a lot more subtle details to consider in practice
how numerically stable and robust is AD?
Automatic Differentiation of Physics Can Give Incorrect Derivatives
You might also wanna read
DeepSeek-V4: Hybrid Sparse-Attention Architecture Enables Efficient Million-Token Context Inference
DeepSeek-V4 introduces a hybrid sparse-attention architecture combined with on-policy distillation across domain specialists, enabling 1M-to
artgor.medium.com·7h ago
Rotary GPU: Enabling Large Mixture-of-Experts Models on Consumer Laptop GPUs with Limited Memory
This paper presents Rotary GPU, an exploratory approach to running large Mixture-of-Experts (MoE) language models on consumer-grade hardware
arxiv.org·1d ago
LinkedIn cuts GPU training hours by 65% with Generative Recommender system optimizations
LinkedIn has developed a Generative Recommender (GR) system that models user activity as token sequences, offering richer long-context perso
startuphub.ai·3d ago
PromptEmbedder: A Dual-LLM Framework for Efficient, Architecture-Agnostic Text Embedding
The article presents PromptEmbedder, a novel dual-LLM framework for efficient and transferable text embedding. It addresses the bottleneck o
arxiv.org·3d ago
Rank-Aware Decomposition Technique Reduces Computation in Recommender Systems by 87.5%
This paper presents a rank-aware decomposition technique for deep ranking models in industrial recommender systems. The key insight is that
arxiv.org·3d ago
Unified Framework for Variational Quantum Knowledge Graph Embeddings on NISQ Devices
This paper introduces a unified framework for variational quantum algorithms (VQAs) applied to knowledge graph embeddings on near-term NISQ
arxiv.org·3d ago