Skip to content
Author: yuantianle

📊 Optimization Methods Classification⚓︎

方法类型 方法名称 特性 应用领域 公式
常微分方程数值解法 龙格-库塔法 - 高精度
- 稳定性好
- 计算量相对较大		 - 常微分方程求解
- 动力学系统模拟
- 工程和物理问题		 4阶龙格-库塔公式（常见形式）：
\(y_{n+1} = y_n + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)\)
\(k_1 = hf(x_n, y_n) ， k_2 = hf(x_n + \frac{h}{2}, y_n + \frac{k_1}{2})\) …
欧拉法 - 计算简单
- 精度较低
- 时间步长敏感		 - 常微分方程求解
- 简单物理系统模拟		 显式欧拉法:
\(y_{n+1} = y_n + hf(x_n, y_n)\)
中点法 - 使用中间点的斜率
- 精度比欧拉法更高
- 稳定性较好		 - 常微分方程求解
- 中等精度要求问题		 中点法公式：
\(y_{n+1} = y_n + hf\left(x_n + \frac{h}{2}, y_n + \frac{h}{2} f(x_n, y_n)\right)\)
偏微分方程数值解法 有限差分法 - 将微分方程转化为代数方程
- 适用于偏微分方程
- 网格划分对结果影响大		 - 偏微分方程求解
- 数值天气预报
- 流体力学		 二阶中心差分近似（用于偏导数）：
\(f''(x) \approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^2}\)
有限元方法 - 将问题域划分为小区域
- 适用于复杂几何和边界条件
- 精度较高		 - 偏微分方程求解
- 结构工程
- 热传导问题		 泛化的有限元方法公式：
\(\int_{\Omega} (\nabla u \cdot \nabla v - fv) \, d\Omega = 0\)
优化问题方法 最速下降法 - 使用函数的一阶导数
- 实现简单，适用性广
- 可能收敛速度慢		 - 非线性方程组求解
- 机器学习参数优化
- 最小化成本函数		 迭代公式：
\(x_{n+1} = x_n - \alpha_n \nabla f(x_n)\)
牛顿法 - 使用函数的一阶和二阶导数
- 收敛速度快
- 需要计算Hessian矩阵		 - 非线性方程组求解
- 优化问题		 迭代公式：
\(x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}\)
共轭梯度法 - 不需要存储Hessian矩阵
- 适用于大规模问题
- 仅适用于二次函数		 - 非线性方程组求解
- 大规模优化问题
- 稀疏系统		 迭代公式：
\(x_{n+1} = x_n + \alpha_n p_n\)
其中，\(p_n\) 是共轭方向。
二分法 - 简单且稳定
- 收敛速度相对较慢
- 需要函数在区间两端取不同符号		 - 线性方程 & 非线性方程
- 求解实数根
- 连续函数的区间搜索		 迭代公式：
\(c = \frac{a+b}{2}\)
其中，a 和 b 是当前区间的端点。
高斯-塞德尔法 - 收敛速度通常比雅可比法快
- 适用于大型稀疏系统		 - 线性方程组
- 数值线性代数		 迭代公式：
\(x_i^{(k+1)} = \frac{1}{a_{ii}} \left(b_i - \sum_{j=1}^{i-1} a_{ij} x_j^{(k+1)} - \sum_{j=i+1}^{n} a_{ij} x_j^{(k)} \right)\)
雅可比法 - 简单迭代法
- 对角线元素须非零
- 收敛速度可能较慢		 - 线性方程组
- 数值线性代数		 迭代公式：
\(x_i^{(k+1)} = \frac{1}{a_{ii}} \left(b_i - \sum_{\substack{j=1 \\ j \neq i}}^{n} a_{ij} x_j^{(k)} \right)\)
数值积分方法 梯形法 - 简单且稳定
- 适用于初等积分
- 精度一般		 - 数值积分
- 近似求解定积分		 梯形法公式：
\(\int_{a}^{b} f(x) \, dx \approx \frac{h}{2} [f(x_0) + 2f(x_1) + 2f(x_2) + \cdots + 2f(x_{n-1}) + f(x_n)]\)
其中， \(h = \frac{b-a}{n}\) 。
Method Type Method Name Features Application Fields Formula
Numerical Methods for ODEs Runge-Kutta Method - High accuracy
- Good stability
- Relatively large computational cost		 - Solving ordinary differential equations
- Dynamical system simulations
- Engineering and physics problems		 4th order Runge-Kutta formula (common form):
\(y_{n+1} = y_n + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)\)
\(k_1 = hf(x_n, y_n), k_2 = hf(x_n + \frac{h}{2}, y_n + \frac{k_1}{2})\) …
Euler's Method - Simple computation
- Low accuracy
- Sensitive to time step		 - Solving ordinary differential equations
- Simulating simple physical systems		 Explicit Euler method:
\(y_{n+1} = y_n + hf(x_n, y_n)\)
Midpoint Method - Uses the slope at the midpoint
- More accurate than Euler's method
- Better stability		 - Solving ordinary differential equations
- Problems requiring moderate accuracy		 Midpoint method formula:
\(y_{n+1} = y_n + hf\left(x_n + \frac{h}{2}, y_n + \frac{h}{2} f(x_n, y_n)\right)\)
Numerical Methods for PDEs Finite Difference Method - Converts differential equations to algebraic equations
- Suitable for partial differential equations
- Grid division greatly affects the result		 - Solving partial differential equations
- Numerical weather prediction
- Fluid mechanics		 Second-order central difference approximation (for partial derivatives):
\(f''(x) \approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^2}\)
Finite Element Method - Divides the problem domain into small regions
- Suitable for complex geometries and boundary conditions
- High accuracy		 - Solving partial differential equations
- Structural engineering
- Heat conduction problems		 Generalized finite element method formula:
\(\int_{\Omega} (\nabla u \cdot \nabla v - fv) \, d\Omega = 0\)
Optimization Methods Steepest Descent Method - Uses the first derivative of the function
- Simple to implement and widely applicable
- May converge slowly		 - Solving nonlinear equations
- Parameter optimization in machine learning
- Minimizing cost functions		 Iterative formula:
\(x_{n+1} = x_n - \alpha_n \nabla f(x_n)\)
Newton's Method - Uses the first and second derivatives of the function
- Fast convergence
- Requires the Hessian matrix		 - Solving nonlinear equations
- Optimization problems		 Iterative formula:
\(x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}\)
Conjugate Gradient Method - Does not require storing the Hessian matrix
- Suitable for large-scale problems
- Only works for quadratic functions		 - Solving nonlinear equations
- Large-scale optimization problems
- Sparse systems		 Iterative formula:
\(x_{n+1} = x_n + \alpha_n p_n\)
where \(p_n\) is the conjugate direction.
Bisection Method - Simple and stable
- Relatively slow convergence
- Requires the function to have different signs at the interval ends		 - Linear & nonlinear equations
- Solving real roots
- Interval search for continuous functions		 Iterative formula:
\(c = \frac{a+b}{2}\)
where a and b are the current interval endpoints.
Gauss-Seidel Method - Generally faster convergence than the Jacobi method
- Suitable for large sparse systems		 - Solving linear equations
- Numerical linear algebra		 Iterative formula:
\(x_i^{(k+1)} = \frac{1}{a_{ii}} \left(b_i - \sum_{j=1}^{i-1} a_{ij} x_j^{(k+1)} - \sum_{j=i+1}^{n} a_{ij} x_j^{(k)} \right)\)
Jacobi Method - Simple iterative method
- Diagonal elements must be non-zero
- May have slow convergence		 - Solving linear equations
- Numerical linear algebra		 Iterative formula:
\(x_i^{(k+1)} = \frac{1}{a_{ii}} \left(b_i - \sum_{\substack{j=1 \\ j \neq i}}^{n} a_{ij} x_j^{(k)} \right)\)
Numerical Integration Methods Trapezoidal Rule - Simple and stable
- Suitable for elementary integrals
- Moderate accuracy		 - Numerical integration
- Approximating definite integrals		 Trapezoidal rule formula:
\(\int_{a}^{b} f(x) \, dx \approx \frac{h}{2} [f(x_0) + 2f(x_1) + 2f(x_2) + \cdots + 2f(x_{n-1}) + f(x_n)]\)
where \(h = \frac{b-a}{n}\)