Introduction

Optimization is one of the most important aspects of deep learning. It is non-convex, highly stochastic, and strongly anisotropic, which creates a gap between clean optimization theory and the practice of training large neural networks. Standard first-order methods such as stochastic gradient descent are often insufficient for modern large-scale training [Amari 1993].

The default optimizer in many deep learning pipelines is Adam or AdamW [Kingma and Ba 2014]. Adam is powerful and robust, but it is mostly structure-agnostic: it treats parameters coordinate-wise and ignores the fact that most weights in modern networks live inside linear layers. Fisher-based optimizers try to use this matrix structure to approximate natural-gradient updates without forming the full Fisher matrix.

We consider a discriminative task of the form

where $w \in \mathbb{R}^D$ are model parameters and $f_w(x)$ are logits. For a negative log-likelihood loss,

the optimization problem can be viewed as maximum likelihood estimation.

Natural Gradient

The Fisher Information Matrix [Amari and Nagaoka 2000] for the joint distribution $p_w(y,x)=p_w(y\mid x)p(x)$ is

Since the data distribution $p(x)$ does not depend on $w$, this becomes:

The Fisher matrix is positive semidefinite and measures local sensitivity of the predictive distribution to parameter perturbations. It also gives the second-order approximation of KL divergence:

for sufficiently small $\Delta w$.

If we treat the model as a statistical manifold with suitable regularity conditions, $F(w)$ defines a Riemannian metric, called the Fisher metric [Amari and Nagaoka 2000]. The steepest descent direction under a Fisher-metric constraint is the natural gradient [Amari 1998]:

In practice, neural networks are overparameterized, so the Fisher matrix is often singular or nearly singular. Still, the natural-gradient idea is useful: updates are rescaled according to their effect on the predictive distribution. Directions that strongly change the output distribution are damped more, while insensitive directions are damped less.

It is also useful to separate the roles of loss gradient and Fisher matrix. The label information enters through $\nabla \mathcal{L}(w)$, while the Fisher matrix captures how parameters affect the model distribution.

Approximation Setup

Computing and inverting a $D\times D$ Fisher matrix is infeasible for modern models. The expectation over all model outputs can also be expensive. A common replacement is the empirical Fisher:

The empirical Fisher replaces the model-output expectation with the observed training labels. This makes it much easier to estimate because $\nabla_w \log p_w(y\mid x)$ is already available from ordinary backpropagation.

Typical scalable approximations are:

• Replace the true Fisher with the empirical Fisher.
• Ignore cross-layer interactions and keep block-diagonal Fisher blocks for linear layers.
• Further approximate each linear-layer Fisher block so that inversion or inverse-root multiplication is tractable.

Kronecker Product

For matrices $A\in\mathbb{R}^{n_1\times m_1}$ and $B\in\mathbb{R}^{n_2\times m_2}$, the Kronecker product is

The main identities used below are:

and

If $A,B\succ 0$, then $(A\otimes B)^\alpha=A^\alpha\otimes B^\alpha$ for any real $\alpha$.

Empirical Fisher for Linear Layers

Consider a weight matrix $W\in\mathbb{R}^{D_{\mathrm{out}}\times D_{\mathrm{in}}}$ with output $y=Wx$. Let $\delta:=\nabla_y l$ be the sample-wise gradient with respect to the layer output. For one sample,

The full gradient is

The empirical Fisher block for $W$ is:

This form makes matrix-vector products feasible, but direct inversion is still too expensive.

For convolutional layers, the same idea applies after patch extraction. If

then local patches can be unfolded into

The convolution kernel becomes a matrix

so the same Fisher approximations can be applied.

KFAC

Inverting the empirical Fisher from Eq. 3 directly is infeasible. KFAC addresses this with a Kronecker factorization [Martens and Grosse 2015]:

KFAC assumes approximate independence between inputs $x$ and output gradients $\delta$:

The inverse action is tractable:

Thus the preconditioned matrix gradient has the form

Intuitively, $F_y$ decorrelates rows while $F_x$ decorrelates columns. KFAC was one of the first scalable natural-gradient approximations for neural networks.

EKFAC

Eigenvalue-corrected KFAC, or EKFAC, improves KFAC by estimating eigenvalues more accurately [George et al. 2018]. Let

KFAC implies

EKFAC keeps the Kronecker eigenbasis but corrects the eigenvalues:

where $\Sigma^*$ is diagonal. Since

the corrected diagonal can be computed as

This avoids forming the full Fisher matrix. EKFAC is also guaranteed to be no worse than KFAC in Frobenius-norm approximation error [George et al. 2018]:

Shampoo

Shampoo was introduced from online convex optimization rather than natural gradient [Gupta et al. 2018], but it has a useful heuristic connection to Fisher preconditioning.

Starting from Eq. 3, consider a rough square of the Fisher:

After expanding and swapping the scalar terms $x^T\hat{x}$ and $\delta^T\hat{\delta}$ between Kronecker factors, one obtains the heuristic approximation

This suggests:

So Shampoo’s left and right preconditioners, $GG^T$ and $G^TG$, can be interpreted as a structured Fisher-style approximation.

The original Shampoo paper proves an $O(\sqrt{T})$ regret bound in online convex optimization, which corresponds to an $O(1/\sqrt{T})$ stochastic optimization rate [Gupta et al. 2018]. In practice, EMA averaging with coefficient $\beta_2$ is often preferred.

SOAP

Shampoo has a similar eigenvalue issue as KFAC: the eigenbasis may be useful, but the implied eigenvalues can be poor. SOAP applies an EKFAC-like eigenvalue correction to Shampoo [Vyas et al. 2024].

The ideal corrected eigenvalues would be

but this requires per-sample gradients. SOAP instead uses the full gradient:

and accumulates this quantity as a second moment via EMA.

SOAP then performs an Adam-like update in Shampoo’s eigenbasis:

• Accumulate momentum $M$.
• Rotate both momentum $M$ and gradient $G$ to the Shampoo eigenbasis.
• Accumulate the second moment $\Sigma^*$ in that eigenbasis.
• Apply Adam-style normalization in the eigenbasis.
• Rotate the update back to the original parameter space.

SOAP also avoids expensive repeated eigendecomposition. It uses randomized SVD-style iterations

which converge to the eigenvectors of $A$ under standard assumptions. If eigenvectors change slowly, one QR iteration every $k$ steps is often enough.

Muon

Bernstein observed that if Shampoo does not accumulate preconditioning matrices, equivalently $\beta_2=0$, then the update becomes [Bernstein 2024]:

If $G=U\Sigma V^T$ is an SVD, the update reduces to

which is the matrix sign or polar factor [Higham 2008]. This object can be interpreted as:

• Orthogonalization:

• Steepest descent under a spectral-norm constraint:

Muon, short for Momentum Orthogonalized with Newton-Schulz, builds on this idea [Jordan et al. 2024]. Subsequent work studied how to scale Muon to large language model training [Liu et al. 2025].

Here $\mathrm{NS5}$ denotes five Newton-Schulz iterations for approximate orthogonalization. This line of work is still developing, with variants such as Polar Express [Amsel et al. 2025] and Gram Newton-Schulz [Zhang et al. 2026]. Muon is currently a practical optimizer for LLM linear layers and is increasingly used as an AdamW alternative.

Experiments

Fisher Matrix Visualization

The first experiment visualizes Fisher matrices for a small MNIST CNN with roughly 3800 parameters. Fisher approximations were computed after the 10th epoch, when the model was already well fitted.

Figure 2 shows six matrices using logarithms of absolute values. The first row contains the true Fisher (Eq. 1), KFAC (Eq. 4), and Shampoo (Eq. 6). The second row contains the empirical Fisher (Eq. 2), EKFAC (Eq. 5), and SOAP.

The empirical Fisher is visually close to the true Fisher. This is plausible because the model is already well fitted and the model distribution is close to the data distribution. The small number of classes also helps. For LLM-like tasks with large vocabularies, the true Fisher can differ much more from the empirical Fisher.

KFAC and Shampoo show visible block structure. Shampoo has lower intensity, which may indicate poor eigenvalue approximation. SOAP and EKFAC improve these eigenvalues and smooth the block structure.

LLM Experiment

The second experiment pretrains a small GPT-like model with roughly 10M parameters on Shakespeare-char. AdamW from Andrej Karpathy’s baseline was compared with Fisher-based optimizers. All optimizers that use SVD, meaning all except Muon and AdamW, use preconditioning every 10 steps.

KFAC was run with power $-1/2$ instead of $-1$ because the latter was unstable and performed poorly. A suffix “G” indicates grafting, where the update is rescaled to have the Frobenius norm of the Adam update. This stabilizes the step size.

Shampoo was tested with powers $-1/4$, as in the original version, and $-1/2$, which corresponds more closely to inverting the Fisher instead of using its inverse square root. In this experiment, $-1/2$ performed significantly better. Muon performed similarly to Shampoo with power $-1/4$, but was substantially faster.

SOAP performed best among the considered optimizers. KFACSOAP is a SOAP modification where eigenvectors are computed using the KFAC approximation from Eq. 4 rather than the Shampoo approximation. It outperformed original SOAP in this experiment, suggesting that the KFAC approximation can be more accurate than Shampoo’s approximation in this setting.

Overall, SOAP-style optimizers performed best, supporting the idea that running Adam in an approximate Fisher eigenbasis is a strong practical recipe.

Conclusion

Fisher-based optimizers can be viewed as attempts to approximate natural-gradient preconditioning in ways that fit modern neural networks. KFAC and EKFAC use activation and output-gradient statistics. Shampoo and SOAP use gradient-matrix structure. Muon can be understood as an efficient orthogonalized update arising from a limiting Shampoo-like view.

The experiments here suggest that SOAP-based methods are the most sample-efficient among the tested optimizers. A promising direction is improving eigenvector approximations. In the Shakespeare-char experiment, a KFAC-style eigenbasis worked better than Shampoo’s eigenbasis, which suggests that the choice of structured Fisher approximation still matters.

References

Amari, S. (1993). Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5), 185-196.

Kingma, D. P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv:1412.6980.

Amari, S., and Nagaoka, H. (2000). Methods of Information Geometry. American Mathematical Society.

Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251-276.

Martens, J., and Grosse, R. (2015). Optimizing Neural Networks with Kronecker-factored Approximate Curvature. ICML.

George, T., Laurent, C., Bouthillier, X., Ballas, N., and Vincent, P. (2018). Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis. NeurIPS.

Gupta, V., Koren, T., and Singer, Y. (2018). Shampoo: Preconditioned Stochastic Tensor Optimization. ICML.

Vyas, N., Morwani, D., Zhao, R., Kwun, M., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. (2024). SOAP: Improving and Stabilizing Shampoo using Adam. arXiv:2409.11321.

Bernstein, J., and Newhouse, L. (2024). Old Optimizer, New Norm: An Anthology. arXiv:2409.20325.

Higham, N. J. (2008). Functions of Matrices: Theory and Computation. SIAM.

Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks. Blog post.

Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., Qin, Y., Xu, W., Lu, E., Yan, J., Chen, Y., Zheng, H., Liu, Y., Liu, S., Yin, B., He, W., Zhu, H., Wang, Y., Wang, J., Dong, M., Zhang, Z., Kang, Y., Zhang, H., Xu, X., Zhang, Y., Wu, Y., Zhou, X., and Yang, Z. (2025). Muon is Scalable for LLM Training. arXiv:2502.16982.

Amsel, N., Persson, D., Musco, C., and Gower, R. M. (2025). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. arXiv:2505.16932.

Zhang, J., Amsel, N., Chen, B., and Dao, T. (2026). Gram Newton-Schulz: A Fast, Hardware-Aware Newton-Schulz Algorithm for Muon. Blog post and implementation note.

Statistical Interpretation

We start by introducing Fisher Information Matrix and describing its statistical properties which help understand the intuition behind the word “information” in the name.

First, let us define the set of distributions that we will be working with. We understand distribution $p_{\theta}: \mathbb{R}^{d} \to \mathbb{R}$ as a map such that $\int p_{\theta}(x)\,dx = 1$ and $p_{\theta}(x) \ge 0$ for all $x$. Each $\theta \in \Theta \subset \mathbb{R}^{n}$ defines its own distribution and is called parameter. The open set $\Theta$ is called a parameter space. The set of distributions that are formed by all possible parameters is called a statistical model with dimension $n$:

Strictly speaking, the mapping $\theta \leftrightarrow p_\theta$ should be bijective and $\Theta$ should be an open set in $\mathbb{R}^n$ for the correct definition of the statistical model. However, due to the common case of overparameterization in neural networks, we omit this requirement in the section on singularity, where we analyze the connection between overparameterization and Fisher matrix.

We will make the following assumptions about the statistical model and the distributions:

Here we denote

and call it a score function.

The first assumption is classical for statistical models. We typically require sufficiently smooth dependence on $\theta$ for derivatives to exist.

The second assumption implies that the support of the distribution does not depend on $\theta$. It is an important requirement for interchanging $\frac{\partial}{\partial \theta}$ and $\int$ and filters out distributions such as Uniform $U[0, \theta]$.

The third assumption requires existence of the (element-wise) integrals and ability to interchange $\frac{\partial}{\partial \theta}$ and $\int$. More strictly, for a function $f_\theta(x) \in { \frac{\partial }{\partial \theta }p_\theta(x), \ldots }$ we need:

The fourth assumption is required for the existence of the Fisher Information Matrix, which will be introduced later.

The score function $s_\theta(x)$ plays an important role in machine learning. Using chain rule, we can express density derivative in terms of the score:

The main property of the score function is having zero mean:

Let us define the Fisher Information Matrix, the main object of the current document. We will also call it Fisher matrix or just Fisher.

Fisher matrix is also the covariance matrix of the score function due to zero mean of the latter. By definition, it is positive semidefinite.

Proof

Using the score property:

Differentiating by $\theta$ on both sides (and using regularity) gives:

Rewriting the score by the definition on the left side and substituting the identity $\frac{\partial}{\partial \theta}p_\theta(x) = s_\theta(x)p_\theta(x)$ on the right side proves the statement.

Next, we would like to discuss the connection of the Fisher matrix with sufficient statistics. By statistic we mean any smooth function $T: \mathbb{X} \to \mathbb{R}^m$. We assume sufficient smoothness $T(x) \in C^{\infty}(\mathbb{X})$. If we denote $X$ as a random variable with distribution $p_\theta(x)$, then we denote $T(X)$ as a random variable with distribution:

We call $T(X)$ a sufficient statistic if $p_\theta (x) / q_\theta (T(x))$ does not depend on $\theta$, or in other words $p_\theta(x) = c(x)q_\theta(T(x))$. In fact, it is sufficient to find any positive function $g_\theta$, such that $p_\theta(x) = c(x)g_\theta(T(x))$. This equivalence is called factorization theorem. It can be easily proved with the following observation:

So we can rewrite factorization with any function $g_\theta$ as follows:

Proof

For any value $t$ of the statistic:

Differentiate with respect to $\theta$:

Now, dividing both sides by $q_\theta(t)$ and using the definition of conditional expectation, we get:

For any fixed $t$:

Now we take expectation with respect to $T$:

After integrating with positive factor $q_\theta(t)$, the covariance matrix still remains positive semidefinite. We also substitute $\widetilde{s}_\theta(t)$ for the second term:

Hence

Now note that equality to zero means that the expectation of a positive semidefinite matrix is zero, thus equivalently for all $t$:

Equivalently, after fixing $T = t$, $s_\theta(x)$ becomes a constant. We can express this as:

For fixed $t$ and $x$ with $T(x) = t$, we can differentiate $a_\theta$ to obtain:

So $a_\theta$ can be expressed as a gradient of some scalar function. Reparameterize:

Then

This results in $p_\theta(x) = c(x) g_\theta(T(x))$ and completes the proof by the factorization theorem.

This theorem shows us an intuitive interpretation of Fisher matrix: when performing any transformation $T(x)$, we lose “information” about the parameter $\theta$. However, when $T(x)$ is a sufficient statistic, it keeps all “information” about $\theta$.

We call a statistic $T: \mathbb{X} \to \mathbb{R}^n$ locally unbiased at point $\theta_0$ if for any $\theta$ from some neighborhood of $\theta_0$ we have:

Now comes the famous Cramér–Rao inequality which provides a lower bound on the estimation error for an unbiased estimator.

Proof

It is easy to show that for a locally unbiased estimator which satisfies $\mathbb{E}_{\widetilde{\theta}}\left[T\right] = \widetilde{\theta}$ for some neighborhood of $\theta$:

Using the score function property $\mathbb{E}\theta s\theta = 0$, we can subtract $0 = \theta \mathbb{E}\theta s\theta$:

If we consider the stacked random vector $\begin{bmatrix} T - \theta \ s_\theta \end{bmatrix}$ and take its covariance matrix, we get:

Schur complement with respect to $F(\theta)$ gives:

which proves the theorem.

Cramér–Rao inequality allows us to interpret Fisher matrix as a lower bound on the uncertainty of estimation. If $F(\theta)$ has eigenvalues close to $0$, its inverse has large eigenvalues and therefore estimation is very poor. On the contrary, large eigenvalues of $F(\theta)$ provide a lot of “information” about the parameter for the estimator. The estimators that achieve equality in Cramér–Rao are called efficient.

Next property allows us to understand how Fisher behaves when distribution consists of independent variables and is factorized by them.

Proof

When considering full parameter $\theta$, we have:

Now, Fisher matrix has form:

where off-block-diagonal entries vanish due to expectation factorization and the zero-mean property of the score function.

So, independence of the parameters is nicely reflected in the Fisher matrix. The inverse implication, however, is generally false: block-diagonal Fisher matrix does not imply independence. Next, when all independent distributions depend on the same parameter, we also get a specific form of the joint Fisher matrix.

Proof

Score function of the joint distribution takes form:

where

Then:

where the second term with $i \ne j$ vanishes due to the zero-mean property of the score function.

Intuitively, this means that when independent variables are combined, their “information” is added.

When considering estimation with $N$ independent samples from distribution $p_{\theta}$ with Fisher matrix $F(\theta)$, we get the simplification:

Now, due to Cramér–Rao inequality, a locally unbiased estimator $T(x_1, \ldots, x_N)$ for the joint distribution $p(x_1)\cdots p(x_N)$ satisfies:

This provides a bound on the convergence of a locally unbiased estimator when $N \to \infty$.

Fisher Metric

Let us introduce the inner product between parameter changes $d\theta_1, d\theta_2 \in \mathbb{R}^n$ induced by the Fisher matrix:

Let us call $\langle \cdot , \cdot \rangle_F$ the Fisher metric. Obviously, it is not a metric in the usual elementary sense. However, it has a connection to differential geometry.

One of the main properties of the Fisher metric is its invariance with respect to reparameterization. For theoretical rigor, we only consider diffeomorphisms. By diffeomorphism we mean a $C^1$ map between $\mathbb{R}^n$ and $\mathbb{R}^n$ which is bijective and whose inverse is also $C^1$.

Proof

Let

be the Jacobian of the reparameterization. By the chain rule:

Fisher matrix in the new coordinates $\eta$ can be written as:

Now let $d\eta_1, d\eta_2$ be infinitesimal parameter changes in $\eta$ space. Then:

Hence:

So, Fisher metric does not depend on the particular choice of parameterization, making it an effective inner product between infinitesimal distribution changes. In fact, when $F \succ 0$, Fisher metric is a Riemannian metric on the statistical manifold.

Fisher metric has connections to the KL divergence

Consider a point $\theta \in \mathbb{R}^n$ and a small parameter change $\Delta \theta \in \mathbb{R}^n$. KL divergence between $\theta$ and $\theta + \Delta \theta$ is:

We would like to write the Taylor expansion up to second order. The first term in KL divergence is fixed, and we differentiate with respect to the second term. For that, we compute:

The first derivative vanishes at point $\theta$ due to the zero-mean property of the score function. The second derivative becomes $F(\theta)$ by the identity proved earlier.

Thus:

where the norm $|\cdot |_F := \sqrt{\langle \cdot, \cdot \rangle_F}$ is induced by the Fisher metric.

So, Fisher metric provides a second-order approximation of the KL divergence. This makes it suitable for trust-region methods with KL constraint.

Next, let us derive an object closely related to the Fisher metric. We can view $\langle \cdot, \cdot \rangle_F$ as an ordinary Euclidean scalar product for transformed vectors: $v \mapsto F^{1/2}v$. The volume change induced by this transformation can be expressed with determinant $\det(F^{1/2}) = \det(F)^{1/2}$.

Jeffreys prior is a prior distribution on parameters $\theta$ that assigns probability mass proportionally to “information volume”:

Here, for correct definition, we consider the case where $F(\theta) \succ 0$ for all $\theta$.

Proof

From the proof of Fisher metric invariance:

Taking determinants, we obtain

hence

For reparameterization of the distribution, we know that:

Now it is easy to see that:

Thus, Jeffreys prior is invariant under smooth reparameterization.

Singularity analysis

Usually, statistics books consider the statistical model to always have positive definite Fisher matrix $F \succ 0$, however we would like to provide interpretation of its singularity.

First, let us slightly weaken the definition of the statistical model: $\Theta$ is a set in $\mathbb{R}^n$ (not necessarily open), and the map $\theta \mapsto p_\theta$ at each point $x$ does not have to be bijective, though it is still $C^1(\Theta)$.

Let us call the statistical model locally overparameterized at point $\theta \in \Theta$ if there exists a direction $v \in \mathbb{R}^n$ that does not change a distribution in first order:

We can equivalently express this in terms of the score function using directional derivative:

Proof

This simple criterion allows us to intuitively understand that zero eigenvalues of the Fisher matrix correspond to directions that do not change the distribution locally. Hence, they do not contain any “information” about the distribution locally.

When strictly defining a statistical model, local overparameterization does not exist.

Proof

Fix any $\theta \in \Theta$. Suppose the model is locally overparameterized at $\theta$. Then there exists a nonzero direction $v \in \mathbb{R}^n$, $v \ne 0$, such that

Denote by $\Phi(\theta) = p_\theta$ the map from parameters to distributions. Since $\theta \leftrightarrow p_\theta$ is a diffeomorphism, it can never have any directional derivative equal to zero. Therefore,

which is a contradiction.

Proof

Suppose $T(x)$ is a locally unbiased estimator at $\theta_0$ and also the model is locally overparameterized at $\theta_0$: there exists a direction $v \ne 0$ such that for all $x$,

As the estimator is unbiased in some neighborhood of $\theta_0$, for sufficiently small $t$ we have:

This implies:

Dividing by $t$ and taking the limit $t \to 0$:

This contradiction proves the statement.

In neural networks, typically there is redundancy of parameters in a sense that model dimension can be reduced. The final statement of this section naturally explains the behavior of the Fisher matrix for such case.

Proof

From the chain rule, similarly to the reparameterization formula, we have:

Because $\eta(\theta) \leftrightarrow p_{\eta(\theta)}$ is a diffeomorphism, the $m \times m$ Fisher matrix $F_{\eta}(\eta(\theta))$ is positive definite, and therefore has full rank $m$.

Now the following sequence of equivalences shows that $\ker F(\theta) = \ker J(\theta)$ and therefore their ranks are equal:

Obviously, $\operatorname{rank}(J(\theta)) \le m$ because the matrix is $m \times n$. So the statement follows.

This provides a good way to understand Fisher matrix rank. If the statistical model induced by parameters has dimension $m$, then in any higher-dimensional parameterization the rank will be at most $m$.

Let us consider an example which shows how a “bad” parameterization can lead to Fisher matrix with rank less than $m$. Take a point $\theta = (\theta_1, \theta_2) \in \mathbb{R}^2$ that defines the normal distribution $\mathcal{N}(\theta_1^3, 1)$:

This distribution can be naturally parameterized with $\eta(\theta) = \theta_1^3$, and it will have constant Fisher matrix:

Now, let us compute the score and original Fisher matrix:

Now, we can see that at point $\theta_1 = 0$ Fisher matrix becomes zero with rank $0$, and the Jacobian

also has rank $0$ at that point.

It is easy to show that if we perform a one-to-one (non-diffeomorphic) mapping $\theta_1 \leftrightarrow \theta_1^3$, then the new model will always have rank-$1$ Jacobian and therefore Fisher will also always have rank $1$.

This example shows that for some strangely parameterized models the Fisher matrix can have rank even less than the dimension of the statistical model. But usually they are the same, and rank of Fisher matrix equals the dimension of the statistical model.

References

• S. Amari and H. Nagaoka, Methods of Information Geometry, 2000.
• H. Jeffreys, “An invariant form for the prior probability in estimation problems,” Proceedings of the Royal Society A, 1946.

Currently supported:

• Sparsity Patterns: Unstructured and Semi-structured N:M
• Model Families: OPT (facebook/opt)

Repository nn-pruning

To validate the toolkit, I reproduced the benchmarks for the OPT model family across three different sparsities: Unstructured (50%), Semi-structured 2:4, and 4:8.

WikiText-2 Perplexity Results (Calibration: 128 C4 sequences, 2048 tokens each. Sparsity applies to Attention and MLP linear weights.)

Download: frog-technical-overview.pdf

Technical Overview

Download: unstructured-pruning-methods.pdf