Implement TensorKit's tsvd! reverse-rule for :fixed differentiation mode

[pbrehmer]

Here we adapt tsvd! and FixedSVD such that the :fixed differentiation mode can also use TensorKit's SVD adjoint. For context, the TensorKit.tsvd! reverse-rule requires the full, untruncated SVD which is currently not passed along the CTMRG algorithm. To change this, tsvd! needs to return the untruncated decomposition as well. Hopefully, TensorKit's reverse-rule will eliminate some of the instabilities of KrylovKit's SVD reverse-rule.

[codecov]

Codecov Report

Attention: Patch coverage is 95.83333% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/utility/svd.jl	93.84%	4 Missing ⚠️

Files with missing lines	Coverage Δ
src/Defaults.jl	85.71% <ø> (ø)
src/algorithms/ctmrg/projectors.jl	86.53% <100.00%> (-0.74%)	⬇️
src/algorithms/ctmrg/simultaneous.jl	98.27% <100.00%> (+0.09%)	⬆️
...rithms/optimization/fixed_point_differentiation.jl	93.07% <100.00%> (+0.83%)	⬆️
src/algorithms/optimization/peps_optimization.jl	95.65% <ø> (+5.11%)	⬆️
src/utility/svd.jl	91.17% <93.84%> (+3.26%)	⬆️

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pbrehmer]

This is still in a rough shape but I think once I figure out how to gauge-fix the full U, S and V, I hope this works! (The gauge transformation only really acts on the truncated singular vectors, so perhaps an identity on the rest is fine?)

[pbrehmer]

I'll hopefully finish this up tomorrow!

[Confusio]

This is still in a rough shape but I think once I figure out how to gauge-fix the full U, S and V, I hope this works! (The gauge transformation only really acts on the truncated singular vectors, so perhaps an identity on the rest is fine?)

I have a naive question regarding gauge fixing in the :fixed mode. If we have already fixed U, S, and V, why is additional gauge fixing necessary? My understanding is that gauge fixing merely sets the relative phases of the singular vectors. Once U, S, and V are determined, the corresponding projectors should be fixed, and the environment tensors should converge element-wise. Did I misunderstand what extra role gauge fixing plays in this process?

[pbrehmer]

If we have already fixed U, S, and V, why is additional gauge fixing necessary? My understanding is that gauge fixing merely sets the relative phases of the singular vectors. Once U, S, and V are determined, the corresponding projectors should be fixed, and the environment tensors should converge element-wise.

It is true that the gauge fixing sets the relative (and global) phases of all environment tensors. The point is that two consecutive CTMRG iterations have to be comparable in the sense that the element-wise differences of all environment tensors should be zero at the fixed point. That's also how the gauge fixing works here: based on a given environment, we perform a CTMRG iteration and fix the Us, Ss and Vs of that iteration relative to the given environment. In order to achieve element-wise convergence, we also need to supply the correct, gauge-fixed environment that fits the gauge-fixed SVD.

In case you want to have a look at the code implementing this, see the _rrule for the :fixed mode.

For further details on the implemented gauge fixing algorithm itself, you can take a look at Appendix A here.

[pbrehmer]

I think this should work now but there are some things where I could use some input @lkdvos @leburgel:

• In order to hook into the tsvd rrule with FixedSVD the whole SVD wrapper and its rrules got a bit messier. Since that will be refactored at some point anyway it might be fine for now but I'm still happy about suggestions to clean this up a little.
• The TensorKit svd_pullback! function (in the ChainRulesCore extension) has a tolerance for gauge dependency warnings and I'm not yet sure what a sensible way would be to control this from the PEPSKit side.

[leburgel]

I'll try to have a proper look later today. As for the gauge dependency warnings tolerance, maybe it's good to let it depend on the magnitude of the smallest singular value that is retained (e.g. divide this by a factor 100 or something to set the tol)?

[lkdvos]

Overall looks quite good to me. Left a couple small remarks

[Confusio]

If we have already fixed U, S, and V, why is additional gauge fixing necessary? My understanding is that gauge fixing merely sets the relative phases of the singular vectors. Once U, S, and V are determined, the corresponding projectors should be fixed, and the environment tensors should converge element-wise.

It is true that the gauge fixing sets the relative (and global) phases of all environment tensors. The point is that two consecutive CTMRG iterations have to be comparable in the sense that the element-wise differences of all environment tensors should be zero at the fixed point. That's also how the gauge fixing works here: based on a given environment, we perform a CTMRG iteration and fix the Us, Ss and Vs of that iteration relative to the given environment. In order to achieve element-wise convergence, we also need to supply the correct, gauge-fixed environment that fits the gauge-fixed SVD.

In case you want to have a look at the code implementing this, see the _rrule for the :fixed mode.

For further details on the implemented gauge fixing algorithm itself, you can take a look at Appendix A here.

I understand why we need gauge-fixed U, S, and V when the environment are given, but my question is even more naive. After one CTMRG step, we have U, S, and V—and consequently the projectors. We keep them (no gauge fixing), and then perform CTMRG iterations. Does this process produce element-wise convergence environments? The current strategy is to find out the gauge for given environment. How about under certain gauge (once U, S, V is given, the gauge is implicitly fixed), find out the environment?

In other words, in current implementation for gauge fixing, the environment is given first, and we then find the corresponding compatible U, S, and V and then fix it. My question is: since we already have U, S, and V fixed, how exactly do we determine the compatible environment? Is one CTMRG step sufficient for this adjustment, or are multiple iterations required?

In some sense, this is related to the symmetry in PEPS. When starting from a generic PEPS, it can be challenging to determine whether the state is symmetric because one must first identify the correct gauge to reveal its symmetry—even if the state is known to be symmetric in principle. In contrast, if one begins with a symmetric PEPS and maintains its symmetry during the gauge-fixing process, the gauge is essentially fixed. Furthermore, if one does not require the sparse structure, any gauge choice is acceptable. In other words, keep-track of gauge is easier.

[pbrehmer]

I added an adaptive pullback gauge sensitivity tolerance similar to #155 but perhaps by adapting it directly to the minimal singular value the tolerance is too high now - at least it gets rid of the warnings in the (hopefully) unproblematic cases. Let me know whether we should keep it or not and then we can merge!

[pbrehmer]

After one CTMRG step, we have U, S, and V—and consequently the projectors. We keep them (no gauge fixing), and then perform CTMRG iterations. Does this process produce element-wise convergence environments?

This will not work, since $U$, $S$, and $V$ were obtained by applying the CTMRG iteration $f$ to the previous environment $x_{n-1}$ to obtain the current environment $f(A, x_{n-1}) = x_n$ (where $A$ is the PEPS). Even if $x_{n-1}$ and $x_n$ are at a fixed point in terms of singular values of the corners and edges, in general the elements will not have converged. Applying that same $f_{U,S,V}$ with the fixed SVD to $x_n$ again will rotate the gauge degrees of freedom such that $f_{U,S,V}(A, x_n) = \Lambda x_n$ where $\Lambda$ is some gauge transformation. I think I've also tried this in practice and it will not lead to element-wise convergence.

My question is: since we already have U, S, and V fixed, how exactly do we determine the compatible environment? Is one CTMRG step sufficient for this adjustment, or are multiple iterations required?

The compatible environment is determined by first gauge fixing the environment obtained by an extra CTMRG step (see the gauge_fix function) and, with the same gauge signs, fixing the SVD. The extra step is required to obtain the gauge signs.

Conceptually, two steps are required for proper gauge-fixing since two consecutive environments have to live in the same gauge. It is important to remember that the entire fixed-point differentiation is based on the equation $f(A, x_n) = x_n$. Practically, there are also ways where the gauge is fixed based on one step only, e.g. by fixing the sign of SVDs. However, this can break down (I believe) in cases with degeneracy and is not guaranteed to work.

I hope this makes it somewhat clearer! I don't find it very easy to explain the details, so in case you need further context, feel free to check out the reference and the code itself. But of course I'm happy to answer any questions.

[leburgel]

Looks good to me.

[lkdvos]

Should I first tag the other version by the way?

[leburgel]

Should I first tag the other version by the way?

You mean 0.4.2? Sure, can't hurt.

[pbrehmer]

Sure, then let's tag v0.4.2 first and then I'll merge