basefold commit & opening refactor & cleanup

[hero78119]

This PR take basefold mpcs as pilot to take new mpcs api in preparation for p3 library.

Change scope

• remove plonky2 dependency from mpcs/poseidon
• mpcs: remove basecode random code as we only need keep reed solomon encoding as our major use case
• make everything work in evaluation (lagrange domain) and avoid unnessesary coefficient format conversion
• mpcs: remove basefold commit with extension field as we dont have this scenario.
• mpcs: use plonky3 dft + mmcs for basefold batch-commit

proof size

New design also introduce a minor optimisation to reduce 5% proof size : during opening phase, we open 2 value of merkle tree on leafs[j], leafs[j+ni], but one of the value can be carry over from previous layer interpolation result, thus prover can skip one of them. With that, we can make this check happened naturally with smaller size of proof.

• Fibonacci 2^20 19.07mb -> 17.99mb => 5.66% proof reduction
• Fibonacci 2^21 20.06mb -> 18.94mb => 5.58% proof reduction
• Fibonacci 2^22 21.10mb -> 19.94mb => 5.49% proof reduction

e2e Latency

on AMD EPYC 32 cores
Fibonacci 2^20

                        time:   [3.6361 s 3.6584 s 3.6883 s]
                        change: [-8.5478% -7.9507% -7.1226%] (p = 0.00 < 0.05)
                        Performance has improved.

Fibonacci 2^21

fibonacci_max_steps_2097152/prove_fibonacci/fibonacci_max_steps_2097152
                        time:   [5.9022 s 5.9411 s 5.9762 s]
                        change: [-12.483% -11.835% -11.205%] (p = 0.00 < 0.05)
                        Performance has improved.

Fibonacci 2^22

fibonacci_max_steps_4194304/prove_fibonacci/fibonacci_max_steps_4194304
                        time:   [11.157 s 11.189 s 11.216 s]
                        change: [-12.124% -11.493% -10.773%] (p = 0.00 < 0.05)
                        Performance has improved.

on AMD 5800X 16 cores

Fibonacci 2^20

fibonacci_max_steps_1048576/prove_fibonacci/fibonacci_max_steps_1048576
                        time:   [5.9030 s 6.1262 s 6.3665 s]
                        change: [-18.746% -15.770% -12.351%] (p = 0.00 < 0.05)
                        Performance has improved.

Fibonacci 2^21

fibonacci_max_steps_2097152/prove_fibonacci/fibonacci_max_steps_2097152
                        time:   [10.681 s 10.710 s 10.738 s]
                        change: [-21.564% -21.252% -20.958%] (p = 0.00 < 0.05)
                        Performance has improved.

[kunxian-xia]

I think we should change this to

pub RSCodeParams {
   pub num_queries: usize,
   pub log_blowup: usize,
   pub log_basecode_msg_size: usize,
}

impl RSCodeParams {
   pub fn standard_fast() -> Self {
        standard_fri_params_with_100_bits_conjectured_security(1)
    }

    pub fn standard_with_100_bits_conjectured_security(log_blowup: usize) -> Self {
        standard_fri_params_with_100_bits_conjectured_security(log_blowup)
    }
}

impl Default for RSCodeParams {
   fn default() -> Self {
      Self::standard_fast()
   }
}

The rationale is that we can use different param for different scenarios. For example, the base layer we will log_blowup = 1 for maximal performance, while for the last layer in recursion we use log_blowup = 3 for minimal proof size.

[kunxian-xia]

what's the difference between pcs_commit and pcs_batch_commit?

[kunxian-xia]

great job!