optimize local status

[efiop]

#!/bin/bash                                   
                                              
set -e                                        
set -x                                        
                                              
rm -rf test_corentin                          
mkdir test_corentin                           
cd test_corentin                              
                                              
if [ ! -d "data" ]; then                      
        mkdir data                            
        for i in $(seq 1 400000); do          
                echo $i > data/$i             
        done                                  
fi                                            
                                              
rm -rf myrepo                                 
mkdir myrepo                                  
cd myrepo                                     
                                              
git init                                      
dvc init                                      
                                              
dvc config cache.type hardlink                
dvc config core.analytics false               

echo '*.py' > .dvcignore

cp ../data -a data                            
dvc add data                                  
dvc status                                    
python -mcProfile -o status.prof -m dvc status

Last dvc status results in ~50sec(way over 1minute if there is no unpacked dir) on current master and ~8sec with this PR.

The main issue that was causing this is relpath that we were calling for each file that we check for match in dvcignore. Plus some alog-the-way optimizations like os.path.join and saving stats on is_protected. Also removing unpacked dir optimization, as it is no longer needed, mainly because of the recent optimizations that are done.

The issue was reported by the user https://discordapp.com/channels/485586884165107732/563406153334128681/713338211128311808 and my reproduction script is based on the dataset structure that he has.

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here. If the CLI API is changed, I have updated tab completion scripts.

• ❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[efiop]

Will look if I can add a benchmark like that to dvc-bench before this is merged.

[efiop]

Also, we've discussed this before, but all of these methods don't take dvcignore into account, which is wrong.

[courentin]

Thank you @efiop !
I can confirm that the dvc status is faster now !

Here is the call graph:

Also, I noticed that the more I have entries in my .dvcignore, the more it takes time to complete (tiny diff though less than 2sec)

[efiop]

@courentin Thanks for the feedback! Yeah, we are definitely not fully done optimizing it, but this seems like a good start 😉

[efiop]

it is too expensive for match to try to resolve paths like this.

[pared]

LGTM