Profiling build defaults

[parsonsmatt]

@bitemyapp's Makefiles are a treasure trove of build tips. Here's a trick for profiling builds:

stack build --profile --work-dir .stack-work-profiling

This tremendously improves build caching, at the expense of disk space.

Can this be a default behavior, or at least, enabled via a global config flag or similar? It is soul crushing to have to rebuild all of the deps for a large project for profiling, and rebuild them again for normal mode.

[roman]

I do this all the time, Do you know if it is also required to have a --stack-root as well to make sure the snapshot dependencies are also profiled?

[snoyberg]

Only downside I can see is screwing up gitignore files, but that seems a small price to pay. What about doing it as a subdir of .stack-work? I don't even know if that will work.

[bitemyapp]

@snoyberg could be a good idea in part because of we always suffix the profiling version of a build then anyone making custom use of workdirs like I was would benefit from this behavior across the board.

Not sure if this is the ideal approach or not, though.

[gavwhela]

Working on this from the hackathon: haskell-hackathon/#5.

[snoyberg]

Hey @gavwhela, just bumping this. Any questions on the codebase?

[buecking]

I just experienced this pain and agree --profile should, at the very least, be a sticky flag. I agree there needs to be a way to customize stack at some level. Has there been any thought into adding custom synonyms i.e. profile=build --profile --work-dir .stack-work-profiling.

[ntc2]

@snoyberg

Only downside I can see is screwing up gitignore files, but that seems a small price to pay. What about doing it as a subdir of .stack-work? I don't even know if that will work

I think using a subdir of .stack-work for the new profiling builds default is a great idea. It's not just .gitignore files that are affected by adding another top-level Stack related scratch dir, because various tools have hardcoded special treatment of standard Haskell build dirs (.stack-work, dist, dist-newstyle). For example, haskell-mode and projectile in Emacs ignore these build dirs by default when generating TAGS. Making the profiling cache a subdir of .stack-work would allow all of the existing special treatment to keep working unchanged.

However, using a subdir of .stack-work fails for me. For example, on one of my local projects:

stack --work-dir .stack-work/.stack-work-profile build --profile                    
flexdis86-0.1.2: configure (lib + exe)
flexdis86-0.1.2: build (lib + exe)                                                                     
macaw-base-0.3: configure (lib)                            
macaw-base-0.3: build (lib)                                
what4-0.4.0: configure (lib)                               
what4-0.4.0: build (lib)                                   
flexdis86-0.1.2: copy/register                             
macaw-base-0.3: copy/register                              
what4-0.4.0: copy/register                
Progress 8/27             
No directory could be located matching the supplied path: /tmp/stack13129/regex-1.0.0.0/.stack-work
No directory could be located matching the supplied path: /tmp/stack13129/s-cargot-0.1.4.0/.stack-work
No directory could be located matching the supplied path: /tmp/stack13129/located-base-0.1.1.1/.stack-work
No directory could be located matching the supplied path: /tmp/stack13129/monadLib-3.7.3/.stack-work
No directory could be located matching the supplied path: /tmp/stack13129/limp-0.3.2.2/.stack-work

A little experimentation makes me suspect the problem is that Stack doesn't do the equivalent mkdir --parents when creating the work dir. For example,

stack --work-dir no-such-parent-dir/.stack-work build

fails with

No directory could be located matching the supplied path: /home/conathan/brittle/sfe.git/no-such-parent-dir

Should be trivial to fix that.

[flip111]

I'm a bit worried that choosing a new working directory will duplicate more than the absolute necessary. It's already the case that after stack clean or just delete the .stack-work directory and then rebuilding leaves a smaller .stack-work directory. So some dirty artifacts remain which are not needed for the rebuild. I hope this doesn't increase the amount of unneeded files. Also see #2244

[tdietert]

Any update on this issue? This would be a lovely feature to have, and think that @ntc2 makes some good points regarding the implementation.

[snoyberg]

Afaict, this issue is ready for someone to work on now. It seems like no one has volunteered to do it yet

[tdietert]

Ahh, so do you think it's safe to assume @gavwhela has stopped working on it?

[snoyberg]

Totally forgot that he'd been working on it and didn't see it when reviewing this issue :)

Yeah, I think it's safe to assume that at this point.