Strip secrets from .binlog

[KirillOsenkov]

[aolszowka]

We are also interested in the feasibility of this to start sharing some high quality large (3,000+ Project) Build Logs to troubleshoot performance issues when viewing such logs. It is difficult to create this volume of high quality data synthetically.

To prevent this from being "+1" post here are several related cases (I almost opened a new one):

• Binary Logging in VSTS (CI) builds #175 - Asks for Similar ability with concern to Environment Variables
• Keep track of which global properties/env variables were actually used dotnet/msbuild#3432 - The MSBuild Ask for the above

I have started a check list for areas of concern:

• Environment Variables
• Paths / File Names - (But we need to preserve "shape" in case its a long path issue)
• Messages - This is probably an impossible ask?

[KirillOsenkov]

I think the most feasible approach for now is rewriting .binlogs via a command-line tool. The API to read and write binlogs is very easy, here's an example:
http://msbuildlog.com/#api

I would imagine that you would write a tool that reads a binlog, iterates over all records, rewrites each record (or drops it entirely) and writes a stripped binlog. You could probably have an extensible set of rewriting rules.

For environment variables you could probably write an analyzer: for each variable, check if it is being used elsewhere in the build, and if not, delete it.

[japj]

@KirillOsenkov If you can save all targets/props (and environment) content to disk from a binlog, I wonder if it would be also possible to “re-load” modified content back into the binlog.

that way you could use just an offline regular text search&replace to strip secure content and prep a binlog for analysis by someone else

[KirillOsenkov]

I guess a binlog rewriter is simpler than that:

1. read all records using

   MSBuildStructuredLog/src/StructuredLogger/BinaryLogger/BinLogReader.cs

   Line 33 in 901b4d8

   public void Replay(string sourceFilePath)

2. create a new instance of BinaryLogger: https://github.com/KirillOsenkov/MSBuildStructuredLog/blob/master/src/StructuredLogger/BinaryLogger/BinaryLogger.cs
3. stream all event args from BinLogReader to BinaryLogger
4. transform each event args arbitrarily on the way

[KirillOsenkov]

A prototype was implemented by @JanKrivanek:
#711

[KirillOsenkov]

Let’s close this issue and file new ones for any remaining work.