Skip to content

Conversation

@BruceForstall
Copy link
Contributor

@BruceForstall BruceForstall commented Sep 15, 2020

  1. Update Azure storage collections location to a new location
    where we have appropriate permissions to manage the data.
  2. Update the Azure storage upload/download implementation
    to the current version of the Azure Python API.
  3. Add JIT-EE interface GUID to path of Azure stored
    collections. When downloading collections, use the appropriate
    collection for your JIT-EE interface GUID. This is all done
    by adding a "-printJITEEVersion" option to the SuperPMI
    MCS tool. Thus, to determine the JIT-EE version, we assume
    that MCS is built with the same version as the JIT (which
    will be true in a normal build), and that MCS is available
    and able to be run -- this typically requires a Core_Root
    location be available. The user can specify the JIT-EE
    version explicitly with the new -jit_ee_version argument.
  4. Simplify the Azure storage format: there is no longer
    a JSON mapping of name to MCH file. Instead, there is just
    a directory full of files. By default, all files are downloaded
    and used for replay/asmdiffs, by that can be filtered with
    the new -filter argument.
  5. The -mch_files (previously -mch_file) argument used
    by replay, asmdiffs, and upload, now accepts a list
    of directories and files, and for each directory all MCH
    files included in that directory, recursively, are used.
  6. Also upload MCT (TOC) files with the MCH files.
  7. A --force_download argument is added to allow forcing
    re-download of the Azure collections to the local cache.
  8. PMI.dll is also looked for on the PATH before downloading
    a cached version from Azure storage.
  9. Some of the lesser-used arguments were renamed to simplify them.
  10. Various bugs were fixed and code simplification/reorganization was done.
    E.g., some of the commonality between replay and asmdiffs was
    factored out.
  11. More code documentation was added.
  12. The superpmi.md documentation was re-written and simplified.

@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 15, 2020
@BruceForstall
Copy link
Contributor Author

@kunalspathak Here's my current set of changes. I need to do a little more testing, update the superpmi.md documentation, and write a summary of the changes for the commit message.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

micro nit

Suggested change
return_code = 1;
return_code = 1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what happens when you let a C programmer touch Python :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

micro nit:

Suggested change
if self.coreclr_args.log_file != None:
if self.coreclr_args.log_file is not None:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# 3. A copy of coredistools. If, when doing asm diffs, a copy of the coredistools
# 3. A copy of coredistools. If, when doing asm diffs, a copy of the coredistools

1. Update Azure storage collections location to a new location
where we have appropriate permissions to manage the data.
2. Update the Azure storage upload/download implementation
to the current version of the Azure Python API.
3. Add JIT-EE interface GUID to path of Azure stored
collections. When downloading collections, use the appropriate
collection for your JIT-EE interface GUID. This is all done
by adding a "-printJITEEVersion" option to the SuperPMI
MCS tool. Thus, to determine the JIT-EE version, we assume
that MCS is built with the same version as the JIT (which
will be true in a normal build), and that MCS is available
and able to be run -- this typically requires a Core_Root
location be available. The user can specify the JIT-EE
version explicitly with the new `-jit_ee_version` argument.
4. Simplify the Azure storage format: there is no longer
a JSON mapping of name to MCH file. Instead, there is just
a directory full of files. By default, all files are downloaded
and used for replay/asmdiffs, by that can be filtered with
the new `-filter` argument.
5. The `-mch_files` (previously `-mch_file`) argument used
by `replay`, `asmdiffs`, and `upload`, now accepts a list
of directories and files, and for each directory all MCH
files included in that directory, recursively, are used.
6. Also upload MCT (TOC) files with the MCH files.
7. A `--force_download` argument is added to allow forcing
re-download of the Azure collections to the local cache.
8. PMI.dll is also looked for on the PATH before downloading
a cached version from Azure storage.
9. Some of the lesser-used arguments were renamed to simplify them.
10. Various bugs were fixed and code simplification/reorganization was done.
E.g., some of the commonality between replay and asmdiffs was
factored out.
11. More code documentation was added.
12. The superpmi.md documentation was re-written and simplified.
@BruceForstall BruceForstall force-pushed the UpdateSupermiAutomation branch from fab1f22 to 5cc1845 Compare September 18, 2020 01:48
@BruceForstall BruceForstall changed the title Many improvements to superpmi.py Improve superpmi.py scripting Sep 18, 2020
@BruceForstall BruceForstall marked this pull request as ready for review September 18, 2020 01:50
@BruceForstall
Copy link
Contributor Author

This is ready for review.

@kunalspathak PTAL
cc @dotnet/jit-contrib

@BruceForstall
Copy link
Contributor Author

Btw, none of this is used in the product, and there are a lot of changes, so you can choose how detailed a review you think is warranted.

Copy link
Contributor

@kunalspathak kunalspathak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BruceForstall BruceForstall merged commit 22d206c into dotnet:master Sep 18, 2020
@BruceForstall BruceForstall deleted the UpdateSupermiAutomation branch September 18, 2020 22:32
@ghost ghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants