Skip to content

#1 Performance measurement#2

Merged
dhendriks merged 18 commits intocom-github-javabdd:masterfrom
magoorden:1-performance-measurement
Oct 25, 2021
Merged

#1 Performance measurement#2
dhendriks merged 18 commits intocom-github-javabdd:masterfrom
magoorden:1-performance-measurement

Conversation

@magoorden
Copy link
Copy Markdown
Contributor

This merge requests adds code to collect performance measurements.

Closes #1.

Beside adding the code, I did the following:
- Minor style changes
- Add MemoryStats class
- Add statistics recording in ZDD classes
As system memory usages fluctuates over time, tool implementation, hardware, etc., measuring these statistics are 'removed' by commenting the out. If one (who knows what he is doing) still wants to measure the system memory usage, the code is still available.
By removing system memory statistics, the statistics output is completely deterministic for the same input and bdd operations.
Now the caller of the BDD package should determine how to format the stats printing. We now just return the stats.
@dhendriks
Copy link
Copy Markdown
Member

dhendriks commented Sep 26, 2021

You changed BDDFactory. There are 5 classes that extend BDDFactory. But of the 5 only JFactory actually implements the new measurements? Should the other ones be updated as well?

@dhendriks
Copy link
Copy Markdown
Member

I scrolled through it a bit and left some comments. Please feel free to address these already. I'll review it in more detail later.

@dhendriks
Copy link
Copy Markdown
Member

You did a bunch of things, besides adding platform-independent performance statistics. You also changed the ints to longs, added opAccess to CacheStats, etc. The changelog is thus incomplete. I also wonder whether in hindsight whether it might have been better to change these separately. That would have made reviewing easier.

@dhendriks
Copy link
Copy Markdown
Member

I've finished my review, I think. Let me know what you think or if you have any questions.

@dhendriks
Copy link
Copy Markdown
Member

Hi @magoorden. Do you plan to work on this further? Or did my feedback scare you away? I'd love to hear from you.

@magoorden
Copy link
Copy Markdown
Contributor Author

Hi @magoorden. Do you plan to work on this further? Or did my feedback scare you away? I'd love to hear from you.

You did not scare me at all. I'm just at holiday now. So when I'm back, I'll take a closer look at your feedback.

@dhendriks
Copy link
Copy Markdown
Member

OK, no problem. Enjoy your holidays!

@magoorden
Copy link
Copy Markdown
Contributor Author

You changed BDDFactory. There are 5 classes that extend BDDFactory. But of the 5 only JFactory actually implements the new measurements? Should the other ones be updated as well?

Good question, haven't thought about that. Let me investigate it. It might be the case that all changes from BDDFactory move to JFactory in the end.

@magoorden
Copy link
Copy Markdown
Contributor Author

I've finished my review, I think. Let me know what you think or if you have any questions.

Are your comments only the onces about the BDDFactory vs JFactory and the changelog? If there is more, I am not able to find them...

@dhendriks
Copy link
Copy Markdown
Member

Good question, haven't thought about that. Let me investigate it. It might be the case that all changes from BDDFactory move to JFactory in the end.

That would be the least work. Not sure however whether that is the most logical choice from the JavaBDD perspective.

@dhendriks
Copy link
Copy Markdown
Member

Are your comments only the onces about the BDDFactory vs JFactory and the changelog? If there is more, I am not able to find them...

I'm looking at #2 and I see 5 comments, then an indication that there are 15 hidden conversations, and then 5 more comments, as part of my review. The first one is about the changelog file, but there are thus many more. I then see some additional standalone comments, including about BDDFactory, int vs long, etc. Do you not see all this on this page?

@magoorden
Copy link
Copy Markdown
Contributor Author

I guess I don't see the hidden conversations 😆

@dhendriks
Copy link
Copy Markdown
Member

I guess I don't see the hidden conversations 😆

You don't see something like this when you scroll up from these comments we're now exchanging:

image

@magoorden
Copy link
Copy Markdown
Contributor Author

You don't see something like this when you scroll up from these comments we're now exchanging:

No, I don't. My comment counter at the top of the pages only counts 13.

@dhendriks
Copy link
Copy Markdown
Member

No, I don't. My comment counter at the top of the pages only counts 13.

Mine is now also only 14 comments (one more than yours I guess due to you adding another comment). The screenshot that I shared is part of the review, not of the comments.

What happens if you go to the 'Files changed' tab. Do you see my 'review comments' then?

@dhendriks
Copy link
Copy Markdown
Member

That would be the least work. Not sure however whether that is the most logical choice from the JavaBDD perspective.

I can imagine only updating JFactory if we remove the other factories. Let's discuss as part of issue #3.

@magoorden
Copy link
Copy Markdown
Contributor Author

Mine is now also only 14 comments (one more than yours I guess due to you adding another comment). The screenshot that I shared is part of the review, not of the comments.

What happens if you go to the 'Files changed' tab. Do you see my 'review comments' then?

I'm still not able to make your review comments visible. They are not being shown in between the code, nor anywhere at the top in a form of a list.

Screenshot 2021-10-17 at 12 18 01

Comment thread CHANGES.md Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/JFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/JFactory.java
Comment thread src/main/java/com/github/javabdd/JFactory.java
Comment thread src/main/java/com/github/javabdd/JFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/JFactory.java Outdated
@dhendriks
Copy link
Copy Markdown
Member

I'm still not able to make your review comments visible. They are not being shown in between the code, nor anywhere at the top in a form of a list.

It turns out I had to 'submit' my review. I think you can see the comments now?

@magoorden
Copy link
Copy Markdown
Contributor Author

It turns out I had to 'submit' my review. I think you can see the comments now?

Yes, I can see them 😃

@magoorden
Copy link
Copy Markdown
Contributor Author

@dhendriks I have been through all your comments. Most of them are resolved. I left some of them open for further discussion.

Once all comments are resolved, I plan to update CHANGES.md with all changes.

Comment thread src/main/java/com/github/javabdd/BDDFactory.java
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/JFactory.java
Comment thread src/main/java/com/github/javabdd/JFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/JFactory.java
@dhendriks
Copy link
Copy Markdown
Member

You changed BDDFactory. There are 5 classes that extend BDDFactory. But of the 5 only JFactory actually implements the new measurements? Should the other ones be updated as well?

I think the basic statistics classes best fit in BDDFactory such that all implementations of that class should handle statistics collection.

Looking at our discussion in #3, the changes in this merge request might be suffiecient, as we (most likely) get rid of the other factories. In case we want to keep some other factories, I would suggest to open new issues to implement the statistics collection in those factories as well.

I agree.

@dhendriks dhendriks closed this Oct 23, 2021
@dhendriks
Copy link
Copy Markdown
Member

I'm wondering about the maximum memory usage. The fact that Java employs garbage collection makes it difficult to correctly measure the maximum memory usage, as the peak may have between just before a garbage collection, between two measurements that we do. See for instance https://cruftex.net/2017/03/28/The-6-Memory-Metrics-You-Should-Track-in-Your-Java-Benchmarks.html for various approaches and why it is so difficult to get it right.

So my question is: what is the practical purpose of this? Where would you use this for? And couldn't a user just use a tool like VisualVM for this?

Maybe we should remove the memory statistics and let users do that themselves, using external means, where they control the environment and circumstances? Then we only count the platform-independent ones in the library itself.

@dhendriks dhendriks reopened this Oct 23, 2021
@dhendriks
Copy link
Copy Markdown
Member

I've completed my review.

- Allow stat measurements to be disabled
- More consistent naming of stat objects and methods.
- Move stats measurements at several places to a separate method.
@magoorden
Copy link
Copy Markdown
Contributor Author

I'm wondering about the maximum memory usage. The fact that Java employs garbage collection makes it difficult to correctly measure the maximum memory usage, as the peak may have between just before a garbage collection, between two measurements that we do. See for instance https://cruftex.net/2017/03/28/The-6-Memory-Metrics-You-Should-Track-in-Your-Java-Benchmarks.html for various approaches and why it is so difficult to get it right.

So my question is: what is the practical purpose of this? Where would you use this for? And couldn't a user just use a tool like VisualVM for this?

Maybe we should remove the memory statistics and let users do that themselves, using external means, where they control the environment and circumstances? Then we only count the platform-independent ones in the library itself.

Good point. I have removed the memory measurement part.

@magoorden
Copy link
Copy Markdown
Contributor Author

@dhendriks Thanks for the review. I have adapted the code based on it. There are two comments unresolved. For those I have the suggestion to move your raised points to new issues to be dealt with later. I wonder whether you would agree or not.

Comment thread src/main/java/com/github/javabdd/BDDFactory.java
Comment thread src/main/java/com/github/javabdd/BDDFactory.java
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/BDDFactory.java Outdated
Comment thread src/main/java/com/github/javabdd/JFactory.java
Comment thread src/main/java/com/github/javabdd/JFactory.java Outdated
@dhendriks
Copy link
Copy Markdown
Member

I've completed another review round. I think we're getting close.

There are two comments unresolved. For those I have the suggestion to move your raised points to new issues to be dealt with later. I wonder whether you would agree or not.

I've added comments to those. I think those are resolved now.

@magoorden
Copy link
Copy Markdown
Contributor Author

Your comments have been resolved. I also updated the CHANGES.md file.

Copy link
Copy Markdown
Member

@dhendriks dhendriks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've completed another review. I've added some suggestions for the changelog.

Comment thread CHANGES.md Outdated
Comment thread CHANGES.md Outdated
Comment thread CHANGES.md Outdated
Comment thread CHANGES.md Outdated
Comment thread CHANGES.md
Comment thread CHANGES.md Outdated
@dhendriks dhendriks merged commit 072a81d into com-github-javabdd:master Oct 25, 2021
@magoorden magoorden deleted the 1-performance-measurement branch October 30, 2021 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Performance measuring of data-based synthesis

2 participants