Skip to content

Comments

Fix Issue 4582 - distinct field names constraint for std.typecons.Tuple#5725

Merged
dlang-bot merged 3 commits intodlang:masterfrom
RazvanN7:Issue_4582
Oct 12, 2017
Merged

Fix Issue 4582 - distinct field names constraint for std.typecons.Tuple#5725
dlang-bot merged 3 commits intodlang:masterfrom
RazvanN7:Issue_4582

Conversation

@RazvanN7
Copy link
Collaborator

@RazvanN7 RazvanN7 commented Sep 7, 2017

I also did some testing and it seems that if Tuple has a list of types longer that 500 you get a recursive expansion error due to the fact that dmd does not support to expand more than 500 times [1]. Accordingly I did some measurements with and without the distincFieldNames constraints and it seems that the performance is not affected by this diff. The worst time without this patch was 3,263s and with patch 3,347s. I think that this is a negligible difference considering that the error message will be very helpful this way. Thanks to bearophile_hugs for his solution which is basically copy-pasted.

[1] https://github.com/dlang/dmd/blob/master/src/ddmd/dtemplate.d#L7438

@dlang-bot
Copy link
Contributor

Thanks for your pull request, @RazvanN7! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.

Some tips to help speed things up:

  • smaller, focused PRs are easier to review than big ones

  • try not to mix up refactoring or style changes with bug fixes or feature enhancements

  • provide helpful commit messages explaining the rationale behind each change

Bear in mind that large or tricky changes may require multiple rounds of review and revision.

Please see CONTRIBUTING.md for more information.

Bugzilla references

Auto-close Bugzilla Description
4582 distinct field names constraint for std.typecons.Tuple

@PetarKirov
Copy link
Member

PetarKirov commented Sep 7, 2017

I just had an idea for a shorter and more efficient implementation that piggybacks the compiler's symbol hash table. Try this:

enum allDistinct(string[] names) = __traits(compiles, 
{
    static foreach (name; names)
        mixin("enum int" ~ name ~ " = 0;");
});

static assert( allDistinct!([]));
static assert( allDistinct!(["a"]));
static assert(!allDistinct!(["a", "a"]));
static assert( allDistinct!(["a", "b"]));

Edit1: I did a quick benchmark:

private template Iota(int stop)
{
    static if (stop <= 0)
        alias AliasSeq!() Iota;
    else
        alias AliasSeq!(Iota!(stop-1), stop-1) Iota;
}

private bool distinctFieldNames1(T...)()
{
    enum int tlen = T.length;
    foreach (i1; Iota!(tlen))
        static if (is(typeof(T[i1]) : string))
            foreach (i2; Iota!(tlen))
                static if (i1 != i2 && is(typeof(T[i2]) : string))
                    if (T[i1] == T[i2])
                        return false;
    return true;
}

private bool distinctFieldNames2(T...)()
{
    enum int tlen = T.length;
    foreach (i1; staticIota!(0, tlen))
        static if (is(typeof(T[i1]) : string))
            foreach (i2; staticIota!(0, tlen))
                static if (i1 != i2 && is(typeof(T[i2]) : string))
                    if (T[i1] == T[i2])
                        return false;
    return true;
}

enum bool allDistinct(string[] names) = __traits(compiles,
{
    static foreach (name; names)
        mixin("enum int" ~ name ~ " = 0;");
});

import std.algorithm.iteration : map;
import std.conv : to;
import std.range : array, iota;
import std.meta : AliasSeq, aliasSeqOf;
import std.typecons : staticIota;

enum gen(size_t n) = n.iota.map!(i => "a" ~ i.to!string).array;
static immutable string[] s = gen!300;
alias s2 = aliasSeqOf!s;
  • The initial version (tested by pragma (msg, distinctFieldNames1!s2);) takes 21.24s:
$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 12.70
        System time (seconds): 8.43
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.24
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 11981784
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 3002200
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
  • pragma (msg, distinctFieldNames2!s2); takes 21.16:
$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 12.56
        System time (seconds): 8.57
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:21.16
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 11985728
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 3003188
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
  • pragma (msg, allDistinct!s2); takes 0.16s:
$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 0.09
        System time (seconds): 0.07
        Percent of CPU this job got: 106%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.16
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 47584
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 12170
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

When I increase the the size to 500 (by using gen!500) I got:

  • pragma (msg, distinctFieldNames1!s2); fails:
$ /usr/bin/time -v dmd -o- test_perf.d
test_perf.d(6): Error: template instance test_perf.Iota!0 recursive expansion
Command exited with non-zero status 1
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 0.14
        System time (seconds): 0.06
        Percent of CPU this job got: 100%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.20
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 62372
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 15868
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 1
  • pragma (msg, distinctFieldNames2!s2); takes 56.44s:
$ /usr/bin/time -v dmd -o- test_perf.d
Command terminated by signal 11
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 30.00
        System time (seconds): 25.48
        Percent of CPU this job got: 98%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:56.44
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 13994220
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 6712430
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
  • pragma (msg, allDistinct!s2); takes 0.28:
$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 0.14
        System time (seconds): 0.06
        Percent of CPU this job got: 71%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.28
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 61164
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 15572
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

When increasing the size to 10000, allDistinct becomes the only method that actually finishes in a reasonable amount of time:

$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 4.73
        System time (seconds): 0.92
        Percent of CPU this job got: 100%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.65
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1083884
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 271753
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Edit2: here's an a bit more optimal distinctFieldNames3 implementation:

enum bool distinctFieldNames3(names...) =
    distinctFieldNames3Impl([names]);

private bool distinctFieldNames3Impl(string[] names)
{
    foreach (i; 0 .. names.length)
        foreach (j; i + 1 .. names.length)
            if (names[i] == names[j])
                return false;
    return true;
}

For 1000 I get:

$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 1.09
        System time (seconds): 0.20
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.29
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 373620
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 93840
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Though it scales poorly w.r.t. memory, here's the result for 10000:

$ /usr/bin/time -v dmd -o- test_perf.d
Command terminated by signal 11
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 81.26
        System time (seconds): 24.62
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:46.34
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 14017064
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 6647563
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

On the other hand, if there are actually duplicates (tested by changing gen to enum gen(size_t n) = n.iota.map!(i => "a").array), distinctFieldNames3 is the fastest:

$ /usr/bin/time -v dmd -o- test_perf.d
false
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 0.35
        System time (seconds): 0.23
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.59
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 455504
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 114346
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

In comparison, allDistinct is much slower because it is not lazy:

$ /usr/bin/time -v dmd -o- test_perf.d
false
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 6.14
        System time (seconds): 1.09
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.24
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1800004
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 451258
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

For a direct comparison between allDistinct and distinctFieldNames3, if we focus on the slightly more realistic case with 500 distinct names, we get:

  • For allDistinct:
$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 0.12
        System time (seconds): 0.07
        Percent of CPU this job got: 95%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.21
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 61188
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 15578
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
  • For distinctFieldNames3:
$ /usr/bin/time -v dmd -o- test_perf.d
true
        Command being timed: "dmd -o- test_perf.d"
        User time (seconds): 0.32
        System time (seconds): 0.10
        Percent of CPU this job got: 98%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.44
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 129276
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 32633
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

So, it's a close tie. What do you guys think we should pick?

Copy link
Member

@PetarKirov PetarKirov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use a more efficient implementation - see #5725 (comment).

std/typecons.d Outdated
alias sharedToString = field;
}

private template Iota(int stop)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we need to have staticIota with 1, 2 and 3 parameters in std.meta but that's a battle for another day. In this case there already something similar defined in this module, which you should be able to reuse. IIRC, it's more optimal because it uses log(n) instances, instead of n, i.e. it should hit the 500 limit harder.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are at least 2 staticIota-a in our std library. Time to draft a public API?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incidentally, last night and this morning I was working on a faster version of staticIota. When I'm done, I'll post a PR for std.meta.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward! Otherwise you could also move this to someone and make it package(std) for now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before static foreach, foreach (const i; aliasSeqOf!(iota(0, n))) was the common pattern, now it's simply:

void main(string[] args)
{
        static foreach (i; 0..3)
    	    pragma(msg, i);
}

https://run.dlang.io/is/QWxTYt

So thinking about it: at least for this PR we don't need Iota at all..

@RazvanN7
Copy link
Collaborator Author

RazvanN7 commented Sep 7, 2017

@ZombineDev Your allDistinct implementation is really cool. I'm really annoyed I didn't think of it. As I see it , the Tuple implementation is limited to 500 names (passing more than 500 names will result in compilation failure in the innards of Tuple), so we should go with allDistinct. What do you think?

@PetarKirov
Copy link
Member

Yes, I like it better because it shorter and easier to review at a glance.

@PetarKirov
Copy link
Member

PetarKirov commented Sep 7, 2017

The only disadvantage is that part of performance comes from committing the check for is(T[i] == string) (which requires using something like allSatisfy!(isConvertibleToString, T)), which may or may not improve the error message.

Edit: with

enum bool allDistinct(names...) = allDistinctImpl!([names]);

enum bool allDistinctImpl(string[] names) = __traits(compiles,
{
    static foreach (name; names)
        mixin("enum int" ~ name ~ " = 0;");
});

If you call allDistinct!(3, s2) you get:

Error: incompatible types for ((3) : ("a0")): 'int' and 'immutable(string)'

And with allDistinct!(int, s2) you get:

Error: type int has no value

And no trace from where exactly the error comes from.

@RazvanN7
Copy link
Collaborator Author

RazvanN7 commented Sep 7, 2017

Yes, but that check is important, since otherwise, you end up with valid cases being rejected. For example, a Tuple instantiation : Tuple(int, "one", int, "two") will fail.

@PetarKirov
Copy link
Member

This can be solved like this:

import std.meta : Stride;
enum bool allDistinct(names...) =
    allDistinctImpl!([ Stride!(2, names) ]);

But the original issue still stands.

@RazvanN7
Copy link
Collaborator Author

RazvanN7 commented Sep 7, 2017

Although I agree that your solution is more elegant I do not think that performance is an issue in this case since Tuple is limited to 500 value names. In terms of readability I find bearophile's code really nice (and to make allDistinct work with the current Tuple implementation I think things might get nasty)

Copy link
Contributor

@wilzbach wilzbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few style nits

std/typecons.d Outdated
private bool distinctFieldNames(T...)()
{
enum int tlen = T.length;
foreach (i1; Iota!(tlen))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static foreach (i; 0 .. tlen)

std/typecons.d Outdated
enum int tlen = T.length;
foreach (i1; Iota!(tlen))
static if (is(typeof(T[i1]) : string))
foreach (i2; Iota!(tlen))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static foreach (j; 0 .. then)

std/typecons.d Outdated
Specs = A list of types (and optionally, member names) that the `Tuple` contains.
*/
template Tuple(Specs...)
template Tuple(Specs...) if (distinctFieldNames!(Specs)())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put this in a separate line, indent-aligned with template

@RazvanN7
Copy link
Collaborator Author

ping @ZombineDev @MetaLang is this ok?

Copy link
Member

@MetaLang MetaLang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though it's a private symbol it should still have a unit test.

@RazvanN7
Copy link
Collaborator Author

RazvanN7 commented Oct 2, 2017

@MetaLang I haven't seen any private symbols getting unittested in phobos. How about this : I add a unittest for Tuple which tests that tuples that do not have distinct field names fail to compile.

@MetaLang
Copy link
Member

MetaLang commented Oct 2, 2017

It depends on what code you've been looking at, probably. Personally, anytime I add a new symbol - public or private - I add unit tests that try to exercise it as thoroughly as possible. If we're not doing that currently in Phobos, we should, because tests will ensure that we know when something breaks.

@wilzbach
Copy link
Contributor

wilzbach commented Oct 2, 2017

It depends on what code you've been looking at, probably. Personally, anytime I add a new symbol - public or private - I add unit tests that try to exercise it as thoroughly as possible. If we're not doing that currently in Phobos, we should, because tests will ensure that we know when something breaks

Yes. Apart from guaranteeing that the function won't break in the future, they help future reader as documentation - and for your current reviewers as well, of course!

@RazvanN7
Copy link
Collaborator Author

@ZombineDev @MetaLang All should be good now

@MetaLang
Copy link
Member

@RazvanN7 thanks!

@MetaLang
Copy link
Member

MetaLang commented Oct 12, 2017

++ bash install.sh dmd-2.068.2 --activate
curl: (28) Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
curl: (28) Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
curl: (28) Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
curl: (28) Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
curl: (28) Operation too slow. Less than 1024 bytes/sec transferred the last 30 seconds
Failed to download 'http://code.dlang.org/download/LATEST'

Unfortunately, I don't have permissions to trigger a rebuild on CircleCI.

EDIT: never mind, I just had to authenticate via Github. It's running now.

@dlang-bot dlang-bot merged commit f24da9d into dlang:master Oct 12, 2017
@wilzbach
Copy link
Contributor

Failed to download 'http://code.dlang.org/download/LATEST'

FWIW the PR to fix this has been open for half a year: dlang/installer#218

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants