static array litteral: [1,2].asStatic + overloads for CT ranges, RT ranges, explicit types

[timotheecour]

(moved from dlang/druntime#2093)

usage

// from litteral
auto a = [1,2].asStatic; // int[2]
auto a = [1,2].asStatic!byte; // byte[2]

// from range
auto a = 2.iota.asStatic!2; // int[2]
auto a = 2.iota.asStatic!(byte[2]); // byte[2]

// from alias
auto a = asStatic!(2.iota); // int[2]
auto a = asStatic!(byte, 2.iota); // byte[2]

motivation

places like here: dlang/dub#1377 (comment)

question for reviewers

not sure how one wants to treat cases where ForEachType != ElementType, eg:
"ab∫".staticArray currently is immutable(char[5]) instead of immutable(dchar[3])

links

https://issues.dlang.org/show_bug.cgi?id=16745 Add template helper for creating static arrays with the size inferred
https://issues.dlang.org/show_bug.cgi?id=8008 Issue 8008 - Syntax for fixed size array literals like [1,2,3]s (edit)
https://issues.dlang.org/show_bug.cgi?id=481 Issue 481 - Fixed-length arrays with automatically computed length (edit)

#4936 Fix Issue 16745 - Add template helper for creating static arrays with the size inferred #4936
#4090 Better static array support in std.array #4090
#6214 Overload for static arrays for std.array.array #6214

not related

https://issues.dlang.org/show_bug.cgi?id=9165 Issue 9165 - Auto conversion from dynamic array to fixed size array at return (edit)

---

This change is 

[dlang-bot]

Thanks for your pull request and interest in making D better, @timotheecour! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the annotated coverage diff directly on GitHub with CodeCov's browser extension
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Auto-close	Bugzilla	Severity	Description
✓	16745	enhancement	Add template helper for creating static arrays with the size inferred

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + phobos#6178"

[JackStouffer]

There's no reason to break up std.array. I suggest you leave that change out.

[n8sh]

If U is a struct you need to check if it has an elaborate destructor and in that case not void-initialize ret because the destructor for ret[i] will be run before a[i] is blitted over it. An alternative could be to use emplace.

[timotheecour]

done, used emplaceRef wherever applicable

[timotheecour]

There's no reason to break up std.array. I suggest you leave that change out.

added motivation for it in intro: std.array is a large package and std.array.static_array and std.array.util don't call each other, so that makes a perfect candidate for splitting.

PTAL, I've improved API to allow all cases (see updated top-level msg)

[wilzbach]

don't call each other, so that makes a perfect candidate for splitting.

So do many other methods in std.algorithm and std.range - the splitting seems to me a bit artificial. You need a lot stronger motivation for this and I really don't want Phobos to end up with one module for every function.

[timotheecour]

/cc @wilzbach
PTAL, all comments addressed, i unsplit std.array (although i disagree with this change but whatever)

the only failure is unrelated:

std.socket.SocketOSException@std\socket.d(2801): Unable to connect socket: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
----------------
0x00411BC8
0x0040F05B
0x0040EFDD
0x0040EE78
0x0040C0DF
0x75BB343D in BaseThreadInitThunk
0x77B99832 in RtlInitializeExceptionChain
0x77B99805 in RtlInitializeExceptionChain
Connecting to www.dlang.org on port 80...
win32.mak:24: recipe for target 'htmlget' failed
gmake: *** [htmlget] Error 1

[JackStouffer]

imports go at the top of modules

[timotheecour]

done

[JackStouffer]

Please avoid using attributes like this, they are a huge pain later on and Andrei has expressed desire for removing all such uses from Phobos.

[timotheecour]

done

[JackStouffer]

superfluous

[timotheecour]

done

[JackStouffer]

assertThrown doesn't work?

[timotheecour]

not nothrow

[JackStouffer]

Don't use undocumented functions in public examples.

Better to just manually enter in both tests from assertSame here for each call.

[timotheecour]

done

[JackStouffer]

ditto only works if it's the only thing in the ddoc-ed comment. If you want this to have its own docs, you have to include all the information because the ddox version shows all functions on separate pages, and not in sequential order like ddoc.

[timotheecour]

done

[JackStouffer]

needs Params: and Returns: sections. same throughout

[timotheecour]

done i think

[JackStouffer]

D20180214T203015

What is this? A date? What for?

[timotheecour]

done (was meant as a simple GUID to cross-reference; but removed it)

[wilzbach]

This shouldn't be part of the public documentation

[timotheecour]

done

[wilzbach]

FYI: There's already https://dlang.org/phobos/std_exception.html#ifThrown

auto foo() {
   throw new Exception("test");
   return true;
}

ifThrown(foo(), false).writeln;

https://run.dlang.io/is/ua4DpZ

[timotheecour]

not nothrow (and results in uglier syntax, see PR in version(none) block)

[wilzbach]

This is a new public symbol. The tester failed for a good reason. We don't want to use undocumented functionality in public examples.

[timotheecour]

removed its used from documented unittests

[wilzbach]

: iota - for public tests.

[dukc]

Honestly I saw this only after doing my own which does pretty much the same: #6214. Otherwise I probably would have thought something else.

[timotheecour]

/cc @dukc

Honestly I saw this only after doing my own which does pretty much the same: #6214. Otherwise I probably would have thought something else.

I personally don't care which version gets in; but yours seems lacking functionality (See unitttests)

[timotheecour]

waiting for @andralex approval, @wilzbach already approved it

[wilzbach]

Here's an idea to avoid being blocked on Andrei's approval for new symbols: we merge this as is to std.experimental and keep it there for one release. We then only need Andrei's approval once every two months when we merge everything from std.experimental to the standard.
This would also help in marking the new addition as unstable and help to avoid breakage if any issues are discovered.
We haven't done this before, but I think we could give it a try.
If this is going to experimental we could immediately merge this and imho having it wait in experimental is a lot better than it sitting in the PR queue forever.

[wilzbach]

More ideas and a summary:

1) std.experimental.staging

As discussed a module in std.experimental for upcoming symbols.
There's a bit of a precedent with e.g. https://dlang.org/phobos/std_experimental_typecons.html

Pros:

• easy to find out which symbols are experimentally used
• easy to find which symbols are currently pending

Cons:

• source code needs to be modified when the symbol gets stabilized, but alias + deprecations could be used

It's debatable whether it should be std.experimental.<moduleName> or std.experimental.staging as both have slight advantages/disadvantages.
The one is a bit more explicit, but requires more typing and interaction from the user. As it also requires more effort on our side to create a module every time a new symbol is introduced, I thing a generic catch-all staging module is a good trade-off.

2) Mark it in the documentation as experimental

see e.g. rust-lang/rust#44015

#[stable(feature = "indirect_hasher_impl", since = "1.22.0")]
impl<T: ?Sized + Hasher> Hasher for Box<T> {

Pros:

• source code doesn't need to be modified when the symbol gets stabilized

Cons:

• accidental usage
• if someone uses an experimental symbol, this might not be clear to reviewers of the user-code

3) Only expose it via a -experimental flag (or -version=StdExperimental) for that matter

version(StdExperimental)
{
...
}

The flag would be added to the default build flag in the Phobos Makefiles, but the symbol wouldn't be accessible without this flag to the user.

Pros:

• source code doesn't need to be modified when the symbol gets stabilized
• explicit opt-in into experimental symbols once

Cons:

• a bit more overhead to maintain which symbols are currently unstable, but we could use an @@@STABLE_2.082@@@ pattern like we do for deprecations (or just maintain a file in which unstable symbols are listed)
• might require a bit of documentation to explain that -version(StdExperimental) needs to be used (but that could be part of a default $(UNSTABLE) macro.

4) Put experimental functions into a stdx dub package

Example: https://github.com/dlang-stdx/collections

Pros:

• doesn't break user code if new symbols are added
• even easier to merge as it's completely decoupled from Phobos

Cons:

• a bit of overhead in creating the build tools
• might not be as visible as Phobos
• there are still some people who don't like dub (or don't want to/can't use dub)
• might introduce a bit of overhead in keeping it working with newer compiler versions (but there's the project tester for this)

---

Personally I would be fine with any of these options as long as we make progress and don't let PRs die in the queue forever.

[dukc]

I prefer option 2. If it's good enough for web api (https://developer.mozilla.org/en-US/docs/Web/API) I don't see why it wouldn't be for us. And probably causes the least "drag" in getting stuff in.

[andralex]

Since we have pipe.array to create arrays from pipes, we should probably call this staticArray.

[andralex]

It's not necessarily about function calls. The matter at hand is asStatic returns by value, so the text should just remind that. "Note: asStatic returns by value, so expressions involving large arrays may be inefficient."

[timotheecour]

done

[andralex]

Please merge with the previous overload and use static if inside. The user doesn't care that the implementations are different.

[timotheecour]

I can replace with

template staticArray(U...) if (U.length == 0 || U.length == 1 && is(U[0]))
{
    static if (U.length == 0)
    {
        pragma(inline, true) T[n] staticArray(T, size_t n)(auto ref T[n] a) nothrow @safe pure @nogc
        {
            return a;
        }
    }
    else
    {
        alias U0 = U[0];
        U0[n] staticArray(T, size_t n)(auto ref T[n] a)
        {
            import std.conv : emplaceRef;

            U0[n] ret = void;
            static foreach (i; 0 .. n)
            {
                emplaceRef!U0(ret[i], cast(U0) a[i]);
            }
            return ret;
        }
    }
}

but I don't see in what ways it improves over what I had:

pragma(inline, true) T[n] staticArray(T, size_t n)(auto ref T[n] a) nothrow @safe pure @nogc
{
    return a;
}

/// ditto
U[n] staticArray(U, T, size_t n)(auto ref T[n] a)
if (!is(U == T))
{
    import std.conv : emplaceRef;
    U[n] ret = void;
    static foreach (i; 0 .. n)
    {
        emplaceRef!U(ret[i], cast(U) a[i]);
    }
    return ret;
}

(more lines of code, more complexity, less readable IMO)

[andralex]

This function has a pernicious bug: if construction throws an exception, ret will be destroyed and destructors will be called for uninitialized objects. Recall that statically-sized arrays on the stack insert one destructor call for each member.

[timotheecour]

avoided this bug by making these overloads as nothrow, so a throwing ctor will cause CT error instead of potential runtime bug.

support for throwing constructors can always be addressed in a future PR (without breaking backward compatibility), maybe something along these lines:

U[n] staticArray(U, T, size_t n)(auto ref T[n] a)
if (!is(U == T))
{
     import std.conv : emplaceRef;

     U[n] ret = void;
    size_t i;
    try
     {
        for (; i < n; i++)
        {
            emplaceRef!U(ret[i], cast(U) a[i]);
        }
    }
    catch(Exception t) // catch(Throwable t) causes errors with D linter; but we really want Throwable here for correctness
    {
        for (; i < n; i++)
        {
            emplaceRef!U(ret[i], U.init);
        }
        throw t;
     }
     return ret;
 }

[andralex]

Template constraint?

[timotheecour]

done

[andralex]

This code duplicates implementation (and bug) from a different overload.

[timotheecour]

done

[andralex]

constraint?

[timotheecour]

done

[andralex]

constraint?

[timotheecour]

done

[andralex]

why is a cast needed?

[timotheecour]

otherwise:
Error: template std.array.staticArray cannot deduce function from argument types !(length)(Result)

[andralex]

constraint

[timotheecour]

done

[andralex]

nice

[timotheecour]

ack

[timotheecour]

PTAL, all comments addressed

/cc @andralex not sure how one wants to treat cases where ForEachType != ElementType, eg:
"ab∫".staticArray currently is immutable(char[5]) but not sure whether we'd want immutable(dchar[3]) instead.

• immutable(char[5])
  this requires no special casing in generic code
  least surprising: immutable(char[]) => immutable(char[5])
  maybe more efficient, since no decoding involved

• immutable(dchar[3])
  this is behavior of "ab∫".array (it's returning dchar[])

[andralex]

@timotheecour on strings we should follow the rule of least astonishment - i.e. if you start with a narrow string you get a static string of the same narrowness. Qualifier on the characters should be the same, but no need to make the entire result qualified, i.e. immutable(char)[5], not immutable(char[5]).

[wilzbach]

Can we please merge this to std.experimental (or similar)?

There are tons of places within Phobos where this would be very useful and we can work out the exact specific and nitpick the name while we get experience using it.

I think it goes without saying, but it's a very powerful tool for convenient @nogc usage.

[andralex]

Multiple major issues remaining.

[andralex]

Template constraint should include that T can be cast to U.

[andralex]

I'd accept the argument "let's leave it nothrow and worry about it later) if (a) this PR were urgent, and/or (b) the right code would be difficult to implement. Neither is applicable. The right code is (stylized):

U[n] staticArray(U, T, size_t n)(auto ref T[n] a)
{
    union Temp { U[n] payload; }
    Temp result = void;
    size_t i = 0;
    static if (hasElaborateDestructor!U) scope (exit)
    {
        foreach (j; 0 .. i)
            destroy(result.payload[j]);
    }
    for (; i != n; ++i)
        emplaceRef!U(result.payload[i], cast(U) a[i]);
    return result.payload;
}

That's awkward and disables named return value optimization, so here's what I think would be a better approach:

U[n] staticArray(U, T, size_t n)(auto ref T[n] a)
{
    U[n] result = void;
    size_t i = 0;
    static if (hasElaborateDestructor!U) scope (failure)
    {
        foreach (j; i .. n)
            emplace(result[j]); // initialize with T.init to make sure is destroyable
    }
    for (; i != n; ++i)
        emplaceRef!U(result[i], cast(U) a[i]);
    return result;
}

This is cute and hopefully not too much so - it comes from the opposite direction: in case of an exception, it initializes (in a nothrow manner) the uninitialized part of the result and then leaves the rest to the compiler. This allows NRVO, is efficient on the nothrow path, and is simple.

[andralex]

A cast here is dangerous, especially since it's virtually undocumented. I understand the necessity is to create arrays of short, float etc. from literals. But we can't afford to allow innocent syntax like [1, 2, 3].asStatic!Object to disastrously pass.

I'll "ask for changes" on account of this.

I recall you had a variant asStaticCast that took care of this. We may need to return to that, but probably we could find a better idea. We could special-case numbers at most, but not allow unchecked casts. Another possibility is to use to.

This is the kind of decision that can be deferred - I suggest you remove this overload entirely, get this in, then add something later.

[andralex]

same discussion of nothrow

[andralex]

DEFINITELY no cast here

[andralex]

This is definitely unsatisfactory. Code shouldn't rely on bounds checking to be correct (see https://dlang.org/spec/arrays.html#bounds), and by the time the assert is even looked at the code has trampled well into infested territory.

Too short is not good, either, because there'd be portions of the array that are left uninitialized! This function is thoroughly problematic.

One solution I suggested to @n8sh: define an overload taking an out size_t informing the user on how many elements have been initialized, and initialize the rest to U.init. To address the other problem (input range too long), we could probably accept truncation. After all the user only seems to be interested in a fragment of the range. They could check the length externally if they cared. All told:

auto staticArray(Un : U[n], U, size_t n, T)(T a, out size_t filled)
{
    U[n] result = void;
    size_t i = 0;
    // same exception safety mechanism as above
    foreach (ref ai; a)
    {
        if (i == n) break;
        emplace(...);
    }
    filled = i;
    return result;
}

auto staticArray(Un : U[n], U, size_t n, T)(auto ref T a)
{
    size_t unused;
    return staticArray!(Un, U, n, T)(a, unused);
}

[andralex]

Forgot to mention, some ranges return an rvalue from front so using foreach is problematic with those. It's simplest to use for.

[dukc]

@andralex to speed things up, please tell us that once those are resolved, do you want to recheck the code yourself, or is it enough for the other reviewers to accept it?

[andralex]

@dukc I'd like to take a last look, but the more good reviews the better.

[andralex]

@timotheecour why the closure? this was a few touches away from merging.

[timotheecour]

I may reopen if I find some time later, but right now I don't have time so I don't want to pollute the PR queue

[andralex]

@timotheecour cool, thanks. Would you allow someone else to take it over? Perhaps @dukc ?

[dukc]

I was actually considering. I also have more work now than before, but not so much that it's out of question. If this is still waiting the next time I decide to do something, I'll take this. And, of course, if Timothee doesn't want others to do it, I'll resubmit range.array!length.

[andralex]

@dukc thanks!

[timotheecour]

@andralex @dukc

@timotheecour cool, thanks. Would you allow someone else to take it over? Perhaps @dukc ?

of course