Using `#[AsConverter]` in with valinor bundle exceeds memory

[norkunas]

My code is running in cli, and i do not see proper stacktrace, here what i get only:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 12288 bytes) in /app/vendor/cuyz/valinor/src/Mapper/Tree/Builder/ValueConverterNodeBuilder.php on line 51
PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 32768 bytes) in /app/vendor/symfony/var-dumper/Cloner/Data.php on line 373

or

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 12288 bytes) in /app/vendor/cuyz/valinor/src/Mapper/Tree/Builder/ValueConverterNodeBuilder.php on line 56
PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 32768 bytes) in /app/vendor/symfony/var-dumper/Cloner/Data.php on line 373

[romm]

Hi @norkunas, I'll need more information to see what's going on here. Could you post the full backtrace, and maybe a reproducible example as well?

[norkunas]

Hey. This is all i am getting, even with --no-debug flag

[romm]

This is still not enough: I have no idea what your app does, and how/where it uses this library. Have you tried increasing the memory_limit setting?

[norkunas]

This command just locates the fixtures files, instantiates a mapper, then tries to map to objects with valinor.
Increasing memory limit of course is an option, but I strongly try avoid it.

[romm]

Well it could just be that there are many fixtures and using the mapper for each one in the same process is just too much for the memory you provided to PHP. I need to know if increasing the memory limit solves the issue, or if there could be an infinite loop somewhere.

Could also be bound to #667 — depending on the objects that are instantiated, a lot of useless information (≈methods unused by the mapper) could be put in memory but not used. We have to improve this part.

[norkunas]

I just map to readonly objects which don't have any methods. Will check with bigger memory limit

[norkunas]

Ok so it uses at least 888M, with bigger limit it works.
my loop does not use the objects at all, just instantiates them and continues

but the thing is that i have working code just without using #[AsConverter] but with custom ClosureModifier which allows to configure data normalization callbacks on properties, which I wanted to replace to #[AsConverter]

[romm]

Have you an estimation on the number of objects created?

Also, what is the memory usage when using ClosureModifier (I don't know what this is btw)?

[norkunas]

Have you an estimation on the number of objects created?

134122

Also, what is the memory usage when using ClosureModifier (I don't know what this is btw)?

Only 36M

I borrowed code from valinor for this:

/**
 * @implements IteratorAggregate<mixed>
 */
final class ClosureModifier implements IteratorAggregate
{
    /**
     * @var array<mixed>
     */
    private array $source;

    /**
     * @param iterable<mixed> $source
     * @param array<Closure>  $map
     */
    public function __construct(
        iterable $source,
        array $map,
    ) {
        $this->source = $this->replace($source, $this->prepareMappings($map));
    }

    /**
     * @param iterable<mixed>       $source
     * @param array<ClosureMapping> $mappings
     *
     * @return array<mixed>
     */
    private function replace(iterable $source, array $mappings, int $depth = 0): array
    {
        $out = [];

        foreach ($source as $key => $value) {
            /** @var int|string $key */
            $newMappings = array_filter($mappings, static fn (ClosureMapping $mapping) => $mapping->matches($key, $depth));

            $modifier = $this->findMapping($newMappings, $depth, $key);

            if (is_array($value)) {
                $out[$key] = $this->replace($value, $newMappings, $depth + 1);

                continue;
            }

            if ($modifier !== null) {
                $out[$key] = $modifier($value, $source);
            } else {
                $out[$key] = $value;
            }
        }

        return $out;
    }

    public function getIterator(): Traversable
    {
        yield from $this->source;
    }

    /**
     * @param array<Closure> $map
     *
     * @return list<ClosureMapping>
     */
    private function prepareMappings(array $map): array
    {
        $mappings = [];

        foreach ($map as $from => $to) {
            $mappings[] = new ClosureMapping(explode('.', (string) $from), $to);
        }

        return $mappings;
    }

    /**
     * @param array<ClosureMapping> $mappings
     */
    private function findMapping(array $mappings, int $atDepth, int|string $key): ?Closure
    {
        foreach ($mappings as $mapping) {
            $mappedKey = $mapping->findMappedKey($key, $atDepth);

            if ($mappedKey !== null) {
                return $mappedKey;
            }
        }

        return null;
    }
}

final class ClosureMapping
{
    /**
     * @var array<string>
     */
    private array $keys;

    private Closure $to;

    private int $depth;

    /**
     * @param array<string> $keys
     */
    public function __construct(array $keys, Closure $to)
    {
        $this->keys = $keys;
        $this->to = $to;
        $this->depth = count($keys) - 1;
    }

    public function matches(int|string $key, int $atDepth): bool
    {
        $from = $this->keys[$atDepth] ?? null;

        return $from === (string) $key || $from === '*';
    }

    public function findMappedKey(int|string $key, int $atDepth): ?Closure
    {
        if ($atDepth < $this->depth || !$this->matches($key, $atDepth)) {
            return null;
        }

        return $this->to;
    }
}

So I can use like this:

$mapper->map(MyObject::class, new ClosureModifier(new TrimModifier(new LineBreakModifier($data)), [
    'description' => static fn (mixed $value) => is_string($value) && $value !== '' ? $value : null,
]));

So my first basic use case is to implement EmptyStringToNull converter and later more advanced ones

[romm]

Alright, thanks for all the info, I should have enough to start searching for the leak!

[norkunas]

Thanks