[azure-storage-file-datalake] Checking for a non-existent file induces memory leak

[hallmeier]

• Package Name: azure-storage-file-datalake
• Package Version: 12.23.0 (current)
• Operating System: Linux and Darwin
• Python Version: 3.14.3 (current)

Describe the bug
When the existence of a non-existent file is checked with the Azure file client, objects in parent scopes are not garbage collected anymore.

To Reproduce
I reproduced this on Linux machines running in Azure Kubernetes and a Mac both in and outside of a Docker container. MWE:

import os

from azure.storage.filedatalake import FileSystemClient
from time import sleep

account_name = os.environ["ACCOUNT_NAME"]
file_system_name = os.environ["FILE_SYSTEM_NAME"]
sas_token = os.environ["SAS_TOKEN"]
directory_name = os.environ["DIRECTORY_NAME"]
file_name = os.environ["FILE_NAME"]


file_system_client = FileSystemClient(account_url=f"https://{account_name}.dfs.core.windows.net",
                                      file_system_name=file_system_name, credential=sas_token)
directory_client = file_system_client.get_directory_client(directory_name)


def check_file():
    file_client = directory_client.get_file_client(file_name)
    file_client.exists()
    big_list = list(range(10000000))


i = 0
while True:
    i += 1
    print(f"Iteration {i}")
    check_file()
    sleep(0.5)

This fills about 1 GB of memory every 5 iterations, eventually leading to an out-of-memory error.

Expected behavior
This should be able to run forever, as it is when the file existence check is omitted or the existence of an existent file is checked.

[weirongw23-msft]

Hi @hallmeier, thanks for reaching out. We've tested the script provided. We observed the memory may increase per iteration, but it ultimately goes down after the Python garbage collector runs every 5-10 iterations. It is highly unlikely that calling exists() will affect memory usage of the code outside of the SDK. For further investigations, please provide detailed profiler results showing a memory leak.

[hallmeier]

Hm, very puzzling. I have yet to find a scenario where I can not reproduce the memory overflow. I don't understand it either, but it's definitely real and causes problems in my application. I try to provide some more information. My dockerfile:

FROM python:3.14

ENV PYTHONUNBUFFERED=1
ENV PIP_NO_CACHE_DIR=1

RUN pip install azure-storage-file-datalake
COPY ./fill_memory.py /fill_memory.py

I simplified the original script by removing the directory_client, set the file_name to "missingfile", and tracked the maximum resident set size:

import os

from azure.storage.filedatalake import FileSystemClient
from resource import getrusage, RUSAGE_SELF
from time import sleep

account_name = os.environ["ACCOUNT_NAME"]
file_system_name = os.environ["FILE_SYSTEM_NAME"]
sas_token = os.environ["SAS_TOKEN"]
file_name = "missingfile"


file_system_client = FileSystemClient(account_url=f"https://{account_name}.dfs.core.windows.net",
                                      file_system_name=file_system_name, credential=sas_token)


def check_file():
    file_client = file_system_client.get_file_client(file_name)
    file_client.exists()
    big_list = list(range(10000000))


i = 0
while True:
    i += 1
    print(f"Iteration {i}")
    check_file()
    maxrss = getrusage(RUSAGE_SELF).ru_maxrss
    print(f"Max RSS: {maxrss}")
    sleep(0.5)

I limit Docker to 8 GB of memory and run the script with docker run --rm --env-file .env fill_memory python -m fill_memory.py:

Iteration 1
Max RSS: 444848
Iteration 2
Max RSS: 836940
Iteration 3
Max RSS: 1228864
Iteration 4
Max RSS: 1620896
Iteration 5
Max RSS: 2012828
Iteration 6
Max RSS: 2404760
Iteration 7
Max RSS: 2796888
Iteration 8
Max RSS: 3188808
Iteration 9
Max RSS: 3580680
Iteration 10
Max RSS: 3972604
Iteration 11
Max RSS: 4364668
Iteration 12
Max RSS: 4756604
Iteration 13
Max RSS: 5148520
Iteration 14
Max RSS: 5540964
Iteration 15
Max RSS: 5540964
Iteration 16
Max RSS: 5540964
Iteration 17
Max RSS: 5540964
Iteration 18
Max RSS: 5540964
Iteration 19
Max RSS: 5540964
Iteration 20
Max RSS: 5540964
Iteration 21
Max RSS: 5540964
Iteration 22
Max RSS: 5540964
Iteration 23
Max RSS: 5540964
Iteration 24
Max RSS: 5540964
Iteration 25
Max RSS: 5540964
Iteration 26
Max RSS: 5540964
Iteration 27
Max RSS: 5540964
Iteration 28
Max RSS: 5541048
Iteration 29
Max RSS: 5932924
Iteration 30
Max RSS: 6324792
Iteration 31
Max RSS: 6716668
Iteration 32
Max RSS: 7108544
Iteration 33
Max RSS: 7453332
Iteration 34
Max RSS: 7453332
Iteration 35
Max RSS: 7457932
Iteration 36
Max RSS: 7613432
Iteration 37

echo $? shows exit code 137. When I omit file_client.exists(), it just looks like this:

Iteration 1
Max RSS: 442024
Iteration 2
Max RSS: 442060
Iteration 3
Max RSS: 442080
Iteration 4
Max RSS: 442104
Iteration 5
Max RSS: 442112
[...]

Here I add some tracemalloc:

i = 0
tracemalloc.start()
old_snapshot = tracemalloc.take_snapshot()
while True:
    i += 1
    print(f"Iteration {i}")
    check_file()
    snapshot = tracemalloc.take_snapshot()
    comparison = snapshot.compare_to(old_snapshot, "lineno")
    maxrss = getrusage(RUSAGE_SELF).ru_maxrss
    print(f"Max RSS: {maxrss}")
    print("Top 3 differences:")
    for diff in comparison[:3]:
        print(diff)
    old_snapshot = snapshot
    sleep(0.5)

Iteration 1
Max RSS: 3410004
Top 3 differences:
/fill_memory.py:21: size=381 MiB (+381 MiB), count=9999743 (+9999743), average=40 B
<frozen importlib._bootstrap_external>:511: size=62.7 KiB (+62.7 KiB), count=555 (+555), average=116 B
/usr/local/lib/python3.14/site-packages/azure/core/pipeline/policies/_retry.py:93: size=48.6 KiB (+48.6 KiB), count=6 (+6), average=8300 B
Iteration 2
Max RSS: 4282436
Top 3 differences:
/usr/local/lib/python3.14/tracemalloc.py:115: size=30.8 KiB (+30.8 KiB), count=358 (+358), average=88 B
/usr/local/lib/python3.14/tracemalloc.py:129: size=25.4 KiB (+25.4 KiB), count=361 (+361), average=72 B
/usr/local/lib/python3.14/tracemalloc.py:193: size=19.5 KiB (+19.5 KiB), count=356 (+356), average=56 B
Iteration 3
Max RSS: 7579384
Top 3 differences:
/fill_memory.py:21: size=763 MiB (+381 MiB), count=19999492 (+9999745), average=40 B
/usr/local/lib/python3.14/site-packages/azure/core/pipeline/policies/_retry.py:93: size=97.3 KiB (+48.6 KiB), count=12 (+6), average=8300 B
/usr/local/lib/python3.14/tracemalloc.py:193: size=60.3 KiB (+40.9 KiB), count=1103 (+747), average=56 B
Iteration 4

And exit code 137. tracemalloc contributes to filling up the memory. Without file_client.exists():

Iteration 1
Max RSS: 1485756
Top 3 differences:
/usr/local/lib/python3.14/urllib/parse.py:889: size=4312 B (+4312 B), count=2 (+2), average=2156 B
/fill_memory.py:20: size=3224 B (+3224 B), count=100 (+100), average=32 B
/usr/local/lib/python3.14/site-packages/azure/storage/filedatalake/_generated/_utils/serialization.py:1387: size=1664 B (+1664 B), count=4 (+4), average=416 B
Iteration 2
Max RSS: 1490984
Top 3 differences:
/usr/local/lib/python3.14/tracemalloc.py:115: size=5368 B (+5368 B), count=61 (+61), average=88 B
/usr/local/lib/python3.14/tracemalloc.py:129: size=4680 B (+4680 B), count=65 (+65), average=72 B
/usr/local/lib/python3.14/tracemalloc.py:193: size=3640 B (+3640 B), count=65 (+65), average=56 B
Iteration 3
Max RSS: 1491092
Top 3 differences:
/usr/local/lib/python3.14/tracemalloc.py:193: size=11.3 KiB (+7952 B), count=207 (+142), average=56 B
/usr/local/lib/python3.14/tracemalloc.py:125: size=4608 B (+4608 B), count=64 (+64), average=72 B
/usr/local/lib/python3.14/tracemalloc.py:129: size=936 B (-3744 B), count=13 (-52), average=72 B
Iteration 4
Max RSS: 1491148
Top 3 differences:
/usr/local/lib/python3.14/tracemalloc.py:193: size=14.3 KiB (+3080 B), count=262 (+55), average=56 B
/usr/local/lib/python3.14/site-packages/azure/storage/filedatalake/_generated/_utils/serialization.py:1387: size=6656 B (+1664 B), count=16 (+4), average=416 B
/usr/local/lib/python3.14/site-packages/azure/storage/filedatalake/_generated/_utils/serialization.py:529: size=6656 B (+1664 B), count=16 (+4), average=416 B
Iteration 5
Max RSS: 1491244
Top 3 differences:
/usr/local/lib/python3.14/site-packages/azure/storage/filedatalake/_generated/_utils/serialization.py:1387: size=8320 B (+1664 B), count=20 (+4), average=416 B
/usr/local/lib/python3.14/site-packages/azure/storage/filedatalake/_generated/_utils/serialization.py:529: size=8320 B (+1664 B), count=20 (+4), average=416 B
/usr/local/lib/python3.14/site-packages/azure/storage/blob/_generated/_utils/serialization.py:1387: size=7920 B (+1584 B), count=10 (+2), average=792 B

I've also just made sure the memory overflow can still be reproduced in Azure Kubernetes. Is there any more information I can help you with?

[nateprewitt]

Hi @hallmeier, we have some leads on what's potentially happening. I have a few clarifying questions though.

• Is the case you hit this with also using highly numerical data (ints or floats)?
• If you add a forced gc collection (gc.collect()) between calls, do you still see growth or is Python able to free the memory?
• How much memory does the docker container that's going OOM have available?

Once we have those answered, we should be able to confirm the current hypothesis. Thanks!

[hallmeier]

Hi, thank you for investigating.

• Yes, in my application most of the occupied memory comes from lists of floats.
• This does help, memory does not grow anymore. Great idea, I didn't think of this.
• I gave it 8 GB to exceed the limit faster, but if I set it to 32 GB or run the script outside a container, the OOM also happens eventually.

[nateprewitt]

Thanks for the quick response! That's really helpful. I'll pass that on to Python team but I think we've got a good idea of what's happening now.