python: Future object cannot be converted to PyBytes

[gaby]

I'm trying to write a PoC for Async Reading/Writing files using opendal python bindings and ran into an error when trying to use the reader as an async context.

import asyncio
import opendal

async def main():
	op = opendal.AsyncOperator("fs", root="./git/test/opendal")

	async with op.open_reader("1g.img") as reader:
		await op.write("new.img", await reader.read(size=1024*1024))

asyncio.run(main())

Output:

$ python3 async-writter.py 
Traceback (most recent call last):
  File "/home/ubuntu/Desktop/git/test/opendal/async-writter.py", line 11, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/ubuntu/Desktop/git/test/opendal/async-writter.py", line 9, in main
    await op.write("new.img", reader.read(size=1024*1024))
TypeError: argument 'bs': 'Future' object cannot be converted to 'PyBytes'

How is this suppose to work? The docs state reader.read() returns bytes, and the Write() function also takes bytes. Somehow there's a conflict of types here for a simple example.

Also, it seems when running async with op.open_reader("1g.img") alone it reads the whole file into memory before doing anything.

Extra information:

$ python3 --version
Python 3.10.6
$ pip3 list | grep -i opendal
opendal                 0.38.0

[gaby]

Seems like I was missing await reader.read() :-)

[gaby]

The memory usage is still an issue, I'm loading a 1GB file and RAM usage goes from 1.89GB to 7.5GB. Is this something the new Sink writer would help, or is it a logic issue?

[gaby]

Normally the following would work with other libraries, but it also fails with the same error as above:

import asyncio
import opendal

async def stream(op):
	chunk = await op.read(size=1024*1024)

	while chunk:
		yield chunk
		chunk = await op.read(size=1024*1024)

async def main():
	op = opendal.AsyncOperator("fs", root="./git/test/opendal")

	async with op.open_reader("1g.img") as reader:
		await op.write("new.img", stream(reader))

asyncio.run(main())

Which makes sense since the async def stream() is an async_generator. So, I'm not really sure how do you Stream a file 1MB a time using OpenDAL, without reading the whole thing into memory?

[Xuanwo]

op.write does load all content in memory to construct to upload. We should expose Writer in python binding like what we do for Reader.

[gaby]

@Xuanwo That would defenitely help. I tried doing this for over 1hr and gave up. It either reads everything into memory or doesnt accept the generator 😂

[Xuanwo]

I tried doing this for over 1hr and gave up. It either reads everything into memory or doesnt accept the generator

Apologies for the inconvenience. We will work to resolve this issue as quickly as possible.

[gaby]

I tried doing this for over 1hr and gave up. It either reads everything into memory or doesnt accept the generator

Apologies for the inconvenience. We will work to resolve this issue as quickly as possible.

Thank you! Looking forward.

[gaby]

@Xuanwo Any update on this?

[Xuanwo]

@Xuanwo Any update on this?

Sorry for the late. We are working some internal refactor for bindings and we will add Writer after that.

[Xuanwo]

Hi, @gaby, we have finally resolved all the issues that were preventing us from exposing the writer in Python. We will begin by releasing opendal v0.40 and then proceed with the work on the Python side. I anticipate releasing this feature in either v0.41 or v0.40.1 (depends the changes we included).

[gaby]

@Xuanwo Thanks for the update, looking forward!

[Xuanwo]

File and AsyncFile has been added in opendal v0.42, let's close. Sorry for the long waiting:

async with await op.open(filename, "rb") as r:
    content = await r.read()