-
Notifications
You must be signed in to change notification settings - Fork 57
Open
Description
Is your feature request related to a problem? Please describe.
Currently Linea can extract ("slice") source code required to produce some variable.
i.e. linea.slice("c") in
a = 2
b = 2
c = a + b
d = c / a
would produce
a = 2
b = 2
c = a + b
as a does not depend ond
However, one may want to output the entire code, including d
Describe the solution you'd like
"Slicing" the entire code would produce the code identical to the input, so what is the point?
In the trivial example above - none, however, leveraging additional capabilities of Linea we can produce useful outcomes:
- Airflow (Perfect. Argo, Ray ...) DAG encompassing the entire code
- Modularization for effectiveness and efficiency - we may output 2 separate functions producing
aandd, which can be run parallel, restarted/cache independently etc. - Replacing in-mem pandas object by distributed datasets (Dask, Spar, Ray ...) for scalability.
Metadata
Metadata
Assignees
Labels
No labels