Skip to content

feat: support batching#10

Merged
simosho merged 2 commits intowaylay_mainfrom
feat/support_batching
Sep 20, 2023
Merged

feat: support batching#10
simosho merged 2 commits intowaylay_mainfrom
feat/support_batching

Conversation

@simosho
Copy link
Copy Markdown

@simosho simosho commented Sep 14, 2023

  • support sending logs in batches to loki
  • drop support for lokiEmitterV0

note: test coverage still very limited.. But it's also tested E2E with a local loki server in https://github.com/waylayio/function-logger-py/pull/6 (batch_interval = 1)

@simosho simosho self-assigned this Sep 14, 2023

def __init__(self, interval: float, capacity: int = LOKI_MAX_BATCH_BUFFER_SIZE, **kwargs):
super().__init__(capacity, **kwargs)
self.interval = interval
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that in LokiQueueHandler the batch_interval is optional but when it gets propagated to LokiQueueHandler we don't consider interval as being optional. Maybe I'm missing something. My worries are related to (time.time() - self._last_flush_time >= self.interval) condition used in shouldFlush fn because we'll get a TypeError if self.interval is None.

Copy link
Copy Markdown
Author

@simosho simosho Sep 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if batch_interval:
self.handler = LokiBatchHandler(batch_interval, target=loki_handler)
else:
self.handler = loki_handler

In the LokiQueueHandler constructor there is check wether or not batch_interval is set? This should be sufficient, right?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I see this chunky code i'm gonna change it to use a ternary operator

Copy link
Copy Markdown
Member

@plankthom plankthom Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably too late for this library, but I would (here and elsewhere) only use dependency injection in the__init__ methods, and not create objects from properties (like the target handler in this case) .. the init should only be a factory method for the class itself, not its members ...

You can then still have other static builder/factory methods that create the objects and stitches them together.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love to add dependency injection as well, but since this repo is a fork, I just want to make minimal changes

@simosho simosho requested a review from a team as a code owner September 15, 2023 08:56
Copy link
Copy Markdown
Member

@plankthom plankthom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

self._last_flush_time = time.time()

def flush(self) -> None:
self.acquire()
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this acquired lock imply that all logs made during the flush are lost because of the

if not self._lock.acquire(blocking=False):
            return

in __call__?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NVM, it's a lock on another object ... (the emitter, not the handler)

Copy link
Copy Markdown
Author

@simosho simosho Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, the handler lock is to prevent the entire buffer of logs to be flushed/sent twice

The other lock used in the emitter is to prevent the emitter from creating an infinite loop:
when it's forwarding a log line to loki, we don't want to forward any newly created log lines in the emit methods itself (e.g. from urrlib/httpx/requests because of the request to loki)

@plankthom
Copy link
Copy Markdown
Member

Had a late response too #10 (comment)

@simosho simosho merged commit 1cfbdbe into waylay_main Sep 20, 2023
@simosho simosho deleted the feat/support_batching branch September 20, 2023 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants