Skip to content

KAFKA-10289 fix failed connect_distributed_test.py (ConnectDistribute…#9673

Merged
hachikuji merged 1 commit intoapache:trunkfrom
chia7712:KAFKA-10289
Dec 9, 2020
Merged

KAFKA-10289 fix failed connect_distributed_test.py (ConnectDistribute…#9673
hachikuji merged 1 commit intoapache:trunkfrom
chia7712:KAFKA-10289

Conversation

@chia7712
Copy link
Copy Markdown
Member

@chia7712 chia7712 commented Dec 2, 2020

issue: https://issues.apache.org/jira/browse/KAFKA-10289

In Python 3, filter functions return iterators rather than list so it can traverse only once. Hence, the following loop only see "empty" and then validation fails.

        src_messages = self.source.committed_messages() # return iterator
        sink_messages = self.sink.flushed_messages()) # return iterator
        for task in range(num_tasks):
            # only first task can "see" the result. following tasks see empty result
            src_seqnos = [msg['seqno'] for msg in src_messages if msg['task'] == task]

reference: https://portingguide.readthedocs.io/en/latest/iterators.html#new-behavior-of-map-and-filter

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@chia7712
Copy link
Copy Markdown
Member Author

chia7712 commented Dec 6, 2020

@hachikuji Could you take a look? I have test this patch on my local. It pass.

command

docker rm -f ducker01 ducker02 ducker03 ducker04 ducker05 ducker06 ducker07 ducker08 ducker09 ducker10 ducker11 ducker12 ducker13 ducker14 \
; export _DUCKTAPE_OPTIONS="--subset 0 --subsets 2" \
&& TC_PATHS="tests/kafkatest/tests/connect/connect_distributed_test.py" /bin/bash tests/docker/run_tests.sh

result

SESSION REPORT (ALL TESTS)
ducktape version: 0.8.0
session_id:       2020-12-06--005
run time:         35 minutes 42.481 seconds
tests run:        30
passed:           30
failed:           0
ignored:          0

command

docker rm -f ducker01 ducker02 ducker03 ducker04 ducker05 ducker06 ducker07 ducker08 ducker09 ducker10 ducker11 ducker12 ducker13 ducker14 \
; export _DUCKTAPE_OPTIONS="--subset 1 --subsets 2" \
&& TC_PATHS="tests/kafkatest/tests/connect/connect_distributed_test.py" /bin/bash tests/docker/run_tests.sh

result

SESSION REPORT (ALL TESTS)
ducktape version: 0.8.0
session_id:       2020-12-06--003
run time:         29 minutes 57.771 seconds
tests run:        30
passed:           30
failed:           0
ignored:          0

Copy link
Copy Markdown
Contributor

@hachikuji hachikuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hachikuji hachikuji merged commit 6e15937 into apache:trunk Dec 9, 2020
kkonstantine pushed a commit that referenced this pull request Jan 15, 2021
…edTest.test_bounce) (#9673)

In Python 3, `filter` functions return iterators rather than `list` so it can traverse only once. Hence, the following loop will only see "empty" and then validation fails.

```python
        src_messages = self.source.committed_messages() # return iterator
        sink_messages = self.sink.flushed_messages()) # return iterator
        for task in range(num_tasks):
            # only first task can "see" the result. following tasks see empty result
            src_seqnos = [msg['seqno'] for msg in src_messages if msg['task'] == task]
```

Reference: https://portingguide.readthedocs.io/en/latest/iterators.html#new-behavior-of-map-and-filter.

Reviewers: Jason Gustafson <jason@confluent.io>
Gerrrr pushed a commit to confluentinc/kafka that referenced this pull request Mar 21, 2022
…edTest.test_bounce) (apache#9673)

In Python 3, `filter` functions return iterators rather than `list` so it can traverse only once. Hence, the following loop will only see "empty" and then validation fails.

```python
        src_messages = self.source.committed_messages() # return iterator
        sink_messages = self.sink.flushed_messages()) # return iterator
        for task in range(num_tasks):
            # only first task can "see" the result. following tasks see empty result
            src_seqnos = [msg['seqno'] for msg in src_messages if msg['task'] == task]
```

Reference: https://portingguide.readthedocs.io/en/latest/iterators.html#new-behavior-of-map-and-filter.

Reviewers: Jason Gustafson <jason@confluent.io>
Gerrrr added a commit to confluentinc/kafka that referenced this pull request Mar 22, 2022
…edTest.test_bounce) (apache#9673) (#687)

In Python 3, `filter` functions return iterators rather than `list` so it can traverse only once. Hence, the following loop will only see "empty" and then validation fails.

```python
        src_messages = self.source.committed_messages() # return iterator
        sink_messages = self.sink.flushed_messages()) # return iterator
        for task in range(num_tasks):
            # only first task can "see" the result. following tasks see empty result
            src_seqnos = [msg['seqno'] for msg in src_messages if msg['task'] == task]
```

Reference: https://portingguide.readthedocs.io/en/latest/iterators.html#new-behavior-of-map-and-filter.

Reviewers: Jason Gustafson <jason@confluent.io>

Co-authored-by: Chia-Ping Tsai <chia7712@gmail.com>
@chia7712 chia7712 deleted the KAFKA-10289 branch March 25, 2024 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants