Skip to content

KAFKA-16684: Remove cache in responseData #16532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
soarez merged 5 commits into from
Jul 9, 2024
Merged

KAFKA-16684: Remove cache in responseData #16532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e32700b
remove FetchResponse cache
m1a2st Jul 5, 2024
e396bcd
add new test for fetcher
m1a2st Jul 5, 2024
2eecce2
simplify cache test
m1a2st Jul 6, 2024
a3ace40
change to use Collections.emptyMap()
m1a2st Jul 7, 2024
12784e1
assert size first
m1a2st Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 12 additions & 24 deletions clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,6 @@ public class FetchResponse extends AbstractResponse {
public static final int INVALID_PREFERRED_REPLICA_ID = -1;

private final FetchResponseData data;
// we build responseData when needed.
private volatile LinkedHashMap<TopicPartition, FetchResponseData.PartitionData> responseData = null;

@Override
public FetchResponseData data() {
Expand All @@ -99,29 +97,19 @@ public Errors error() {
}

public LinkedHashMap<TopicPartition, FetchResponseData.PartitionData> responseData(Map<Uuid, String> topicNames, short version) {
if (responseData == null) {
synchronized (this) {
if (responseData == null) {
// Assigning the lazy-initialized `responseData` in the last step
// to avoid other threads accessing a half-initialized object.
final LinkedHashMap<TopicPartition, FetchResponseData.PartitionData> responseDataTmp =
new LinkedHashMap<>();
data.responses().forEach(topicResponse -> {
String name;
if (version < 13) {
name = topicResponse.topic();
} else {
name = topicNames.get(topicResponse.topicId());
}
if (name != null) {
topicResponse.partitions().forEach(partition ->
responseDataTmp.put(new TopicPartition(name, partition.partitionIndex()), partition));
}
});
responseData = responseDataTmp;
}
final LinkedHashMap<TopicPartition, FetchResponseData.PartitionData> responseData = new LinkedHashMap<>();
data.responses().forEach(topicResponse -> {
String name;
if (version < 13) {
name = topicResponse.topic();
} else {
name = topicNames.get(topicResponse.topicId());
}
}
if (name != null) {
topicResponse.partitions().forEach(partition ->
responseData.put(new TopicPartition(name, partition.partitionIndex()), partition));
}
});
return responseData;
}

Expand Down
17 changes: 17 additions & 0 deletions clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2884,6 +2884,11 @@ public void handleError(Throwable t) {
verifySessionPartitions();
handler.handleError(t);
}

@Override
public Map<Uuid, String> sessionTopicNames() {
return handler.sessionTopicNames();
}

// Verify that session partitions can be traversed safely.
private void verifySessionPartitions() {
Expand Down Expand Up @@ -3665,6 +3670,18 @@ public void testWhenFetchResponseReturnsALeaderShipChangeErrorAndNewLeaderInform
// Validate subscription is still valid & fetch-able for tp1.
assertTrue(subscriptions.isFetchable(tp1));
}

@Test
public void testFetcherDontCacheAnyData() {
Copy link
Copy Markdown
Member

@chia7712 chia7712 Jul 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is good but it's gone too far. Maybe we can create a FetchResponse and then test the method responseData?

Copy link
Copy Markdown
Collaborator Author

@m1a2st m1a2st Jul 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will simplify this test

short version = 17;
Copy link
Copy Markdown
Contributor

@apoorvmittal10 apoorvmittal10 Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any existing test where version < 13? If not then can we please add one.

Copy link
Copy Markdown
Collaborator Author

@m1a2st m1a2st Jul 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apoorvmittal10, Thanks for your comments, In testFetchWithNoTopicId have been test for version 12, Should I add a test only for FetchResponse#responseData this method to test version 12? WDYT

Copy link
Copy Markdown
Member

@soarez soarez Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, org.apache.kafka.clients.consumer.internals.FetcherTest#testFetchWithNoTopicId is testing org.apache.kafka.common.requests.FetchResponse#responseData with version = 12, so I don't think we need a new test.

Copy link
Copy Markdown
Contributor

@apoorvmittal10 apoorvmittal10 Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thansk for looking into, if there already exists a test then we can skip new one.

FetchResponse fetchResponse = fetchResponse(tidp0, records, Errors.NONE, 100L, -1L, 0L, 0);
LinkedHashMap<TopicPartition, FetchResponseData.PartitionData> responseData = fetchResponse.responseData(topicNames, version);
assertEquals(topicNames.size(), responseData.size());
responseData.forEach((topicPartition, partitionData) -> assertEquals(records, partitionData.records()));
LinkedHashMap<TopicPartition, FetchResponseData.PartitionData> nonResponseData = fetchResponse.responseData(emptyMap(), version);
assertEquals(emptyMap().size(), nonResponseData.size());
nonResponseData.forEach((topicPartition, partitionData) -> assertEquals(MemoryRecords.EMPTY, partitionData.records()));
Copy link
Copy Markdown
Member

@FrankYang0529 FrankYang0529 Apr 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is never executed. We can remove this assertion.

Copy link
Copy Markdown
Member

@chia7712 chia7712 Apr 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice find

Copy link
Copy Markdown
Collaborator

@TaiJuWu TaiJuWu Apr 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can also replace emptyMap().size() with 0.

Copy link
Copy Markdown
Contributor

@Parkerhiphop Parkerhiphop Apr 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've opened a minor PR #19376 to remove it.

Thanks @FrankYang0529 for pointing it out!

}

private OffsetsForLeaderEpochResponse prepareOffsetsForLeaderEpochResponse(
TopicPartition topicPartition,
Expand Down