Skip to content

prevent npe on mismatch between number of kafka partitions and task count #5139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pjain1 merged 1 commit into from
Dec 20, 2017
Merged

prevent npe on mismatch between number of kafka partitions and task count #5139

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ka-indexing-service/src/main/java/io/druid/indexing/kafka/supervisor/KafkaSupervisor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2117,7 +2117,7 @@ private Map<Integer, Long> getLagPerPartition(Map<Integer, Long> currentOffsets)
&& latestOffsetsFromKafka.get(e.getKey()) != null
&& e.getValue() != null
? latestOffsetsFromKafka.get(e.getKey()) - e.getValue()
: null
: Integer.MIN_VALUE
Copy link
Copy Markdown
Contributor

@jihoonson jihoonson Dec 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe is the below better because we don't have to add unnecessary lag values?

private Map<Integer, Long> getLagPerPartition(Map<Integer, Long> currentOffsets)
  {
    if (latestOffsetsFromKafka == null) {
      return ImmutableMap.of();
    }

    return currentOffsets
        .entrySet()
        .stream()
        .filter(e -> latestOffsetsFromKafka.get(e.getKey()) != null && e.getValue() != null)
        .collect(
            Collectors.toMap(
                Map.Entry::getKey,
                e -> latestOffsetsFromKafka.get(e.getKey()) - e.getValue()
            )
        );
  }

Copy link
Copy Markdown
Member Author

@pjain1 pjain1 Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't wanted to filter so that its visible that there is mismatch between task count and number of available Kafka partitions. Currently, whenever total lag is calculated, x -> Math.max(x, 0) is used so setting lag to Integer.MIN_VALUE won't add to the total.

Copy link
Copy Markdown
Member Author

@pjain1 pjain1 Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, if you prefer this then we can do this as well.

Copy link
Copy Markdown
Contributor

@jihoonson jihoonson Dec 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both looks good to me. If you think the current patch is better to check there is mismatch between task count and number of partitions, please go for it.

)
);
}
Expand Down