Add custom configuration to nginx/metrics

[toms-place]

Proposed commit message

This PR adds the possibility to add custom configuration to the nginx/metrics input.
This allows to add custom conditions to nginx metrics, in order to use http://${kubernetes.pod.ip} in the hosts field on fleet managed agents.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Discuss if this is also needed for logs

How to test this PR locally

Add ingress nginx to a k8s cluster, deploy a fleet agent on an ingress nginx node and scrape the metrics directly from the pod.

Related issues

With this PR the headless service is not needed anymore.

• Closes [NGINX]: DNS Lookup of multiple ingress pods via headless service #12746

Screenshots

[muthu-mps]

• Why do we apply the condition only for stubstatus datastream and not for the access and error logs?
• Is this applicable for Nginx ingress controller and for Nginx on Kubernetes? Because this condition is already applied for the Nginx_Ingress_Controller integration.
  

  integrations/packages/nginx_ingress_controller/data_stream/access/manifest.yml

  Line 47 in fc127eb

  - name: condition

[toms-place]

Hi @muthu-mps :)
Thanks for reviewing!

The nginx_ingress_contoller package only supports logs. It currently does not support metrics/stubstatus.
That's why I adapted the nginx package.

Adding the condition only is also an option, but I thought having more control by just using a custom field is appreciated.

I also have an open bullet point to discuss, if we want to have this for acces/error logs as well in the nginx package. What do you think?

[muthu-mps]

@toms-place - As the logs can be fetched using the nginx_ingress_controller integration, we can adopt the condition for the metrics.

[toms-place]

So do you want me to change it to condition only with an empty string as default , instead of using a custom field?

[muthu-mps]

As we try to fetch data by applying condition, we could probably change this to condition instead of custom input without applying default variable.

@toms-place - Can we try applying the config and process the data using stubstatus datastream to make sure that it process the expected metrics.

[toms-place]

I changed it to condition now ;)

[toms-place]

@muthu-mps can this be merged now? :)

[muthu-mps]

@toms-place - Sorry for the delay here, I would like to understand if this is going to be a redundant feature when we enable this data stream to nginx-ingress-controller integration. Will it be beneficial either way when used in both the integrations? Can we add the metrics data stream to the ingress-controller and adopt this change? WDYT?

[toms-place]

@muthu-mps - Ah I think the confusion here is that I am talking about metrics. the nginx-ingress-controller integration only handles logs atm.

[toms-place]

But it could be beneficial there as well I guess.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

[toms-place]

Please reopen it.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ishleenk17]

/test

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 0b3c314

Failed CI Steps

• :pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

[ishleenk17]

@toms-place : Would you like to resume it ? I would take it to completion.

[toms-place]

@toms-place : Would you like to resume it ? I would take it to completion.

@ishleenk17 - Yes, what is still left here?

[ishleenk17]

@toms-place : Would you like to resume it ? I would take it to completion.

@ishleenk17 - Yes, what is still left here?

Some merge conflict resolutions. Then we should be GTG.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[toms-place]

@toms-place : Would you like to resume it ? I would take it to completion.

@ishleenk17 - Yes, what is still left here?

Some merge conflict resolutions. Then we should be GTG.

All resolved, please review again :)

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!