Graphite Emitter : unable to receive metrics via Pickle protocol

[itaiy]

I'm using Druid version 0.9.1.1, and enabled the Graphite Emitter.
Metrics are sent, but do not appear in Graphite because apparently Druid metrics' names include a forward slash and a colon, which are both not allowed in Graphite.
Has anyone encountered this issue?

Some details :
I'm using a hosted Graphite service, and since the events are pickled, I've configured the port to 2004.
My Druid settings for this are :
druid.emitter=graphite
druid.emitter.graphite.hostname=
druid.emitter.graphite.port=2004
druid.emitter.graphite.eventConverter={"type":"all", "namespacePrefix": ".druid.test"}

I changed my Druid service log level to DEBUG, and it seems the data is actually sent to graphite, but doesn't appear there :
2016-09-29T08:29:07,327 INFO [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - trying to connect to graphite server
...
2016-09-29T08:29:07,434 DEBUG [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - sent [.druid.test.druid/coordinator.:8081.segment/overShadowed/count] with value [0] and time [1475137718]
...
2016-09-29T08:29:07,451 DEBUG [GraphiteEmitter-0] com.codahale.metrics.graphite.PickledGraphite - Wrote 43 metrics

[itaiy]

Just to test, I've replaced the current implementation of GraphiteEmitter.sanitize() with the following:

protected static String sanitize(String namespace)
{
    Pattern DOT_OR_WHITESPACE_OR_COLON = Pattern.compile("[\\s]+|[.]+|[:]+");
    String interimSanitizedStr = DOT_OR_WHITESPACE_OR_COLON.matcher(namespace).replaceAll("_");
    Pattern FORWARD_SLASH = Pattern.compile("[/]+");
    return FORWARD_SLASH.matcher(interimSanitizedStr).replaceAll(".");
}

And it seems to work as expected, which is - replacing ':' with '_' (e.g 127.0.0.1:8081 with 127_0_0_1_8081) and '/' with '.' (e.g segment/count with segment.count).
Before I continue working in that direction: @b-slim, @himanshug - you were both involved in adding #1978. Does the above change makes sense to you?

Thanks!

[b-slim]

@itaiy i don't think that's the issue. According to the carbon server doc . and should be replaced but don't think you need to replace/or: it is working for me and also plenty of other ppls. I am wondering if you have a configuration issue with the carbon server ?

[itaiy]

Thanks for the quick response, @b-slim!
As mentioned above, I'm using a hosted Graphite service, and they told me those characters are not allowed by Graphite.
I couldn't find something specific in Graphite's documentation, but here are some results from around the web that support this statement (at least for forward slash):

1. Graphite metrics should clean metrics names dropwizard/metrics#637
2. http://stackoverflow.com/questions/29674488/whitespace-in-statsd-metric-name (the title says "whitespace" but it also refers to slash in the content).
3. https://answers.launchpad.net/graphite/+question/171766
4. http://mingbowan.blogspot.co.il/2012/08/enable-special-character-support-in.html
   The last two links are also linked from the StackOverflow question mentioned above.

Given these results, I'm not sure how it even works...
Did you have to apply some specific configuration in your own Graphite to make it work?
Also - can you refer me to any specific documentation you've seen in regards to this issue?

Thanks again!

[b-slim]

@itaiy i get your point but i am pretty sure with recent version of carbon sever it worked (as you can see the module has been used by other as well).
now it might be that the hosted version won't accept / and : so i guess the best way to fix this is to inject the pattern of the sanatize regex as part of the config. This will ensure that it is at least backward compatible. Will be happy to see the changes and review the PR ! Thanks for reporting the issue and looking forward for the fix !

[b-slim]

btw / need to be replace by . thought ! since it will be part of metric path.

[itaiy]

Just wanted to update on this issue :
So the hosted-graphite guys did some more research, and there seems to be a lot of ambiguity around this issue.
Bottom line - they decided to automatically convert forward slashes to periods in metric names (for Pickle protocol only at the moment), since in practice Graphite allows forward slashes.
Colons work fine BTW (as opposed to what we initially thought).

As this is the case and considering @b-slim's comment, I'm closing this issue.

Thanks @b-slim for your answers!

[b-slim]

@itaiy great to hear that !