Problem with twitter example: bad credentials

[rjurney]

I am certain my twitter credentials are correct, having entered them multiple times to check. When I run the twitter example server, I get this as a result:

twitter

Your twitter user login name and pw is needed
twitter username? rjurney
twitter password? ls: /Users/rjurney/Software/druid/examples/bin/lib/druid-examples-*-selfcontained.jar: No such file or directory
Running command:

• java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=/Users/rjurney/Software/druid/examples/bin/examples/twitter/twitter_realtime.spec -classpath /Users/rjurney/Software/druid/examples/bin/examples/twitter:/Users/rjurney/Software/druid/examples/bin/../target/druid-examples-0.4.32.2-SNAPSHOT-selfcontained.jar:/Users/rjurney/Software/druid/examples/bin/../config::/Users/rjurney/Software/druid/examples/bin/config druid.examples.RealtimeStandaloneMain
  2013-06-12 08:30:04,344 INFO [main] com.metamx.druid.initialization.Initialization - Loading properties from runtime.properties
  2013-06-12 08:30:04,345 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.host] as [127.0.0.1]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.processing.numThreads] as [1]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.request.logging.dir] as [/tmp/example/log]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.service] as [example]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[com.metamx.emitter.logging] as [true]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.processing.buffer.sizeBytes] as [10000000]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.port] as [8080]
  2013-06-12 08:30:04,346 INFO [main] com.metamx.druid.initialization.Initialization - Loaded(runtime.properties) Property[druid.processing.formatString] as [processing_%s]
  2013-06-12 08:30:04,346 WARN [main] com.metamx.druid.initialization.Initialization - property[druid.zk.service.host] not set, skipping ZK-specified properties.
  2013-06-12 08:30:04,369 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public synchronized void com.metamx.druid.realtime.RealtimeNode.start() throws java.lang.Exception] on object[com.metamx.druid.realtime.RealtimeNode@538d7ace].
  2013-06-12 08:30:04,384 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [10000000] for [druid.processing.buffer.sizeBytes] on [com.metamx.druid.DruidProcessingConfig#intermediateComputeSizeBytes()]
  2013-06-12 08:30:04,386 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [1] for [druid.processing.numThreads] on [com.metamx.common.concurrent.ExecutorServiceConfig#getNumThreads()]
  2013-06-12 08:30:04,386 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [processing_%s] for [druid.processing.formatString] on [com.metamx.common.concurrent.ExecutorServiceConfig#getFormatString()]
  2013-06-12 08:30:04,454 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [50000] for [druid.query.groupBy.maxIntermediateRows] on [com.metamx.druid.query.group.GroupByQueryEngineConfig#getMaxIntermediateRows()]
  2013-06-12 08:30:04,456 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.query.groupBy.singleThreaded] on [com.metamx.druid.query.group.GroupByQueryRunnerFactoryConfig#isSingleThreaded()]
  2013-06-12 08:30:04,704 INFO [main] druid.examples.twitter.TwitterSpritzerFirehoseFactory - maxEventCount=500000
  2013-06-12 08:30:04,704 INFO [main] druid.examples.twitter.TwitterSpritzerFirehoseFactory - maxRunMinutes=120
  2013-06-12 08:30:04,732 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [PT60s] for [com.metamx.metrics.emitter.period, com.metamx.druid.emitter.period] on [com.metamx.metrics.MonitorSchedulerConfig#getEmitterPeriod()]
  2013-06-12 08:30:04,740 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [8080] for [druid.port] on [com.metamx.druid.initialization.ServerConfig#getPort()]
  2013-06-12 08:30:04,741 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [10] for [druid.http.numThreads] on [com.metamx.druid.initialization.ServerConfig#getNumThreads()]
  2013-06-12 08:30:04,741 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.http.maxIdleTimeMillis] on [com.metamx.druid.initialization.ServerConfig#getMaxIdleTimeMillis()]
  2013-06-12 08:30:04,749 INFO [main] org.mortbay.log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
  2013-06-12 08:30:04,792 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.HttpClient.start()] on object[com.metamx.http.client.HttpClient@785e8d7d].
  2013-06-12 08:30:04,794 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.core.LoggingEmitter.start()] on object[com.metamx.emitter.core.LoggingEmitter@25a41cc7].
  2013-06-12 08:30:04,794 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.druid.realtime.RealtimeManager.start() throws java.io.IOException] on object[com.metamx.druid.realtime.RealtimeManager@2151b0a5].
  2013-06-12 08:30:04,795 INFO [main] com.metamx.druid.realtime.RealtimeManager - Calling the FireDepartment and getting a Firehose.
  2013-06-12 08:30:04,835 INFO [main] druid.examples.twitter.TwitterSpritzerFirehoseFactory - returned from sample()
  2013-06-12 08:30:04,835 INFO [Twitter Stream consumer-1[initializing]] twitter4j.TwitterStreamImpl - Establishing connection.
  2013-06-12 08:30:04,837 INFO [main] com.metamx.druid.realtime.RealtimeManager - Firehose acquired!
  2013-06-12 08:30:04,837 INFO [main] com.metamx.druid.realtime.RealtimeManager - Someone get us a plumber!
  2013-06-12 08:30:04,838 INFO [main] com.metamx.druid.realtime.plumber.RealtimePlumberSchool - Creating plumber using rejectionPolicy[serverTime-PT3M]
  2013-06-12 08:30:04,839 INFO [main] com.metamx.druid.realtime.RealtimeManager - We have our plumber!
  2013-06-12 08:30:04,839 INFO [chief-twitterstream] com.metamx.druid.realtime.RealtimeManager - FireChief[twitterstream] state ok.
  2013-06-12 08:30:04,840 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public synchronized void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@52234265].
  2013-06-12 08:30:04,842 INFO [main] com.metamx.druid.realtime.RealtimeNode - Starting Jetty
  2013-06-12 08:30:04,842 INFO [main] org.mortbay.log - jetty-6.1.x
  2013-06-12 08:30:04,874 INFO [chief-twitterstream] com.metamx.druid.realtime.plumber.RealtimePlumberSchool - Expect to run at [2013-06-12T09:03:00.000Z]
  2013-06-12 08:30:04,932 INFO [main] org.mortbay.log - Started SelectChannelConnector@0.0.0.0:8080
  2013-06-12 08:30:04,933 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.druid.http.FileRequestLogger.start()] on object[com.metamx.druid.http.FileRequestLogger@ffeef1].
  2013-06-12 08:30:05,415 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - 401:Authentication credentials (https://dev.twitter.com/docs/auth) were missing or incorrect. Ensure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync. <title>Error 401 Unauthorized</title>

  HTTP ERROR: 401

  Problem accessing '/1/statuses/sample.json'. Reason:

      Unauthorized

2013-06-12 08:30:05,415 INFO [Twitter Stream consumer-1[Establishing connection]] druid.examples.twitter.TwitterSpritzerFirehoseFactory - Disconnect_from_Twitter
2013-06-12 08:30:05,415 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Waiting for 10000 milliseconds

[fjy]

I have a feeling this is because of Twitter retiring their v1 API and upgrading to v1.1. We will need to fix this.

[cheddar]

Hrm... yes. https://dev.twitter.com/blog/api-v1-is-retired

I wonder if they still provide the spritzer...

[fjy]

AFAIK they didn't mention anything about removing the spritzer.

On Wed, Jun 12, 2013 at 2:51 PM, cheddar notifications@github.com wrote:

Hrm... yes. https://dev.twitter.com/blog/api-v1-is-retired

I wonder if they still provide the spritzer...

—
Reply to this email directly or view it on GitHubhttps://github.com/metamx/druid/issues/161#issuecomment-19358118
.

[fjy]

http://blog.twitter4j.org/2013/06/12/33

On Wed, Jun 12, 2013 at 2:55 PM, Fangjin Yang fangjin@metamarkets.comwrote:

AFAIK they didn't mention anything about removing the spritzer.

On Wed, Jun 12, 2013 at 2:51 PM, cheddar notifications@github.com wrote:

Hrm... yes. https://dev.twitter.com/blog/api-v1-is-retired

I wonder if they still provide the spritzer...

—
Reply to this email directly or view it on GitHubhttps://github.com/metamx/druid/issues/161#issuecomment-19358118
.

[cheddar]

Hrm, yeah, it looks like if we just update the twitter4j dependency to 3.0 it'll start working again. Can you try that out?

[fjy]

We'll probably need to update the tutorial links as well.

[swang]

I don't think just updating to twitter4j 3.0 is going to work. Twitter no longer accepts basic-auth to access it's stream. You need to have an application only token: https://dev.twitter.com/docs/auth/application-only-auth

[rjurney]

I have updated to twitter4j 3.0.3 in this pull request. It is step 1 to fixing the problem, but did not itself fix the problem. We have to ask the user to go here https://twitter.com/oauth_clients/new and register a new app. This gives them 4 key values they enter into twitter4j.properties like so:

debug=true
oauth.consumerKey=*********************
oauth.consumerSecret=******************************************
oauth.accessToken=**************************************************
oauth.accessTokenSecret=******************************************

Then I think it should 'just work.' Testing now. The shell scripts need reworking.

See pull request 164: https://github.com/metamx/druid/pull/164

[rjurney]

I have resolved this issue with the new twitter4j.properties. New pull request coming, that one was borked.

[rjurney]

I have resolved this issue in https://github.com/metamx/druid/pull/165

Still need to update the scripts to ask for the keys, and explain what to do to get them.