Auth error: ETIMEDOUT

[jamesholcomb]

Environment details

Note: App is running external from GCP

• OS: Ubuntu 16.04.3 LTS
• Node.js version: 8.9.1
• npm version: 5.5.1
• @google-cloud/pubsub version: 0.14.5

Steps to reproduce

UNKNOWN

Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Auth error:Error: connect ETIMEDOUT 172.217.4.109:443
Dec 31, 2017 7:05 AM ERROR  { message: 'Getting metadata from plugin failed with error: con
nect ETIMEDOUT 172.217.4.109:443',
  stack: 'Error: Getting metadata from plugin failed with error: connect ETIMEDOUT 172.217.4.109:44
3\n    at ClientDuplexStream.onConnectionStatus (/home/jholcomb/ridealert.server/node_modules/@goog
le-cloud/pubsub/src/connection-pool.js:270:21)\n    at emitOne (events.js:116:13)\n    at ClientDup
lexStream.emit (events.js:211:7)\n    at ClientDuplexStream._emitStatusIfDone (/home/jholcomb/ridea
lert.server/node_modules/grpc/src/node/src/client.js:260:10)\n    at ClientDuplexStream._receiveSta
tus (/home/jholcomb/ridealert.server/node_modules/grpc/src/node/src/client.js:233:8)\n    at /home/
jholcomb/ridealert.server/node_modules/grpc/src/node/src/client.js:757:12',
  code: 16 }

This error occured after the server app had been connected and processing messages for several days. Is it expected that the pubsub lib automatically reconnects (eventually) or does a reconnect in this situation have to be handled by the client?

[stephenplusplus]

Sorry for the delay @jamesholcomb. @callmehiphop do you know?

[jamesholcomb]

On a possibly related note, I started getting this error after updating to 0.16.2.

Sending messages seems to continue working. But I have to restart the app in order to receive messages on subscriptions.

Jan 17, 2018 7:31 PM ERROR { message: 'Retry total timeout exceeded before anyresponse was received', stack: 'Error: Retry total timeout exceeded before anyresponse was received at repeat (/Users/jamesholcomb/code/ridealert.server/node_modules/google-gax/lib/api_callable.js:224:18) at Timeout._onTimeout (/Users/jamesholcomb/code/ridealert.server/node_modules/google-gax/lib/api_callable.js:256:13) at ontimeout (timers.js:475:11) at tryOnTimeout (timers.js:310:5) at Timer.listOnTimeout (timers.js:270:5)' }

[theacodes]

Related: googleapis/google-auth-library-python#211

This is due to Compute Engine's metadata server being occasionally unavailable. We should move this to the nodejs auth library and set a reasonable timeout and retry when talking to the GCE metadata server.

In the meantime, you can use a service account private key file as describe in cloud.google.com/docs/authentication/getting-started

[theacodes]

@stephenplusplus can you handle moving this to the appropriate place?

[stephenplusplus]

Hmm, I'm not sure which library of code is making the request, whether it's our handwritten layer, the gapic, or the auth library itself. @callmehiphop do you know?

[callmehiphop]

@stephenplusplus hard to say, when creating the gapic clients we attempt to get the project id from the auth library, after which all requests are made by the gapic. If this error occurs multiple times, I would think it was from the gapic since we cache the project id.

[stephenplusplus]

@jamesholcomb do you know the minimum required code to track down the method that is making the request that returns that error? Not so much concerned with reproducing, but just knowing which method is making the call.

[jamesholcomb]

@stephenplusplus I've only encountered this error once. The server where it was running did not have full debug/trace enabled at the time, so I don't have much to go on.

[ctavan]

@stephenplusplus @callmehiphop I have encountered a similar error (see below) while running my branch from #65 (which is based on current master). I'll base my branch on the v0.16.2 tag and test again.

Here's the error, I'm getting ECONNRESET instead of ETIMEDOUT:

{ Error: Getting metadata from plugin failed with error: connect ECONNRESET 169.254.169.254:80
    at /app/node_modules/grpc/src/client.js:554:15 code: 16, metadata: Metadata { _internal_repr: {} } } },
  id: '06f12d14-5870-4aea-ad1c-f76ccf5d17fe' }

[stephenplusplus]

This issue was moved to googleapis/google-auth-library-nodejs#283.