Logging: cannot log on GCE

[Lance0404]

I have a web api deployed with docker image on GCP GCE. However failed to output any log to stackdriver logging under the tag which my VM stands for.

Most importantly, the same api deployed in my dev env (IDC) can sent log to GCP stackdriver logging successfully, which can be seen in stackdriver logging under the tag 'global'. Of course, with GOOGLE_APPLICATION_CREDENTIALS included.

Environment details

1. Python 3.6.7
2. ubuntu 18.04
3. Cloud API access scopes: Allow full access to all Cloud APIs
4. google related module

google-api-core==1.9.0
google-auth==1.6.3
google-cloud-core==0.29.1
google-cloud-logging==1.10.0
googleapis-common-protos==1.5.9

5. the service account creating this VM has the role Logs Writer added.

Steps to reproduce

1. deploy api with docker-compose within that GCE

Code example

• this is how i prepare my cloud handler

client = google.cloud.logging.Client()
cloud_handler = CloudLoggingHandler(client, name='cloud')
logging.getLogger('cc').addHandler(cloud_handler)

Really appreciate any support. Let me know if further info are needed.

[timfanda35]

You can try to set the log level.

I write a sample code to reproduce the problem:

import logging
import google
from google.cloud.logging.handlers.handlers import CloudLoggingHandler

client = google.cloud.logging.Client()
cloud_handler = CloudLoggingHandler(client, name='cloud')
logger = logging.getLogger('cc')
# logger.setLevel(logging.INFO)
logger.addHandler(cloud_handler)

logger.info("Hello World!")

Uncomment the # logger.setLevel(logging.INFO), and execute sample code.
You should see the logs in Stackdriver Logging global/cloud after seconds.

Reference:
https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#cloud-logging-handler

[Lance0404]

Hi @timfanda35 ,

Thank you so much for the fast reply and really sorry for my delayed response. I have modified my code in your way as:

client = google.cloud.logging.Client()
cloud_handler = CloudLoggingHandler(client, name='cloud')
# logging.getLogger('cc').addHandler(cloud_handler)
logger = logging.getLogger('cc')
logger.addHandler(cloud_handler)
logger.info('Hello World!')

However, yet to succeed.

The code locates in a file mylogging.py, which will be imported at the start of my flask app.
An easy way to test if my app is able to log onto stackdriver logging.

python manage.py shell

output message as below:

2019-04-23 02:01:33,090 - google.cloud.logging.handlers.transports.background_thread:background_thread:_thread_main:133 - DEBUG - 25 - - - Background thread started.
2019-04-23 02:01:33,090 - cc:mylogging:<module>:140 - INFO - 25 - - - Hello World!
2019-04-23 02:01:33,090 - cc:mylogging:<module>:140 - INFO - 25 - - - Hello World!
2019-04-23 02:01:33,120 - google.auth.transport.requests:requests:__call__:117 - DEBUG - 25 - - - Making request: POST https://oauth2.googleapis.com/token
2019-04-23 02:01:33,129 - urllib3.connectionpool:connectionpool:_new_conn:813 - DEBUG - 25 - - - Starting new HTTPS connection (1): oauth2.googleapis.com:443
2019-04-23 02:01:33,210 - urllib3.connectionpool:connectionpool:_make_request:393 - DEBUG - 25 - - - https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
2019-04-23 02:01:33,218 - flask.app:factory:create_app:16 - INFO - 25 - - - flask app is up by Lance!
2019-04-23 02:01:33,219 - flask.app:factory:create_app:30 - DEBUG - 25 - - - app.config['CLEAN_TABLE'] False
2019-04-23 02:01:33,301 - google.cloud.logging.handlers.transports.background_thread:background_thread:_safely_commit_batch:123 - DEBUG - 25 - - - Submitted 1 logs
2019-04-23 02:01:33,603 - flask.app:manage:<module>:24 - INFO - 25 - - - enter flask shell mode
>>>

From above, it seems like the code is functional or log shouldn't be shown as the format i designed. And it looks like some log was sent to gcp with status code as 200. But there is still no log shown in the stackdriver logging under the tag "GCE VM instance 'my instance name'".

How come it can be done in machine outside GCE, like my local machine, but not on GCE.

Thank you for the help.

[Lance0404]

Hi @timfanda35 ,

Sorry I didn't notice that you told me to check stackdriver logging under the tag 'global/cloud'. The log sure did show up under this tag. Thank you so much.

I was always looking under the tag "GCE VM instance 'my instance name'", which was linked directly from the GCE GUI.

Is there any way to direct the log to stackdriver logging "GCE VM instance 'my instance name'"? Which is more straightforward.

Thank you for the help.

[mnguyen25]

Hi @Lance0404, is there any update on this?

[Lance0404]

Hi @mnguyen25 , sorry that I don't have breakthrough for this, cause I changed to using AWS from this May due to my job changed. Thanks for asking. Would also be glad to know if someone figure it out.

[mnguyen25]

@Lance0404, lately today, I figured it out, still don't know how to distinguish between cloud env and local env however

from google.cloud.logging.resource import Resource
import requests

res = Resource(
	type="gce_instance",
	labels={
		"instance_id": requests.get("http://metadata/computeMetadata/v1/instance/id", headers={"Metadata-Flavor": "Google"}).text
        },
)

client = google.cloud.logging.Client()

handler = client.get_default_handler(resource=res)

This will fail locally cause the request would hang there and fail eventually.