Fixing SES backend - source (mail_from) must be string. #16166

satishmittal111 · 2021-05-30T06:22:44Z

The Source in send_raw_email must be a String type. mail_from=None is failing the SES backend for airflow alerts.

Currently SES Backend is failing with error -

  File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/models/taskinstance.py", line 1852, in email_alert
    send_email(self.task.email, subject, html_content)
  File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/utils/email.py", line 52, in send_email
    return backend(
  File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/providers/amazon/aws/utils/emailer.py", line 39, in send_email
    hook.send_email(
  File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/providers/amazon/aws/hooks/ses.py", line 95, in send_email
    return ses_client.send_raw_email(
  File "/Users/username/Library/Python/3.8/lib/python/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  ......
  File "/Users/username/Library/Python/3.8/lib/python/site-packages/botocore/validate.py", line 293, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Source, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Refer - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ses.html#SES.Client.send_raw_email

Bug fixing for SES backend for the emails sent by Airflow.

You can see the information about how to configure email backend with AWS SES: [https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html]
Here is the class you should modify: airflow/providers/amazon/aws/utils/emailer.py
Tests for email class are in: tests/providers/amazon/aws/utils/test_emailer.py
Run the tests in Local Virtualenv: LOCAL_VIRTUALENV.rst
Run the tests in Breeze: BREEZE.rst
Run airflow webserver and scheduler and configure it to send emails and connect to a real AWS account and test if the email is sent with the SES Backend

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

The Source in send_raw_email must be a String type. mail_from=None is failing the SES backend for airflow alerts. Error - File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/models/taskinstance.py", line 1852, in email_alert send_email(self.task.email, subject, html_content) File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/utils/email.py", line 52, in send_email return backend( File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/providers/amazon/aws/utils/emailer.py", line 39, in send_email hook.send_email( File "/Users/username/Library/Python/3.8/lib/python/site-packages/airflow/providers/amazon/aws/hooks/ses.py", line 95, in send_email return ses_client.send_raw_email( File "/Users/username/Library/Python/3.8/lib/python/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/Users/username/Library/Python/3.8/lib/python/site-packages/botocore/client.py", line 648, in _make_api_call request_dict = self._convert_to_request_dict( File "/Users/username/Library/Python/3.8/lib/python/site-packages/botocore/client.py", line 696, in _convert_to_request_dict request_dict = self._serializer.serialize_to_request( File "/Users/username/Library/Python/3.8/lib/python/site-packages/botocore/validate.py", line 293, in serialize_to_request raise ParamValidationError(report=report.generate_report()) botocore.exceptions.ParamValidationError: Parameter validation failed: Invalid type for parameter Source, value: None, type: <class 'NoneType'>, valid types: <class 'str'> Refer - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ses.html#SES.Client.send_raw_email

satishmittal111 · 2021-05-30T07:04:12Z

airflow/providers/amazon/aws/utils/emailer.py

    """Email backend for SES."""
    hook = SESHook(aws_conn_id=conn_id)
+
+    mail_from = conf.get('smtp', 'SMTP_MAIL_FROM')


Please guide if this is right approach or should we update the EmailOperator to have input for "from".

It looks good. Can you add tests to avoid regression?

Thanks @mik-laj, I will update the Pr with tests.

@satishmittal111 Are you able to update this PR with tests?

uranusjr · 2021-05-30T13:32:37Z

I think we should just require the user to pass in email_from manually instead.

satishmittal111 · 2021-05-30T14:32:32Z

@uranusjr currently we have smtp, sendgrid and aws SES as airflow backend. Sendgrid and smtp backend are using SENDGRID_MAIL_FROM and SMTP_MAIL_FROM environment variables respectively for "from" email. Should we create environment variable SES_MAIL_FROM for this (as per pattern)? cc - @mik-laj

potiuk · 2021-07-03T11:36:49Z

@uranusjr currently we have smtp, sendgrid and aws SES as airflow backend. Sendgrid and smtp backend are using SENDGRID_MAIL_FROM and SMTP_MAIL_FROM environment variables respectively for "from" email. Should we create environment variable SES_MAIL_FROM for this (as per pattern)? cc - @mik-laj

While this is likely not perfect, I think it's better to make it consistent.

This is what sendgrid uses currently:

from_email = kwargs.get('from_email') or os.environ.get('SENDGRID_MAIL_FROM')
from_name = kwargs.get('from_name') or os.environ.get('SENDGRID_MAIL_SENDER')**

It allows for both kwargs and environment variables set, but with standard emails sent by airflow (alert/sla) the kwargs are not used anyway. However the documentation in https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/email-config.rst does not mention those env vars. It mentions few other '[email]' options but it does not mention any of the [smtp] options which are used by the standard "send_email_smtp" backend.

[smtp]
smtp_user = airflow
smtp_password = airflow
smtp_mail_from = airflow@example.com

So we have a bit of a mess to be honest. Proposal:

How about making it consistent and update the documentation at the same time between the three backends.

For all backends we could use:

SMTP:

from_email = kwargs.get('from_email') or os.environ.get('SMTP_MAIL_FROM') or conf.get('smtp', 'smtp_mail_from')
from_name = kwargs.get('from_name') or os.environ.get('SMTP_MAIL_SENDER') or conf.get('smtp', 'smtp_mail_sender')

SENDGRID:

from_email = kwargs.get('from_email') or os.environ.get('SENDGRID_MAIL_FROM') or conf.get('sendgrid', 'sendgrid_mail_from')
from_name = kwargs.get('from_name') or os.environ.get('SENDGRID_MAIL_SENDER') or conf.get('sendgrid', 'sendgrid_mail_sender')

SES:

from_email = kwargs.get('from_email') or os.environ.get('SES_MAIL_FROM') or conf.get('ses', 'ses_mail_from')
from_name = kwargs.get('from_name') or os.environ.get('SES_MAIL_SENDER') or conf.get('ses', 'ses_mail_sender')

And update the email documentation about it (also mentioning that you need the corresponding providers to use it?

potiuk · 2021-07-10T10:59:10Z

Hey @satishmittal111 - are you going to improve the PR ? I am preparing to release the next wave of providers and it would be great to get it merged before.

potiuk · 2021-07-11T21:17:40Z

hello?

satishmittal111 · 2021-07-12T02:50:29Z

@potiuk Sorry for the late response. The proposal looks good. Changes -

Update documentation with Env Variables.
Add Env Variables for SES backend.

I will be able to complete the PR by end of this week. Will it work for you?

potiuk · 2021-07-12T06:52:23Z

I will be able to complete the PR by end of this week. Will it work for you?

Sure. I plan to release providers before, but I can also be released separately/next time :)

jzuhusky · 2021-07-19T20:57:48Z

@potiuk @satishmittal111 @mik-laj

Any updates here? I just ran into this issue. Seems like the fix is done? Any idea when this might be released?

potiuk · 2021-07-23T19:02:17Z

Not yet - seems that @satishmittal111 has to complete it yet :)

ignaski · 2021-09-02T08:29:22Z

This PR seems to be stuck :(

potiuk · 2021-09-02T08:37:08Z

This PR seems to be stuck :(

Maybe you would like to take it over @ignaski ? Seems the solution is there, only some tests need to be added.

uranusjr · 2021-09-02T08:37:09Z

Since we’re very close to 2.2 now, anyone interested please feel free to open a separate PR inheriting works in here. I’m also adding the pending-response label so @satishmittal111 please respond if you are able to finish this soon.

ignaski · 2021-09-02T11:07:05Z

This PR seems to be stuck :(

Maybe you would like to take it over @ignaski ? Seems the solution is there, only some tests need to be added.

I will try to. I've never contributed to an open source project, thus it will take time to setup and get familiar.

satishmittal111 · 2021-09-02T11:26:33Z

Apologies. I was not able to take it further due to personal priority. Please take over the changes.

github-actions · 2021-10-19T00:06:28Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

uranusjr · 2021-10-19T00:40:48Z

Continuing in #18042.

boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels May 30, 2021

satishmittal111 mentioned this pull request May 30, 2021

Fixing SES backend - source (From) must be string. #16165

Closed

satishmittal111 commented May 30, 2021

View reviewed changes

satishmittal111 requested a review from mik-laj May 30, 2021 14:44

potiuk mentioned this pull request Jul 3, 2021

Airflow module for email backend using AWS SES not resolving mail_from parameter from airflow config and is defaulted as None #16778

Closed

potiuk referenced this pull request Jul 3, 2021

Add aws ses email backend for use with EmailOperator. (#13986)

01049dd

uranusjr mentioned this pull request Jul 14, 2021

Airflow + AWS SES HTTP Api: Error Invalid type for parameter Source, value: None, type: <class 'NoneType'>, valid types: <class 'str'> #16938

Closed

uranusjr added the pending-response label Sep 2, 2021

ignaski mentioned this pull request Sep 6, 2021

Fixing ses email backend #18042

Merged

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Oct 19, 2021

uranusjr closed this Oct 19, 2021

jedcunningham mentioned this pull request Feb 18, 2022

Status of testing of Apache Airflow 2.2.4rc1 #21669

Closed

40 tasks

Fixing SES backend - source (mail_from) must be string. #16166

Fixing SES backend - source (mail_from) must be string. #16166

Uh oh!

Conversation

satishmittal111 commented May 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

satishmittal111 May 30, 2021

Choose a reason for hiding this comment

Uh oh!

mik-laj May 30, 2021

Choose a reason for hiding this comment

Uh oh!

satishmittal111 May 30, 2021

Choose a reason for hiding this comment

Uh oh!

ashb Sep 3, 2021

Choose a reason for hiding this comment

Uh oh!

uranusjr commented May 30, 2021

Uh oh!

satishmittal111 commented May 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potiuk commented Jul 3, 2021

Uh oh!

potiuk commented Jul 10, 2021

Uh oh!

potiuk commented Jul 11, 2021

Uh oh!

satishmittal111 commented Jul 12, 2021

Uh oh!

potiuk commented Jul 12, 2021

Uh oh!

jzuhusky commented Jul 19, 2021

Uh oh!

potiuk commented Jul 23, 2021

Uh oh!

ignaski commented Sep 2, 2021

Uh oh!

potiuk commented Sep 2, 2021

Uh oh!

uranusjr commented Sep 2, 2021

Uh oh!

ignaski commented Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

satishmittal111 commented Sep 2, 2021

Uh oh!

github-actions bot commented Oct 19, 2021

Uh oh!

uranusjr commented Oct 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

satishmittal111 commented May 30, 2021 •

edited

Loading

satishmittal111 commented May 30, 2021 •

edited

Loading

ignaski commented Sep 2, 2021 •

edited

Loading