Skip to content

[AIRFLOW-2524] Update SageMaker hook, operator and sensor for training, tuning and transform #4091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ashb merged 7 commits into from
Nov 1, 2018
Merged

[AIRFLOW-2524] Update SageMaker hook, operator and sensor for training, tuning and transform #4091

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fda468a
Update SageMaker hook, operator and sensor for training, tuning and t…
yangaws Oct 25, 2018
b279c00
Add unimplemented error and improve codestyle
yangaws Oct 25, 2018
4fe1c61
Fix S3hook for bucket creation
yangaws Oct 26, 2018
2249849
Address new comments from Ash
yangaws Oct 31, 2018
2b3d71c
Fix typo of describe_training_job_with_log()
yangaws Oct 31, 2018
e621963
fix flake8
yangaws Oct 31, 2018
438aa8a
Update time conversion to utc and add aws_conn_id to awshook in opera…
yangaws Oct 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions airflow/contrib/hooks/aws_hook.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
# specific language governing permissions and limitations
# under the License.


import boto3
import configparser
import logging
Expand Down Expand Up @@ -164,17 +163,17 @@ def _get_credentials(self, region_name):
aws_session_token=aws_session_token,
region_name=region_name), endpoint_url

def get_client_type(self, client_type, region_name=None):
def get_client_type(self, client_type, region_name=None, config=None):
session, endpoint_url = self._get_credentials(region_name)

return session.client(client_type, endpoint_url=endpoint_url,
verify=self.verify)
config=config, verify=self.verify)

def get_resource_type(self, resource_type, region_name=None):
def get_resource_type(self, resource_type, region_name=None, config=None):
session, endpoint_url = self._get_credentials(region_name)

return session.resource(resource_type, endpoint_url=endpoint_url,
verify=self.verify)
config=config, verify=self.verify)

def get_session(self, region_name=None):
"""Get the underlying boto3.session."""
Expand All @@ -191,3 +190,16 @@ def get_credentials(self, region_name=None):
# secret key separately can lead to a race condition.
# See https://stackoverflow.com/a/36291428/8283373
return session.get_credentials().get_frozen_credentials()

def expand_role(self, role):
"""
Expand an IAM role name to an IAM role ARN. If role is already an IAM ARN,
no change is made.

:param role: IAM role name or ARN
:return: IAM role ARN
"""
if '/' in role:
return role
else:
return self.get_client_type('iam').get_role(RoleName=role)['Role']['Arn']
Loading