AWS: Ensure "snapshot_tags" is present on AMIs

[JAVGan]

This commit changes the get_image_by_tags call on publish method for
AWSService so it will search for the snapshot tags alonside the
requested tags.

We need that to ensure the snapshot was properly created in order to
avoid retrieving the wrong AMI during the search.

Refers to SPSTRAT-615

[JAVGan]

@lslebodn @rbikar PTAL

[codecov-commenter]

Codecov Report

❌ Patch coverage is 44.44444% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.09%. Comparing base (57e906f) to head (a042462).
⚠️ Report is 10 commits behind head on master.

Files with missing lines	Patch %	Lines
cloudimg/aws.py	44.44%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #43      +/-   ##
==========================================
- Coverage   97.44%   96.09%   -1.35%     
==========================================
  Files           3        3              
  Lines         586      614      +28     
==========================================
+ Hits          571      590      +19     
- Misses         15       24       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rbikar]

LGTM, @JAVGan only a little question, the "snapshot_tags" is kind of hardcoded here. Is it something required by aws API or rather is it required minimum for tags set on image/snapshot?

[lslebodn]

There are couple of tag parameters passed to he publish method via metada from the pubtools-marketplacesvm.

1. tags

• https://github.com/release-engineering/pubtools-marketplacesvm/blob/eb063b4890b9e2cf72f7c2308a9d1f55ebfb6e8c/src/pubtools/_marketplacesvm/cloud_providers/aws.py#L434
  which are detected from image metadata
• https://github.com/release-engineering/pubtools-marketplacesvm/blob/eb063b4890b9e2cf72f7c2308a9d1f55ebfb6e8c/src/pubtools/_marketplacesvm/cloud_providers/aws.py#L434

They are automatically extended by "custom_tags" provided by user via starmap-mappings.

2. ami_tags && snapshot_tags passed here

• https://github.com/release-engineering/pubtools-marketplacesvm/blob/eb063b4890b9e2cf72f7c2308a9d1f55ebfb6e8c/src/pubtools/_marketplacesvm/tasks/community_push/command.py#L415-L417

So far they are the same and just contain billing code

• https://github.com/release-engineering/pubtools-marketplacesvm/blob/eb063b4890b9e2cf72f7c2308a9d1f55ebfb6e8c/src/pubtools/_marketplacesvm/tasks/community_push/command.py#L507-L510

Different objects should be find by following tags:

• s3 object should be bound by tags
• snapshot should by find by union of tags and snapshot_tags
• ami should by find by union of tags and ami_tags

We tried to merge tags via add_tags local function for snapshot
(BTW when I return back to the code; it always take me a minute to understand how add_tags works :-) )

• cloudimg/cloudimg/aws.py

  Line 618 in 641e5bf

  add_tags("tags", extra_kwargs)

• cloudimg/cloudimg/aws.py

  Line 618 in 641e5bf

  add_tags("tags", extra_kwargs)

But it was not done for for ami_tags.

So I would say get_image_by_tags needn't have new parameter. We should just pass right values there via current parameter tags

Anyway, It would be nice to improve logging a bit. We automatically fallback to finding image by tags if it cannot be find by name. But it is impossible to identify from logs.
It will be much simpler to analyze issue when one could see a message.

DEBUG ... Cannot find ami by name {metadata.image_name}. Going to try by tags ...

[lslebodn]

TY

[lslebodn]

Cloudimg support python3.6+

• cloudimg/setup.py

  Line 13 in 641e5bf

  python_requires=">=3.6",

Python3.9 introduced simple way of creating a new dictionary with the merged keys (dict1 | dict2)

• https://docs.python.org/3/library/stdtypes.html#typesmapping

It would allow "oneliner" fix

• cloudimg/cloudimg/aws.py

  Line 629 in 641e5bf

  add_tags("snapshot_tags", extra_kwargs)

-            self.get_image_by_tags(metadata.tags)
+            self.get_image_by_tags(metadata.tags | metadata.snapshot_tags)

One would just need to ensure both are dict (none None)

[JAVGan]

@lslebodn I've changed this PR so instead of using a new parameter for get_image_by_tags it passes the snapshot_tags from metadata alongside the tags, thanks a lot for the suggestion!

Since this is not using Python3.9 exclusively I've opted out for a solution which is widely compatible instead of the operator |.

I'll add a new commit on top of it to improve the logs.

@rbikar FYI

[JAVGan]

@lslebodn PTAL again

[lslebodn]

LGTM

@rbikar please re-review

[JAVGan]

@rbikar if it looks good to you feel free to merge & release as well

[lslebodn]

We already have the same message here

• cloudimg/cloudimg/aws.py

  Line 610 in 641e5bf

  log.info('Snapshot does not exist: %s', metadata.snapshot_name)

  
  and

• cloudimg/cloudimg/aws.py

  Line 1006 in 641e5bf

  log.info('Snapshot (%s) does not exist', metadata.snapshot_id)

There was just missing info in get_snapshot_by_id :-)

[JAVGan]

Patched

[lslebodn]

TYVM