Admin and User Metadata for transactional ressources and users (#5897)

[fthobe]

Closes #5897
Concerns #5951

Breaking Changes

This feature breaks sqlite versions predating to December 2023, verify your requirements before migrating.

Introduction

Overview

This PR introduces the concept of metadata fields to solidus. The purpose of metadata fields is to allow customers and admins alike to provide additional information to transactions to either extend the informations contained or allow to insert additional data to correlate transactions with third party systems.

Examples for data can be:

• any transaction itself such as orders, RMAs, payments and stock movements;
• individual items contained inside an order (line items);
• match users with 3rd party systems.

Possible Applications on a very high level are:

• customers that want to add Purchase Order information to an order;
• administrators that want to add a ticket number to an RMA.

The current scope of meta data are following ressources:

• Users
• Orders
• Customer Returns and any from of correlated transactions
• Payments
• Shipments
• Line Items

How is the additional information stored?

for each of the above resources two columns have been created:

• customer_metadata
• admin_metadata

The resources store the information as jsonb.

What is the access model?

Cancancan has been used to allow users to write data with the current scope just extended by the additional fields:

• Users can create / delete / alter metadata on orders only till the payment itself is processed,
• Admins have universal access R/W access to metadata at all stages of transactions.

Testing

Extensive test cases have been added.

Detailed Overview

Types of Meta Data and usage suggestions

Overview

Meta data can be broken down in three large topics visible in the table below:

	Use Case
Administrative Metadata	Information required to correlate all transactions to 3rd party systems.
Customer Metadata	Information allowing customers to customize orders (eg options to store information like a text string or link to an image) or correlate transactions (such as POs).

Administrative Metadata

Transactional resources are either by nature integrated with 3rd party systems (eg Payments, Billing in Italy with eInvoicing) or by operational requirement (ERP, CRMs like Salesforce,...). Meta data as an extension on admin site for all transactional resources to connect the transactional resources to any of such systems. Given that more and more countries will switch to a form of eInvoicing in the coming years meta data allows to fetch or store additional information in those transactions.

Customer Meta Data

Customer meta data allows users to customize a cart with either administrative (PO numbers,...) or customisation information (eg an URL to a picture to customize a shirt, text strings for incisions or in our case IMEIs to relate to a subscription) allowing to significantly extend the features without changing the underlying model. Fields should be accessible on front-end and backend.

Schema

Meta Data should net a reliable outcome:
"key1" : "value1" should be as applicable as "key2" : "value2", without restriction other than the quantity of storable key value pairs and limitations of the dimension of the value given that json as a format does not pose limits we should do that server side.

Future models to reserve or block certain key names might be possible.

Scopes & Arrays

Nature of the Values

The system is agnostic to the value contained. A decision has been made to store the data in additional columns in a jsonb which by now is relatively widely supported. Some preexisting installations might have to upgrade DBs, older sqlite versions dating before Jan 2024 are therefor not supported and need to be updated.

"key1" : "value1" should lead to the same behaviour as "key2" : "value2" without requiring modification inside the code. Reasonable provisions should be taken inside the code base to edit the quantities and content size of key value pairs and appropriate errors should be returned where rate or size limits are exceeded.

Administrative Meta Data

Allow appropriate roles to customize resources with private fields not end-user accessible.
R/W = Read and Write Access

	Admin	Use Case
Order	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Line Item	R/W	Add information to customize object such as link to an image for customization of a product or relate to a 3rd party system reference
RMA	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Payment	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Order	R/W	Add 3rd party system reference fields such as PO-Order or Dropshipping reference
Shipment	R/W	Add 3rd party system reference fields such as a delivery note
Payment	R/W	Add 3rd party system reference fields for payment conciliation
User	R/W	Add a 3rd party reference

Customer Meta Data

Allowing users to customize an incomplete order (or I'd say a cart) and call the information after a completed order. Admins should in an ideal case be allowed to edit those information after purchase, but I feel that can wait if technical limitations are currently in place.

	User	Admin	Use Case
Order	R/W	R/W	Add 3rd party system reference fields such as PO-Order,
Line Item	R/W	R/W	Add information to customize object such as link to an image for customization of a product or relate to a 3rd party system reference
User	R/W	R/W	Add additional User Fields or References

[codecov]

Codecov Report

Attention: Patch coverage is 70.24070% with 136 lines in your changes missing coverage. Please review.

Project coverage is 88.73%. Comparing base (0ef2e29) to head (3dfbeb6).
Report is 46 commits behind head on main.

Files with missing lines	Patch %	Lines
...els/spree/permission_sets/configuration_display.rb	0.00%	30 Missing ⚠️
.../models/spree/permission_sets/dashboard_display.rb	0.00%	19 Missing ⚠️
core/lib/spree/permission_sets.rb	0.00%	5 Missing ⚠️
core/lib/spree/permission_sets/base.rb	0.00%	5 Missing ⚠️
...lib/spree/permission_sets/configuration_display.rb	0.00%	5 Missing ⚠️
.../spree/permission_sets/configuration_management.rb	0.00%	5 Missing ⚠️
...ore/lib/spree/permission_sets/dashboard_display.rb	0.00%	5 Missing ⚠️
core/lib/spree/permission_sets/default_customer.rb	0.00%	5 Missing ⚠️
core/lib/spree/permission_sets/order_display.rb	0.00%	5 Missing ⚠️
core/lib/spree/permission_sets/order_management.rb	0.00%	5 Missing ⚠️
... and 11 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6118      +/-   ##
==========================================
- Coverage   89.28%   88.73%   -0.55%     
==========================================
  Files         813      830      +17     
  Lines       17906    18062     +156     
==========================================
+ Hits        15987    16028      +41     
- Misses       1919     2034     +115     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tvdeyen]

Thanks. I like the feature. The only request I have is the coordination of commits in this PR. Tests should be included with the commit that makes the introduction/change. You could also add the tests first, make them pending and then implement commit by commit, removing the pending flag, but this is your decision.

[fthobe]

@shahmayur001 it was discussed with Thomas to squash everything at the end.

[tvdeyen]

@shahmayur001 it was discussed with Thomas to squash everything at the end.

To be fully transparent I offered to squash merge this if @shahmayur001 is not able to rewrite the git history to our standards.

[fthobe]

@tvdeyen this should be all, all comments are resolved, the failed tests are either related to

• flaky interfaces
• Fix new Rubocop offences #6128

And are outside of the impact zone of this PR.

[JustShah]

@kennyadsl @tvdeyen Thank you so much for your guidance and support in helping us become more familiar with the Solidus community. As you know, we are completely new contributors, and we’ve likely made quite a few mistakes but we truly appreciate your patience. We’re actively learning from your feedback, taking notes, and doing our best to avoid repeating the same mistakes in the future. Your patience and insights are truly appreciated!

[tvdeyen]

Thank you for your patience and the adjusting of your commits. This is very much appreciated