Skip to content

Transactional Data Reimport Test #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nnikolopoulos merged 32 commits into from
Jul 11, 2023
Merged

Transactional Data Reimport Test #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
a112d49
Add ReimportTest notebook
Mar 27, 2023
0e6fa19
Restructure ReimportTest nb and add notes
Mar 28, 2023
bc0ad4f
Implement all requested cases, add data and assertions
Mar 30, 2023
7397513
Merge branch 'develop' into TransactionalDataReimport
Mar 30, 2023
4da2a21
Merge branch 'develop' into TransactionalDataReimport
Apr 18, 2023
f40bbcd
WIP in all cases except use case 3
Apr 18, 2023
c7d82ae
Merge branch 'develop' into TransactionalDataReimport
May 3, 2023
748e5f1
fix syntax in Instance()
May 3, 2023
f56ba03
More work in case 3 in imports, queries and assertions
May 4, 2023
20c66cc
Add new case and make ReimportTest pass
May 4, 2023
98e05b6
Cleanup
May 4, 2023
9aecc6f
Fix assertions
May 4, 2023
350530b
Add ReimportTest in Tests
May 5, 2023
30ab846
Merge branch 'develop' into TransactionalDataReimport
May 5, 2023
6624a04
Fix the used memory markdown 28Gb RAM
May 5, 2023
590b3a1
Merge branch 'develop' into TransactionalDataReimport
May 10, 2023
f4d0260
Fix case 3 assertion and clean up
May 10, 2023
b2de2ae
Merge branch 'develop' into TransactionalDataReimport
May 15, 2023
471e963
Clean up, implement feedback and update RAM md in Tests
May 15, 2023
fd943d1
Clean up x2
May 15, 2023
5352558
Feedback
May 16, 2023
c38a057
Merge branch 'develop' into TransactionalDataReimport
May 19, 2023
0d81cac
Restructuring and strictly keeping the "reimport" cases.
May 19, 2023
172a13f
Tests RAM markdown
May 19, 2023
027c01f
Merge branch 'develop' into TransactionalDataReimport
May 26, 2023
10d1b66
Feedback + test case proposal
May 26, 2023
aa541c5
Merge branch 'develop' into TransactionalDataReimport
Jun 28, 2023
cfd59c7
Merge branch 'develop' into TransactionalDataReimport
Jun 29, 2023
05060c9
Merge remote-tracking branch 'origin/develop' into TransactionalDataR…
Jul 5, 2023
bbbe255
Merge branch 'develop' into TransactionalDataReimport
Jul 9, 2023
aa9b40a
RAM md in Tests
Jul 11, 2023
57b31f7
Correct RAM in Tests
Jul 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
347 changes: 347 additions & 0 deletions ifrs17-template/Test/ReimportTest.ipynb
View file
Open in desktop
Comment thread
nnikolopoulos marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,347 @@
{
"metadata": {
"authors": [],
"id": "VOS67A7-B0Omk1s4TFC3LQ",
"kernelspec": {
"display_name": "Formula Framework",
"language": "C#",
"name": "C#"
},
"language_info": {
"file_extension": ".cs",
"mimetype": "text/plain",
"name": "C#"
}
},
"nbformat": 4,
"nbformat_minor": 5,
"cells": [
{
"cell_type": "markdown",
"source": [
"<p style=\"font-weight:bold;\"> <span style=\"font-size: 36px\"> Reimport of Transactional Data Test</span> </p>"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#!eval-notebook \"../Initialization/InitSystemorphBaseToMemory\""
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Check Method"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"public async Task<ActivityLog> ReimportAndCompare((string transactionalData, string importFormat) importData, DataNodeState[] dataNodeStates)",
"\n{",
"\n var workspace = Workspace.CreateNew();",
"\n workspace.Initialize(x => x.FromSource(DataSource).DisableInitialization<DataNodeState>().DisableInitialization<IfrsVariable>().DisableInitialization<RawVariable>());",
"\n await workspace.UpdateAsync<DataNodeState>(dataNodeStates);",
"\n",
"\n Activity.Start();",
"\n var importActivity = await Import.FromString(importData.transactionalData).WithFormat(importData.importFormat).WithTarget(workspace).ExecuteAsync();",
"\n if(importActivity.RepeatOnce().ToArray().HasErrors()) return importActivity;",
Comment thread
nnikolopoulos marked this conversation as resolved.
"\n ",
"\n var firstImportRawVariable = workspace.Query<RawVariable>().ToArray();",
"\n var firstImportIfrsVariable = workspace.Query<IfrsVariable>().ToArray();",
"\n",
"\n var reimportActivity = await Import.FromString(importData.transactionalData).WithFormat(importData.importFormat).WithTarget(workspace).ExecuteAsync();",
"\n if(reimportActivity.RepeatOnce().ToArray().HasErrors()) return reimportActivity;",
Comment thread
nnikolopoulos marked this conversation as resolved.
"\n",
"\n var secondImportRawVariable = workspace.Query<RawVariable>().ToArray();",
"\n var secondImportIfrsVariable = workspace.Query<IfrsVariable>().ToArray();",
"\n",
"\n List<string> errors = new List<string>();",
"\n var missingRawVariables = firstImportRawVariable.Except(secondImportRawVariable, RawVariableComparer.Instance(ignoreValues: false));",
"\n var missingIfrsVariables = firstImportIfrsVariable.Except(secondImportIfrsVariable, IfrsVariableComparer.Instance(ignoreValues: false));",
"\n var extraRawVariables = secondImportRawVariable.Except(firstImportRawVariable, RawVariableComparer.Instance(ignoreValues: false));",
"\n var extraIfrsVariables = secondImportIfrsVariable.Except(firstImportIfrsVariable, IfrsVariableComparer.Instance(ignoreValues: false));",
"\n",
"\n if(missingRawVariables.Any()) errors.Add($\"Missing RawVariables after the reimport: {string.Join(\"\\n\", missingRawVariables.ToList())}.\");",
"\n if(missingIfrsVariables.Any()) errors.Add($\"Missing IfrsVariables after the reimport: {string.Join(\"\\n\", missingIfrsVariables.ToList())}.\");",
"\n if(extraRawVariables.Any()) errors.Add($\"Extra RawVariables after the reimport: {string.Join(\"\\n\", extraRawVariables.ToList())}.\");",
"\n if(extraIfrsVariables.Any()) errors.Add($\"Extra IfrsVariables after the reimport: {string.Join(\"\\n\", extraIfrsVariables.ToList())}.\");",
"\n ",
"\n if(errors.Any()) ApplicationMessage.Log(Error.Generic, string.Join(\"\\n\", errors));",
"\n",
"\n ApplicationMessage.Log(Warning.Generic, \"Count of IfrsVariables in workspace: \" + secondImportIfrsVariable.Count());",
"\n ApplicationMessage.Log(Warning.Generic, \"Count of RawVariables in workspace: \" + secondImportRawVariable.Count());",
"\n return Activity.Finish();",
"\n}"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Data Preparation"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Create:",
"\n- Import args for current year (2021)",
"\n- Import args for previous year (2020)",
"\n- Partition by Reporting Node (using details from most recent args)"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var args = new ImportArgs(\"CH\", 2021, 3, Periodicity.Quarterly, null, null);",
"\nvar previousArgs = new ImportArgs(\"CH\", 2020, 12, Periodicity.Quarterly, null, null);",
"\n",
"\nvar partitionByReportingNode = new PartitionByReportingNode { Id = (Guid)(await DataSource.Partition.GetKeyForInstanceAsync<PartitionByReportingNode>(args)), ReportingNode = args.ReportingNode };"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Create 5 data node states:",
"\n- dt11StatePrevious",
"\n- dtr11StatePrevious",
"\n- dt11State",
"\n- dtr11State",
"\n- dt13State"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var dt11StatePrevious = new DataNodeState { DataNode = \"DT1.1\", State = State.Active, Year = previousArgs.Year, Month = previousArgs.Month, Partition = partitionByReportingNode.Id };",
"\n",
"\nvar dtr11StatePrevious = new DataNodeState { DataNode = \"DTR1.1\", State = State.Active, Year = previousArgs.Year, Month = previousArgs.Month, Partition = partitionByReportingNode.Id };",
"\n",
"\nvar dt11State = new DataNodeState { DataNode = \"DT1.1\", State = State.Active, Year = args.Year, Month = args.Month, Partition = partitionByReportingNode.Id };",
"\n",
"\nvar dtr11State = new DataNodeState { DataNode = \"DTR1.1\", State = State.Active, Year = args.Year, Month = args.Month, Partition = partitionByReportingNode.Id };",
"\n",
"\nvar dt13State = new DataNodeState { DataNode = \"DT1.3\", State = State.Active, Year = args.Year, Month = args.Month, Partition = partitionByReportingNode.Id };"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Test Transactional Data"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Reimport Openings for DataNode at Inception"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Create openings for a data node at inception (DT1.1) and import more than once."
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var openings = @\"",
"\n@@Main,,,,",
"\nReportingNode,Year,Month,,",
"\nCH,2021,3,,",
"\n@@Opening,,,,",
"\nDataNode,EstimateType,AmountType,AccidentYear,Value",
"\nDT1.1,C,,,72.2",
"\nDT1.1,AA,PR,,-1.5",
"\nDT1.1,OA,PR,,1.5",
"\n\";"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var activity = await ReimportAndCompare((openings, ImportFormats.Opening), new [] { dt11State, dtr11State });",
"\nactivity"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"activity.Status.Should().Be(ActivityLogStatus.Succeeded);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Reimport Actuals for Both: DataNode at Inception and DataNode Not at Inception"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Create an actuals file that contains data for both a data node at inception (DT1.3) and a data node not at inception (DT1.1). ",
"\n",
"\nImport this file more than once."
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var actuals = @\"",
"\n@@Main,,,,,",
"\nReportingNode,Year,Month,,,",
"\nCH,2021,3,,,",
"\n@@Actual,,,,,",
"\nDataNode,AocType,AmountType,EstimateType,AccidentYear,Value",
"\nDT1.1,CF,PR,A,,90",
"\nDT1.1,CF,ICO,A,,-6",
"\nDT1.1,CF,NIC,A,,-70",
"\nDT1.1,CF,ACA,A,,-10",
"\nDT1.1,CF,AEA,A,,-5",
"\nDT1.3,CF,PR,A,,90",
"\nDT1.3,CF,ICO,A,,-6",
"\nDT1.3,CF,NIC,A,,-70",
"\nDT1.3,CF,ACA,A,,-10",
"\nDT1.3,CF,AEA,A,,-5\";"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var activity = await ReimportAndCompare((actuals, ImportFormats.Actual), new[] {dt11StatePrevious, dtr11StatePrevious, dt13State});",
"\nactivity"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"activity.Status.Should().Be(ActivityLogStatus.Succeeded);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Reimport Cashflows for Linked Reinsurance Not at Inception"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Create a cashflows file that contains data for a data node not at inception (DTR1.1). ",
"\n",
"\nImport this file more than once."
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var cashflows = @\"",
"\n@@Main,,,,,,,,,,,,,,,,,,,,,,,,,,",
"\nReportingNode,Year,Month,,,,,,,,,,,,,,,,,,,,,,,,",
"\nCH,2021,3,,,,,,,,,,,,,,,,,,,,,,,,",
"\n@@Cashflow,,,,,,,,,,,,,,,,,,,,,,,,,,",
"\nDataNode,AmountType,EstimateType,AocType,Novelty,AccidentYear,Values0,Values1,Values2,Values3,Values4,Values5,Values6,Values7,Values8,Values9,Values10,Values11,Values12,Values13,Values14,Values15,Values16,Values17,Values18,Values19,Values20",
"\nDTR1.1,PR,BE,CL,C,,0,-57.5,0,0,-57.5,0,0,-57.5,0,0,-57.5,0,0,0,0,0,0,0,0,0,0",
"\nDTR1.1,NIC,BE,CL,C,,15,15,15,15,15,15,15,15,15,15,15,15,0,0,0,0,0,0,0,0,0",
"\nDTR1.1,,RA,CL,C,,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,1.25,0,0,0,0,0,0,0,0,0\";"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var activity = await ReimportAndCompare((cashflows, ImportFormats.Cashflow), new[] {dt11StatePrevious, dtr11StatePrevious});",
"\nactivity"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"activity.Status.Should().Be(ActivityLogStatus.Succeeded);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
""
],
"metadata": {},
"execution_count": 0,
"outputs": []
}
]
}
11 changes: 10 additions & 1 deletion ifrs17-template/Test/Tests.ipynb
View file
Open in desktop
Comment thread
nnikolopoulos marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"cell_type": "markdown",
"source": [
"Comprehensive collection of tests executed on top of the Systemorph use cases (initialization).",
"\n<br> Execute this Notebook using at least 26Gb RAM."
"\n<br> Execute this Notebook using at least 32Gb RAM."
],
"metadata": {},
"execution_count": 0,
Expand All @@ -44,6 +44,15 @@
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#!eval-notebook \"ReimportTest\""
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
Expand Down