Skip to content

Fix opening importer #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcolleoni merged 7 commits into from
Jan 26, 2023
Merged

Fix opening importer #199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1fd40c5
improving import: fix opening and Fix CommitToAsync to deal with phys…
dcolleoni Jan 26, 2023
2e276c5
Merge remote-tracking branch 'origin/develop' into FixOpeningImporter
dcolleoni Jan 26, 2023
9cfe5b9
clean up
dcolleoni Jan 26, 2023
ba0852d
Merge remote-tracking branch 'origin/develop' into FixOpeningImporter
dcolleoni Jan 26, 2023
05c968a
clean up
dcolleoni Jan 26, 2023
bf86c8c
add test for cf
dcolleoni Jan 26, 2023
d3bd8b7
rename file and correct memory usage
dcolleoni Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions ifrs17-template/Test/Data/NominalCashflows_CH_2020_12_DT1.1NoPrem.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
@@Main,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
ReportingNode,Year,Month,Scenario,,,,,,,,,,,,,,,,,,,,,,,,,,
CH,2020,12,,,,,,,,,,,,,,,,,,,,,,,,,,,
@@Cashflow,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
DataNode,AmountType,EstimateType,AocType,Novelty,AccidentYear,Values0,Values1,Values2,Values3,Values4,Values5,Values6,Values7,Values8,Values9,Values10,Values11,Values12,Values13,Values14,Values15,Values16,Values17,Values18,Values19,Values20,Values21,Values22,Values23
DT1.1,NIC,BE,BOP,N,,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25
DT1.1,,CU,BOP,N,,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-3
DT1.1,,RA,BOP,N,,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5
DT1.1,NIC,BE,CL,C,,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25,-25
DT1.1,,CU,CL,C,,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-10,-5,-5,-5,-5,-5,-5,-5,-5,-5,-5,-3
DT1.1,,RA,CL,C,,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5,-2.5
198 changes: 198 additions & 0 deletions ifrs17-template/Test/ReimportWithDifferentScopeTest.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
{
"metadata": {
"authors": [],
"kernelspec": {
"display_name": "Formula Framework",
"language": "C#",
"name": "C#"
},
"language_info": {
"file_extension": ".cs",
"mimetype": "text/plain",
"name": "C#"
}
},
"nbformat": 4,
"nbformat_minor": 5,
"cells": [
{
"cell_type": "markdown",
"source": [
"<p style=\"font-weight:bold;\"> <span style=\"font-size: 36px\"> Reimport With Different Scope Test </span> </p>"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#!eval-notebook \"../Initialization/InitSystemorphBaseToMemory\""
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Cash flow Import",
"\n",
"\nImporting a file with N GICs and GRICs."
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"await Import.FromFile(\"../Files/TransactionalData/NominalCashflows_CH_2020_12.csv\")",
"\n .WithFormat(ImportFormats.Cashflow)",
"\n .WithTarget(DataSource)",
"\n .WithActivityLog(Session, DataSource, DataSetReader)",
"\n .ExecuteAsync()"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var ifrsVars1 = await DataSource.Query<IfrsVariable>().ToArrayAsync();",
"\nvar dn1 = ifrsVars1.Select(x => x.DataNode).ToHashSet();",
"\nvar dt11Prem1 = ifrsVars1.Where(x => x.DataNode == \"DT1.1\" && x.AmountType == \"PR\");",
"\n(ifrsVars1.Count(), dn1.Count(), dt11Prem1.Count())"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Cash flow Import - restatement",
"\nImporting a file for the same period as previous import but it only contains 1 GIC: DT1.1 where one amount type (Premium) has been removed"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"await Import.FromFile(\"Data/NominalCashflows_CH_2020_12_DT1.1NoPrem.csv\")",
"\n .WithFormat(ImportFormats.Cashflow)",
"\n .WithTarget(DataSource)",
"\n .WithActivityLog(Session, DataSource, DataSetReader)",
"\n .ExecuteAsync()"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"var ifrsVars2 = await DataSource.Query<IfrsVariable>().ToArrayAsync();",
"\nvar dn2 = ifrsVars2.Select(x => x.DataNode).ToHashSet();",
"\nvar dt11Prem2 = ifrsVars2.Where(x => x.DataNode == \"DT1.1\" && x.AmountType == \"PR\");",
"\n(ifrsVars2.Count(), dn2.Count(), dt11Prem2.Count())"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Test"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Import generate variables"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"ifrsVars1.Count().Should().NotBe(0);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"ifrsVars2.Count().Should().NotBe(0);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Count of data nodes does not change"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"dn2.Count().Should().Be(dn1.Count());"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Premiums are present with the first import and then deleted"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"dt11Prem1.Count().Should().NotBe(0);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"dt11Prem2.Count().Should().Be(0);"
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
""
],
"metadata": {},
"execution_count": 0,
"outputs": []
}
]
}
11 changes: 10 additions & 1 deletion ifrs17-template/Test/Tests.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"cell_type": "markdown",
"source": [
"Comprehensive collection of tests executed on top of the Systemorph use cases (initialization).",
"\n<br> Execute this Notebook using at least 12Gb RAM."
"\n<br> Execute this Notebook using at least 14Gb RAM."
],
"metadata": {},
"execution_count": 0,
Expand All @@ -54,6 +54,15 @@
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#!eval-notebook \"ReimportWithDifferentScopeTest\""
],
"metadata": {},
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"source": [
Expand Down
23 changes: 13 additions & 10 deletions ifrs17/Import/Importers.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,9 +254,11 @@
{
"cell_type": "code",
"source": [
"public static async Task CleanAsync<T> (this IDataSource dataSource, Expression<Func<T, bool>> filter = null) where T : class",
"public static async Task CleanAsync<T> (this IDataSource dataSource, Guid partitionId = default, Expression<Func<T, bool>> filter = null) where T : class, IPartitioned",
"\n{",
"\n var loadData = await dataSource.Query<T>().Where(filter?? (Expression<Func<T, bool>>)(x => true)).ToListAsync();",
"\n var loadData = partitionId != (Guid)default",
"\n ? await dataSource.Query<T>().Where(x => x.Partition == partitionId ).Where(filter?? (Expression<Func<T, bool>>)(x => true)).ToListAsync()",
"\n : await dataSource.Query<T>().Where(filter ?? (Expression<Func<T, bool>>)(x => true)).ToListAsync();",
"\n await dataSource.DeleteAsync(loadData);",
"\n}"
],
Expand All @@ -282,11 +284,11 @@
"\nwhere TData : class, IPartitioned",
"\nwhere TPartition : IfrsPartition",
"\n{",
"\n if(partitionId != new Guid()) {",
"\n if(partitionId != (Guid)default) {",
"\n await target.Partition.SetAsync<TPartition>(partitionId);",
"\n await source.Partition.SetAsync<TPartition>(partitionId);",
"\n }",
"\n if(snapshot) await CleanAsync<TData>(target, filter);",
"\n if(snapshot) await CleanAsync<TData>(target, partitionId, filter);",
"\n await target.UpdateAsync<TData>( await source.Query<TData>().ToArrayAsync() );",
"\n await target.CommitAsync();",
"\n}"
Expand Down Expand Up @@ -847,8 +849,10 @@
"\n }",
"\n",
"\n // Remove data nodes which are unaffected by the updated yield curves",
"\n await workspaceToCompute.DeleteAsync( await workspaceToCompute.Query<RawVariable>()",
"\n .Where(x => !(dataNodesToUpdate.Contains(x.DataNode) && (x.Partition == targetPartition || x.Partition == defaultPartition))).ToArrayAsync() );",
"\n // TODO : Reintroduce this functionality. Note all UpdateAsync/DeleteAsync performed to the workspaceToCompute are then trasferred to the DataSource.",
"\n // This is way this functionality should be written in a different way. ",
"\n // await workspaceToCompute.DeleteAsync( await workspaceToCompute.Query<RawVariable>()",
"\n // .Where(x => !(dataNodesToUpdate.Contains(x.DataNode) && (x.Partition == targetPartition || x.Partition == defaultPartition))).ToArrayAsync() );",
"\n",
"\n importLog = importLog.Merge(await ComputeAsync(args, workspace, workspaceToCompute, false));",
"\n if(importLog.Errors.Any()) return Activity.Finish().Merge(importLog);",
Expand Down Expand Up @@ -1400,7 +1404,7 @@
"\n log = log.Merge(await ComputeAsync(args, workspace, workspaceToCompute, false)); ",
"\n if(log.Errors.Any()) return Activity.Finish().Merge(log);",
"\n }",
"\n if (!log.Errors.Any()) await workspaceToCompute.CommitToTargetAsync(DataSource);",
"\n await workspaceToCompute.CommitToTargetAsync(DataSource);",
"\n return Activity.Finish().Merge(log);",
"\n});"
],
Expand Down Expand Up @@ -1507,7 +1511,6 @@
"\n var targetDataNodes = workspace.Query<IfrsVariable>().Select(v => v.DataNode).Distinct().ToArray();",
"\n await workspace.CommitToAsync<IfrsVariable,PartitionByReportingNodeAndPeriod>(DataSource, partitionId, snapshot : true,",
"\n filter : x => targetDataNodes.Contains(x.DataNode));",
"\n ",
"\n return Activity.Finish().Merge(parsingLog);",
"\n});"
],
Expand Down Expand Up @@ -1545,9 +1548,9 @@
"\n workspaceToCompute.Initialize(x => x.FromSource(DataSource));",
"\n var calculationLog = await ComputeAsync(args, workspace, workspaceToCompute, false); ",
"\n if(calculationLog.Errors.Any()) return Activity.Finish().Merge(calculationLog);",
"\n await workspaceToCompute.CommitToAsync<IfrsVariable,PartitionByReportingNodeAndPeriod>(DataSource, partitionId);",
"\n await workspaceToCompute.CommitToTargetAsync(DataSource, x => x.SnapshotMode<IfrsVariable>());",
"\n return Activity.Finish().Merge(parsingLog).Merge(calculationLog);",
"\n});"
"\n})"
],
"metadata": {},
"execution_count": 0,
Expand Down