Migrate from ReadStreamAsDT() to ReadStream().ToDataTable() generates BadUsageException

[killergege]

I just upgraded some code I inherited from 3.1 to 3.2 and my call to FileHelperEngine.ReadStreamAsDT(..) appears as deprecated.
So i followed the recommandation and used ReadStream(..).ToDataTable().

The problem is that the same code generates the exception "The record class Object must be marked with the [DelimitedRecord] or [FixedLengthRecord] Attribute" because I use a generated delimited class.
Doesn't the CreateRecordClass an object with the proper attributes?

Here an example of code that generates the problem :

        var builder = new DelimitedClassBuilder("temp") { IgnoreFirstLines = 0, IgnoreEmptyLines = true, Delimiter = ";" };
        var sr = new StreamReader(fileStream, Encoding.Default);
        var headerArray = sr.ReadLine().Split(';');
        foreach (var header in headerArray)
        {
            builder.AddField(header, typeof(string));
            builder.LastField.FieldQuoted = true;
            builder.LastField.QuoteChar = '"';
        }
        var engine = new FileHelperEngine(builder.CreateRecordClass());
        //return engine.ReadStreamAsDT(sr);  //Works
        return engine.ReadStream(sr).ToDataTable(); //Throws exception

[netniV]

Does the ReadStream(sr) without the ToDataTable() work? Do you see the same errors?

var engineType = builder.CreateRecordClass();
var engine = new FileHelperEngine(engineType);
//return engine.ReadStreamAsDT(sr);  //Works
var streamResults = engine.ReadStream(sr)
var tableResults = streamResults.ToDataTable(); //Throws exception?
return tableResults;

Do you have the full Exception / StrackTrace?

You can also check the attributes against a class, by using ...

var attributes = engineType.GetAtttributes()

There is also Type.GetCustomAttributes() but I'm not sure which you would need as I'm writing this without coding it ;-)

[killergege]

Yes sorry, that wasn't a very good bug report :).
It happens on the ToDatabase() call, and generates this exception :

L'exception FileHelpers.BadUsageException s'est produite
  HResult=-2146233088
  Message=The record class Object must be marked with the [DelimitedRecord] or [FixedLengthRecord] Attribute
  Source=FileHelpers
  StackTrace:
       à FileHelpers.RecordInfo.InitRecordFields()
       à FileHelpers.RecordInfo..ctor(Type recordType)
       à FileHelpers.RecordInfo.RecordInfoFactory.Resolve(Type type)
       à FileHelpers.RecordInfo.Resolve(Type type)
       à FileHelpers.ExtensionsFileHelpers.ToDataTable[T](T[] records)

CustomAttributes says that the Type contains a DelimitedRecordAttribute

But looking at the code, there's something that feels wrong : ReadStream(sr) returns an array of objects, so the type inference generates a call to ToDatatable<object>() and is not using my type.
You won't be able to use generics without knowing the type at the compilation time.
ToDatatable() might need an overload where you can specify the type in parameter.

You could get the type using records.First().GetType(); but it would only work if there are rows, so the behavior would not be consistent (it would crash with the BadUsage exception above if empty, but work if there are rows)...

[netniV]

I think the reason for that may be that it isn't a typed engine (eg, FileHelpersEngine<T>). I will have to take a further look tomorrow but you could be correct that its down to the array being returned ReadStream()being used by ToDataTable()

[netniV]

Turns out that I was half right. The main reason for this problem is because you are using the plain FileHelpersEngine() class which now inherits from FileHelpersEngine using object as the T type. This reduces the duplicate of code for the majority of cases.

Unfortunately, to get around this, you are going to need some not so nice looking code. To help, I created an example using a comma delimited file I had so you will need to modify this to your needs:

    try
    {
	var fileInfo = new FileInfo(@"C:\TEMP\Binding.csv");
	var fileStream = fileInfo.OpenRead();
	var builder = new DelimitedClassBuilder("temp") { IgnoreFirstLines = 0, IgnoreEmptyLines = true, Delimiter = "," };
	var sr = new StreamReader(fileStream, Encoding.Default);
	var headerArray = sr.ReadLine().Split(',');
	foreach (var header in headerArray)
	{
	    var field = header;
	    if (field.StartsWith("\"") && field.EndsWith("\""))
		field = field.Substring(1, field.Length - 2);

	    builder.AddField(field, typeof(string));
	    builder.LastField.FieldQuoted = true;
	    builder.LastField.QuoteChar = '"';
	}

	// Get the class type read to build our generic FileHelpersEngine<type>()
	var classType = builder.CreateRecordClass();

	// Build an array of dynamic parameters
	var classTypes = new Type[] { classType };

	// Construct our generic type
	var engineGenericType = typeof(FileHelperEngine<>);
	var engineType = engineGenericType.MakeGenericType(classTypes);

	var engine = Activator.CreateInstance(engineType);

	//return engine.ReadStream(sr)
	var engineReadStreamMethod = engineType.GetMethod("ReadStream", new Type[] { typeof(TextReader) });
	var engineReadStreamArgs = new object[] { sr };
	var engineReadStreamResult = engineReadStreamMethod.Invoke(engine, engineReadStreamArgs);

	//return engine.ToDataTable()
	var toDataTableType = typeof(FileHelpers.ExtensionsFileHelpers);
	var toDataTableMethod = toDataTableType.GetMethod("ToDataTable");
	var toDataTableGeneric = toDataTableMethod.MakeGenericMethod(new Type[] { classType });
	var toDataTableArgs = new object[] { engineReadStreamResult };
	var toDataTableResult = toDataTableGeneric.Invoke(null, toDataTableArgs);

	DataTable testTable = toDataTableResult as DataTable;
	Console.WriteLine($"DataTable created with {testTable.Rows.Count} rows and {testTable.Columns.Count}");
	foreach (DataRow testRow in testTable.Rows)
	{
	    bool firstColumn = true;
	    foreach (DataColumn testColumn in testTable.Columns)
	    {
		string columnPadding = firstColumn ? "" : ", ";
		Console.Write($"{columnPadding}{testColumn.ColumnName}: {testRow[testColumn]}");
		firstColumn = false;
	    }
	    Console.WriteLine();
	}
    }
    catch (Exception ex)
    {
	Console.WriteLine($"{ex.GetType().FullName}:\r\n{ex.Message}\r\n{ex.StackTrace}");
    }

[killergege]

Indeed, not a very pretty code :).
So I guess there no supported way to read a CSV file you don't know the structure beforehand with FileHelpers and put the values into a datatable ?
Maybe I'm just not using the right way (it was before but it seems that it's not anymore).

[netniV]

Have the above code as a function, and it's using the correct methods. It's supported but it's just not so easy to understand the ReadDataTableFromStream function :)

I think there may need to be a case where if the array type is object[], that the class of the items are used to work out the class type. I'll take a look tomorrow at that.

 try
    {
	var fileInfo = new FileInfo(@"C:\TEMP\Binding.csv");
	var fileStream = fileInfo.OpenRead();
	var builder = new DelimitedClassBuilder("temp") { IgnoreFirstLines = 0, IgnoreEmptyLines = true, Delimiter = "," };
	var sr = new StreamReader(fileStream, Encoding.Default);
	var headerArray = sr.ReadLine().Split(',');
	foreach (var header in headerArray)
	{
	    var field = header;
	    if (field.StartsWith("\"") && field.EndsWith("\""))
		field = field.Substring(1, field.Length - 2);

	    builder.AddField(field, typeof(string));
	    builder.LastField.FieldQuoted = true;
	    builder.LastField.QuoteChar = '"';
	}

	// Get the class type read to build our generic FileHelpersEngine<type>()
	var classType = builder.CreateRecordClass();
        DataTable testTable = ReadDataTableFromStream(classType, sr);
	Console.WriteLine($"DataTable created with {testTable.Rows.Count} rows and {testTable.Columns.Count}");
	foreach (DataRow testRow in testTable.Rows)
	{
	    bool firstColumn = true;
	    foreach (DataColumn testColumn in testTable.Columns)
	    {
		string columnPadding = firstColumn ? "" : ", ";
		Console.Write($"{columnPadding}{testColumn.ColumnName}: {testRow[testColumn]}");
		firstColumn = false;
	    }
	    Console.WriteLine();
	}
    }
    catch (Exception ex)
    {
	Console.WriteLine($"{ex.GetType().FullName}:\r\n{ex.Message}\r\n{ex.StackTrace}");
    }

    public DataTable ReadDataTableFromStream(Type classType, StreamReader sr)
    {
	// Build an array of dynamic parameters
	var classTypes = new Type[] { classType };

	// Construct our generic type
	var engineGenericType = typeof(FileHelperEngine<>);
	var engineType = engineGenericType.MakeGenericType(classTypes);

	var engine = Activator.CreateInstance(engineType);

	//return engine.ReadStream(sr)
	var engineReadStreamMethod = engineType.GetMethod("ReadStream", new Type[] { typeof(TextReader) });
	var engineReadStreamArgs = new object[] { sr };
	var engineReadStreamResult = engineReadStreamMethod.Invoke(engine, engineReadStreamArgs);

	//return engine.ToDataTable()
	var toDataTableType = typeof(FileHelpers.ExtensionsFileHelpers);
	var toDataTableMethod = toDataTableType.GetMethod("ToDataTable");
	var toDataTableGeneric = toDataTableMethod.MakeGenericMethod(new Type[] { classType });
	var toDataTableArgs = new object[] { engineReadStreamResult };
	var toDataTableResult = toDataTableGeneric.Invoke(null, toDataTableArgs);

	return DataTable testTable = toDataTableResult as DataTable;
    }

[killergege]

Yes, this solves the problem, but having to use reflection for a "basic" use case feels a bit like a workaround (or maybe it's not so common ? I often use FileHelpers to load files provided by users so the application need to validate it with explicit error messages - or use a format specified in database or configuration, not "hard coded").

What was the reason for making ReadStreamAsDT deprecated ?

But I'll do what you suggest in the meantime ;)

[netniV]

Not sure as I am not the original author, just a guy who likes to help fix problems ;-)

Technically, you could still use ReadStreamAsDT() because whilst marked as obsolete it's still just a warning not an error (unless you have that flag of warnings are errors).

[killergege]

Ok thanks a lot then :)
I annoy a lot of people because I don't want warnings in their code so... I won't keep those in mine ;)

[gwsampso]

What about changing your engine to a DelimitedFileEngine?

[tippmar]

@gwsampso Good idea, but that doesn't work either. Just tried it. I'm going back to using ReadStreamAsDT() unless/until this bug gets fixed or ReadStreamAsDT() gets removed from the class.

[netniV]

That was my implied suggestion above. You can even put #pragma warning disable <code> and #pragma warning restore <code> messages around the code that you know is going to error with an obsolete/depreciated warning so that they don't appear.