Fix garbled characters in history view with non-ascii characters (fixes#124) #136

shiena · 2017-07-13T18:31:33Z

fixes #124
fixes #35

Depends on:

Switch to blocking IO model for reading output from processes #181

I replaced the reading of standard output from asynchronous to synchronous to eliminate garbled characters.

Synchronous reading of both standard output and standard error may cause deadlock. So I replaced standard output only.
Reference: https://stackoverflow.com/questions/7160187/standardoutput-readtoend-hangs

Also, I'm worried about whether performance will be bad.

Since the ReadLine() loop blocks I/O, use Stream::BeginRead instead.

reference from https://github.com/msysgit/msysgit/wiki/Git-for-Windows-Unicode-Support

shiena · 2017-07-18T12:10:51Z

@StanleyGoldman
The appveyor build failed with an error occurred in nuget restore GitHub.Unity.sln.
Can you build it again?

shiena · 2017-07-22T11:44:56Z

@StanleyGoldman
A pull request has been completed. I will summarize the changes.

Using Process::StandardOutput::BaseStream::BeginRead instead of Process::BeginOutputReadLine
Also change StandardError
However, since .NET 4.6 has no problem in character encoding, use the original code
Add options to git command according to Git for Windows Unicode support

StanleyGoldman · 2017-07-31T14:54:43Z

Hey @shiena sorry I've been MIA, @shana and I have been on vacation.
We will take a look at this soon.

StanleyGoldman · 2017-08-02T16:40:12Z

Hey @shiena, sorry it's taken so long to get to this.

Processes are executed in a separate thread, so because of the bug in Unity we can definitely go with synchronous calls here. Also until Unity removes support for their .net 3.5 profile we will not compile this library in .net 4.6.

So when you get the chance remove all the #if/#endif and keep the .net 3.5 side. I think there are a few more changes to make so after that I will review again.

StanleyGoldman

Remove all the #if/#endif with NET_4_6

shiena · 2017-08-02T17:09:21Z

@StanleyGoldman Thank you for review.
I kept the .net 3.5 side according to the compilation policy.

StanleyGoldman · 2017-08-02T17:22:57Z

💨... Thanks for the quick changes ...
I'm testing your changes against the https://github.com/unity3d-jp/piranhan repo now.

StanleyGoldman · 2017-08-02T18:03:53Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

+                    {
+                        var encoded = outputEncoding.GetString(outputBuffer, 0, bytesRead);
+                        outputContents.Append(encoded);
+                        Process.StandardOutput.BaseStream.BeginRead(outputBuffer, 0, outputBuffer.Length, outputCallback, Process.StandardOutput);


~~I believe this line of code should not be here.~~

It seems I am mistaken

StanleyGoldman · 2017-08-02T18:04:14Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

+                    {
+                        var encoded = errorEncoding.GetString(errorBuffer, 0, bytesRead);
+                        errorContents.Append(encoded);
+                        Process.StandardError.BaseStream.BeginRead(errorBuffer, 0, errorBuffer.Length, errorCallback, Process.StandardError);


~~I think the same goes for this line.~~

Mistaken here too

StanleyGoldman · 2017-08-02T18:18:48Z

I understand why you've done what you've done... 🤔... I'm thinking about it

StanleyGoldman

I understand what happened now, you worked real hard to keep both async and sync methods possible. Since we are going to go sync, you can rewrite the way the output is being handled to be simpler & easier to follow.

StanleyGoldman · 2017-08-02T18:30:26Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

-                Process.BeginOutputReadLine();
+            {
+                outputReset.Reset();
+                Process.StandardOutput.BaseStream.BeginRead(


Use a loop here to read the bytes and encode the string with the correct encoding. After that you can read the lines and call the appropriate output method. This way we won't need the complicated AsyncCallback that has a reference to itself.

If you read the standard output using a loop there, it blocks processing.
Because RunCredentialHelper writes to the standard input of git-credential-wincred command when signing in to GitHub.

Unity/src/GitHub.Api/Git/GitCredentialManager.cs

Lines 147 to 172 in 1476044

Logger.Trace("RunCredentialHelper helper:\"{0}\" action:\"{1}\"", credHelper, action);

SimpleProcessTask task;

if (credHelper.StartsWith('!'))

{

// it's a separate app, run it as such

task = new SimpleProcessTask(taskManager.Token, credHelper.Substring(1).ToNPath(), action);

}

else

{

var args = $"credential-{credHelper} {action}";

task = new SimpleProcessTask(taskManager.Token, args);

}

task.Configure(processManager, true);

task.OnStartProcess += proc =>

{

foreach (var line in lines)

{

proc.StandardInput.WriteLine(line);

}

proc.StandardInput.Close();

};

return task;

For that reason, I think that we need an asynchronous BeginRead.
Please tell me if there is a better way.

StanleyGoldman · 2017-08-02T18:31:03Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

-                Process.BeginErrorReadLine();
+            {
+                errorReset.Reset();
+                Process.StandardError.BaseStream.BeginRead(


Do the same thing here.

StanleyGoldman · 2017-08-02T18:33:29Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

+            if (Process.StartInfo.RedirectStandardOutput)
+                outputReset.WaitOne();
+            if (Process.StartInfo.RedirectStandardError)
+                errorReset.WaitOne();


That way we no longer need these ManualResetEvent objects.

…iew-with-non-ascii-characters-fixes-124

…rs-in-history-view-with-non-ascii-characters-fixes-124 # Conflicts: # src/GitHub.Api/NewTaskSystem/ProcessTask.cs

StanleyGoldman · 2017-08-11T16:16:42Z

@shiena there is another branch based off this, could you test it out for me.

https://github.com/github-for-unity/Unity/tree/fixes/returning-lines-sooner

StanleyGoldman · 2017-08-11T16:18:25Z

LFS Verification checks are preventing me from pushing to your repo

shiena · 2017-08-11T16:30:25Z

@StanleyGoldman Japanese also displayed correctly for that branch.

# Conflicts: # src/GitHub.Api/NewTaskSystem/ProcessTask.cs

StanleyGoldman · 2017-08-11T16:49:32Z

Edited the pull request description to show what this PR depends on

shiena · 2017-08-11T16:55:16Z

@StanleyGoldman I think that #35 has also been fixed.

StanleyGoldman · 2017-08-11T17:02:37Z

You may be right, let's see

StanleyGoldman · 2017-08-11T17:05:47Z

You are correct.

StanleyGoldman · 2017-08-11T19:30:31Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

-                    if (token.IsCancellationRequested)
+                    var outputStream = Process.StandardOutput.BaseStream;
+                    var outputBuffer = new byte[bufferSize];
+                    var outputEncoding = Process.StartInfo.StandardOutputEncoding ?? Console.Out.Encoding;


Can Process.StartInfo.StandardOutputEncoding ever be null?

It could not be null by tracing the call hierarchy. So delete the null check.

StanleyGoldman · 2017-08-11T19:30:54Z

src/GitHub.Api/NewTaskSystem/ProcessTask.cs

-        }
+                    var errorStream = Process.StandardError.BaseStream;
+                    var errorBuffer = new byte[bufferSize];
+                    var errorEncoding = Process.StartInfo.StandardErrorEncoding ?? Console.Error.Encoding;


Same question about Process.StartInfo.StandardErrorEncoding ...

Also delete null check.

shiena · 2017-08-22T17:48:22Z

@StanleyGoldman @shana
A process encoding bug has been avoided as #181 has been merged. (Process.StandardOutput.ReadLine correctly encodes)
However, if the user sets encoding options other than utf8 in git config, the history view garbled.
Therefore, I recommend setting -C logoutputencoding=utf8 etc for git command line option.

Even if you need the git command line option, this pull request is redundant, so I would like to create another pull request.

shana · 2017-08-22T18:15:35Z

@shiena Yup, we were literally just discussing that 10 minutes ago and came to the same conclusion 😄 👍

StanleyGoldman · 2017-08-22T18:15:44Z

Hey @shiena we were about to say the same thing to you...

Fix garbled characters in history view with non-ascii characters

1b37248

StanleyGoldman added the work-in-progress label Jul 13, 2017

shiena added 6 commits July 18, 2017 03:20

Add null terminator

334187e

Wait for thread termination

6ee7b85

Add reason not to use Process.OutputDataReceived

a0d8fcd

.NET 4.6 has not encoding problem

298f8f9

Read process output asynchronously

3168f6d

Since the ReadLine() loop blocks I/O, use Stream::BeginRead instead.

Unicode support setting

e229e8e

reference from https://github.com/msysgit/msysgit/wiki/Git-for-Windows-Unicode-Support

Overwrite i18n.commitencoding to unify character encoding with utf8

8950cff

shiena and others added 4 commits July 26, 2017 21:15

Merge branch 'master' into fix-history-view-encoding

37803d6

Merge branch 'master' into fix-history-view-encoding

0ab3746

Merge branch 'master' into fix-history-view-encoding

9395d31

Merge branch 'master' into fix-history-view-encoding

04520ea

Merge branch 'master' into fix-history-view-encoding

277b43f

StanleyGoldman self-requested a review August 2, 2017 16:40

StanleyGoldman suggested changes Aug 2, 2017

View reviewed changes

Remove all the #if/#endif with NET_4_6

05fc0c3

StanleyGoldman suggested changes Aug 2, 2017

View reviewed changes

StanleyGoldman added 4 commits August 10, 2017 18:57

Reading stream inline while leaving input to be routed via event

83bb4c5

Merge branch 'master' into pr/136-fix-garbled-characters-in-history-v…

cd35477

…iew-with-non-ascii-characters-fixes-124

Merge branch 'fixes/process-wrapper' into pr/136-fix-garbled-characte…

5290aa6

…rs-in-history-view-with-non-ascii-characters-fixes-124 # Conflicts: # src/GitHub.Api/NewTaskSystem/ProcessTask.cs

Mixing both changes after the merge

5bef6a2

Adding NewlineSplitStringBuilder

00ed6f1

StanleyGoldman added 4 commits August 11, 2017 12:32

Fixing conditionals

5aeb275

Merge branch 'fixes/process-wrapper' into fix-history-view-encoding

5fab3cd

# Conflicts: # src/GitHub.Api/NewTaskSystem/ProcessTask.cs

Fixing error redirection

d12c143

Merge branch 'fixes/process-wrapper' into fix-history-view-encoding

a282a16

StanleyGoldman mentioned this pull request Aug 11, 2017

Switch to blocking IO model for reading output from processes #181

Merged

StanleyGoldman added ready-for-review and removed work-in-progress labels Aug 11, 2017

Fixing error

f828b6e

Removing DoNotRunOnAppVeyor attribute

89fba73

StanleyGoldman previously approved these changes Aug 11, 2017

View reviewed changes

StanleyGoldman requested a review from shana August 11, 2017 17:06

shiena dismissed StanleyGoldman’s stale review via 89fba73 August 11, 2017 17:08

StanleyGoldman reviewed Aug 11, 2017

View reviewed changes

shiena added 2 commits August 13, 2017 23:20

Remove redundant null check

c9109fa

Merge branch 'master' into fix-history-view-encoding

d8b4535

shiena mentioned this pull request Aug 22, 2017

Fix garbled characters in history view with non-ascii characters (fixes #124)(fixes #35) #220

Merged

StanleyGoldman closed this Aug 22, 2017

shiena deleted the fix-history-view-encoding branch August 23, 2017 03:07

	Logger.Trace("RunCredentialHelper helper:\"{0}\" action:\"{1}\"", credHelper, action);

	SimpleProcessTask task;
	if (credHelper.StartsWith('!'))
	{
	// it's a separate app, run it as such
	task = new SimpleProcessTask(taskManager.Token, credHelper.Substring(1).ToNPath(), action);
	}
	else
	{
	var args = $"credential-{credHelper} {action}";
	task = new SimpleProcessTask(taskManager.Token, args);
	}

	task.Configure(processManager, true);

	task.OnStartProcess += proc =>
	{
	foreach (var line in lines)
	{
	proc.StandardInput.WriteLine(line);
	}
	proc.StandardInput.Close();
	};

	return task;

Fix garbled characters in history view with non-ascii characters (fixes#124) #136

Fix garbled characters in history view with non-ascii characters (fixes#124) #136

Uh oh!

Conversation

shiena commented Jul 13, 2017 • edited by shana Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shiena commented Jul 18, 2017

Uh oh!

shiena commented Jul 22, 2017

Uh oh!

StanleyGoldman commented Jul 31, 2017

Uh oh!

StanleyGoldman commented Aug 2, 2017

Uh oh!

StanleyGoldman left a comment

Choose a reason for hiding this comment

Uh oh!

shiena commented Aug 2, 2017

Uh oh!

StanleyGoldman commented Aug 2, 2017

Uh oh!

StanleyGoldman Aug 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StanleyGoldman Aug 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StanleyGoldman commented Aug 2, 2017

Uh oh!

StanleyGoldman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StanleyGoldman commented Aug 11, 2017

Uh oh!

StanleyGoldman commented Aug 11, 2017

Uh oh!

shiena commented Aug 11, 2017

Uh oh!

StanleyGoldman commented Aug 11, 2017

Uh oh!

shiena commented Aug 11, 2017

Uh oh!

StanleyGoldman commented Aug 11, 2017

Uh oh!

StanleyGoldman commented Aug 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shiena commented Aug 22, 2017

Uh oh!

shana commented Aug 22, 2017

Uh oh!

StanleyGoldman commented Aug 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

shiena commented Jul 13, 2017 •

edited by shana

Loading

StanleyGoldman Aug 2, 2017 •

edited

Loading

StanleyGoldman Aug 2, 2017 •

edited

Loading