Fix material icon & diff highlight

[wxiaoguang]

No description provided.

[wxiaoguang]

Before:

• Some files like "php" were not recognized by the material icon rules.
• The full-file diff highlight is not quite right

After:

[wxiaoguang]

Next: Fix file icon mapping #33855

[lunny]

There is a regression caused #35504

[wxiaoguang]

There is a regression caused #35504

The regression is from this PR, or indeed from " Full-file syntax highlighting for diff pages #33766 "?

[lunny]

There is a regression caused #35504

The regression is from this PR, or indeed from " Full-file syntax highlighting for diff pages #33766 "?

I used git bisect to identify the issue, and it points to this PR. When I check out commit 657239b, I can reproduce the bug. However, checking out the previous commit, c102492, everything works correctly.

[wxiaoguang]

diff --git a/services/gitdiff/gitdiff.go b/services/gitdiff/gitdiff.go
index 6e15f71609..1a82c7fb7f 100644
--- a/services/gitdiff/gitdiff.go
+++ b/services/gitdiff/gitdiff.go
@@ -1337,6 +1337,7 @@ func GetDiffForRender(ctx context.Context, repoLink string, gitRepo *git.Reposit
 }
 
 func highlightCodeLines(diffFile *DiffFile, isLeft bool, content string) map[int]template.HTML {
+       content, _ = charset.ToUTF8(util.UnsafeStringToBytes(content), charset.ConvertOpts{KeepBOM: false})
        highlightedNewContent, _ := highlight.Code(diffFile.Name, diffFile.Language, content)
        splitLines := strings.Split(string(highlightedNewContent), "\n")
        lines := make(map[int]template.HTML, len(splitLines))

Can you try this?

[lunny]

diff --git a/services/gitdiff/gitdiff.go b/services/gitdiff/gitdiff.go
index 6e15f71609..1a82c7fb7f 100644
--- a/services/gitdiff/gitdiff.go
+++ b/services/gitdiff/gitdiff.go
@@ -1337,6 +1337,7 @@ func GetDiffForRender(ctx context.Context, repoLink string, gitRepo *git.Reposit
 }
 
 func highlightCodeLines(diffFile *DiffFile, isLeft bool, content string) map[int]template.HTML {
+       content, _ = charset.ToUTF8(util.UnsafeStringToBytes(content), charset.ConvertOpts{KeepBOM: false})
        highlightedNewContent, _ := highlight.Code(diffFile.Name, diffFile.Language, content)
        splitLines := strings.Split(string(highlightedNewContent), "\n")
        lines := make(map[int]template.HTML, len(splitLines))

Can you try this?

Looks like it works