[critical] xray + wireguard confuses certificates when opening multiple tabs simultaneously.

[devopg]

Integrity requirements

• I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
• I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
• I searched issues and did not find any similar issues.
• The problem can be successfully reproduced in the latest Release

Description

v24.11.11

After 26 hours of and 5 updates (#4022) to my investigation, I found the problem!

The main problem is that with high parallelism, xray + wireguard are mixing up the site certificates.

Here's what the problem looks like. I opened about 30 websites simultaneously without caches. As we can see, a couple of websites returned an error stating that the certificates do not match.

The interesting thing about the error on this site is that the certificate actually belongs to a neighboring site:

After countless combinations and usage variations, I found a scenario in which this bug appears.

This is a variation when using the inbound WireGuard protocol + sniffing. I try with 3 clients (WG official | WireSock | ASUS WG).

---

Additional information and interesting points that will also help you understand that the problem with xray + wireguard.

1. During the tests, I created a combination of WG Inbound with "replace IP to domain" (unchecking the "Route only" option) -> outbound to another XRAY server to VLESS Inbound. On the first server, I set a rule to block all ".ru" domains, and on the second server, I enabled logging and noticed something strange: from time to time, domains with ".ru" appeared in the logs, which should never have been there in the this place.

This indicates that the xray sniffer + WireGuard correctly identifies the domains (IP <-> DOMAIN), BUT it incorrectly processes the domains cannot properly parse the packets/frames (I'm not sure what it's called) and make the correct routing.

2. Another clue that will convince you that the problem lies specifically with the xray sniffer + WireGuard.
   I replaced the WireGuard Inbound in the above setup with VLESS without changing anything else. And you know what? Everything started working perfectly; there were no ".ru" domains in the logs on SERVER2. It worked flawlessly when opening more than 100 tabs simultaneously.

---

In summary,

The problem in the parsing/processing/routing of packets/frames (I'm not sure what it's called) when using WireGuard Inbound with Sniffer. You should take a look at the piece of code that is responsible for this.

Reproduction Method

For a quick understanding of whether this has been fixed, refer to the image with two server above. The first server has WireGuard inbound with a sniffer and blocking domains like ".com," and there should be no ".com" domains in the logs on the second server. This will already be a sign that you are fixing the issue in the right place.

Client config

Details

[Interface]
PrivateKey = WBN8MdW2A=
Address = 10.0.0.2/32
DNS = 1.1.1.1, 1.0.0.1
MTU = 1420

# wg_in-1
[Peer]
PublicKey = oQRrVxhEgU=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = 192.168.50.254:31108

Server config

Details

`{
  "id": 1,
  "userId": 0,
  "up": 976856060,
  "down": 19195323780,
  "total": 0,
  "remark": "wg_in",
  "enable": true,
  "expiryTime": 0,
  "listen": "",
  "port": 39108,
  "protocol": "wireguard",
  "settings": "{\n  \"mtu\": 1420,\n  \"secretKey\": \"+APdM50E=\",\n  \"peers\": [\n    {\n      \"privateKey\": \"WBN8dW2A=\",\n      \"publicKey\": \"9+IdOQtwkMxJEc=\",\n      \"allowedIPs\": [\n        \"10.0.0.2/32\"\n      ],\n      \"keepAlive\": 0\n    },\n    {\n      \"privateKey\": \"uKQ+/a1UIQnvtsWHgdZet/5B1nJtxFfm4l5FrKMHEUs=\",\n      \"publicKey\": \"gw9HqqG5GPCxk=\",\n      \"allowedIPs\": [\n        \"10.0.0.3/32\"\n      ],\n      \"keepAlive\": 0\n    }\n  ],\n  \"noKernelTun\": true\n}",
  "streamSettings": "",
  "tag": "inbound-39108",
  "sniffing": "{\n  \"enabled\": true,\n  \"destOverride\": [\n    \"http\",\n    \"tls\",\n    \"quic\"\n  ],\n  \"metadataOnly\": false,\n  \"routeOnly\": false\n}",
  "clientStats": []
}`
routing for direct domains:
[ { "type": "field", "inboundTag": [ "api" ], "outboundTag": "api" }, { "type": "field", "outboundTag": "direct", "domain": [ "geosite:category-ru", "regexp:.*\\.ru" ] }, { "type": "field", "outboundTag": "direct", "ip": [ "geoip:ru" ] } ]

Client log

Details

no errors 

Server log

Details

no errors 

[Fangliding]

The investigation is good, but I didn't see the configuration and log

[devopg]

@Fangliding Everything is set to default there; it's easier to look at the picture.
added some info

[Fangliding]

Anyway, it is needed because writing another configuration is annoying and may miss some information (but just assume that this problem does not exist)
It should be a context design issue that leads to the leakage of routing information between different connections. I'm starting to doubt if it's necessary to keep this problematic inbound, perhaps it should be removed

[devopg]

If you are using x3-ui, then indeed, by looking at the picture, you can literally reproduce the environment. Everything is set to default there.

I think you need this, the complete config. I only trimmed the serverUrl and Keys.

Details

{
  "api": {
    "services": [
      "HandlerService",
      "LoggerService",
      "StatsService"
    ],
    "tag": "api"
  },
  "burstObservatory": null,
  "dns": null,
  "fakedns": null,
  "inbounds": [
    {
      "allocate": null,
      "listen": "127.0.0.1",
      "port": 62789,
      "protocol": "dokodemo-door",
      "settings": {
        "address": "127.0.0.1"
      },
      "sniffing": null,
      "streamSettings": null,
      "tag": "api"
    },
    {
      "allocate": {
        "concurrency": 3,
        "refresh": 5,
        "strategy": "always"
      },
      "listen": null,
      "port": 39108,
      "protocol": "wireguard",
      "settings": {
        "mtu": 1420,
        "noKernelTun": true,
        "peers": [
          {
            "allowedIPs": [
              "10.0.0.2/32"
            ],
            "keepAlive": 0,
            "privateKey": "WBN8p6VSS2nNMdW2A=",
            "publicKey": "9+IdOxJEc="
          }],
        "secretKey": "+Anh8heDdI4PdM50E="
      },
      "sniffing": {
        "destOverride": [
          "http",
          "tls",
          "quic"
        ],
        "enabled": true,
        "metadataOnly": false,
        "routeOnly": true
      },
      "streamSettings": null,
      "tag": "inbound-39108"
    }
  ],
  "log": {
    "access": "none",
    "dnsLog": false,
    "error": "",
    "loglevel": "warning",
    "maskAddress": ""
  },
  "observatory": null,
  "outbounds": [
    {
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "",
            "port": 443,
            "users": [
              {
                "encryption": "none",
                "flow": "xtls-rprx-vision",
                "id": "fb5df8ed9a-2cf501ac7153"
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "tcp",
        "realitySettings": {
          "fingerprint": "chrome",
          "publicKey": "bSFDU",
          "serverName": "cloudflare.com",
          "shortId": "3d9a",
          "spiderX": "/"
        },
        "security": "reality",
        "tcpSettings": {
          "header": {
            "type": "none"
          }
        }
      },
      "tag": "vl_main-base_vl"
    },
    {
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "UseIP",
        "noises": [],
        "redirect": ""
      },
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    }
  ],
  "policy": {
    "levels": {
      "0": {
        "statsUserDownlink": true,
        "statsUserUplink": true
      }
    },
    "system": {
      "statsInboundDownlink": true,
      "statsInboundUplink": true,
      "statsOutboundDownlink": true,
      "statsOutboundUplink": true
    }
  },
  "reverse": null,
  "routing": {
    "domainStrategy": "AsIs",
    "rules": [
      {
        "inboundTag": [
          "api"
        ],
        "outboundTag": "api",
        "type": "field"
      },
      {
        "ip": [
          "geoip:private"
        ],
        "outboundTag": "blocked",
        "type": "field"
      },
      {
        "outboundTag": "blocked",
        "protocol": [
          "bittorrent"
        ],
        "type": "field"
      },
      {
        "ip": [
          "geoip:telegram"
        ],
        "outboundTag": "direct",
        "type": "field"
      },
      {
        "domain": [
          "geosite:category-ru",
          "regexp:.*\\.ru"
        ],
        "outboundTag": "direct",
        "type": "field"
      },
      {
        "ip": [
          "geoip:ru"
        ],
        "outboundTag": "direct",
        "type": "field"
      }
    ]
  },
  "stats": {},
  "transport": null
}

I will provide and explain everything you need; feel free to ask.

[hr567]

It seems that it is the same problem I've had before. #3948 @Fangliding
I got around this issue on Linux by using the kernel's wireguard module + tproxy. The configuration is not very complicated but not applicable to other platforms.
I am having some issues with the wireguard + tproxy solution in a docker environment. Configurations and iptables rules that work fine on host are not proxying ipv4 traffic in docker. I haven't found a specific reason yet. However, this solution is currently running well on my server host.
If you're also using Linux, I think wiregurad + tproxy would be a good (temporary) solution until this issue is resolved in Xray.

[devopg]

@hr567
We are a startup and are deploying a large, complex multi-server project. We would like to avoid additional layers or manual, tedious configurations, so temporary solutions are not suitable for us.

[Fangliding]

dup