上海 9929 优选低延迟路径： GOST MTCP + ECMP 启动预热

测试机器环境：利群上海 9929 、xTom 软银 JP 。

0.优选后测试结果截图

RTT 延迟不抽卡，维持稳定不波动不断流。单线程测速能跑起来。

1. 问题背景

部分跨境/跨运营商线路 [本文特指上海 9929 宽带，存在多条 4837 & 9929 混合 ECMP： 据传联通将该业务外包给第三方公司，在后端接入了多条 AS4837 （普通联通骨干网）和 AS9929 （联通精品网）线路] 存在明显的 ECMP （ Equal-Cost Multi-Path ）：ICMP ping 稳定，但不同 TCP 连接的 RTT 会落在两个甚至多个明显不同的档位（例如一部分 ~33ms ，一部分 ~51ms ）。每次新建 TCP 都可能重新参与 ECMP 哈希，导致普通端口转发下不同业务连接分别落在快路/慢路，延迟随连接波动。

2. 核心思路

在 GOST MTCP 启动阶段选出一条低延迟外层 TCP ，后续所有业务长期复用这一条连接，而不是在运行期间持续探测、清理慢连接。

任意 TCP 客户端 → 前置 GOST :12000 → 单个长期 MTCP session （ 4 MiB Stream / 32 MiB Receive ）

  → 启动阶段检测并只保留低延迟外层 TCP → 远端 GOST :11000 → 目标 TCP 服务

GOST MTCP 把多个逻辑 TCP stream 复用在同一条外层 TCP 上，因此 ECMP 抽路的影响从"每个业务连接抽一次"收敛为"MTCP 外层连接建立时抽一次"。

本人曾尝试同时维持多条 MTCP TCP 、定期探测 RTT 、ss -K 踢慢路，但存在两个问题：一是 ss -K 会影响该 session 上承载的所有逻辑 stream ；二是踢掉后重建的连接会重新参与 ECMP ，仍可能抽到慢路，形成"抽路—清理"死循环。最终方案改为：只配置一个 MTCP node ，正常只允许一个 outer TCP ，仅在启动阶段做 ECMP 预热，命中快路后停止干预，运行期间不跑周期检测、不定时 kill 、不主动重建。

3. MTCP 缓冲区参数

单 stream 缓冲区过小会限制吞吐，需按 BDP 估算。例如 RTT≈33ms 、目标带宽≈400Mbps 时，BDP ≈ 400Mbps × 0.033s ≈ 1.65MiB ，因此 4 MiB Stream Buffer 较为宽裕：

mux.maxFrameSize: 32768        # 32 KiB

mux.maxReceiveBuffer: 33554432 # 32 MiB

mux.maxStreamBuffer: 4194304   # 4 MiB

4. 远端 GOST 配置

远端只需监听 :11000，实际转发目标由前置 GOST 的 forwarder 指定。

/root/gost/remote.yaml：

services:

- name: mtcp-server

  addr: :11000

  handler:

    type: relay

  listener:

    type: mtcp

    metadata:

      mux.version: 2

      mux.keepaliveDisabled: false

      mux.keepaliveInterval: 10s

      mux.keepaliveTimeout: 30s

      mux.maxFrameSize: 32768

      mux.maxReceiveBuffer: 33554432

      mux.maxStreamBuffer: 4194304

5. 前置 GOST 配置

前置监听 :12000，通过 chain-mtcp 走 MTCP dialer 连到远端。

services:

- name: tcp-entry

  addr: :12000

  handler:

    type: tcp

    chain: chain-mtcp

  listener:

    type: tcp

  forwarder:

    nodes:

    - name: target

      addr: 127.0.0.1:2345



chains:

- name: chain-mtcp

  hops:

  - name: remote

    nodes:

    - name: remote-mtcp

      addr: 203.0.113.20:11000

      connector:

        type: relay

      dialer:

        type: mtcp

        metadata:

          mux.version: 2

          mux.keepaliveDisabled: false

          mux.keepaliveInterval: 10s

          mux.keepaliveTimeout: 30s

          mux.maxFrameSize: 32768

          mux.maxReceiveBuffer: 33554432

          mux.maxStreamBuffer: 4194304

最终链路：TCP 客户端 → 前置:12000 → GOST TCP Handler → GOST MTCP Dialer → 203.0.113.20:11000 → 远端 GOST Relay → 127.0.0.1:2345（示例目标地址，可替换为任意内网/本机地址）。本质是 TCP → MTCP 隧道 → TCP。

6. 判断 ECMP 快路 / 慢路

ss -tin "dst 203.0.113.20:11000"

重点看 minrtt: 字段而非当前 rtt（当前 RTT 会受排队、突发流量、重传、ACK 延迟等临时因素影响，minrtt 更能反映这条 TCP 最初命中的基础路径）。例如某线路快路 32~34ms 、慢路 50~52ms ，可设 THRESH=40：minrtt < THRESH 判为 FAST ，否则判为 SLOW 。

7. 启动预热脚本 mtcp-prewarm.sh

脚本只负责启动阶段选路，逻辑要点：

• 目标地址上 ESTAB 连接数 = 0：主动 connect 触发 GOST 建立 MTCP ，等待。


• ESTAB = 1：读取 minrtt；< THRESH 则连续确认 STABLE_REQUIRED 次后判定 PREWARM SUCCESS；>= THRESH 则 ss -K 踢掉这条唯一的慢连接，等它彻底消失后重新触发连接，让 GOST 重新参与一次 ECMP 。


• ESTAB > 1：直接 PREWARM ERROR 退出，不自动挑一条 kill。因为设计目标是"一个 MTCP node + 一个 outer TCP"，出现多条说明运行态已偏离预期（可能是配置 reload 、dialer 被重新实例化、旧 session 未完全退出等），脚本不猜测该删哪条，避免误伤正在承载业务的 session ，提示手动 systemctl restart 对应 GOST 服务。

完整代码（支持多目标、支持自定义触发命令 PREWARM_CMD，含 {DST} {PORT} {TRIGGER} 占位符，各阈值/超时均可用环境变量覆盖）：

#!/usr/bin/env bash



# GOST MTCP ECMP 启动预热

#

# 默认用法：

#   ./mtcp-prewarm.sh <目标 IP> <目标端口>

#

# 多目标：

#   ./mtcp-prewarm.sh <IP1> <PORT1> <IP2> <PORT2> ...

#

# 为不同目标指定不同的本地触发入口：

#   ./mtcp-prewarm.sh \

#     203.0.113.20:[email protected]:12000 \

#     198.51.100.20:[email protected]:13000

#

# 默认触发入口：

#   PREWARM_ADDR=127.0.0.1:12000

#

# 自定义触发命令：

#   PREWARM_CMD='your command' ./mtcp-prewarm.sh ...

#

# PREWARM_CMD 可使用占位符：

#   {DST}      当前目标地址

#   {PORT}     当前目标端口

#   {TRIGGER}  当前本地触发入口

#

# 设计目标：

#   只在启动/维护阶段运行。

#   严格单连接模式：目标地址同时出现超过 1 条 ESTAB TCP 时立即报错退出，

#   不自动清理任何连接，避免把异常运行态误判为预热成功。

#   如果唯一 MTCP 外层 TCP 的 minrtt >= THRESH ，则断开并主动触发重建。

#   连续 STABLE_REQUIRED 次确认唯一连接低于阈值后退出，不再碰现有连接。



set -uo pipefail



THRESH="${THRESH:-40}"

MAX_ROUNDS="${MAX_ROUNDS:-30}"

SETTLE="${SETTLE:-2}"

EMPTY_WAIT="${EMPTY_WAIT:-1}"

STABLE_REQUIRED="${STABLE_REQUIRED:-2}"

PREWARM_ADDR="${PREWARM_ADDR:-127.0.0.1:12000}"

PREWARM_HOLD="${PREWARM_HOLD:-0.3}"

TRIGGER_TIMEOUT="${TRIGGER_TIMEOUT:-3}"

KILL_WAIT="${KILL_WAIT:-2}"

PREWARM_CMD="${PREWARM_CMD:-}"



ts() {

    date '+%F %T'

}



log() {

    echo "[$(ts)] $*"

}



usage() {

    cat <<'USAGE'

用法：

  ./mtcp-prewarm.sh <目标 IP> <目标端口>

  ./mtcp-prewarm.sh <IP1> <PORT1> <IP2> <PORT2> ...

  ./mtcp-prewarm.sh <目标 IP:端口 @触发 IP:端口> [...]



示例：

  ./mtcp-prewarm.sh 203.0.113.20 11000



  ./mtcp-prewarm.sh \

    203.0.113.20 11000 \

    198.51.100.20 11000



  ./mtcp-prewarm.sh \

    203.0.113.20:[email protected]:12000 \

    198.51.100.20:[email protected]:13000



可配置环境变量：

  THRESH=40

  MAX_ROUNDS=30

  SETTLE=2

  EMPTY_WAIT=1

  STABLE_REQUIRED=2

  PREWARM_ADDR=127.0.0.1:12000

  PREWARM_HOLD=0.3

  TRIGGER_TIMEOUT=3

  KILL_WAIT=2

  PREWARM_CMD='自定义触发命令'



PREWARM_CMD 占位符：

  {DST} {PORT} {TRIGGER}

USAGE

}



is_fast() {

    awk -v r="$1" -v t="$THRESH" 'BEGIN { exit !(r < t) }'

}



split_trigger() {

    local trigger="$1"

    TRIGGER_HOST="${trigger%:*}"

    TRIGGER_PORT="${trigger##*:}"



    if [[ -z "$TRIGGER_HOST" || -z "$TRIGGER_PORT" || "$TRIGGER_HOST" == "$TRIGGER_PORT" ]]; then

        return 1

    fi



    case "$TRIGGER_PORT" in

        ''|*[!0-9]*) return 1 ;;

    esac



    return 0

}



trigger_reconnect() {

    local dst="$1"

    local port="$2"

    local trigger="$3"

    local cmd=""



    log "${dst}:${port} trigger=${trigger}"



    if [[ -n "$PREWARM_CMD" ]]; then

        cmd="${PREWARM_CMD//\{DST\}/$dst}"

        cmd="${cmd//\{PORT\}/$port}"

        cmd="${cmd//\{TRIGGER\}/$trigger}"

        timeout "$TRIGGER_TIMEOUT" bash -lc "$cmd" >/dev/null 2>&1 || true

        return

    fi



    if ! split_trigger "$trigger"; then

        log "${dst}:${port} 无效触发入口: ${trigger}"

        return

    fi



    timeout "$TRIGGER_TIMEOUT" bash -c \

        "exec 3<>/dev/tcp/${TRIGGER_HOST}/${TRIGGER_PORT}; sleep ${PREWARM_HOLD}" \

        >/dev/null 2>&1 || true

}



count_established() {

    local dst="$1"

    local port="$2"



    ss -ntH "dst ${dst}:${port}" 2>/dev/null |

        awk '$1 == "ESTAB" { n++ } END { print n + 0 }'

}



show_established() {

    local dst="$1"

    local port="$2"



    ss -ntpH "dst ${dst}:${port}" 2>/dev/null |

    awk '$1 == "ESTAB" {

        printf("  local=%s peer=%s", $4, $5)

        for (i=6; i<=NF; i++) printf(" %s", $i)

        printf("\n")

    }'

}



get_flows() {

    local dst="$1"

    local port="$2"

    local lport=""

    local line=""

    local local_ep=""



    ss -tinH "dst ${dst}:${port}" 2>/dev/null |

    while IFS= read -r line; do

        if [[ "$line" == ESTAB* ]]; then

            local_ep=$(awk '{print $4}' <<<"$line")

            lport="${local_ep##*:}"

            continue

        fi



        if [[ -n "$lport" && "$line" =~ minrtt:([0-9.]+) ]]; then

            echo "${lport} ${BASH_REMATCH[1]}"

            lport=""

        fi

    done

}



kill_flow() {

    local dst="$1"

    local port="$2"

    local lport="$3"

    local minrtt="$4"



    log "${dst}:${port} KILL sport=${lport} minrtt=${minrtt}ms"

    ss -K "dst ${dst}:${port} sport = :${lport}" >/dev/null 2>&1 || true

}



prewarm_one() {

    local dst="$1"

    local port="$2"

    local trigger="$3"

    local stable=0

    local round total

    local flow lport minrtt

    local lock lockfd

    local socket_count kill_deadline



    case "$port" in

        ''|*[!0-9]*)

            log "错误: ${dst}:${port} 端口必须是数字"

            return 2

            ;;

    esac



    lock="/tmp/mtcp-prewarm-${dst//[^a-zA-Z0-9]/_}-${port}.lock"



    exec {lockfd}>"$lock"



    if ! flock -n "$lockfd"; then

        log "${dst}:${port} 已有预热任务运行，跳过"

        exec {lockfd}>&-

        return 0

    fi



    log "============================================================"

    log "MTCP ECMP prewarm start"

    log "target=${dst}:${port}"

    log "trigger=${trigger}"

    log "fast threshold=<${THRESH}ms"

    log "max rounds=${MAX_ROUNDS}"

    log "============================================================"



    for ((round=1; round<=MAX_ROUNDS; round++)); do



        socket_count="$(count_established "$dst" "$port")"



        if (( socket_count > 1 )); then

            log "============================================================"

            log "${dst}:${port} PREWARM ERROR"

            log "${dst}:${port} expected exactly 1 MTCP outer TCP, found ${socket_count}"

            log "${dst}:${port} multiple MTCP sessions detected"

            log "${dst}:${port} restart the corresponding GOST service"

            log "${dst}:${port} current established sockets:"



            show_established "$dst" "$port" |

            while IFS= read -r line; do

                log "$line"

            done



            log "============================================================"



            exec {lockfd}>&-

            return 3

        fi



        if (( socket_count == 0 )); then

            stable=0



            log "${dst}:${port} round=${round} no MTCP established flow"



            trigger_reconnect "$dst" "$port" "$trigger"



            sleep "$EMPTY_WAIT"

            continue

        fi



        mapfile -t flows < <(get_flows "$dst" "$port")



        total=${#flows[@]}



        if (( total == 0 )); then

            stable=0



            log "${dst}:${port} round=${round} one ESTAB flow exists but minrtt is not ready"



            sleep "$EMPTY_WAIT"

            continue

        fi



        if (( total > 1 )); then

            log "============================================================"

            log "${dst}:${port} PREWARM ERROR"

            log "${dst}:${port} inconsistent ss result"

            log "${dst}:${port} socket_count=${socket_count}, parsed_flows=${total}"

            log "${dst}:${port} no connection was modified"

            log "============================================================"



            exec {lockfd}>&-

            return 4

        fi



        flow="${flows[0]}"

        lport="${flow%% *}"

        minrtt="${flow##* }"



        log "---------------- ${dst}:${port} round ${round} ----------------"



        if is_fast "$minrtt"; then



            log "${dst}:${port} FAST sport=${lport} minrtt=${minrtt}ms"

            log "${dst}:${port} flows=1 fast=1 slow=0"



            ((stable++))



            log "${dst}:${port} single flow FAST, stable=${stable}/${STABLE_REQUIRED}"



            if (( stable >= STABLE_REQUIRED )); then



                socket_count="$(count_established "$dst" "$port")"



                if (( socket_count != 1 )); then

                    log "============================================================"

                    log "${dst}:${port} PREWARM ERROR"

                    log "${dst}:${port} final verification expected 1 MTCP outer TCP, found ${socket_count}"



                    if (( socket_count > 1 )); then

                        log "${dst}:${port} restart the corresponding GOST service"



                        show_established "$dst" "$port" |

                        while IFS= read -r line; do

                            log "$line"

                        done

                    fi



                    log "============================================================"



                    exec {lockfd}>&-

                    return 3

                fi



                log "${dst}:${port} PREWARM SUCCESS"

                log "${dst}:${port} verified single MTCP outer TCP with minrtt < ${THRESH}ms"



                exec {lockfd}>&-

                return 0

            fi



            sleep "$SETTLE"

            continue

        fi



        stable=0



        log "${dst}:${port} SLOW sport=${lport} minrtt=${minrtt}ms"

        log "${dst}:${port} flows=1 fast=0 slow=1"



        socket_count="$(count_established "$dst" "$port")"



        if (( socket_count != 1 )); then

            log "============================================================"

            log "${dst}:${port} PREWARM ERROR"

            log "${dst}:${port} flow count changed before kill"

            log "${dst}:${port} expected 1, found ${socket_count}"

            log "${dst}:${port} no connection was modified"



            if (( socket_count > 1 )); then

                log "${dst}:${port} restart the corresponding GOST service"



                show_established "$dst" "$port" |

                while IFS= read -r line; do

                    log "$line"

                done

            fi



            log "============================================================"



            exec {lockfd}>&-

            return 3

        fi



        kill_flow "$dst" "$port" "$lport" "$minrtt"



        kill_deadline=$((SECONDS + KILL_WAIT))



        while (( SECONDS < kill_deadline )); do



            socket_count="$(count_established "$dst" "$port")"



            if (( socket_count == 0 )); then

                break

            fi



            sleep 0.1

        done



        socket_count="$(count_established "$dst" "$port")"



        if (( socket_count > 1 )); then

            log "============================================================"

            log "${dst}:${port} PREWARM ERROR"

            log "${dst}:${port} multiple sessions appeared after kill: ${socket_count}"

            log "${dst}:${port} restart the corresponding GOST service"



            show_established "$dst" "$port" |

            while IFS= read -r line; do

                log "$line"

            done



            log "============================================================"



            exec {lockfd}>&-

            return 3

        fi



        if (( socket_count == 1 )); then

            log "============================================================"

            log "${dst}:${port} PREWARM ERROR"

            log "${dst}:${port} killed flow did not disappear within ${KILL_WAIT}s"

            log "${dst}:${port} no reconnect was triggered"

            log "${dst}:${port} check/restart the corresponding GOST service"



            show_established "$dst" "$port" |

            while IFS= read -r line; do

                log "$line"

            done



            log "============================================================"



            exec {lockfd}>&-

            return 5

        fi



        trigger_reconnect "$dst" "$port" "$trigger"



        log "${dst}:${port} waiting ${SETTLE}s for GOST reconnect..."



        sleep "$SETTLE"

    done



    log "============================================================"

    log "${dst}:${port} PREWARM TIMEOUT"



    socket_count="$(count_established "$dst" "$port")"



    log "${dst}:${port} established=${socket_count}"



    if (( socket_count > 1 )); then

        log "${dst}:${port} multiple MTCP sessions detected"

        log "${dst}:${port} restart the corresponding GOST service"



        show_established "$dst" "$port" |

        while IFS= read -r line; do

            log "$line"

        done

    fi



    mapfile -t flows < <(get_flows "$dst" "$port")



    for flow in "${flows[@]}"; do

        log "${dst}:${port} sport=${flow%% *} minrtt=${flow##* }ms"

    done



    log "============================================================"



    exec {lockfd}>&-

    return 1

}



declare -a TARGETS=()



if (( $# == 0 )); then

    usage

    exit 2

fi



if [[ "$1" == *"@"* ]]; then



    for spec in "$@"; do



        if [[ "$spec" != *"@"* ]]; then

            echo "错误: 混合参数格式不受支持"

            usage

            exit 2

        fi



        target="${spec%%@*}"

        trigger="${spec#*@}"



        dst="${target%:*}"

        port="${target##*:}"



        TARGETS+=("${dst}|${port}|${trigger}")

    done



else



    if (( $# % 2 != 0 )); then

        echo "错误: IP/端口参数必须成对出现"

        usage

        exit 2

    fi



    while (( $# > 0 )); do

        TARGETS+=("$1|$2|${PREWARM_ADDR}")

        shift 2

    done

fi



failed=0



for item in "${TARGETS[@]}"; do



    IFS='|' read -r dst port trigger <<<"$item"



    if ! prewarm_one "$dst" "$port" "$trigger"; then

        failed=1

    fi

done



if (( failed != 0 )); then

    log "one or more targets failed"

    exit 1

fi



log "all targets PREWARM SUCCESS"



exit 0

保存后校验语法：

chmod +x /root/mtcp-prewarm.sh

bash -n /root/mtcp-prewarm.sh

8. 使用方法

单目标手动执行：

THRESH=40 \

PREWARM_ADDR=127.0.0.1:12000 \

/root/mtcp-prewarm.sh 203.0.113.20 11000

若首次抽到慢路，脚本会 SLOW → KILL → 等旧连接消失 → 重新触发 直到命中快路并连续确认两次，输出 PREWARM SUCCESS。

多目标一次执行：

./mtcp-prewarm.sh \

  203.0.113.20 11000 \

  198.51.100.20 11000

不同目标使用不同本地触发入口：

./mtcp-prewarm.sh \

  203.0.113.20:[email protected]:12000 \

  198.51.100.20:[email protected]:13000

若普通 TCP connect 无法触发实际业务链，可用 PREWARM_CMD 自定义触发命令：

PREWARM_CMD="timeout 1 bash -c 'exec 3<>/dev/tcp/127.0.0.1/12000; sleep 0.3'" \

./mtcp-prewarm.sh 203.0.113.20 11000

不同线路 RTT 档位不同时，建议用独立 systemd service 分别设置各自的 THRESH（例如线路 A 快路 33ms/慢路 51ms 用 THRESH=40，线路 B 快路 46ms/慢路 60ms 用 THRESH=52）。

9. systemd 集成

长期运行建议用 systemd 管理 GOST：开机启动、异常退出自动拉起、日志统一记录，并在每次启动后自动执行一次 ECMP 预热（ExecStartPost）。

前置 GOST （主线路）：

cat >/etc/systemd/system/gost-mtcp-main.service <<'EOF'

[Unit]

Description=GOST MTCP Main

After=network-online.target

Wants=network-online.target



[Service]

Type=simple

WorkingDirectory=/root/gost



ExecStart=/root/gost/gost -D -C /root/gost/cn.yaml



Environment="PREWARM_ADDR=127.0.0.1:12000"

Environment="THRESH=40"



ExecStartPost=/bin/bash -lc '/root/mtcp-prewarm.sh 203.0.113.20 11000 || true'



Restart=on-failure

RestartSec=2

TimeoutStartSec=120



KillSignal=SIGTERM



StandardOutput=journal

StandardError=journal



[Install]

WantedBy=multi-user.target

EOF

启用与查看：

systemctl daemon-reload

systemctl enable --now gost-mtcp-main

systemctl status gost-mtcp-main --no-pager

journalctl -u gost-mtcp-main -n 100 -l --no-pager

ExecStartPost 末尾的 || true 是为了防止预热失败被 systemd 判定为启动失败，进而触发 Restart=on-failure 形成"启动→预热失败→service failed→重启→再次预热失败"的循环；加上 || true 后预热失败只写入 journal ，GOST 主进程照常运行，留给人工检查。

10. 验证与常见问题

验证只有一个 MTCP 且命中快路：

ss -ntp "dst 203.0.113.20:11000"   # ESTAB 数量应为 1

ss -tin "dst 203.0.113.20:11000"   # minrtt < THRESH

为什么 flows > 1 不自动处理？ 严格模式下同一目标出现 ≥2 条 outer TCP 属于异常运行态（而非单纯"抽到慢路"），脚本报 PREWARM ERROR 并停止，不猜测删哪条；更安全的处理是 systemctl restart 对应服务，让 ExecStartPost 重新走一次干净预热。

为什么不再用 cron 周期性踢慢路？ 一条 MTCP TCP 上可能承载多个业务 stream ，运行期间 ss -K 相当于主动重置整个 session 。原则改为：启动阶段允许检测/淘汰/重抽，正常运行阶段不主动干预；确需重建时用 systemctl restart 触发完整的启动预热流程。

运行一段时间后 RTT 变高怎么办？ 需区分两种情况：若还是原来那条 MTCP TCP （本地端口不变）且 minrtt 仍处于原来的低延迟档，只是当前 rtt 因拥塞/重传等临时抖动升高，无需处理；若本地端口已变化，说明 MTCP 外层 TCP 已重新建立并可能重新抽到慢路，此时也不应重新引入周期性 FastPath ，而是通过 systemctl restart 重新走一次启动预热。

为什么 mtr -T 仍可能看到多条路径？ mtr -T 每次探测都是新建 TCP 五元组，会重新参与 ECMP ，不会复用已建立的 GOST MTCP 连接，因此不能用它判断 MTCP 实际路径，应以 ss -tin 的 minrtt 为准。

文中公网 IP 、端口均为示例，实际使用请按自己线路的 RTT 分布调整。